Service registry lookup

[mboersma]

Come up with simple design for new object models / dynamic config / envvars to be Deis' service registry and attachment workflow.

[gabrtv]

I've been working on a design for this with the Opscode guys. I hope to post a specification here shortly and get a PR together that implements a manual service registry that we can later make more dynamic (late binding, multi-tenancy, etc.)

[bacongobbler]

@gabrtv, do you still have that spec hiding around somewhere for this issue? :)

[bacongobbler]

Standalone Service Registry

The service registry is a wrapper to common web services for
centralized service creation (known as provisioning). It can work with
common on-premise services such as a local mysql database or a Redis server
for a "private" or on-premise service registry, or with public SaaS applications
such as MongoLab or ElephantSQL for a public service offering.

Technical Overview

This project will be a generic wrapper for web services, giving administrators
the power to make their consumable service readily available to any PaaS that
uses this service registry.

Similar to Heroku Add-ons, customers use the service registry to provision an
addon offered by a third-party provider for their applications. When this
happens, the service registry sends a request to a providers's service gateway,
which then creates a new private addon. This addon represents an instance of
the provider's service.

Target Market

The service registry aims to target the PaaS industry, where new PaaS vendors
are looking to add data services to their PaaS offering (similar to Heroku
Add-Ons). Private PaaS solutions may also implement this solution by
connecting to external databases or services instead, whether those services
are internal to a clustered environment or are in a large datacenter in another
region.

It also targets the SaaS market, where new providers are looking to get some
public visibility of their product through third parties that implement
this service in their offering.

Administrator's Spec

An administrator of the registry can manage all of the registered gateways.
You can view the gateways that are bound to the registry with the
deis services:list command:

$ deis services:list
builtin-mysql
builtin-postgresql
amazon-rds-oracledb
mongolab

Registering a Gateway

Adds the gateway to the registry. The gateway is disabled by default.

$ deis services:add builtin-mysql --endpoint http://my-gateway-fqdn.com:port/
Adding builtin-mysql...done
Use `deis services:enable builtin-mysql` to enable this service

Unregistering a Gateway

Removes the gateway from the registry. Any addons created with this gateway
will NOT be deprovisioned when this gateway is removed.

$ deis services:remove builtin-mysql
Are you sure you want to remove builtin-mysql? (y or n)
Removing builtin-mysql...done

Disabling a Gateway

Disables a gateway, denying any calls to provision a new addon.

$ deis services:disable builtin-mysql

Enabling a Gateway

Enables a Gateway for use.

$ deis services:enable builtin-mysql

Developer Spec

A developer can manage their addons through the command line interface. You can view your current addons that are bound to your application with the deis addons:list command:

$ deis addons:list
builtin-mysql:free
builtin-postgresql:free
amazon-rds-oracledb:small
mongolab:enterprise

Adding an Addon

Adds the addon to the application, and forces a new release.

$ deis addons:add builtin-redis --plan=free
Adding builtin-redis:free to myapp...done
Use `deis addons:docs builtin-redis --plan=free` to view documentation

Removing an Addon

Removes the addon from the application, and forces a new release.

$ deis addons:remove builtin-redis --plan=free
Removing builtin-redis:free from myapp...done

Switching Plans

Changes the addon credentials within the application to reflect the upgraded (or downgraded) plan's new endpoint. Forces a new release.

$ deis addons:update builtin-redis --plan=paid
Upgrading builtin-redis:paid to myapp... done
You are now being billed for this addon at $0.06/container/hr
Use `deis addons:docs builtin-redis --plan=paid` to view documentation

Opening an Addon's Dashboard

Opens a management page on the provider's website for the addon, giving the user administrative access to the addon.

$ deis addons:open builtin-redis
Opening builtin-redis:free for myapp.

Read Documentation on an Addon

Opens documentation on how to use the addon with the application.

$ deis addons:docs builtin-redis:free
Opening documentation on builtin-redis:free for myapp.

Registry Endpoints

GET     /gateways
    retrieve information about all gateways available

POST    /gateways
    attach a gateway to the registry

GET     /gateways/:gateway_name
    get information about this specific gateway

PUT     /gateways/:gateway_name
    update configuration for a gateway

DELETE  /gateways/:gateway_name
    unregister a gateway. also unprovisions all addons provisioned with this gateway

GET     /addons
    get information on all addons available

GET     /addons?user=:username
    get all addons provisioned by :username

GET     /addons?gateway=:gateway_name
    get all addons provisioned by a specific gateway

POST    /addons
    provisions a new addon

GET     /addons/:addon_name
    get information about the addon

PUT     /addons/:addon_name
    update an addon's config

DELETE /addons/:addon_name
    deprovision an addon

Architecture Diagram

+-------------+    deis addons:add mysql --plan=free    +--------+
|             |-----------------------------------------| Client |
|  Controller |-----+                                   +--------+
|             |     |
+-------------+     |   POST /addons { type: mysql, plan: free }
                    |   RETURN { status: 200, creds: { MYSQL_URL: ... }, name: mysql-746017aa }
192.168.0.1         |                   |
+------------+      | <-----------------+
|            |      |
|  Registry  |------+
|            |
|            |         CREATE DATABASE 746017aa; GRANT USER ...;
+------------+                                   |
      |                                          |
      | POST / { plan: free }                    |
      | RETURN { status: 200, creds: {...} }     |
      |            |                             |
      |            |           192.168.0.2       |      192.168.0.3:3306
      |            |        +---------------+    |      +---------------+
      |            V        |               |    V      |               |
      +---------------------|    Gateway    |-----------|   MySQL       |
      |                     |               |           |               |
      |                     +---------------+           +---------------+
      |                        192.168.0.4              192.168.0.3:5432
      |                     +---------------+           +---------------+
      |                     |               |           |               |
      +---------------------|    Gateway    |-----------|   PostgreSQL  |
      |                     |               |           |               |
      |                     +---------------+           +---------------+
      |                        192.168.0.5              http://mongolab.com/
      |                     +---------------+           +---------------+
      |                     |               |           |               |
      +---------------------|    Gateway    |-----------|   MongoLab    |
                            |               |           |               |
                            +---------------+           +---------------+

[gabrtv]

@bacongobbler This looks great. Let's flesh out the developer UX next. For example, how does a developer actually attach a database to an existing application using the CLI?

With Heroku this looks like heroku addons:add <provider>:<plan>. Maybe for Deis this means deis services:add <provider>:<plan>? What about the other parts of the lifecycle? Let's flesh that out, taking inspiration from Heroku's command-line workflow..

[bacongobbler]

optionally, you should also be able to make this scriptable by specifying the --application=<appname> option on any command, or whatever name makes most sense to make this command apply to a specific application.

[gabrtv]

Thanks for this @bacongobbler. After much chin stroking and IRC discussion, here are the next steps before we can start implementation:

• Merge Upgrade/Downgrade into a single 'Change' or 'Update' function
• Change developer commands to deis addons:action -- for Heroku compatibility
• Define ops/admin commands with deis services:action -- how does the admin inject services into the registry and manage them?
• Revamp the spec into a single comment, so we can see the latest version all at once

Once we agree on moving forward, I suggest we start with a PostgreSQL reference implementation!

[bacongobbler]

Features to discuss before this spec is considered final. These may also be cut from the final spec (as we don't need to concern ourselves with SPOF for now):

• optimally, the registry should have no Single Points Of Failure. This means that if the registry went down in the cluster, the entire system should still be usable.
• gateways and registries should have a shared secret key for authentication. We don't want someone to attach a rogue gateway to our registry, and we don't want to be the culprit of a MITM attack! This can be done using OAuth tokens.
• health management of the services. We should know when MongoLab is down or when our internal postgresql database is undergoing maintenance.
• more clarity on how the registry and the gateway communicate with each other (through bi-directional http API requests and OAuth tokens).
• how does a developer create their own custom service gateway?

[bacongobbler]

Work in progress at https://github.com/bacongobbler/service-registry

[bacongobbler]

It's very much alpha in it's current state, but I have a working proof of concept where the service registry will create and delete new MySQL databases, as well as return a URI for use. Documentation (and tests!) has still yet to be written, but you can try it out and see what kind of changes you guys would like to see.

Instructions on getting started:

1. install the dependencies required for MySQL-python
2. install rabbitmq for celery tasks
3. run the following:

$ git clone git://github.com/bacongobbler/service-registry.git -b develop
$ cd service-registry
$ virtualenv venv && source venv/bin/activate
$ pip install -r requirements.txt
$ python manage.py syncdb --noinput
$ python manage.py runserver

4. go to http://127.0.0.1:8000/api/v1/providers/, add a MySQL provider, enable it by ticking the checkbox, and click POST
5. go to http://127.0.0.1:8000/api/v1/services/, add a new service with the name asdfqwerty, the new provider we just created, the free or paid plan, leave the URI field or write in it (will be overwritten later), any url for the dashboard, and any url for the docs.

After clicking POST to save, the entry should have been created, and the new database should be created in your local MySQL service. To verify, take a look at the uri to ensure that it was populated, then open up a mysql shell:

$ mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 5.6.15 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| asdfqwerty         |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)

mysql> select User from mysql.user;
+------------------+
| User             |
+------------------+
| root             |
| root             |
|                  |
|                  |
| 8w685g6qkha3spot |
| root             |
+------------------+
6 rows in set (0.00 sec)

For example, my uri field is now set to mysql://8w685g6qkha3spot:qrpsgu8yk64ig97h@localhost:3306/asdfqwerty, so I can connect trying to use that information:

$ mysql -u 8w685g6qkha3spot -pqrpsgu8yk64ig97h asdfqwerty
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 28
Server version: 5.6.15 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
Empty set (0.00 sec)

[tombh]

So exciting! Great work :)

The code's on the develop branch right?

[bacongobbler]

the code's on the develop branch right?

Sorry, the instructions imply that it's currently on the develop branch. I'll be more explicit next time. You are correct! :)

[tombh]

Just tried it and it worked first time out of the box ;)

What's next then? Is the plan for these endpoints to live on the official Deis API? I see that there'll be new commands for the CLI client. Will the Deis cookbook officially have recipes for certain 'built-in services'?

I'd like to help with this if I can. My Danabox project (formerly Crowdbox) is actually in private beta now and I think having database support is going to help encourage people to try it out.

[bacongobbler]

Just tried it and it worked first time out of the box ;)

I try to keep a green policy for master :)

What's next then? Is the plan for these endpoints to live on the
official Deis API? I see that there'll be new commands for the CLI
client. Will the Deis cookbook officially have recipes for certain
'built-in services'?

The plan is to have this residing on the controller as a standalone web
app for now. Commands like deis addons:add builtin-mysql will
communicate with this webapp through a POST request, which will then
receive the credentials as a response. The client should then inject
those credentials into the container, much like how deis config:set
accomplishes this (see
https://github.com/opdemand/deis/blob/master/client/deis.py#L778).

I'd like to help with this if I can. My Danabox project (formerly
Crowdbox) is actually in private beta http://danabox.io/ now and I
think having database support is going to help encourage people to
try it out.

By all means, please do!

[tombh]

What's the reasoning for having a separate app? Sorry if this has been discussed elsewhere already. As a separate web app is it easy to piggyback off the API's cookie session from the CLI client? Is the idea to have it included in the official cookbook? And will there be recipes for built-in services in the cookbook?

The new commands in the CLI client should be pretty straightforward. So should we start a new branch on Deis to pull all these things together?

[bacongobbler]

What's the reasoning for having a separate app? Sorry if this has been discussed elsewhere already. As a separate web app is it easy to piggyback off the API's cookie session from the CLI client?

Good question. I think a little background where I'm coming from is necessary...

I come from a cloudfoundry background, working on Stackato. Cloudfoundry's cloud_controller_ng has service discovery built in via a message bus called NATS and by using pub/sub techniques. The service registries (named service brokers in cfv2) are all separate sinatra apps, proxied together using an asynchronous proxy server. There is one service broker for each service (i.e. MongoDB, MySQL, Postgres, etc.), so each web app knows how to handle each service. It's convoluted, but it works. It has a couple advantages this way:

• you can put the webapp on a dedicated services node, which you can customize to your liking (i.e. increasing the disk size, memory, etc.)
• It ensures that if your service node goes down, your cloud controller doesn't go down with it.

flynn.io is the same, in a sense. It has multiple micro apps all formed together to make one giant PaaS system, which are all discovered using discoverd, which is another service discovery system. It helps keep different offerings available on multiple servers, and in a perfect world, makes each system fully redundant (well, the service discovery is not necessarily redundant, nor is the controller). Cloudfoundry has this issue too, where they are trying to solve the problem of having no single points of failure in the cluster (currently, the message bus and the controller are SPOFs, though there may be other components that are just as likely to be a SPOF, too).

Unfortunately, Deis does not have service discovery yet (looking forward to seeing this in a future release 😉), so it'd probably make more sense to integrate this into the controller for now.

Is the idea to have it included in the official cookbook? And will there be recipes for built-in services in the cookbook?

The idea is the same with how you can enable or disable formation providers. You should be able to enable or disable certain service providers with a couple changes in api/models.py. If you don't want mongodb available because you don't want any evil aGPL code in your infrastructure, then it should be as easy as flicking a switch in api/models.py and a quick change in the cookbook to remove mongodb from being installed.

The new commands in the CLI client should be pretty straightforward. So should we start a new branch on Deis to pull all these things together?

Yes, a new branch would be good to put all this work back into the controller. dibs on 231-service-registry :)

[tombh]

That makes sense about isolating service discovery, you can't make the service-discoverer the same as one of the services that it is trying to discover.

So I made @bacongobbler's dibbed 231-service-registry branch with a basic integration of the service registry into the controller. Just wondering if it's the right direction to be heading in?

Sorry about the 3 commits. I forgot to remove the '.git' folder when copying in the service-registry repo and git thought it was a submodule ಠ_ಠ

Still to do:

• Add the returned URI to the app's config.
• All the other addons:* commands.
• It still needs to have authentication set up, I'm imaging that it will piggyback off the session from the controller API?
• Cookbook recipe to install MySQL and register it as a provider.

[bacongobbler]

So I made @bacongobbler's dibbed 231-service-registry branch with a basic integration of the service registry into the controller. Just wondering if it's the right direction to be heading in?

I like your approach, but if we're following Deis' current design pattern, the service code should be integrated back into the api app, such as the models and the tasks, and the services.provider should just be moved into a top level services package.

Still to do:

• Add the returned URI to the app's config.
• All the other addons:* commands.
• It still needs to have authentication set up, I'm imaging that it will piggyback off the session from the controller API?
• Cookbook recipe to install MySQL and register it as a provider.

Tack on "add tests" to that list!

It still needs to have authentication set up, I'm imaging that it will piggyback off the session from the controller API?

Yes. It should be the same access controls. You should only be able to add a service to an application you have permission to create new releases for.

Cookbook recipe to install MySQL and register it as a provider.

Maybe that can be set up as a south migration or a database seed? (the provider registration, not installing MySQL)?

[tombh]

I did actually think complete integration was preferable. But then I wondered if it would be harder to extract out when the time came to make it standalone. Okay, so let's do it that way.

So should the models and tasks be appended to api/models.py and api/tasks.py? What do you mean by a 'services package'? Like a services folder inside api/?

Yes, tests of course! I forgot about them.

Yes, same access controls. That's pretty straightforward then if it's being integrated straight into the API app.

Yes, I think a south migration would work for provider registration. Should it also work through the admin/ web interface as well? What about actually installing MySQL on the controller? You know, like with a recipe that does all the apt-get install stuff? So you actually have an a MySQL instance on the node? I'm not so sure about that. Ideally it would be its own node wouldn't it? Or at least you should have the option of putting it on its own node. Already I want to write a postgres and redis providers! And we can't have the controller being filled up with all kinds of services can we?

[bacongobbler]

So should the models and tasks be appended to api/models.py and api/tasks.py?

Yes.

What do you mean by a 'services package'? Like a services folder inside api/?

More like a services folder at the root directory, much like how provider/digitalocean.py is set up with api/tasks.py. It would be the same for services/mysql.py and api/tasks.py.

Should it also work through the admin/ web interface as well?

All of the node providers/flavors are done in the same way, so yes :)

What about actually installing MySQL on the controller? You know, like with a recipe that does all the apt-get install stuff? So you actually have an a MySQL instance on the node? I'm not so sure about that. Ideally it would be its own node wouldn't it? Or at least you should have the option of putting it on its own node. Already I want to write a postgres and redis providers! And we can't have the controller being filled up with all kinds of services can we?

Installing MySQL can be just as easy as creating a new services recipe in https://github.com/opdemand/deis-cookbook :)

I think this needs a bigger discussion, as this is a particularly larger subject all on its own. As this was just a proof of concept, all communication was done locally with a base install of MySQL (you can even see the code here). Ideally, It'd be nice to have this all on a separate deis layer, with the operations engineer being able to specify what services he would like on each node (maybe he wants builtin-mysql and builtin-postgresql on one node, and builtin-redis on another), and then the controller could communicate with the nodes for provisioning requests and whatnot. The communication parts that are hardcoded at https://github.com/bacongobbler/service-registry/blob/master/provider/mysql.py#L12 could easily be moved back up to the Provider model (which would have to be renamed to ServiceProvider, I'd assume).

@gabrtv or @mboersma, any input from you guys on this topic?

[tombh]

More like a services folder at the root directory, much like how provider/digitalocean.py is set up with api/tasks.py. It would be the same for services/mysql.py and api/tasks.py.

Ah, got it!

So, a service layer, that definitely sounds like the right way to go. But yeah part of a bigger discussion. So there'll be CLI commands like deis services:enable mysql for instance? In fact, if that's the approach, then that's the point at which Provider.enabled = true should be set, right? Not with a migration.

Might deis services be a sort of convenience wrapper for deis layer? The thing about a mysql node though would be that there is no point in scaling it. Unless you get into sharding!

I imagine there could be arguments to deis services that specify which node to place the service on.

Definitely +1 for ServiceProvider naming convention, got to try and avoid confusion with the infrastructure Provider.

BTW I'm not wedded to implementing all this, I may not be the best person to do it. I'm just happy to do any grunt work.

[gabrtv]

@bacongobbler @tombh When I get some time to breathe I plan to give this a more thorough review including a deep dive into the code. Here is my early feedback:

1. It works? Awesome!
2. I agree with putting the services package at the top level of the Deis project parallel with provider
3. Until we split out authn/authz into its own micro-service, we can't really modularize the API endpoints. We should host the new API endpoints alongside the existing ones (rather than spinning up a new gateway component).
4. Single-point-of-failure shouldn't be an issue as the Controller will soon be scaleable horizontally on multiple containers across multiple machines.

On a related, service discovery via etcd is coming sooner than you might think! \o/

[tombh]

@gabrtv Thanks for the feedback :) I reckon I've got a good idea now of how to flesh out a prototype for this. I really want this on Danabox, so I'm going to crack on with it today and hopefully get it deployed there for some live testing soon. BTW, point 4 and containerisation in general is sounding awesome. I don't fully understand how etcd fits into the puzzle yet, but sounds exciting.

[onbjerg]

What is the temporary way to add databases etc. and access them via. your application?

[bacongobbler]

@onbjerg http://docs.deis.io/en/latest/using_deis/config-application/#attach-to-backing-services

[onbjerg]

@bacongobbler What if I want to run my database on Deis? Not possible yet?

Also, any news on this? Still researching, or?

[bacongobbler]

No news other than what's already been posted. Lots of feature requests with very little (wo)manpower is all :)

[jwaldrip]

Any chance this is getting priority with the recent EY acquisition?

[cddr]

In the meantime, how would people suggest sharing a development database setup with colleagues. To me, the best options seem to be:-

• setup a hosted instance on something like RDS and config the app to connect to that (would only work with teams small enough not to clobber each other's development databases).
• fork the repo and tweak the deis Vagrantfile to add an instance that provisions a database service and makes it available to the rest of the cluster

Am I missing anything better?

[bacongobbler]

Short from hosting it on your own infrastructure, that sounds correct to me.

[CloudSide]

@bacongobbler Where is the Service Registry repository?

[bacongobbler]

@CloudSide I deleted https://github.com/bacongobbler/service-registry as it was from a pre-production release of Deis (v0.7.0 or something) and is quite out of date. We weren't going to move forward with that branch. The idea now is to implement a service endpoint in the controller, which then will register and consume Cloud Foundry broker APIs since that project has been open-sourced and is used in production.

[CloudSide]

@bacongobbler How do I use Cloud Foundry broker APIs in deis ?

[bacongobbler]

it isn't implemented yet. This issue is to track the progress of that feature. :)

[CloudSide]

I'm looking forward

[pascalschwientek]

Any progress in the last 2 month on this?

[bacongobbler]

@pascalschwientek we've updated the roadmap, but all communication related to this feature will be laid out in this issue. No progress has been made.

[zeiv]

all communication related to this feature will be laid out in this issue

Is this still true? Say I want to create a Design Doc, should that be a separate issue or a continuation of this one?

[bacongobbler]

It will be a continuation of this thread. I will be tackling deis addons support once steward becomes a little more stable for general use. Right now it's in the middle of a major refactor after a face-to-face meeting with users from SIG-service-catalog.

[zeiv]

Ok sweet, thanks for the update.

[deis-admin]

Moved to deis/controller#1220.