This proposal, replacing #7730 and building on #8637, is to bring straightforward, Fig-inspired stack composition to the Docker client. The ultimate goal is to provide a default Docker development experience that is:

1. delightful
2. suitable for production orchestration in the “80% case”
3. compatible with Docker clustering

I’ve already implemented an alpha of the required functionality on my composition branch - though it’s not ready for prime time, everyone is very much encouraged to try it out, especially Fig users. Scroll to the bottom for test builds!

The basic idea of stack composition is that with a very simple configuration file which describes what containers you want your application to consist of, you can type a single command and Docker will do everything necessary to get it running.

Configuration file

A group.yml is used to describe your application. It looks a lot like fig.yml, except it also provides a name for the group that the containers will be created in:

name: rails-example

containers:
  db:
    image: postgres:latest
  web:
    build: .
    command: bundle exec rackup -p 3000
    volumes:
      - .:/myapp
    ports:
      - "3000:3000"
    links:
      - db

A container entry must specify either an image to be pulled or a build directory (but not both). Other configuration options mostly map to their docker run counterparts.

docker up

On top of the docker groups command described in #8637, there’s a new docker up command. It performs the following steps:

1. Parse group.yml
2. For each defined container, checks whether we have an image for it. If not, either build or pull it as needed.
3. Create/update the group and create/recreate the containers. (This is a single, idempotent POST to the group creation API endpoint - see “Group creation” below.)
4. Start the containers in dependency order (based on links and volumes_from declarations in group.yml).
5. Unless docker up was invoked with -d, attach to all containers and stream their aggregated log output until the user sends Ctrl-C, at which point attempt to stop all containers with SIGTERM. Subsequent Ctrl-Cs result in a SIGKILL.

Enhancements to existing CLI commands

A new syntax is introduced as a shorthand for referring to containers, images, build directories and groups. With the docker groups subcommands, : designates the group defined in group.yml. With other commands, :web designates the container named web in group.yml.

Here are some example commands with their longwinded/non-portable equivalents.

List containers in the current group:

$ docker ps rails-example
$ docker ps :

Rebuild the web image:

$ docker build -t rails-example-web .
$ docker build :web

Re-pull the db image:

$ docker pull postgres:latest
$ docker pull :db

Delete the web image:

$ docker rmi rails-example-web
$ docker rmi :web

Kill the web container:

$ docker rm rails-example/web
$ docker rm :web

Remove the whole group:

$ docker groups rm rails-example
$ docker groups rm :

Open a bash shell in the web container:

$ docker exec -ti rails-example/web bash
$ docker exec :web bash

Run a one-off container using web’s image and configuration:

$ docker build -t rails-example-web . && docker run -ti -v `pwd`:/myapp --link rails-example/db:db rails-example-web bash
$ docker run -ti :web bash

Group creation

Groups are created by posting JSON to the daemon with the configuration info for all containers. docker up posts to that endpoint with a JSON payload generated from group.yml, and is responsible for automatically building and pulling images before doing so. The following is essentially equivalent to docker up -d:

$ docker up --parse > group.json
$ docker groups create - < group.json
$ docker groups start <groupname>

Topics for discussion: an inexhaustive list

Including the group name in the file. I’m unsure about making this the default - lots of Fig users want to be able to do it, but I’m worried that it’ll hurt portability (we don’t do it with Dockerfile, and in my opinion it’s better off for it). Alternate approaches include using the basename of the current directory (like Fig does), or generating a name and storing it in a separate, unversioned file.

Clustering and production. People are already deploying single-host production sites with fig up -d, validating this general approach in simple scenarios, but we need to be sure that it’ll port well to a clustered Docker instance.

Scaling. I don't think an equivalent to fig scale is necessary on day one, but it will eventually be needed as Docker becomes a multi-host platform, so there shouldn't be anything in the design that'll make that difficult to implement later.

Test builds

• linux/amd64: http://cl.ly/2c3p40251H11/download/docker-1.3.1-dev-linux
• darwin/386: http://cl.ly/0P0s452H3t0x/download/docker-1.3.1-dev-darwin-386
• darwin/amd64: http://cl.ly/1U1A0W2e0Y3u/download/docker-1.3.1-dev-darwin-amd64

Here's how to test it out if you're running boot2docker. First, replace the binary in your VM:

$ boot2docker ssh
docker@boot2docker:~$ sudo -i
root@boot2docker:~# curl -LO http://cl.ly/2c3p40251H11/download/docker-1.3.1-dev-linux
root@boot2docker:~# /etc/init.d/docker stop
root@boot2docker:~# mv /usr/local/bin/docker ./docker-stable
root@boot2docker:~# mv ./docker-1.3.1-dev-linux /usr/local/bin/docker
root@boot2docker:~# chmod +x /usr/local/bin/docker
root@boot2docker:~# /etc/init.d/docker start
root@boot2docker:~# docker version   # both "Git commit"s should be 56d8c9e
root@boot2docker:~# exit
docker@boot2docker:~$ exit

Next, replace your client binary:

$ curl -LO http://cl.ly/1U1A0W2e0Y3u/download/docker-1.3.1-dev-darwin-amd64
$ mv /usr/local/bin/docker ./docker-stable
$ mv ./docker-1.3.1-dev-darwin-amd64 /usr/local/bin/docker
$ chmod +x /usr/local/bin/docker
$ docker version                     # both "Git commit"s should be 56d8c9e

Not yet implemented

There are a few things left to implement:

• Supporting volumes_from
• An equivalent to fig logs, which streams the aggregated logs of already-running containers - probably docker groups (attach|logs)
• docker groups (pause|unpause|restart), to complete the bulk operations
• Validation of the YAML file
• Nicer output for ps (giving more prominence to names over IDs, and perhaps doing away with the long list of names in the default display)
• Code cleanup
• Test coverage

Example app 1: Python/Redis counter

Here’s a sample app you can try:

app.py:

from flask import Flask
from redis import Redis

app = Flask(__name__)
redis = Redis(host="redis", port=6379)

@app.route('/')
def hello():
    redis.incr('hits')
    return 'Hello World! I have been seen %s times.' % redis.get('hits')

if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

requirements.txt:

flask
redis

Dockerfile:

FROM python:2.7
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt

group.yml:

name: counter

containers:
  web:
    build: .
    command: python app.py
    ports:
      - "5000:5000"
    volumes:
      - .:/code
    links:
      - redis
    environment:
      - PYTHONUNBUFFERED=1
  redis:
    image: redis:latest
    command: redis-server --appendonly yes

If you put those four files in a directory and type docker up, you should see everything start up:

It'll build the web image, pull the redis image, start both containers and stream their aggregated output. If you Ctrl-C, it'll shut them down.

Example app 2: Fresh Rails app

I’ve ported the Rails example from Fig. See composing_rails.md.

Code

To get hacking, check out the composition branch on my fork:

$ git checkout -b composition master
$ git pull [email protected]:aanand/docker.git composition