Using Docker Compose for Python Development

Docker is an amazing tool for developers. It allows us to build and replicate images on any host, removing the inconsistencies of dev environments and reducing onboarding timelines considerably.

To provide an example of how you might move to containerized development, I built a simple todo API with Python, Django REST Framework, and PostgreSQL using Docker Compose for development, testing, and eventually in my CI/CD pipeline.

In a two-part series, I will cover the development and pipeline creation steps. In this post, I will cover the first part: developing and testing with Docker Compose.

This tutorial requires you to have a few items before you can get started.

• Install Docker Community Edition

• Install Docker Compose

• Download Todo app example -- Non-Docker branch

The todo app here is essentially a stand-in, and you could replace it with your own application. Some of the setup here is specific for this application. The needs of your application may not be covered, but it should be a good starting point for you to get the concepts needed to Dockerize your own applications.

Once you have everything set up, you can move on to the next section.

At the foundation of any Dockerized application, you will find a Dockerfile. The Dockerfile contains all of the instructions used to build out the application image. You can set this up by installing Python and all of its dependencies. However, the Docker ecosystem has an image repository with a Python image already created and ready to use.

In the root directory of the application, create a new Dockerfile.

/> touch Dockerfile

Open the newly created Dockerfile in your favorite editor. The first instruction, FROM, will tell Docker to use the prebuilt Python image. There are several choices, but this project uses the python:3.6.1-alpine image. For more details about why I'm using alpine here over the other options, you can read this post.

FROM python:3.6.1-alpine

If you run docker build ., you will see something similar to the following:

Sending build context to Docker daemon  10.53MB
Step 1/1 : FROM python:3.6.1-alpine
3.6.1-alpine: Pulling from library/python
90f4dba627d6: Pull complete
19bc0bb0be9f: Pull complete
e05eff433916: Pull complete
e70196200a87: Pull complete
a6d780959950: Pull complete
Digest: sha256:0945574465b917d524ce9b748479a286c2ed3c5a97311ac5950464907d4d8b53
Status: Downloaded newer image for python:3.6.1-alpine
 ---> ddd6300d05a3
Successfully built ddd6300d05a3

With only one instruction in the Dockerfile, this doesn't do too much, but it does show you the build process without too much happening. At this point, you now have an image created, and running docker images will show you the images you have available:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
python              3.6.1-alpine        ddd6300d05a3        5 weeks ago         88.7MB

The Dockerfile needs more instructions to build out the application. Currently it’s only creating an image with Python installed, but we still need our application code to run inside the container. Let's add some more instructions to do this and build this image again.

This particular Docker file uses RUN, COPY, and WORKDIR. You can read more about those on Docker's reference page to get a deeper understanding.

Let's add the instructions to the Dockerfile now:

FROM python:3.6.1-alpine
RUN apk update \
  && apk add \
    build-base \
    postgresql \
    postgresql-dev \
    libpq
RUN mkdir /usr/src/app
WORKDIR /usr/src/app
COPY ./requirements.txt .
RUN pip install -r requirements.txt
ENV PYTHONUNBUFFERED 1
COPY . .

Here is what is happening:

• Update apk packages and then install a few additional requirements

• Make the directory /usr/src/app

• Set the working directory /usr/src/app

• Copy requirements.txt to the working directory

• Run pip install with the requirements.txt file

• Set the environment variable PYTHONUNBUFFERED to 1

• Copy all the files from the project's root to the working directory

You can now run docker build . again and see the results:

Sending build context to Docker daemon  10.53MB
Step 1/8 : FROM python:3.6.1-alpine
 ---> ddd6300d05a3
Step 2/8 : RUN apk update   && apk add     build-base     postgresql     postgresql-dev     libpq
 ---> Running in 22bfdcf8a0dd
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
v3.4.6-165-g6b9a79f [http://dl-cdn.alpinelinux.org/alpine/v3.4/main]
v3.4.6-160-g14ad2a3 [http://dl-cdn.alpinelinux.org/alpine/v3.4/community]
OK: 5974 distinct packages available
## APK packages installed ##
Executing busybox-1.24.2-r13.trigger
OK: 215 MiB in 65 packages
 ---> 782357cafece
Removing intermediate container 22bfdcf8a0dd
Step 3/8 : ENV PYTHONUNBUFFERED 1
 ---> Running in 0d6de64f5f8b
 ---> 609106526013
Removing intermediate container 0d6de64f5f8b
Step 4/8 : RUN mkdir /usr/src/app
 ---> Running in b30ac1098156
 ---> 17def88f6c5f
Removing intermediate container b30ac1098156
Step 5/8 : WORKDIR /usr/src/app
 ---> 158d43ac2a47
Removing intermediate container fef446ea4ed0
Step 6/8 : COPY ./requirements.txt .
 ---> 2751d1f4c313
Removing intermediate container a49b090cb8e3
Step 7/8 : RUN pip install -r requirements.txt
 ---> Running in eaf9912e4810
## PIP Requirements Installed ##
Installing collected packages: pytz, Django, djangorestframework, psycopg2, django-cors-headers, dj-database-url, gunicorn, virtualenv
Successfully installed Django-1.11.3 dj-database-url-0.4.2 django-cors-headers-2.1.0 djangorestframework-3.6.3 gunicorn-19.7.1 psycopg2-2.7.2 pytz-2017.2 virtualenv-15.1.0
 ---> dfcca42a45b0
Removing intermediate container eaf9912e4810
Step 8/8 : COPY . .
 ---> 61f356fba3ea
Removing intermediate container fda87c3db8cf
Successfully built 61f356fba3ea

You have now successfully created the application image using Docker. Currently however, our app won't do much since we still need a database, and we want to connect everything together. This is where Docker Compose will help us out.

Now that you know how to create an image with a Dockerfile, let's create an application as a service and connect it to a database. Then we can run some setup commands and be on our way to creating that new todo list.

Create the file docker-compose.yml:

/> touch docker-compose.yml

The Docker Compose file will define and run the containers based on a configuration file. We are using compose file version 3 syntax, and you can read up on it on Docker's site.

An important concept to understand is that Docker Compose works at “runtime.” Up until now, we have been building images using docker build . -- this is “buildtime.” This is especially important when we add things like volumes and command because they will override what is set up at “buildtime.”

For example, the usr/src/app directory will be created during the build. We then map that directory to the host machine, and the host machine application code will be used during “runtime.” This allows us to make changes locally, and those are then accessible in the container.

Open your docker-compose.yml file in your editor, and copy paste the following lines:

version: '3'
services:
  web:
    build: .
    command: gunicorn -b 0.0.0.0:8000 todosapp.wsgi:application
    depends_on:
      - postgres
    volumes:
      - .:/usr/src/app
    ports:
      - "8000:8000"
    environment:
      DATABASE_URL: postgres://todoapp@postgres/todos
  postgres:
    image: postgres:9.6.2-alpine
    environment:
      POSTGRES_USER: todoapp
      POSTGRES_DB: todos

This will take a bit to unpack, but let's break it down by service.

The first directive in the web service is to build the image based on our Dockerfile. This will recreate the image we used before, but it will now be named according to the project we are in, name. After that, we are giving the service some specific instructions on how it should operate:

• command: gunicorn -b 0.0.0.0:8000 todosapp.wsgi:application - Once the image is built, and the container is running, this command will start the application.

• depends_on: - This will tell Docker Compose to start up the postgres service when the web service runs.

• volumes: - This section will mount paths between the host and the container.

• .:/usr/src/app - This will mount the root directory to our working directory in the container.

• environment: - The application itself expects the environment variable DATABASE_URL to run.

• ports: - This will publish the container's port, in this case 8000, to the host as port 8000.

The DATABASE_URL is the connection string. postgres://todoapp@postgres/todos connects using the todoapp user, on the host postgres, using the database todos.

Like the Python image we used, the Docker Store has a prebuilt image for PostgreSQL. Instead of using a build directive, we can use the name of the image, and Docker will grab that image for us and use it. In this case, we are using postgres:9.6.2-alpine. We could leave it like that, but it has environment variables to let us customize it a bit.

• environment: - This particular image accepts a couple environment variables so we can customize things to our needs.

• POSTGRES_USER: todoapp - This creates the user todoapp as the default user for PostgreSQL.

• POSTGRES_DB: todos - This will create the default database as todos.

Now that we have our services defined, we can build the application using docker-compose up. This will show the images being built and eventually starting. After the initial build, you will see the names of the containers being created.

Pulling postgres (postgres:9.6.2-alpine)...
9.6.2-alpine: Pulling from library/postgres
cfc728c1c558: Pull complete
b749e72b24f9: Pull complete
0abdb8c9c36b: Pull complete
90ca3848ef7e: Pull complete
3ecf037a5034: Pull complete
9327e3c5554c: Pull complete
3133782bad17: Pull complete
143bac6c8910: Pull complete
d6da9f4bd18e: Pull complete
Digest: sha256:f88000211e3c682e7419ac6e6cbd3a7a4980b483ac416a3b5d5ee81d4f831cc9
Status: Downloaded newer image for postgres:9.6.2-alpine
Building web
...
Creating pythondjangotodoapp_postgres_1 ...
Creating pythondjangotodoapp_postgres_1 ... done
Creating pythondjangotodoapp_web_1 ...
Creating pythondjangotodoapp_web_1 ... done
...
web_1       | [2017-08-03 13:23:18 +0000]  () [INFO] Starting gunicorn 19.7.1
web_1       | [2017-08-03 13:23:18 +0000]  () [INFO] Listening at: http://0.0.0.0:8000 (1)

At this point, the application is running, and you will see log output in the console. You can also run the services as a background process, using docker-compose up -d. During development, I prefer to run without -d and create a second terminal window to run other commands. If you want to run it as a background process and view the logs, you can run docker-compose logs.

At a new command prompt, you can run docker-compose ps to view your running containers. You should see something like the following:

             Name                           Command               State           Ports
------------------------------------------------------------------------------------------------
pythondjangotodoapp_postgres_1   docker-entrypoint.sh postgres    Up      5432/tcp
pythondjangotodoapp_web_1        gunicorn -b 0.0.0.0:8000 t ...   Up      0.0.0.0:8000->8000/tcp

This will tell you the name of the services, the command used to start it, its current state, and the ports. Notice pythondjangotodoapp_web_1 has listed the port as 0.0.0.0:8000->8000/tcp. This tells us that you can access the application using localhost:8000/todos on the host machine.

A small but important step not to overlook is the schema migration for the database. Compose comes with an exec command that will execute a one-off command on a running container. The typical function to migrate schemas is python manage.py migrate. We can run that on the web service using docker-compose exec.

/> docker-compose exec web python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, todos
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying sessions.0001_initial... OK
  Applying todos.0001_initial... OK

Now we can try out the API:

/> curl localhost:8000/todos
[]

The schema and all of the data in the container will persist as long as the postgres:9.6.2-alpine image is not removed. Eventually, however, it would be good to check how your app will build with a clean setup. You can run docker-compose down, which will clear things that are built and let you see what is happening with a fresh start.

Feel free to check out the source code, play around a bit, and see how things go for you.

The application itself includes some integration tests. There are various ways to go about testing with Docker, including creating Dockerfile.test and docker-compose.test.yml files specific for the test environment. That's a bit beyond the current scope of this article, but I want to show you how to run the tests using the current setup.

The current containers are running using the project name pythondjangotodoapp. This is a default from the directory name. If we attempt to run commands, it will use the same project, and containers will restart. This is what we don't want.

Instead, we will use a different project name to run the application, isolating the tests into their own environment. Since containers are ephemeral (short-lived), running your tests in a separate set of containers makes certain that your app is behaving exactly as it should in a clean environment.

In your terminal, run the following command:

/> docker-compose -p tests run -p 8000 --rm web python manage.py test
Starting tests_postgres_1 ... done
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
........
----------------------------------------------------------------------
Ran 8 tests in 0.086s
OK
Destroying test database for alias 'default'...

The docker-compose command accepts several options, followed by a command. In this case, you are using -p tests to run the services under the tests project name. The command being used is run, which will execute a one-time command against a service.

Since the docker-compose.yml file specifies a port, we use -p 8000 to create a random port to prevent port collision. The --rm option will remove the containers when we stop the containers. Finally, we are running in the web service python manage.py test.

At this point, you should have a solid start using Docker Compose for local app development. In the next part of this series about using Docker Compose for Python development, I will cover integration and deployments of this application using Codeship.

Is your team using Docker in its development workflow? If so, I would love to hear about what you are doing and what benefits you see as a result.