Using Docker to Build Debian Packages

Building Debian packages isn't always fun. If you've ever tried to turn some software into a .deb package, chances are you were either overwhelmed by the sheer number of available build tools and file formats or you messed up your system in an attempt to satisfy all of the package's dependencies. There are many things that can, and initially will, go wrong.

Fortunately, the right mix of automation and virtualization can make a big difference. In this article I will tell you the story behind a build system for Debian packages I developed at Jimdo that addresses many pain points we faced over the years.

When I started working at Jimdo in 2013, we had our own share of problems building Debian packages. We've relied on the Debian operating system to power our servers from day one. Debian itself already comes with thousands of packages ready to be installed. However, if the bundled software is outdated, we still need to recompile a newer version ourselves. In addition, we want to distribute our own tools as .deb files too. So being able to build and maintain Debian packages has always been important to us.

In order to create packages back then, engineers had to log into a dedicated build server, aptly named buildhost02, and then run a couple of shell commands as a special "buildcontroller" user. This process had several drawbacks: It was often hard to figure out who built a particular package or why it was built in the first place. And worst of all, we sometimes didn't know how a package was created. Apart from a loose collection of shell scripts, there was no unified way of creating packages, let alone a default location for the sources they stemmed from.

The list of problems makes it obvious that this build system wasn't going to last forever. Our build server, buildhost02, was hard to understand and modify, and setting up another server with the same functionality was quite a hassle (despite its name ending in "02," there was no other server should things go wrong). Moreover, as building usually involves installing multiple dependencies, the host was getting more and more contaminated over time. To sum it up, buildhost02 was what is called a unique snowflake.

In May 2014, I had the chance to start working on what we called the "Packaging 2.0" milestone. The milestone's goal was to develop a new and better build system based on our past experience. Later, this project became known as buildbox.

With buildbox, package builds were supposed to be:

• Automated. Engineers should be able to build package X for distribution Y by running a single command on their local machines.

• Documented. It should be obvious how and why a package was built and who is going to maintain it.

• Repeatable. It should be possible to reproduce every build of a package.

• Simple. Engineers, or really anybody, should be able to do the building.

To complete the milestone, we agreed that I not only had to rebuild all of our 100+ custom Debian packages with buildbox (a process that took a lot of reverse engineering), I also had to redeploy every single package to make sure that everything still worked as expected.

When it comes to packaging software, there's no one-size-fits-all tool that supports every programming language out there.

At Jimdo, we're using a myriad of languages, most prominently PHP, Python, Ruby, and Go. Every language has its own packaging standards and tooling around it (which is sad). As a result, build instructions can be found in many places -- Makefiles, debian/rules files, shell scripts, etc. -- and they often depend on other software being installed beforehand.

This diversity represented a challenge I had to tackle early on. I knew that having to support different build mechanisms would require some form of abstraction.

This is how the buildtasks library came to be. Buildtasks is a Rubygem that allows you to build Debian packages via a set of Rake tasks. It currently supports two ways of building:

• git-buildpackage

• fpm-cookery

The former is ideal for building packages that already come with a debian/ directory (as all Debian upstream packages do). The latter is useful for easily creating packages from all sorts of code that hasn't been "debianized." At Jimdo, we use both methods heavily.

The buildtasks gem provides a convenient Ruby DSL to abstract the actual steps involved in package building. Here's a sample Rakefile utilizing the gem's support for git-buildpackage:

# Rakefile
require "buildtasks"
BuildTasks::GitBuildpackage.define do
  name "periodicnoise"
  version "debian/1.1"
  source "https://github.com/Jimdo/periodicnoise"
end

This Ruby code will automatically generate a handful of Rake tasks to create a Debian package for the given project, periodicnoise. You can list the tasks this way:

$ rake -T
rake build    # Build packages
rake clean    # Remove any temporary products
rake clobber  # Remove any generated file
rake clone    # Clone Git repository and checkout version
rake deps     # Install build dependencies
rake patch    # Apply any patches
rake publish  # Publish built packages

Given these tasks, here's how you would trigger a package build:

$ rake build publish PUBLISH_DIR=/some/path

Under the hood, the buildtasks library will clone the defined Git repository from GitHub, check out the specified version (debian/1.1), install all build dependencies listed in debian/control, and finally invoke git-buildpackage to build the package. Afterwards, the package gets copied to the directory specified by the PUBLISH_DIR environment variable. All of that complexity is hidden from the user, whose job it is to provide a Rakefile with proper attributes.

But buildtasks does not only abstract away the build steps, it also provides a common interface. No matter what build mechanism you choose -- git-buildpackage or fpm-cookery or another tool added in the future -- the generated Rake tasks will always include two operations easy to understand and automate for humans and machines: build and publish. In fact, you don't necessarily need buildtasks to satisfy this interface. You could also provide your own Rake tasks with the same names.

Buildtasks is open source. You can install it via gem install buildtasks and read the full documentation on GitHub.

While buildtasks provides the logic to create Debian packages, it's just one, albeit important, component of buildbox. Another crucial piece is the environment used for building. Above all, this environment needs to be isolated -- a minimal installation of the target OS with as few preinstalled packages as possible. (Package-specific build dependencies should be installed as part of the actual build process, and not before. Only then will you notice broken or missing dependencies.)

For these reasons, it's a good idea to pair buildtasks with some form of server virtualization. Virtual machines are disposable and can usually be provisioned in minutes. More importantly, they provide a clean isolated environment ideal for package building.

When I implemented the first version of buildbox, I decided to use Vagrant together with VirtualBox. Over time it became clear, however, that VMs weren't the best fit for our needs.

I realized this when I finally decided to give Docker a try. Rather than having to wait minutes for a new VM to be ready for work (we used Puppet to provision a base VirtualBox VM), Docker containers can be brought up in a moment's notice. Even better: For each build, the environment is guaranteed to be pristine. We no longer had to resort to ugly workarounds like reusing existing VMs and sharing dependencies between builds to speed up the whole process. Docker was the missing piece allowing us to quickly build packages in a predictable and reproducible way.

hbspt.cta.load(1169977, '11903a5d-dfb4-42f2-9dea-9a60171225ca');

We're equipped with the buildtasks gem, and we know that Docker is the right technology for the task at hand. Now all we need to do is combine both. What follows is a demo showing you how to achieve this with Docker Compose. Compose is a useful tool for describing, building, and running container applications with Docker. It's designed for managing multi-container applications but works equally well with a single container like the one we're going to use for the demo.

First of all, we need to write a Dockerfile defining the environment we will use for package building. Remember that we strive to reduce preinstalled software to a minimum. Therefore, we will only install the buildtasks gem along with git-buildpackage and fpm-cookery, as well as some other required dependencies. For the sake of this demo, our Docker image will be based on the debian:jessie image from the official "debian" repository on Docker Hub. In general, you should use an image of the OS you want to build packages for.

Here's the final Dockerfile:

# Dockerfile
FROM debian:jessie
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y \
    build-essential \
    curl \
    devscripts \
    equivs \
    git-buildpackage \
    git \
    lsb-release \
    make \
    openssh-client \
    pristine-tar \
    rake \
    rsync \
    ruby \
    ruby-dev \
    rubygems \
    wget
RUN echo "gem: --no-ri --no-rdoc" >/etc/gemrc
RUN gem install fpm -v 1.4.0
RUN gem install fpm-cookery -v 0.29.0
RUN gem install buildtasks -v 0.0.1
RUN gem install bundler -v 1.10.0

As the next step, we create a file named docker-compose.yml, which tells Compose what to do. The file looks like this:

# docker-compose.yml
demo:
    build: .
    command: rake build publish PUBLISH_DIR=/pkg
    volumes:
        - .:/data
        - pkg:/pkg
    working_dir: /data

Last but not least, we need a Rakefile with build instructions. For simplicity, we're going to use the one for periodicnoise I showed you earlier:

# Rakefile
require "buildtasks"
BuildTasks::GitBuildpackage.define do
    name "periodicnoise"
    version "debian/1.1"
    source "https://github.com/Jimdo/periodicnoise"
end

With these three files in our hands, we can finally instruct Compose to build and run our demo container application. This process essentially boils down to creating a Docker image and then executing rake build publish inside a container based on that image. By using two data volumes, we ensure that the current Rakefile is available inside the container (at /data/Rakefile) and that all built packages end up in pkg/ on the host system.

This is the output of Compose:

$ docker-compose build
Building demo...
...
Successfully built a5d21c6f8085
$ docker-compose run demo
git clone https://github.com/Jimdo/periodicnoise git-debian-1.1
...
cd git-debian-1.1
git checkout -qf debian/1.1
...
DEBIAN_FRONTEND=noninteractive mk-build-deps -i -r -t 'apt-get -y' debian/control
...
git-buildpackage --git-ignore-branch --git-ignore-new --git-builder=debuild -i.git -I.git -uc -us -b
...
dpkg-deb: building package `periodicnoise' in`../periodicnoise_1.1_amd64.deb'.
...
mkdir -p /pkg
cp periodicnoise_1.1_amd64.deb /pkg

Afterwards, we can verify that the package was successfully built and saved to our local pkg/ folder:

$ ls -1 pkg/
periodicnoise_1.1_amd64.deb

If you want to try this yourself, you can find the demo's source code here.

In this article I showed you the two main ingredients of buildbox -- Jimdo's build system for Debian packages -- and how they can be used together: There's the buildtasks gem, which automates and abstracts the build steps, and there's Docker, which provides an isolated environment instantly available for building.

Buildbox itself isn't open source (yet), but essentially it's a Git repository containing a collection of Dockerfiles, Rakefiles, and Makefiles, as well as shell scripts gluing everything together. With buildbox, it only takes a single command for engineers to build the packages of project X for Debian distribution Y, and just another command to push the results to our internal Debian package repository.

I hope that, after reading this far, you too will be able to develop something that makes package building a bit more fun. The technology is already out there -- it's ready to be put to good use.