Ensuring Containers are Always Running with Docker’s Restart Policy

Editor's Note: This is a guest blog by Ben Cane , published by permission. This post originally appeared on the CloudBees CodeShip website .

Getting a notification that Docker containers are down in production is one of the worst ways to spend your night. In today’s article, we’ll discuss how to use Docker’s restart policy to automatically restart containers and avoid those late-night notifications.

What happens when an application crashes?

Before we get started with Docker’s restart policy, let’s understand a bit more about how Docker behaves when an application crashes. To facilitate this, we’ll create a Docker container that executes a simple bash script named crash.sh.

#/bin/bash
sleep 30
exit 1

The above script is simple; when started, it will sleep for 30 seconds and then it will exit with an exit code of 1 indicating an error.

Building and running a custom container

In order to run this script within a container, we’ll need to build a custom Docker container which includes the crash.sh script. In order to build a custom container, we first need to create a simple Dockerfile .

$ vi Dockerfile

The Dockerfile will contain the following three lines:

FROM ubuntu:14.04
ADD crash.sh /
CMD /bin/bash /crash.sh

The above Dockerfile will build a container based on the latest ubuntu:14.04image . It will also add the crash.sh script into the / directory of the container. The final line tells Docker to execute the crash.sh script when the container is started.

With the Dockerfile defined, we can now build our custom container using the docker build command.

$ sudo docker build -t testing_restarts ./
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu:14.04
---> e36c55082fa6
Step 2 : ADD crash.sh /
---> eb6057d904ef Removing intermediate container 5199db00ba76
Step 3 : CMD /bin/bash /crash.sh
---> Running in 01e6f5e12c3f
---> 0e2f4ac52f19
Removing intermediate container 01e6f5e12c3f
Successfully built 0e2f4ac52f19

This build command created a Docker image with a tagged name of testing_restarts . We can now start a container using the testing_restarts image by executing docker run.

$ sudo docker run -d --name testing_restarts testing_restarts
a35bb16634a029039c8b34dddba41854e9a5b95222d32e3ca5624787c4c8914a

From the above, it appears that Docker was able to start a container named testing_restarts. Let’s check the status of that container by running docker ps.

$ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS

The docker ps command doesn’t show any running containers. The reason for this is because docker ps by default only shows running containers. Let’s take a look at running and non-running containers by using the -a flag.

With the docker ps results, we can see that when an application within a Docker container exits, that container is also stopped. This means that, by default, if an application that is running within a container crashes, the container stops and that container will remain stopped until someone or something restarts it.

Changing Docker’s default behavior

It’s possible to automatically restart crashed containers by specifying a restart policy when initiating the container. To understand restart policies better, let’s see what happens when we use the always restart policy with this same container.

$ sudo docker run -d --name testing_restarts --restart always testing_restarts
8320e96172e4403cf6527df538fb7054accf3a55513deb12bb6a5535177c1f19

In the above command, we specified that Docker should apply the always restart policy to this container via the--restart flag. Let’s see what effect this has on our container by executing a docker ps again.

This time we can see that the container is up and running but only for 21 seconds. If we run docker ps again, we will see something interesting.

The second run shows the container has only been up for 19 seconds. This means that even though our application (crash.sh ) continues to exit with an error, Docker is continuously restarting the container every time it exits.

Now that we understand how restart policies can be used to change Docker’s default behavior, let’s take a look at what restart policies Docker has available.

Docker’s restart policy(ies)

Docker currently has four restart policies:

• no

• on-failure

• unless-stopped

• always

The no policy is the default restart policy and simply does not restart a container under any circumstance.

Restarting on failure but stopping on success

The on-failure policy is a bit interesting as it allows you to tell Docker to restart a container if the exit code indicates error but not if the exit code indicates success. You can also specify a maximum number of times Docker will automatically restart the container.

Let’s try this restart policy out with our testing_restarts container and set a limit of 5 restarts.

$ sudo docker run -d --name testing_restarts --restart on-failure:5 testing_restarts
85ff2f096bac9965a9b8cffbb73c1642bf7b64a2173bbd145961231861b95819

If we run docker ps within a minute of launching the container, we will see that the container is running and has been recently started.

The same will not be true, however, if we run the docker ps command 3 minutes after launching the container.

We can see from the above that after 3 minutes the container is stopped. This is due to the fact that the container has been restarted more than our max-retries setting.

With success

The benefit of on-failures is that when an application exits with a successful exit code, the container will not be restarted. Let’s see this in action by making a quick minor change to the crash.sh script.

$ vi crash.sh

The change will be to set the exit code to 0.

#/bin/bash
sleep 30
exit 0

By setting the script to exit with a 0 exit code, we will be removing the error indicator from the script. Meaning as far as Docker can tell, this script will execute successfully every time.

With the script changed, we will need to rebuild the container before we can run it again.

$ sudo docker build -t testing_restarts ./
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM ubuntu:14.04
---> e36c55082fa6
Step 2 : ADD crash.sh /
---> a4e7e4ad968f
Removing intermediate container 88115fe05456
Step 3 : CMD /bin/bash /crash.sh
---> Running in fc8bbaffd9b9
---> 8aaa3d99f432
Removing intermediate container fc8bbaffd9b9
Successfully built 8aaa3d99f432

With the container image rebuilt, let’s launch this container again with the same** on-failures and max-retries** settings.

$ sudo docker run -d --name testing_restarts --restart on-failure:5 testing_restarts
f0052e0c509dfc1c1b112c3b3717c23bc66db980f222144ca1c9a6b51cabdc19

This time, when we perform a docker ps -a execution, we should see some different results.

Since the crash.sh script exited with a successful exit code (0 ), Docker understood this as a success and did not restart the container.

Always restart the container

If we wanted the container to be restarted regardless of the exit code, we have a couple of restart policies we could use:

• always

• unless-stopped

The always restart policy tells Docker to restart the container under every circumstance. We experimented with the always restart policy earlier, but let’s see what happens when we restart the current container with the always restart policy.

$ sudo docker run -d --name testing_restarts --restart always testing_restarts
676f12c9cd4cac7d3dd84d8b70734119ef956b3e5100b2449197c2352f3c4a55

If we wait for a few minutes and run docker ps -a again, we should see that the container has been restarted even with the exit code showing success.

What’s great about the always restart policy is that even if our Docker host was to crash on boot, the Docker service will restart our container. Let’s see this in action to fully appreciate why this is useful.

$ sudo reboot

By default or even with on-failures , our container would not be running on reboot. Which, depending on what task the container performs, may be problematic.

With the always restart policy, that is not the case. The always restart policy will always restart the container. This is true even if the container has been stopped before the reboot. Let’s look at that scenario in action.

$ sudo docker stop testing_restarts
testing_restarts
$ sudo reboot

Before rebooting our system, we simply stopped the container. This means the container is still there, just not running. Once the system is back up after our reboot however, the container will be running.

The reason our container is running after a reboot is because of the always policy. Whenever the Docker service is restarted, containers using the always policy will be restarted regardless of whether they were running or now.

The problem is that restarting a container that has been previously stopped after a reboot can be a bit problematic. What if our container was stopped for a valid reason, or worse, what if the container is out of date?

The solution for this is the unless-stopped restart policy.

Only stop when Docker is stopped

The unless-stopped restart policy behaves the same as always with one exception. When a container is stopped and the server is rebooted or the Docker service is restarted, the container will not be restarted.

Let’s see this in action by starting the container with the unless-stopped policy and repeating our last example.

$ sudo docker run -d --name testing_restarts --restart unless-stopped testing_restarts
fec5be52b9559b4f6421b10fe41c9c1dc3a16ff838c25d74238c5892f2b0b36

With the container running, let’s stop it and reboot the system again.

$ sudo docker stop testing_restarts
testing_restarts
$ sudo reboot

This time when the system restarts, we should see the container is in a stopped state.

One important item with unless-stopped is that if the container was running before the reboot, the container would be restarted once the system restarted. We can see this in action by restarting our container and rebooting the system again.

$ sudo docker start testing_restarts
testing_restarts
$ sudo reboot

After this reboot, the container should be running.

Selecting the best restart p olicy

The difference between always andunless-stopped may be small, but in some environments this small difference may be a critical decision.

When selecting the best restart policy, it’s important to keep in mind what type of workload the container is performing.

A Redis instance, for example, may be a critical component in your environment which should have an always or unless-stopped policy. On the other hand, a batch-processing application may need to be restarted until the process successfully completes. In this case, it would make sense to use the on-failures policy.

Either way, with Docker’s restart policy you can now rest assured that next time a Docker host mysteriously reboots at 3 a.m., your containers will be restarted.

Additional resources

Build Docker containers for Go applications

Setting up a CD pipeline using Docker containers

Learn more about CD with Jenkins and Docker