Getting a notification that Docker containers are down in production is one of the worst ways to spend your night. In today's article, we'll discuss how to use Docker's restart policy to automatically restart containers and avoid those late-night notifications.
What Happens When an Application Crashes?
Before we get started with Docker's restart policy, let's understand a bit more about how Docker behaves when an application crashes. To facilitate this, we'll create a Docker container that executes a simple bash script named crash.sh.
#/bin/bash sleep 30 exit 1
The above script is simple; when started, it will sleep for 30 seconds, and then it will exit with an exit code of 1 indicating an error.
Building and running a custom container
In order to run this script within a container, we'll need to build a custom Docker container which includes the crash.sh script. In order to build a custom container, we first need to create a simple Dockerfile.
$ vi Dockerfile
The Dockerfile will contain the following three lines:
FROM ubuntu:14.04 ADD crash.sh / CMD /bin/bash /crash.sh
The above Dockerfile will build a container based on the latest ubuntu:14.04 image. It will also add the crash.sh script into the / directory of the container. The final line tells Docker to execute the crash.sh script when the container is started.
With the Dockerfile defined, we can now build our custom container using the docker build command.
$ sudo docker build -t testing_restarts ./ Sending build context to Docker daemon 3.072 kB Step 1 : FROM ubuntu:14.04 ---> e36c55082fa6 Step 2 : ADD crash.sh / ---> eb6057d904ef Removing intermediate container 5199db00ba76 Step 3 : CMD /bin/bash /crash.sh ---> Running in 01e6f5e12c3f ---> 0e2f4ac52f19 Removing intermediate container 01e6f5e12c3f Successfully built 0e2f4ac52f19
This build command created a Docker image with a tagged name of testing_restarts. We can now start a container using the testing_restarts image by executing docker run.
$ sudo docker run -d --name testing_restarts testing_restarts a35bb16634a029039c8b34dddba41854e9a5b95222d32e3ca5624787c4c8914a
From the above, it appears that Docker was able to start a container named testing_restarts. Let's check the status of that container by running docker ps.
$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
The docker ps command doesn't show any running containers. The reason for this is because docker ps by default only shows running containers. Let's take a look at running and non-running containers by using the -a flag.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a35bb16634a0 testing_restarts "/bin/sh -c '/bin/bas" 9 minutes ago Exited (1) 8 minutes ago
With the docker ps results, we can see that when an application within a Docker container exits, that container is also stopped. This means that, by default, if an application that is running within a container crashes, the container stops and that container will remain stopped until someone or something restarts it.
https://js.hscta.net/cta/current.js
hbspt.cta.load(1169977, 'e3e6fa35-122e-4b9e-831b-43cecbca43cc', {});
Changing Docker's Default Behavior
It's possible to automatically restart crashed containers by specifying a restart policy when initiating the container. To understand restart policies better, let's see what happens when we use the always restart policy with this same container.
$ sudo docker run -d --name testing_restarts --restart always testing_restarts 8320e96172e4403cf6527df538fb7054accf3a55513deb12bb6a5535177c1f19
In the above command, we specified that Docker should apply the always restart policy to this container via the --restart flag. Let's see what effect this has on our container by executing a docker ps again.
$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8320e96172e4 testing_restarts "/bin/sh -c '/bin/bas" About a minute ago Up 21 seconds
This time we can see that the container is up and running but only for 21 seconds. If we run docker ps again, we will see something interesting.
$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8320e96172e4 testing_restarts "/bin/sh -c '/bin/bas" About a minute ago Up 19 seconds
The second run shows the container has only been up for 19 seconds. This means that even though our application (crash.sh) continues to exit with an error, Docker is continuously restarting the container every time it exits.
Now that we understand how restart policies can be used to change Docker's default behavior, let's take a look at what restart policies Docker has available.
Docker's Restart Policy(ies)
Docker currently has four restart policies:
noon-failureunless-stoppedalways
The no policy is the default restart policy and simply does not restart a container under any circumstance.
Restarting on failure but stopping on success
The on-failure policy is a bit interesting as it allows you to tell Docker to restart a container if the exit code indicates error but not if the exit code indicates success. You can also specify a maximum number of times Docker will automatically restart the container.
Let's try this restart policy out with our testing_restarts container and set a limit of 5 restarts.
$ sudo docker run -d --name testing_restarts --restart on-failure:5 testing_restarts 85ff2f096bac9965a9b8cffbb73c1642bf7b64a2173bbd145961231861b95819
If we run docker ps within a minute of launching the container, we will see that the container is running and has been recently started.
$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 85ff2f096bac testing_restarts "/bin/sh -c '/bin/bas" About a minute ago Up 8 seconds
The same will not be true, however, if we run the docker ps command 3 minutes after launching the container.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 85ff2f096bac testing_restarts "/bin/sh -c '/bin/bas" 3 minutes ago Exited (1) 20 seconds ago
We can see from the above that after 3 minutes the container is stopped. This is due to the fact that the container has been restarted more than our max-retries setting.
With success
The benefit of on-failures is that when an application exits with a successful exit code, the container will not be restarted. Let's see this in action by making a quick minor change to the crash.sh script.
$ vi crash.sh
The change will be to set the exit code to 0.
#/bin/bash sleep 30 exit 0
By setting the script to exit with a 0 exit code, we will be removing the error indicator from the script. Meaning as far as Docker can tell, this script will execute successfully every time.
With the script changed, we will need to rebuild the container before we can run it again.
$ sudo docker build -t testing_restarts ./ Sending build context to Docker daemon 3.072 kB Step 1 : FROM ubuntu:14.04 ---> e36c55082fa6 Step 2 : ADD crash.sh / ---> a4e7e4ad968f Removing intermediate container 88115fe05456 Step 3 : CMD /bin/bash /crash.sh ---> Running in fc8bbaffd9b9 ---> 8aaa3d99f432 Removing intermediate container fc8bbaffd9b9 Successfully built 8aaa3d99f432
With the container image rebuilt, let's launch this container again with the same on-failures and max-retries settings.
$ sudo docker run -d --name testing_restarts --restart on-failure:5 testing_restarts f0052e0c509dfc1c1b112c3b3717c23bc66db980f222144ca1c9a6b51cabdc19
This time, when we perform a docker ps -a execution, we should see some different results.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f0052e0c509d testing_restarts "/bin/sh -c '/bin/bas" 41 seconds ago Exited (0) 11 seconds ago
Since the crash.sh script exited with a successful exit code (0), Docker understood this as a success and did not restart the container.
Always restart the container
If we wanted the container to be restarted regardless of the exit code, we have a couple of restart policies we could use:
alwaysunless-stopped
The always restart policy tells Docker to restart the container under every circumstance. We experimented with the always restart policy earlier, but let's see what happens when we restart the current container with the always restart policy.
$ sudo docker run -d --name testing_restarts --restart always testing_restarts 676f12c9cd4cac7d3dd84d8b70734119ef956b3e5100b2449197c2352f3c4a55
If we wait for a few minutes and run docker ps -a again, we should see that the container has been restarted even with the exit code showing success.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9afad0ccd068 testing_restarts "/bin/sh -c '/bin/bas" 4 minutes ago Up 22 seconds
What's great about the always restart policy is that even if our Docker host was to crash on boot, the Docker service will restart our container. Let's see this in action to fully appreciate why this is useful.
$ sudo reboot
By default or even with on-failures, our container would not be running on reboot. Which, depending on what task the container performs, may be problematic.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 676f12c9cd4c testing_restarts "/bin/sh -c '/bin/bas" 9 minutes ago Up 2 seconds
With the always restart policy, that is not the case. The always restart policy will always restart the container. This is true even if the container has been stopped before the reboot. Let's look at that scenario in action.
$ sudo docker stop testing_restarts testing_restarts $ sudo reboot
Before rebooting our system, we simply stopped the container. This means the container is still there, just not running. Once the system is back up after our reboot however, the container will be running.
$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 676f12c9cd4c testing_restarts "/bin/sh -c '/bin/bas" 11 minutes ago Up 24 seconds
The reason our container is running after a reboot is because of the always policy. Whenever the Docker service is restarted, containers using the always policy will be restarted regardless of whether they were running or now.
The problem is that restarting a container that has been previously stopped after a reboot can be a bit problematic. What if our container was stopped for a valid reason, or worse, what if the container is out of date?
The solution for this is the unless-stopped restart policy.
Only stop when Docker is stopped
The unless-stopped restart policy behaves the same as always with one exception. When a container is stopped and the server is rebooted or the Docker service is restarted, the container will not be restarted.
Let's see this in action by starting the container with the unless-stopped policy and repeating our last example.
$ sudo docker run -d --name testing_restarts --restart unless-stopped testing_restarts fec5be52b9559b4f6421b10fe41c9c1dc3a16ff838c25d74238c5892f2b0b36
With the container running, let's stop it and reboot the system again.
$ sudo docker stop testing_restarts testing_restarts $ sudo reboot
This time when the system restarts, we should see the container is in a stopped state.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fec5be52b955 testing_restarts "/bin/sh -c '/bin/bas" 2 minutes ago Exited (137) About a minute ago
One important item with unless-stopped is that if the container was running before the reboot, the container would be restarted once the system restarted. We can see this in action by restarting our container and rebooting the system again.
$ sudo docker start testing_restarts testing_restarts $ sudo reboot
After this reboot, the container should be running.
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fec5be52b955 testing_restarts "/bin/sh -c '/bin/bas" 5 minutes ago Up 13 seconds testing_restarts
The difference between always and unless-stopped may be small, but in some environments this small difference may be a critical decision.
Selecting the Best Restart Policy
When selecting the best restart policy, it's important to keep in mind what type of workload the container is performing.
A Redis instance, for example, may be a critical component in your environment which should have an always or unless-stopped policy. On the other hand, a batch-processing application may need to be restarted until the process successfully completes. In this case, it would make sense to use the on-failures policy.
Either way, with Docker's restart policy you can now rest assured that next time a Docker host mysteriously reboots at 3 a.m., your containers will be restarted.
PS: If you liked this article you might also be interested in one of our free eBooks from our Codeship Resources Library. Download it here: Why Containers and Docker are the Future