This HOWTO page describes how to checkpoint and restore a Docker container.
Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.
Checkpoint & Restore is now available in the _experimental_ runtime mode for Docker. Simply start your docker daemon with --experimental to enable the feature.
In addition to installing version 1.13 of Docker, you need CRIU installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are libprotobuf-c and libnl-3. Here's an output of
ldd on my system:
$ ldd `which criu` linux-vdso.so.1 => (0x00007ffc09fda000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000) libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007fd28b0b7000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd28aeb2000) libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fd28ac98000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd28a8d3000) /lib64/ld-linux-x86-64.so.2 (0x000056386bb38000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd28a5cc000)
There's a top level
checkpoint sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.
Here's an example of creating a checkpoint, from a container that simply logs an integer in a loop.
First, we create container:
$ docker run -d --name looper --security-opt seccomp:unconfined busybox \ /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
You can verify the container is running by printings its logs:
$ docker logs looper
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
$ docker checkpoint create looper checkpoint1
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container start call. Here's an example:
$ docker start --checkpoint checkpoint1 looper
If we then print the logs, you should see they start from where we left off and continue to increase.
Restoring into a new container
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. This is done by providing a custom storage path with the
--checkpoint-dir option. Here's a slightly revised example from before:
$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \ /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done' # wait a few seconds to give the container an opportunity to print a few lines, then $ docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2 $ docker create --name looper-clone --security-opt seccomp:unconfined busybox \ /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done' $ docker start --checkpoint-dir=/tmp --checkpoint=checkpoint2 looper-clone
You should be able to print the logs from
looper-clone and see that they start from wherever the logs of
# docker checkpoint create --help Usage: docker checkpoint create [OPTIONS] CONTAINER CHECKPOINT
Create a checkpoint from a running container
Options: --checkpoint-dir string Use a custom checkpoint storage directory --help Print usage --leave-running Leave the container running after checkpoint
# docker start --help Usage: docker start [OPTIONS] CONTAINER [CONTAINER...]
Start one or more stopped containers
Options: -a, --attach Attach STDOUT/STDERR and forward signals --checkpoint string Restore from this checkpoint --checkpoint-dir string Use a custom checkpoint storage directory --detach-keys string Override the key sequence for detaching a container --help Print usage -i, --interactive Attach container's STDIN
CRIU has already been integrated into the lower level components that power Docker, namely runc and containerd. The final step in the process is to integrate with Docker itself. You can track the status of that process in this pull request.
The latest versions of the Docker integration require at least version 2.0 of CRIU in order to work correctly. Additionally, depending on the storage driver being used by Docker, and other factors, there may be other compatibility issues that will attempt to be listed here.
Checkpointing an interactive container is currently not supported.
You'll notice that all of the above examples disable Docker's default seccomp support. In order to use seccomp, you'll need a newer version of the Kernel. **Update Needed with Exact Version**
There is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. The following small kernel patches fix the mount id and symlink target path issue:
155e35d4daby David Howells
df1a085af1by David Howells
f25801ee46by David Howells
4bacc9c923by David Howells
9391dd00d1by Al Viro
Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git cd ubuntu-vivid git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git git remote update git cherry-pick 155e35d4da git cherry-pick df1a085af1 git cherry-pick f25801ee46 git cherry-pick 4bacc9c923 git cherry-pick 9391dd00d1 cp /boot/config-$(uname -r) .config make olddefconfig make -j 8 bzImage modules sudo make install modules_install sudo reboot
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:
External Checkpoint Restore
|Note: External C/R was done as proof-of-concept. Its use is highly discouraged.|
Although it's not recommended, you can also learn more about using CRIU without integrating with docker: Docker_External.