Difference between revisions of "Docker"

From CRIU
Jump to navigation Jump to search
(12 intermediate revisions by 2 users not shown)
Line 5: Line 5:
 
Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.  
 
Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.  
  
== Docker 1.10 ==
+
== Docker Experimental ==
  
The easiest way to try CRIU and Docker together is to install [https://github.com/boucher/docker/releases/tag/v1.10_2-16-16-experimental this pre-compiled version of Docker]. It's based on Docker 1.10, and built with the <code>DOCKER_EXPERIMENTAL</code> build tag.
+
Checkpoint & Restore is now available in the _experimental_ runtime mode for Docker. Simply start your docker daemon with '''--experimental''' to enable the feature.
 
 
To install, download the <code>docker-1.10.0-dev</code> binary to your system. You'll need to start a docker daemon from this binary, and then you can use the same binary to communicate with that daemon. To start a docker daemon, run a command something like this:
 
 
 
    docker-1.10.0-dev daemon -D --graph=/var/lib/docker-dev --host unix:///var/run/docker-dev.sock
 
 
 
The '''graph''' and '''host''' options will prevent colliding with an existing installation of Docker, but you can replace your existing docker if desired. In another shell, you can then connect to that daemon:
 
 
 
    docker-1.10.0-dev --host unix:///var/run/docker-dev.sock run -d busybox top
 
  
 
=== Dependencies ===
 
=== Dependencies ===
  
In addition to downloading the binary above (or compiling one yourself), you need '''CRIU''' installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are '''libprotobuf-c''' and '''libnl-3'''. Here's an output of <code>ldd</code> on my system:
+
In addition to installing version 1.13 of Docker, you need '''CRIU''' installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are '''libprotobuf-c''' and '''libnl-3'''. Here's an output of <code>ldd</code> on my system:
  
    # ldd `which criu`
+
$ ldd `which criu`
 
     linux-vdso.so.1 =>  (0x00007ffc09fda000)
 
     linux-vdso.so.1 =>  (0x00007ffc09fda000)
 
     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000)
 
     libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000)
Line 33: Line 25:
 
=== checkpoint ===  
 
=== checkpoint ===  
  
Creating a checkpoint is a top level Docker command with this new version of Docker. Here's an example that simply logs an integer in a loop.  
+
There's a top level <code>checkpoint</code> sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.
 +
 
 +
Here's an example of creating a checkpoint, from a container that simply logs an integer in a loop.
  
 
First, we create container:
 
First, we create container:
  
    docker run -d --name looper --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
+
$ docker run -d --name looper --security-opt seccomp:unconfined busybox \
 +
          /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
  
 
You can verify the container is running by printings its logs:
 
You can verify the container is running by printings its logs:
  
    docker logs looper
+
$ docker logs looper
  
 
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
 
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
  
    docker checkpoint looper
+
$ docker checkpoint create looper checkpoint1
  
 
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
 
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
Line 51: Line 46:
 
=== restore ===
 
=== restore ===
  
Like '''checkpoint''', '''restore''' is a top level command in this version of Docker. Continuing our example, let's restore the same container:
+
Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container '''start''' call. Here's an example:
  
    docker restore looper
+
$ docker start --checkpoint checkpoint1 looper
  
 
If we then print the logs, you should see they start from where we left off and continue to increase.  
 
If we then print the logs, you should see they start from where we left off and continue to increase.  
Line 59: Line 54:
 
==== Restoring into a '''new''' container ====
 
==== Restoring into a '''new''' container ====
  
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. Right now that is done with the <code>--force</code> option, in conjunction with the <code>--image-dir</code> option. Here's a slightly revised example from before:
+
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. This is done by providing a custom storage path with the <code>--checkpoint-dir</code> option. Here's a slightly revised example from before:
  
    $ docker run -d --name looper2 --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
+
$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
   
+
          /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
    # wait a few seconds to give the container an opportunity to print a few lines, then
+
    $ docker checkpoint --image-dir=/tmp/checkpoint1 looper2
+
# wait a few seconds to give the container an opportunity to print a few lines, then
   
+
$ docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2
    $ docker create --name looper-force --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
+
   
+
$ docker create --name looper-clone --security-opt seccomp:unconfined busybox \
    $ docker restore --force=true --image-dir=/tmp/checkpoint1 looper-force
+
          /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
 +
 +
$ docker start --checkpoint-dir=/tmp --checkpoint=checkpoint2 looper-clone
  
  
You should be able to print the logs from <code>looper-force</code> and see that they start from wherever the logs of <code>looper</code> end.
+
You should be able to print the logs from <code>looper-clone</code> and see that they start from wherever the logs of <code>looper</code> end.
  
 
=== usage ===
 
=== usage ===
  
    # docker checkpoint --help
+
Checkpoint
    Usage: docker checkpoint [OPTIONS] CONTAINER
 
    Checkpoint one or more running containers
 
        --help            Print usage
 
        --image-dir        directory for storing checkpoint image files
 
        --leave-running    leave the container running after checkpoint
 
        --work-dir        directory for storing log file
 
 
 
  
    # docker restore --help
+
  # docker checkpoint create --help
    Usage: docker restore [OPTIONS] CONTAINER
+
  Usage: docker checkpoint create [OPTIONS] CONTAINER CHECKPOINT
    Restore one or more checkpointed containers
 
        --force            bypass checks for current container state
 
        --help            Print usage
 
        --image-dir        directory to restore image files from
 
        --work-dir        directory for restore log
 
  
== Docker 1.12 ==
+
  Create a checkpoint from a running container
  
More detailed instructions on running checkpoint/restore with Docker in version 1.12 will be coming in the future, but in the meantime, you must build the version of Docker available in the '''docker-checkpoint-restore''' branch of [[User:Boucher|Boucher]]'s fork of Docker, [https://github.com/boucher/docker/tree/docker-checkpoint-restore available here]. Make sure to build with the env <code>DOCKER_EXPERIMENTAL=1</code>.
+
  Options:
 +
      --checkpoint-dir string  Use a custom checkpoint storage directory
 +
      --help                    Print usage
 +
      --leave-running          Leave the container running after checkpoint
  
The command line interface has changed from the 1.10 version. <code>docker checkpoint</code> is now an umbrella command for a few checkpoint operations. To create a checkpoint, use the <code>docker checkpoint create</code> command, which takes <code>container_id</code> and <code>checkpoint_id</code> as non-optional arguments. Example:
+
Restore
  
    docker checkpoint create my_container my_first_checkpoint
+
  # docker start --help
 +
  Usage: docker start [OPTIONS] CONTAINER [CONTAINER...]
  
Restoring a container is now performed just as an option to <code>docker start</code>. Although typically you may create and start a container in a single step using <code>docker run</code>, under the hood this is actually two steps: <code>docker create</code> followed by <code>docker start</code>. You can also call <code>start</code> on a container that was previously running and has since been stopped or killed. That looks something like this:
+
  Start one or more stopped containers
  
    docker start --checkpoint my_first_checkpoint my_container
+
  Options:
 +
  -a, --attach                  Attach STDOUT/STDERR and forward signals
 +
      --checkpoint string      Restore from this checkpoint
 +
      --checkpoint-dir string  Use a custom checkpoint storage directory
 +
      --detach-keys string      Override the key sequence for detaching a container
 +
      --help                    Print usage
 +
  -i, --interactive            Attach container's STDIN
  
 
== Integration Status ==  
 
== Integration Status ==  
Line 157: Line 152:
 
* {{torvalds.git|bd9b51e79c}} by Al Viro
 
* {{torvalds.git|bd9b51e79c}} by Al Viro
 
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
 
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
 
 
  
 
== External Checkpoint Restore ==
 
== External Checkpoint Restore ==
  
{{Note| External C/R was done as proof-of-concept.  Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}
+
{{Note| External C/R was done as proof-of-concept.  Its use is highly discouraged.}}
 
 
This approach is called external because it's happening external to the
 
Docker daemon.  After checkpoint, the Docker daemon thinks that the
 
container has exited.  After restore, the Docker daemon doesn't know that
 
the container is running again.  Therefore, commands such as
 
<code>docker ps, stop, kill</code> and <code>logs</code>
 
will not work correctly.
 
 
 
Starting with CRIU 1.3, it is possible to checkpoint and restore a
 
process tree running inside a Docker container.  However, it's
 
important to note that Docker needs native support for checkpoint
 
and restore in order to maintain its parent-child relationship and
 
to correctly keep track of container states.  In other words, while
 
CRIU can C/R a process tree, the restored tree will not become a
 
child of Docker and, from Docker's point of view, the container's
 
state will remain "Exited" (even after successful restore).
 
 
 
It's important to re-emphasize that by checkpointing and restoring
 
a Docker container, we mean C/R of a process tree running inside a
 
container, excluding the Docker daemon itself.  As CRIU currently
 
does not support nested PID namespaces, the C/R process tree cannot
 
include the Docker daemon which runs in the global PID namespace.
 
 
 
=== Command Line Options ===
 
 
 
In addition to the usual CRIU command line options used when
 
checkpointing and restoring a process tree, the following command
 
line options are needed for Docker containers.
 
 
 
==== <code>--root</code> ====
 
 
 
This option has been used in the past only for restore operations
 
that wanted to change the root of the mount namespace.  It was not
 
used for checkpoint operations.
 
 
 
However, because Docker by default uses the AUFS graph driver and
 
the AUFS module in the kernel reveals branch pathnames in
 
<code>/proc/''pid''/map_files</code>, option <code>--root</code>
 
is used to specify the root of the
 
mount namespace.  Once the kernel AUFS module is fixed, it won't
 
be necessary to specify this option anymore.
 
 
 
==== <code>--ext-mount-map</code> ====
 
 
 
This option is used to specify the path of the external bind mounts.
 
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with
 
source files outside the container's mount namespace.  Older versions
 
of Docker also bind mount <code>/.dockerinit</code>.
 
 
 
For example, assuming the default Docker configuration, <code>/etc/hostname</code>
 
in the container's mount namespace is bind mounted from the source
 
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.
 
 
 
==== <code>--manage-cgroups</code> ====
 
 
 
When a process tree exits after a checkpoint operation, the cgroups
 
that Docker had created for the container are removed.  This option
 
is needed during restore to move the process tree into its cgroups,
 
re-creating them if necessary.
 
 
 
==== <code>--evasive-devices</code> ====
 
 
 
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers
 
(i.e., <code>docker run -d ...</code>).  Since earlier versions of Docker used
 
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat
 
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.
 
 
 
==== <code>--inherit-fd</code> ====
 
 
 
For native C/R support, this option tells CRIU to let the restored process "inherit"
 
its specified file descriptor (instead of restoring from checkpoint).
 
 
 
=== Restore Prework for External C/R ===
 
 
 
Docker supports many storage drivers (AKA graph drivers) including
 
AUFS, Btrfs, ZFS, DeviceMapper, OverlayFS, and VFS.  The user can
 
specify his/her desired storage driver via the <code>DOCKER_DRIVER</code>
 
environment variable or the <code>-s (--storage-driver)</code> command
 
line option.
 
 
 
Currently C/R can only be done on containers using either AUFS, OverlayFS, or VFS.
 
In the following example, we assume AUFS.
 
 
 
When Docker notices that the container has exited (due to CRIU dump),
 
it dismantles the container's filesystem.  We need to set up the container's
 
filesystem again before attempting to restore.
 
 
 
=== An External C/R Example ===
 
 
 
Below is an example to show C/R operations for a shell script that
 
continuously appends a number to a file.  You can use tail -f to
 
see the process in action.
 
 
 
As you will see below, after restore, the process's parent is PID
 
1 (init), not Docker.  Also, although the process has been successfully
 
restored, Docker still thinks that the container has exited.
 
 
 
To set up the container's AUFS filesystem before restore, its branch
 
information should be saved before checkpointing the container.
 
For convenience, however, AUFS branch information is saved in the
 
dump.log file.  So we can examine dump.log to set up the filesystem
 
again.
 
 
 
For brevity, the 64-character long container ID is replaced by the
 
string <container_id> in the following lines.
 
 
 
<pre>
 
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'
 
<container_id>
 
$
 
$ docker ps
 
CONTAINER ID  IMAGE          COMMAND          CREATED        STATUS
 
168aefb8881b  busybox:latest  "/bin/sh -c 'i=0; 6 seconds ago  Up 4 seconds
 
$
 
$ sudo criu dump -o dump.log -v4 -t 17810 \
 
-D /tmp/img/<container_id> \
 
--root /var/lib/docker/aufs/mnt/<container_id> \
 
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
 
--ext-mount-map /etc/hosts:/etc/hosts \
 
--ext-mount-map /etc/hostname:/etc/hostname \
 
--ext-mount-map /.dockerinit:/.dockerinit \
 
--manage-cgroups \
 
--evasive-devices
 
$
 
$ sudo grep successful /tmp/img/<container_id>/dump.log
 
(00.020103) Dumping finished successfully
 
$
 
$ docker ps -a
 
CONTAINER ID  IMAGE          COMMAND          CREATED        STATUS
 
168aefb8881b  busybox:latest  "/bin/sh -c 'i=0; 6 minutes ago  Exited (-1) 4 minutes ago
 
$
 
$ sudo mount -t aufs -o br=\
 
/var/lib/docker/aufs/diff/<container_id>:\
 
/var/lib/docker/aufs/diff/<container_id>-init:\
 
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\
 
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\
 
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\
 
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\
 
none /var/lib/docker/aufs/mnt/<container_id>
 
$
 
$ sudo criu restore -o restore.log -v4 -d
 
-D /tmp/img/<container_id> \
 
--root /var/lib/docker/aufs/mnt/<container_id> \
 
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \
 
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \
 
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \
 
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \
 
--manage-cgroups \
 
--evasive-devices
 
$
 
$ sudo grep successful /tmp/img/<container_id>/restore.log
 
(00.424428) Restore finished successfully. Resuming tasks.
 
$
 
$ ps -ef | grep /bin/sh
 
root    18580    1  0 12:38 ?        00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
 
$
 
$ docker ps -a
 
CONTAINER ID  IMAGE          COMMAND          CREATED        STATUS
 
168aefb8881b  busybox:latest  "/bin/sh -c 'i=0; 7 minutes ago  Exited (-1) 5 minutes ago
 
$
 
</pre>
 
 
 
=== External C/R Helper Script ===
 
 
 
As seen in the above examples, the CRIU command line for checkpointing and
 
restoring a Docker container is pretty long.  For restore, there is also
 
an additional step to set up the root filesystem before invoking CRIU.
 
 
 
To automate the C/R process, there is a helper script in the contrib
 
subdirectory of CRIU sources, called docker_cr.sh.  In addition to
 
invoking CRIU, this helper script sets up the root filesystem for AUFS,
 
UnionFS, and VFS for restore.
 
 
 
With docker_cr.sh, all you have to provide is the container ID.
 
If you don't specify a container ID, docker_cr.sh will list all running
 
containers and prompt you to choose one.  Also, as shown in the help
 
output below, by setting the appropriate environment variable, it's
 
possible to tell docker_cr.sh which Docker and CRIU binaries to use,
 
where Docker's home directory is, and where CRIU should save and look
 
for its image files.
 
 
 
<pre>
 
# docker_cr.sh --help
 
Usage:
 
docker_cr.sh -c|-r [-hv] [<container_id>]
 
-c, --checkpoint checkpoint container
 
-h, --help print help message
 
-r, --restore restore container
 
-v, --verbose enable verbose mode
 
 
 
Environment:
 
DOCKER_HOME (default /var/lib/docker)
 
CRIU_IMG_DIR (default /var/lib/docker/criu_img)
 
DOCKER_BINARY (default docker)
 
CRIU_BINARY (default criu)
 
</pre>
 
 
 
Below is an example to checkpoint and restore Docker container 4397:
 
 
 
<pre>
 
# docker_cr.sh -c 4397
 
dump successful
 
# docker_cr.sh -r 4397
 
restore successful
 
</pre>
 
 
 
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code>
 
executes.  For example:
 
 
 
<pre>
 
# docker_cr.sh -c -v 40d3
 
docker binary: docker
 
criu binary: criu
 
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
 
 
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \
 
    --manage-cgroups --evasive-devices \
 
    --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
 
    --ext-mount-map /etc/hosts:/etc/hosts \
 
    --ext-mount-map /etc/hostname:/etc/hostname \
 
    --ext-mount-map /.dockerinit:/.dockerinit \
 
    -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
 
 
dump successful
 
(00.020827) Dumping finished successfully
 
 
 
# docker_cr.sh -r -v 40d3
 
docker binary: docker
 
criu binary: criu
 
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
 
 
mount -t aufs -o
 
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init
 
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721
 
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16
 
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229
 
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
 
none
 
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
 
 
 
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \
 
    -o restore.log --manage-cgroups --evasive-devices \
 
    --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \
 
    --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \
 
    --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \
 
    --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \
 
    -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \
 
    --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid
 
 
 
restore successful
 
(00.408807) Restore finished successfully. Resuming tasks.
 
 
 
root      6206    1  1 10:49 ?        00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
 
</pre>
 
 
 
  
[[Category:HOWTO]]
+
Although it's not recommended, you can also learn more about using CRIU without integrating with docker: [[Docker_External]].

Revision as of 21:07, 24 January 2017

This HOWTO page describes how to checkpoint and restore a Docker container.

Introduction

Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.

Docker Experimental

Checkpoint & Restore is now available in the _experimental_ runtime mode for Docker. Simply start your docker daemon with --experimental to enable the feature.

Dependencies

In addition to installing version 1.13 of Docker, you need CRIU installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are libprotobuf-c and libnl-3. Here's an output of ldd on my system:

$ ldd `which criu`
   	linux-vdso.so.1 =>  (0x00007ffc09fda000)
   	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000)
   	libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007fd28b0b7000)
   	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd28aeb2000)
   	libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fd28ac98000)
   	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd28a8d3000)
   	/lib64/ld-linux-x86-64.so.2 (0x000056386bb38000)
   	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd28a5cc000)

checkpoint

There's a top level checkpoint sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.

Here's an example of creating a checkpoint, from a container that simply logs an integer in a loop.

First, we create container:

$ docker run -d --name looper --security-opt seccomp:unconfined busybox  \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

You can verify the container is running by printings its logs:

$ docker logs looper

If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:

$ docker checkpoint create looper checkpoint1

You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.

restore

Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container start call. Here's an example:

$ docker start --checkpoint checkpoint1 looper

If we then print the logs, you should see they start from where we left off and continue to increase.

Restoring into a new container

Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. This is done by providing a custom storage path with the --checkpoint-dir option. Here's a slightly revised example from before:

$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

# wait a few seconds to give the container an opportunity to print a few lines, then
$ docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2

$ docker create --name looper-clone --security-opt seccomp:unconfined busybox \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

$ docker start --checkpoint-dir=/tmp --checkpoint=checkpoint2 looper-clone


You should be able to print the logs from looper-clone and see that they start from wherever the logs of looper end.

usage

Checkpoint

 # docker checkpoint create --help
 Usage:	docker checkpoint create [OPTIONS] CONTAINER CHECKPOINT
 Create a checkpoint from a running container
 Options:
     --checkpoint-dir string   Use a custom checkpoint storage directory
     --help                    Print usage
     --leave-running           Leave the container running after checkpoint

Restore

  # docker start --help
  Usage:	docker start [OPTIONS] CONTAINER [CONTAINER...]
 Start one or more stopped containers
 Options:
 -a, --attach                  Attach STDOUT/STDERR and forward signals
     --checkpoint string       Restore from this checkpoint
     --checkpoint-dir string   Use a custom checkpoint storage directory
     --detach-keys string      Override the key sequence for detaching a container
     --help                    Print usage
 -i, --interactive             Attach container's STDIN

Integration Status

CRIU has already been integrated into the lower level components that power Docker, namely runc and containerd. The final step in the process is to integrate with Docker itself. You can track the status of that process in this pull request.

Compatibility Notes

The latest versions of the Docker integration require at least version 2.0 of CRIU in order to work correctly. Additionally, depending on the storage driver being used by Docker, and other factors, there may be other compatibility issues that will attempt to be listed here.

TTY

Checkpointing an interactive container is currently not supported.

Seccomp

You'll notice that all of the above examples disable Docker's default seccomp support. In order to use seccomp, you'll need a newer version of the Kernel. **Update Needed with Exact Version**

OverlayFS

There is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. The following small kernel patches fix the mount id and symlink target path issue:

Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:

git clone  git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git
cd ubuntu-vivid
git remote add torvalds  git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git remote update

git cherry-pick 155e35d4da
git cherry-pick df1a085af1
git cherry-pick f25801ee46
git cherry-pick 4bacc9c923
git cherry-pick 9391dd00d1

cp /boot/config-$(uname -r) .config
make olddefconfig
make -j 8 bzImage modules
sudo make install modules_install
sudo reboot

Async IO

If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:

External Checkpoint Restore

Note.svg Note: External C/R was done as proof-of-concept. Its use is highly discouraged.

Although it's not recommended, you can also learn more about using CRIU without integrating with docker: Docker_External.