Docker External

Warning: External C/R was done as a proof-of-concept. Do not use it; please see Docker for a proper way to do checkpoint/restore with Docker. This article is kept here for historical purposes and some background info.

This approach is called external because it's happening external to the Docker daemon. After checkpoint, the Docker daemon thinks that the container has exited. After restore, the Docker daemon doesn't know that the container is running again. Therefore, commands such as docker ps, stop, kill and logs will not work correctly.

Starting with CRIU 1.3, it is possible to checkpoint and restore a process tree running inside a Docker container. However, it's important to note that Docker needs native support for checkpoint and restore in order to maintain its parent-child relationship and to correctly keep track of container states. In other words, while CRIU can C/R a process tree, the restored tree will not become a child of Docker and, from Docker's point of view, the container's state will remain "Exited" (even after successful restore).

It's important to re-emphasize that by checkpointing and restoring a Docker container, we mean C/R of a process tree running inside a container, excluding the Docker daemon itself. As CRIU currently does not support nested PID namespaces, the C/R process tree cannot include the Docker daemon which runs in the global PID namespace.

Command Line Options

In addition to the usual CRIU command line options used when checkpointing and restoring a process tree, the following command line options are needed for Docker containers.

--root

This option has been used in the past only for restore operations that wanted to change the root of the mount namespace. It was not used for checkpoint operations.

However, because Docker by default uses the AUFS graph driver and the AUFS module in the kernel reveals branch pathnames in /proc/pid/map_files, option --root is used to specify the root of the mount namespace. Once the kernel AUFS module is fixed, it won't be necessary to specify this option anymore.

--ext-mount-map

This option is used to specify the path of the external bind mounts. Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with source files outside the container's mount namespace. Older versions of Docker also bind mount /.dockerinit.

For example, assuming the default Docker configuration, /etc/hostname in the container's mount namespace is bind mounted from the source at /var/lib/docker/containers/container_id/hostname.

--manage-cgroups

When a process tree exits after a checkpoint operation, the cgroups that Docker had created for the container are removed. This option is needed during restore to move the process tree into its cgroups, re-creating them if necessary.

--evasive-devices

Docker bind mounts /dev/null on /dev/stdin for detached containers (i.e., docker run -d ...). Since earlier versions of Docker used /dev/null in the global namespace, this option tells CRIU to treat the global /dev/null and the container /dev/null as the same device.

--inherit-fd

For native C/R support, this option tells CRIU to let the restored process "inherit" its specified file descriptor (instead of restoring from checkpoint).

Restore Prework for External C/R

Docker supports many storage drivers (AKA graph drivers) including AUFS, Btrfs, ZFS, DeviceMapper, OverlayFS, and VFS. The user can specify his/her desired storage driver via the DOCKER_DRIVER environment variable or the -s (--storage-driver) command line option.

Currently C/R can only be done on containers using either AUFS, OverlayFS, or VFS. In the following example, we assume AUFS.

When Docker notices that the container has exited (due to CRIU dump), it dismantles the container's filesystem. We need to set up the container's filesystem again before attempting to restore.

An External C/R Example

Below is an example to show C/R operations for a shell script that continuously appends a number to a file. You can use tail -f to see the process in action.

As you will see below, after restore, the process's parent is PID 1 (init), not Docker. Also, although the process has been successfully restored, Docker still thinks that the container has exited.

To set up the container's AUFS filesystem before restore, its branch information should be saved before checkpointing the container. For convenience, however, AUFS branch information is saved in the dump.log file. So we can examine dump.log to set up the filesystem again.

For brevity, the 64-character long container ID is replaced by the string <container_id> in the following lines.

$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'
<container_id>
$ 
$ docker ps
CONTAINER ID  IMAGE           COMMAND           CREATED        STATUS
168aefb8881b  busybox:latest  "/bin/sh -c 'i=0; 6 seconds ago  Up 4 seconds
$ 
$ sudo criu dump -o dump.log -v4 -t 17810 \
	-D /tmp/img/<container_id> \
	--root /var/lib/docker/aufs/mnt/<container_id> \
	--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
	--ext-mount-map /etc/hosts:/etc/hosts \
	--ext-mount-map /etc/hostname:/etc/hostname \
	--ext-mount-map /.dockerinit:/.dockerinit \
	--manage-cgroups \
	--evasive-devices
$
$ sudo grep successful /tmp/img/<container_id>/dump.log
(00.020103) Dumping finished successfully
$
$ docker ps -a
CONTAINER ID  IMAGE           COMMAND           CREATED        STATUS
168aefb8881b  busybox:latest  "/bin/sh -c 'i=0; 6 minutes ago  Exited (-1) 4 minutes ago
$
$ sudo mount -t aufs -o br=\
/var/lib/docker/aufs/diff/<container_id>:\
/var/lib/docker/aufs/diff/<container_id>-init:\
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\
none /var/lib/docker/aufs/mnt/<container_id>
$
$ sudo criu restore -o restore.log -v4 -d
	-D /tmp/img/<container_id> \
	--root /var/lib/docker/aufs/mnt/<container_id> \
	--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \
	--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \
	--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \
	--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \
	--manage-cgroups \
	--evasive-devices
$
$ sudo grep successful /tmp/img/<container_id>/restore.log
(00.424428) Restore finished successfully. Resuming tasks.
$
$ ps -ef | grep /bin/sh
root     18580     1  0 12:38 ?        00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
$
$ docker ps -a
CONTAINER ID  IMAGE           COMMAND           CREATED        STATUS
168aefb8881b  busybox:latest  "/bin/sh -c 'i=0; 7 minutes ago  Exited (-1) 5 minutes ago
$

External C/R Helper Script

As seen in the above examples, the CRIU command line for checkpointing and restoring a Docker container is pretty long. For restore, there is also an additional step to set up the root filesystem before invoking CRIU.

To automate the C/R process, there is a helper script in the contrib subdirectory of CRIU sources, called docker_cr.sh. In addition to invoking CRIU, this helper script sets up the root filesystem for AUFS, UnionFS, and VFS for restore.

With docker_cr.sh, all you have to provide is the container ID. If you don't specify a container ID, docker_cr.sh will list all running containers and prompt you to choose one. Also, as shown in the help output below, by setting the appropriate environment variable, it's possible to tell docker_cr.sh which Docker and CRIU binaries to use, where Docker's home directory is, and where CRIU should save and look for its image files.

# docker_cr.sh --help
Usage:
	docker_cr.sh -c|-r [-hv] [<container_id>]
	-c, --checkpoint	checkpoint container
	-h, --help		print help message
	-r, --restore		restore container
	-v, --verbose		enable verbose mode

Environment:
	DOCKER_HOME		(default /var/lib/docker)
	CRIU_IMG_DIR		(default /var/lib/docker/criu_img)
	DOCKER_BINARY		(default docker)
	CRIU_BINARY		(default criu)

Below is an example to checkpoint and restore Docker container 4397:

# docker_cr.sh -c 4397
dump successful
# docker_cr.sh -r 4397
restore successful

Optionally, you can specify -v to see the commands that docker_cr.sh executes. For example:

# docker_cr.sh -c -v 40d3
docker binary: docker
criu binary: criu
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \
     --manage-cgroups --evasive-devices \
     --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
     --ext-mount-map /etc/hosts:/etc/hosts \
     --ext-mount-map /etc/hostname:/etc/hostname \
     --ext-mount-map /.dockerinit:/.dockerinit \
     -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

dump successful
(00.020827) Dumping finished successfully

# docker_cr.sh -r -v 40d3
docker binary: docker
criu binary: criu
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

mount -t aufs -o
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
none
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \
     -o restore.log --manage-cgroups --evasive-devices \
     --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \
     --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \
     --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \
     --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \
     -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \
     --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid

restore successful
(00.408807) Restore finished successfully. Resuming tasks.

root      6206     1  1 10:49 ?        00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done

See also

• Docker