Open main menu




11,641 bytes removed, 12:42, 2 August 2016
no edit summary
== External Checkpoint Restore ==
{{Note| External C/R was done as proof-of-concept. Its use is highly discouraged and the helper script mentioned below will be deprecated in the near future.}}
This approach is called external because Although it's happening external to theDocker daemon. After checkpoint, the Docker daemon thinks that thecontainer has exited. After restore, the Docker daemon doesn't know thatthe container is running again. Therefore, commands such as<code>docker ps, stop, kill</code> and <code>logs</code>will not work correctly. Starting with CRIU 1.3recommended, it is possible to checkpoint and restore aprocess tree running inside a Docker container. However, it'simportant to note that Docker needs native support for checkpointand restore in order to maintain its parent-child relationship andto correctly keep track of container states. In other words, whileCRIU you can C/R a process tree, the restored tree will not become achild of Docker and, from Docker's point of view, the container'sstate will remain "Exited" (even after successful restore). It's important to re-emphasize that by checkpointing and restoringa Docker container, we mean C/R of a process tree running inside acontainer, excluding the Docker daemon itself. As CRIU currentlydoes not support nested PID namespaces, the C/R process tree cannotinclude the Docker daemon which runs in the global PID namespace. === Command Line Options === In addition to the usual CRIU command line options used whencheckpointing and restoring a process tree, the following commandline options are needed for Docker containers. ==== <code>--root</code> ==== This option has been used in the past only for restore operationsthat wanted to change the root of the mount namespace. It was notused for checkpoint operations. However, because Docker by default uses the AUFS graph driver andthe AUFS module in the kernel reveals branch pathnames in<code>/proc/''pid''/map_files</code>, option <code>--root</code>is used to specify the root of themount namespace. Once the kernel AUFS module is fixed, it won'tbe necessary to specify this option anymore. ==== <code>--ext-mount-map</code> ==== This option is used to specify the path of the external bind mounts.Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets withsource files outside the container's mount namespace. Older versionsof Docker also bind mount <code>/.dockerinit</code>. For example, assuming the default Docker configuration, <code>/etc/hostname</code>in the container's mount namespace is bind mounted from the sourceat <code>/var/lib/docker/containers/''container_id''/hostname</code>. ==== <code>--manage-cgroups</code> ==== When a process tree exits after a checkpoint operation, the cgroupsthat Docker had created for the container are removed. This optionis needed during restore to move the process tree into its cgroups,re-creating them if necessary. ==== <code>--evasive-devices</code> ==== Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<code>/dev/null</code> in the global namespace, this option tells CRIU to treatthe global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device. ==== <code>--inherit-fd</code> ==== For native C/R support, this option tells CRIU to let the restored process "inherit"its specified file descriptor (instead of restoring from checkpoint). === Restore Prework for External C/R === Docker supports many storage drivers (AKA graph drivers) includingAUFS, Btrfs, ZFS, DeviceMapper, OverlayFS, and VFS. The user canspecify his/her desired storage driver via the <code>DOCKER_DRIVER</code>environment variable or the <code>-s (--storage-driver)</code> commandline option. Currently C/R can only be done on containers learn more about using either AUFS, OverlayFS, or VFS.In the following example, we assume AUFS. When Docker notices that the container has exited (due to CRIU dump),it dismantles the container's filesystem. We need to set up the container'sfilesystem again before attempting to restore. === An External C/R Example === Below is an example to show C/R operations for a shell script thatcontinuously appends a number to a file. You can use tail -f tosee the process in action. As you will see below, after restore, the process's parent is PID1 (init), not Docker. Also, although the process has been successfullyrestored, Docker still thinks that the container has exited. To set up the container's AUFS filesystem before restore, its branchinformation should be saved before checkpointing the container.For convenience, however, AUFS branch information is saved in thedump.log file. So we can examine dump.log to set up the filesystemagain. For brevity, the 64-character long container ID is replaced by thestring <container_id> in the following lines. <pre>$ without integrating with docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<container_id>$ $ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds$ $ sudo criu dump -o dump.log -v4 -t 17810 \ -D /tmp/img/<container_id> \ --root /var/lib/docker/aufs/mnt/<container_id> \ --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \ --ext-mount-map /etc/hosts:/etc/hosts \ --ext-mount-map /etc/hostname:/etc/hostname \ --ext-mount-map /.dockerinit:/.dockerinit \ --manage-cgroups \ --evasive-devices$$ sudo grep successful /tmp/img/<container_id>/dump.log(00.020103) Dumping finished successfully$$ docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago$$ sudo mount -t aufs -o br=\/var/lib/docker/aufs/diff/<container_id>:\/var/lib/docker/aufs/diff/<container_id>-init:\/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\none /var/lib/docker/aufs/mnt/<container_id>$$ sudo criu restore -o restore.log -v4 -d -D /tmp/img/<container_id> \ --root /var/lib/docker/aufs/mnt/<container_id> \ --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \ --ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \ --ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \ --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \ --manage-cgroups \ --evasive-devices$$ sudo grep successful /tmp/img/<container_id>/restore.log(00.424428) Restore finished successfully. Resuming tasks.$$ ps -ef | grep /bin/shroot 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done$$ docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago$</pre> === External C/R Helper Script === As seen in the above examples, the CRIU command line for checkpointing andrestoring a Docker container is pretty long. For restore, there is alsoan additional step to set up the root filesystem before invoking CRIU. To automate the C/R process, there is a helper script in the contribsubdirectory of CRIU sources, called In addition toinvoking CRIU, this helper script sets up the root filesystem for AUFS,UnionFS, and VFS for restore. With, all you have to provide is the container ID.If you don't specify a container ID, will list all runningcontainers and prompt you to choose one. Also, as shown in the helpoutput below, by setting the appropriate environment variable, it'spossible to tell which Docker and CRIU binaries to use,where Docker's home directory is, and where CRIU should save and lookfor its image files. <pre># --helpUsage: -c|-r [-hv[Docker_External] [<container_id>] -c, --checkpoint checkpoint container -h, --help print help message -r, --restore restore container -v, --verbose enable verbose mode Environment: DOCKER_HOME (default /var/lib/docker) CRIU_IMG_DIR (default /var/lib/docker/criu_img) DOCKER_BINARY (default docker) CRIU_BINARY (default criu)</pre> Below is an example to checkpoint and restore Docker container 4397: <pre># -c 4397dump successful# -r 4397restore successful</pre> Optionally, you can specify <code>-v</code> to see the commands that <code></code>executes. For example: <pre># -c -v 40d3docker binary: dockercriu binary: criuimage directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bfcontainer root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \ --manage-cgroups --evasive-devices \ --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \ --ext-mount-map /etc/hosts:/etc/hosts \ --ext-mount-map /etc/hostname:/etc/hostname \ --ext-mount-map /.dockerinit:/.dockerinit \ -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf dump successful(00.020827) Dumping finished successfully # -r -v 40d3docker binary: dockercriu binary: criuimage directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bfcontainer root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf mount -t aufs -o/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158none/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \ -o restore.log --manage-cgroups --evasive-devices \ --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \ --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \ --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \ --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \ -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \ --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/ restore successful(00.408807) Restore finished successfully. Resuming tasks. root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done</pre>  [[Category:HOWTO]]