275
edits
Changes
Jump to navigation
Jump to search
mLine 1:
Line 1: − + − + − + + − == Docker 1.10 ==+ − The easiest way to try CRIU and Docker together is to install [https://github.com/boucher/docker/releases/tag/v1.10_2-16-16-experimental this pre-compiled version of Docker]. It's based on Docker 1.10, and built with the <code>DOCKER_EXPERIMENTAL</code> build tag. + + − To install, download the <code>docker-1.10.0-dev</code> binary to your system. You'll need to start a docker daemon from this binary, and then you can use the same binary to communicate with that daemon. To start a docker daemon, run a command something like this:+ − $ docker-1.10.0-dev daemon -D --graph=/var/lib/docker-dev --host unix:///var/run/docker-dev.sock+ − The '''graph''' and '''host''' options will prevent colliding with an existing installation of Docker, but you can replace your existing docker if desired. In another shell, you can then connect to that daemon:+ − $ docker-1.10.0-dev --host unix:///var/run/docker-dev.sock run -d busybox top+ − − === Dependencies === − − In addition to downloading the binary above (or compiling one yourself), you need '''CRIU''' installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are '''libprotobuf-c''' and '''libnl-3'''. Here's an output of <code>ldd</code> on my system: − − $ ldd `which criu` − linux-vdso.so.1 => (0x00007ffc09fda000) − libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000) − libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007fd28b0b7000) − libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd28aeb2000) − libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fd28ac98000) − libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd28a8d3000) − /lib64/ld-linux-x86-64.so.2 (0x000056386bb38000) − libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd28a5cc000) − − === checkpoint === − − Creating a checkpoint is a top level Docker command with this new version of Docker. Here's an example that simply logs an integer in a loop. From this point forward, commands are show using '''docker''' instead of '''docker-dev-1.10''', but if you have not installed this version globally you can use the latter. Line 46:
Line 30: − + Line 52:
Line 36: − Like '''checkpoint''', '''restore''' is a top level command in this version of Docker. Continuing our example, let's restore the same container:+ − + Line 60:
Line 44: − + Line 66:
Line 50: − + − + − + + + + + − You should be able to print the logs from <code>looper-force</code> and see that they start from wherever the logs of <code>looper</code> end.+ − === usage ===+ − # docker checkpoint --help+ − Usage: docker checkpoint [OPTIONS] CONTAINER − − Checkpoint one or more running containers − − --help Print usage − --image-dir directory for storing checkpoint image files − --leave-running leave the container running after checkpoint − --work-dir directory for storing log file − # docker restore --help+ − Usage: docker restore [OPTIONS] CONTAINER − − Restore one or more checkpointed containers − − --force bypass checks for current container state − --help Print usage − --image-dir directory to restore image files from − --work-dir directory for restore log − == Docker 1.12 ==+ − More detailed instructions on running checkpoint/restore with Docker in version 1.12 will be coming in the future, but in the meantime, you must build the version of Docker available in the '''docker-checkpoint-restore''' branch of [[User:Boucher|Boucher]]'s fork of Docker, [https://github.com/boucher/docker/tree/docker-checkpoint-restore available here]. Make sure to build with the env <code>DOCKER_EXPERIMENTAL=1</code>.+ + − The command line interface has changed from the 1.10 version. <code>docker checkpoint</code> is now an umbrella command for a few checkpoint operations. To create a checkpoint, use the <code>docker checkpoint create</code> command, which takes <code>container_id</code> and <code>checkpoint_id</code> as non-optional arguments. Example:+ − docker checkpoint create my_container my_first_checkpoint+ + + + − Restoring a container is now performed just as an option to <code>docker start</code>. Although typically you may create and start a container in a single step using <code>docker run</code>, under the hood this is actually two steps: <code>docker create</code> followed by <code>docker start</code>. You can also call <code>start</code> on a container that was previously running and has since been stopped or killed. That looks something like this:+ − docker start --checkpoint my_first_checkpoint my_container+ + − == Integration Status == + − CRIU has already been integrated into the lower level components that power Docker, namely '''runc''' and '''containerd'''. The final step in the process is to integrate with Docker itself. You can track the status of that process in [https://github.com/docker/docker/pull/22049 this pull request].+ + + + + + + Line 120:
Line 103: − + + Line 164:
Line 148: + − + − == External Checkpoint Restore == − − {{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}} − − This approach is called external because it's happening external to the − Docker daemon. After checkpoint, the Docker daemon thinks that the − container has exited. After restore, the Docker daemon doesn't know that − the container is running again. Therefore, commands such as − <code>docker ps, stop, kill</code> and <code>logs</code> − will not work correctly. − − Starting with CRIU 1.3, it is possible to checkpoint and restore a − process tree running inside a Docker container. However, it's − important to note that Docker needs native support for checkpoint − and restore in order to maintain its parent-child relationship and − to correctly keep track of container states. In other words, while − CRIU can C/R a process tree, the restored tree will not become a − child of Docker and, from Docker's point of view, the container's − state will remain "Exited" (even after successful restore). − − It's important to re-emphasize that by checkpointing and restoring − a Docker container, we mean C/R of a process tree running inside a − container, excluding the Docker daemon itself. As CRIU currently − does not support nested PID namespaces, the C/R process tree cannot − include the Docker daemon which runs in the global PID namespace. − − === Command Line Options === − − In addition to the usual CRIU command line options used when − checkpointing and restoring a process tree, the following command − line options are needed for Docker containers. − − ==== <code>--root</code> ==== − − This option has been used in the past only for restore operations − that wanted to change the root of the mount namespace. It was not − used for checkpoint operations. − − However, because Docker by default uses the AUFS graph driver and − the AUFS module in the kernel reveals branch pathnames in − <code>/proc/''pid''/map_files</code>, option <code>--root</code> − is used to specify the root of the − mount namespace. Once the kernel AUFS module is fixed, it won't − be necessary to specify this option anymore. − − ==== <code>--ext-mount-map</code> ==== − − This option is used to specify the path of the external bind mounts. − Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with − source files outside the container's mount namespace. Older versions − of Docker also bind mount <code>/.dockerinit</code>. − − For example, assuming the default Docker configuration, <code>/etc/hostname</code> − in the container's mount namespace is bind mounted from the source − at <code>/var/lib/docker/containers/''container_id''/hostname</code>. − − ==== <code>--manage-cgroups</code> ==== − − When a process tree exits after a checkpoint operation, the cgroups − that Docker had created for the container are removed. This option − is needed during restore to move the process tree into its cgroups, − re-creating them if necessary. − − ==== <code>--evasive-devices</code> ==== − − Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers − (i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used − <code>/dev/null</code> in the global namespace, this option tells CRIU to treat − the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device. − − ==== <code>--inherit-fd</code> ==== − − For native C/R support, this option tells CRIU to let the restored process "inherit" − its specified file descriptor (instead of restoring from checkpoint). − − === Restore Prework for External C/R === − − Docker supports many storage drivers (AKA graph drivers) including − AUFS, Btrfs, ZFS, DeviceMapper, OverlayFS, and VFS. The user can − specify his/her desired storage driver via the <code>DOCKER_DRIVER</code> − environment variable or the <code>-s (--storage-driver)</code> command − line option. − − Currently C/R can only be done on containers using either AUFS, OverlayFS, or VFS. − In the following example, we assume AUFS. − − When Docker notices that the container has exited (due to CRIU dump), − it dismantles the container's filesystem. We need to set up the container's − filesystem again before attempting to restore. − − === An External C/R Example === − − Below is an example to show C/R operations for a shell script that − continuously appends a number to a file. You can use tail -f to − see the process in action. − − As you will see below, after restore, the process's parent is PID − 1 (init), not Docker. Also, although the process has been successfully − restored, Docker still thinks that the container has exited. − − To set up the container's AUFS filesystem before restore, its branch − information should be saved before checkpointing the container. − For convenience, however, AUFS branch information is saved in the − dump.log file. So we can examine dump.log to set up the filesystem − again. − − For brevity, the 64-character long container ID is replaced by the − string <container_id> in the following lines. − − <pre> − $ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done' − <container_id> − $ − $ docker ps − CONTAINER ID IMAGE COMMAND CREATED STATUS − 168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds − $ − $ sudo criu dump -o dump.log -v4 -t 17810 \ − -D /tmp/img/<container_id> \ − --root /var/lib/docker/aufs/mnt/<container_id> \ − --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \ − --ext-mount-map /etc/hosts:/etc/hosts \ − --ext-mount-map /etc/hostname:/etc/hostname \ − --ext-mount-map /.dockerinit:/.dockerinit \ − --manage-cgroups \ − --evasive-devices − $ − $ sudo grep successful /tmp/img/<container_id>/dump.log − (00.020103) Dumping finished successfully − $ − $ docker ps -a − CONTAINER ID IMAGE COMMAND CREATED STATUS − 168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago − $ − $ sudo mount -t aufs -o br=\ − /var/lib/docker/aufs/diff/<container_id>:\ − /var/lib/docker/aufs/diff/<container_id>-init:\ − /var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\ − /var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\ − /var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\ − /var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\ − none /var/lib/docker/aufs/mnt/<container_id> − $ − $ sudo criu restore -o restore.log -v4 -d − -D /tmp/img/<container_id> \ − --root /var/lib/docker/aufs/mnt/<container_id> \ − --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \ − --ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \ − --ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \ − --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \ − --manage-cgroups \ − --evasive-devices − $ − $ sudo grep successful /tmp/img/<container_id>/restore.log − (00.424428) Restore finished successfully. Resuming tasks. − $ − $ ps -ef | grep /bin/sh − root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done − $ − $ docker ps -a − CONTAINER ID IMAGE COMMAND CREATED STATUS − 168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago − $ − </pre> − − === External C/R Helper Script === − − As seen in the above examples, the CRIU command line for checkpointing and − restoring a Docker container is pretty long. For restore, there is also − an additional step to set up the root filesystem before invoking CRIU. − − To automate the C/R process, there is a helper script in the contrib − subdirectory of CRIU sources, called docker_cr.sh. In addition to − invoking CRIU, this helper script sets up the root filesystem for AUFS, − UnionFS, and VFS for restore. − − With docker_cr.sh, all you have to provide is the container ID. − If you don't specify a container ID, docker_cr.sh will list all running − containers and prompt you to choose one. Also, as shown in the help − output below, by setting the appropriate environment variable, it's − possible to tell docker_cr.sh which Docker and CRIU binaries to use, − where Docker's home directory is, and where CRIU should save and look − for its image files. − − <pre> − # docker_cr.sh --help − Usage: − docker_cr.sh -c|-r [-hv] [<container_id>] − -c, --checkpoint checkpoint container − -h, --help print help message − -r, --restore restore container − -v, --verbose enable verbose mode − − Environment: − DOCKER_HOME (default /var/lib/docker) − CRIU_IMG_DIR (default /var/lib/docker/criu_img) − DOCKER_BINARY (default docker) − CRIU_BINARY (default criu) − </pre> − − Below is an example to checkpoint and restore Docker container 4397: − − <pre> − # docker_cr.sh -c 4397 − dump successful − # docker_cr.sh -r 4397 − restore successful − </pre> − − Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code> − executes. For example: − − <pre> − # docker_cr.sh -c -v 40d3 − docker binary: docker − criu binary: criu − image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − − criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \ − --manage-cgroups --evasive-devices \ − --ext-mount-map /etc/resolv.conf:/etc/resolv.conf \ − --ext-mount-map /etc/hosts:/etc/hosts \ − --ext-mount-map /etc/hostname:/etc/hostname \ − --ext-mount-map /.dockerinit:/.dockerinit \ − -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − − dump successful − (00.020827) Dumping finished successfully − − # docker_cr.sh -r -v 40d3 − docker binary: docker − criu binary: criu − image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − − mount -t aufs -o − /var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − /var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init − /var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721 − /var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16 − /var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229 − /var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158 − none − /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf − − criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \ − -o restore.log --manage-cgroups --evasive-devices \ − --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \ − --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \ − --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \ − --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \ − -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \ − --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid − − restore successful − (00.408807) Restore finished successfully. Resuming tasks. − − root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done − </pre> − − − [[Category:HOWTO]]
→TTY: Update info about support for checkpointing of an interactive container
This HOWTO page describes how to checkpoint and restore a Docker container.
This article describes the status of CRIU integration with Docker, and how to use it.
== Introduction ==
== Docker Experimental ==
Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.
Naturally, Docker wants to manage the full lifecycle of processes running inside its containers, so CRIU should be run by Docker (rather than separately).
This feature is available in the ''experimental'' mode for Docker (since Docker 1.13, so every later version, like Docker 17.03, should work).
To enable experimental features (incl. CRIU), you need to do something like this:
echo "{\"experimental\": true}" >> /etc/docker/daemon.json
systemctl restart docker
In addition to having a recent version of Docker, you need '''CRIU''' 2.0 or later installed on your system (see [[Installation]] for more info).
=== checkpoint ===
There's a top level <code>checkpoint</code> sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.
Here's an example of creating a checkpoint, from a container that simply logs an integer in a loop.
First, we create container:
First, we create container:
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
$ docker checkpoint looper
$ docker checkpoint create looper checkpoint1
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
=== restore ===
=== restore ===
Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container '''start''' call. Here's an example:
$ docker restore looper
$ docker start --checkpoint checkpoint1 looper
If we then print the logs, you should see they start from where we left off and continue to increase.
If we then print the logs, you should see they start from where we left off and continue to increase.
==== Restoring into a '''new''' container ====
==== Restoring into a '''new''' container ====
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. Right now that is done with the <code>--force</code> option, in conjunction with the <code>--image-dir</code> option. Here's a slightly revised example from before:
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. This is done by providing a custom storage path with the <code>--checkpoint-dir</code> option. Here's a slightly revised example from before:
$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
# wait a few seconds to give the container an opportunity to print a few lines, then
# wait a few seconds to give the container an opportunity to print a few lines, then
$ docker checkpoint --image-dir=/tmp/checkpoint1 looper2
$ docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2
$ docker create --name looper-force --security-opt seccomp:unconfined busybox \
$ docker create --name looper-clone --security-opt seccomp:unconfined busybox \
/bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
/bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
$ docker restore --force=true --image-dir=/tmp/checkpoint1 looper-force
$ docker start --checkpoint-dir=/tmp --checkpoint=checkpoint2 looper-clone
You should be able to print the logs from <code>looper-clone</code> and see that they start from wherever the logs of <code>looper</code> end.
=== Passing additional options ===
[[Configuration files]] can be used to set additional CRIU options when performing checkpoint/restore of Docker containers. These options should be added in the file <code>/etc/criu/runc.conf</code> (in order to '''overwrite''' the ones set by runc/Docker). Note that the options stored in <code>~/.criu/default.conf</code> or <code>/etc/criu/default.conf</code> will be '''overwritten''' by the ones set via [[RPC]] by Docker.
For example, in order to checkpoint and restore a container with established TCP connections CRIU requires the <code>--tcp-established</code> option to be set. However, this option is set to false by default and it is currently not possible to change this behaviour via the command-line interface of Docker. This feature can be enabled by adding <code>tcp-established</code> in the file <code>/etc/criu/runc.conf</code>. Note that for this functionality to work, the version of [[https://github.com/opencontainers/runc runc]] must be recent enough to have the commit [[https://github.com/opencontainers/runc/commit/e157963054e1be28bcd6612f15df1ea561c62571 e157963]] applied.
An alternative solution is to use [https://podman.io/ Podman] which has support to specify <code>--tcp-established</code> on the command-line.
=== Synopsis ===
Checkpoint
# docker checkpoint create --help
Usage: docker checkpoint create [OPTIONS] CONTAINER CHECKPOINT
Create a checkpoint from a running container
Options:
--checkpoint-dir string Use a custom checkpoint storage directory
--help Print usage
--leave-running Leave the container running after checkpoint
Restore
# docker start --help
Usage: docker start [OPTIONS] CONTAINER [CONTAINER...]
Start one or more stopped containers
Options:
-a, --attach Attach STDOUT/STDERR and forward signals
--checkpoint string Restore from this checkpoint
--checkpoint-dir string Use a custom checkpoint storage directory
--detach-keys string Override the key sequence for detaching a container
--help Print usage
-i, --interactive Attach container's STDIN
== Compatibility Notes ==
== Compatibility Notes ==
=== TTY ===
=== TTY ===
Checkpointing an interactive container is currently not supported.
Checkpointing an interactive container is supported by CRIU, runc and containerd, but not yet enabled in Docker.
(See [[https://github.com/moby/moby/pull/38405 PR 38405]] for more information.)
=== Seccomp ===
=== Seccomp ===
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
== External checkpoint/restore ==
Although it's not recommended, you can also learn more about using CRIU without integrating with Docker. See [[Docker External]] for more info.