21
edits
Changes
Jump to navigation
Jump to search
Line 1:
Line 1: − {{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. If you just want to experiment with C/R, you can use one of the following Docker versions for your C/R experiments: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + − Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]+ − Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]}} − {{Note| The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). However, there is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. See below for instructions on how to apply the relevant patches.}}+ − {{Note| If your process uses async IO and your kernel is older than 3.19, you need to apply two patches. See below for instructions.}}+ + − + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + − There are two ways to checkpoint and restore a Docker container:+ − '''1. External C/R''' using CRIU directly on the command line as it's typically+ − done for any process tree. − This approach is called external because it's happening external to the+ − Docker daemon. After checkpoint, the Docker daemon thinks that the − container has exited. After restore, the Docker daemon doesn't know that − the container is running again. Therefore, commands such as − <code>docker ps, stop, kill</code> and <code>logs</code> − will not work correctly. − '''2. Native C/R''' using new <code>docker checkpoint</code> and+ − <code>docker restore</code> commands. − This approach is called native because the Docker daemon is involved in both checkpoint and restore.+ − Therefore, its notion of the container state will be corrent. All commands such as − <code>docker ps, stop, kill </code> and <code>logs</code> will work. − This is obviously the preferred method of checkpointing and restoring Docker containers. − Native C/R is work in progress, say pre-alpha quality.+ − − '''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]''' − − Source files for Docker 1.5 C/R are − '''[https://github.com/SaiedKazemi/docker/tree/cr here]''' − and for Docker 1.7 C/R are − '''[https://github.com/boucher/docker/tree/cr-combined here]'''. − The '''[https://github.com/SaiedKazemi/docker/wiki wiki]''' − page provides an overview of the project history. − + − + Line 74:
Line 165: − + Line 81:
Line 172: − + + + + + + + + + + Line 100:
Line 200: − + Line 106:
Line 206: − + Line 119:
Line 219: − + Line 130:
Line 230: − + Line 137:
Line 237: − + Line 144:
Line 244: − + − + Line 164:
Line 264: − + Line 239:
Line 339: − + Line 334:
Line 434: +
Update documentation on using CRIU with Docker
This HOWTO page describes how to checkpoint and restore a Docker container.
This HOWTO page describes how to checkpoint and restore a Docker container.
== Introduction ==
Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.
== Docker 1.10 ==
The easiest way to try CRIU and Docker together is to install [this pre-compiled version of Docker](https://github.com/boucher/docker/releases/tag/v1.10_2-16-16-experimental). It's based on Docker 1.10, and built with the `DOCKER_EXPERIMENTAL` build tag.
To install, download the `docker-1.10.0-dev` binary to your system. You'll need to start a docker daemon from this binary, and then you can use the same binary to communicate with that daemon. To start a docker daemon, run a command something like this:
`docker-1.10.0-dev daemon -D --graph=/var/lib/docker-dev --host unix:///var/run/docker-dev.sock`
The *graph* and *host* options will prevent colliding with an existing installation of Docker, but you can replace your existing docker if desired. In another shell, you can then connect to that daemon:
`docker-1.10.0-dev --host unix:///var/run/docker-dev.sock run -d busybox top`
=== Dependencies ===
In addition to downloading the binary above (or compiling one yourself), you need *CRIU* installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are *libprotobuf-c* and *libnl-3*. Here's an output of `ldd` on my system:
```
# ldd `which criu`
linux-vdso.so.1 => (0x00007ffc09fda000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000)
libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007fd28b0b7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd28aeb2000)
libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fd28ac98000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd28a8d3000)
/lib64/ld-linux-x86-64.so.2 (0x000056386bb38000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd28a5cc000)
```
=== checkpoint ===
Creating a checkpoint is a top level Docker command with this new version of Docker. Here's an example that simply logs an integer in a loop.
First, we create container:
`docker run -d --name looper --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'`
You can verify the container is running by printings its logs:
`docker logs looper`
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
`docker checkpoint looper`
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
=== restore ===
Like `checkpoint`, `restore` is a top level command in this version of Docker. Continuing our example, let's restore the same container:
`docker restore looper`
If we then print the logs, you should see they start from where we left off and continue to increase.
==== Restoring into a *new* container ====
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. Right now that is done with the `--force` option, in conjunction with the `--image-dir` option. Here's a slightly revised example from before:
```
$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
# wait a few seconds to give the container an opportunity to print a few lines, then
$ docker checkpoint --image-dir=/tmp/checkpoint1 looper2
$ docker create --name looper-force --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
$ docker restore --force=true --image-dir=/tmp/checkpoint1 looper-force
```
You should be able to print the logs from `looper-force` and see that they start from wherever the logs of `looper` end.
=== usage ===
```
# docker checkpoint --help
== Introduction ==
Usage: docker checkpoint [OPTIONS] CONTAINER
Checkpoint one or more running containers
--help Print usage
--image-dir directory for storing checkpoint image files
--leave-running leave the container running after checkpoint
--work-dir directory for storing log file
```
```
# docker restore --help
Usage: docker restore [OPTIONS] CONTAINER
Restore one or more checkpointed containers
--force bypass checks for current container state
--help Print usage
--image-dir directory to restore image files from
--work-dir directory for restore log
```
== Docker 1.12 ==
More detailed instructions on running checkpoint/restore with Docker in version 1.12 will be coming in the future, but in the meantime, you must build the version of Docker available in the `docker-checkpoint-restore` branch of `@boucher`'s fork of Docker, [available here](https://github.com/boucher/docker/tree/docker-checkpoint-restore). Make sure to build with the env `DOCKER_EXPERIMENTAL=1`.
The command line interface has changed from the 1.10 version. `docker checkpoint` is now an umbrella command for a few checkpoint operations. To create a checkpoint, use the `docker checkpoint create` command, which takes `container_id` and `checkpoint_id` as non-optional arguments. Example:
`docker checkpoint create my_container my_first_checkpoint`
Restoring a container is now performed just as an option to `docker start`. Although typically you may create and start a container in a single step using `docker run`, under the hood this is actually two steps: `docker create` followed by `docker start`. You can also call `start` on a container that was previously running and has since been stopped or killed. That looks something like this:
`docker start --checkpoint my_first_checkpoint my_container`
== Integration Status ==
CRIU has already been integrated into the lower level components that power Docker, namely *runc* and *containerd*. The final step in the process is to integrate with Docker itself. You can track the status of that process in [this pull request](https://github.com/docker/docker/pull/22049).
== Compatibility Notes ==
The latest versions of the Docker integration require at least version 2.0 of CRIU in order to work correctly. Additionally, depending on the storage driver being used by Docker, and other factors, there may be other compatibility issues that will attempt to be listed here.
=== TTY ===
Checkpointing an interactive container is currently not supported.
=== Seccomp ===
You'll notice that all of the above examples disable Docker's default seccomp support. In order to use seccomp, you'll need a newer version of the Kernel. **Update Needed with Exact Version**
You can watch this short demo
to see how it works.
== OverlayFS ==
=== OverlayFS ===
The following small kernel patches fix the mount id and symlink target path issues noted above:
There is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. The following small kernel patches fix the mount id and symlink target path issue:
* {{torvalds.git|155e35d4da}} by David Howells
* {{torvalds.git|155e35d4da}} by David Howells
</pre>
</pre>
== Async IO (AIO) ==
=== Async IO ===
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
== External C/R ==
== External Checkpoint Restore ==
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}
This approach is called external because it's happening external to the
Docker daemon. After checkpoint, the Docker daemon thinks that the
container has exited. After restore, the Docker daemon doesn't know that
the container is running again. Therefore, commands such as
<code>docker ps, stop, kill</code> and <code>logs</code>
will not work correctly.
Starting with CRIU 1.3, it is possible to checkpoint and restore a
Starting with CRIU 1.3, it is possible to checkpoint and restore a
include the Docker daemon which runs in the global PID namespace.
include the Docker daemon which runs in the global PID namespace.
== Command Line Options ==
=== Command Line Options ===
In addition to the usual CRIU command line options used when
In addition to the usual CRIU command line options used when
line options are needed for Docker containers.
line options are needed for Docker containers.
=== <code>--root</code> ===
==== <code>--root</code> ====
This option has been used in the past only for restore operations
This option has been used in the past only for restore operations
be necessary to specify this option anymore.
be necessary to specify this option anymore.
=== <code>--ext-mount-map</code> ===
==== <code>--ext-mount-map</code> ====
This option is used to specify the path of the external bind mounts.
This option is used to specify the path of the external bind mounts.
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.
=== <code>--manage-cgroups</code> ===
==== <code>--manage-cgroups</code> ====
When a process tree exits after a checkpoint operation, the cgroups
When a process tree exits after a checkpoint operation, the cgroups
re-creating them if necessary.
re-creating them if necessary.
=== <code>--evasive-devices</code> ===
==== <code>--evasive-devices</code> ====
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.
=== <code>--inherit-fd</code> ===
==== <code>--inherit-fd</code> ====
For native C/R support, this option tells CRIU to let the restored process "inherit"
For native C/R support, this option tells CRIU to let the restored process "inherit"
its specified file descriptor (instead of restoring from checkpoint).
its specified file descriptor (instead of restoring from checkpoint).
== Restore Prework for External C/R ==
=== Restore Prework for External C/R ===
Docker supports many storage drivers (AKA graph drivers) including
Docker supports many storage drivers (AKA graph drivers) including
filesystem again before attempting to restore.
filesystem again before attempting to restore.
== An External C/R Example ==
=== An External C/R Example ===
Below is an example to show C/R operations for a shell script that
Below is an example to show C/R operations for a shell script that
</pre>
</pre>
== External C/R Helper Script ==
=== External C/R Helper Script ===
As seen in the above examples, the CRIU command line for checkpointing and
As seen in the above examples, the CRIU command line for checkpointing and
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
</pre>
</pre>
[[Category:HOWTO]]
[[Category:HOWTO]]