Difference between revisions of "Docker"

From CRIU
Jump to: navigation, search
m (rm extra vertical whitespace)
m (TTY: Update info about support for checkpointing of an interactive container)
 
(8 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This HOWTO page describes how to checkpoint and restore a Docker container.
+
This article describes the status of CRIU integration with Docker, and how to use it.
  
== Introduction ==
+
== Docker Experimental ==
  
Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.  
+
Naturally, Docker wants to manage the full lifecycle of processes running inside its containers, so CRIU should be run by Docker (rather than separately).
 +
This feature is available in the ''experimental'' mode for Docker (since Docker 1.13, so every later version, like Docker 17.03, should work).
  
== Docker 1.10 ==
+
To enable experimental features (incl. CRIU), you need to do something like this:
  
The easiest way to try CRIU and Docker together is to install [https://github.com/boucher/docker/releases/tag/v1.10_2-16-16-experimental this pre-compiled version of Docker]. It's based on Docker 1.10, and built with the <code>DOCKER_EXPERIMENTAL</code> build tag.  
+
echo "{\"experimental\": true}" >> /etc/docker/daemon.json
 +
systemctl restart docker
  
To install, download the <code>docker-1.10.0-dev</code> binary to your system. You'll need to start a docker daemon from this binary, and then you can use the same binary to communicate with that daemon. To start a docker daemon, run a command something like this:
+
In addition to having a recent version of Docker, you need '''CRIU''' 2.0 or later installed on your system (see [[Installation]] for more info).
  
$ docker-1.10.0-dev daemon -D --graph=/var/lib/docker-dev --host unix:///var/run/docker-dev.sock
+
=== checkpoint ===
 
 
The '''graph''' and '''host''' options will prevent colliding with an existing installation of Docker, but you can replace your existing docker if desired. In another shell, you can then connect to that daemon:
 
 
 
$ docker-1.10.0-dev --host unix:///var/run/docker-dev.sock run -d busybox top
 
  
=== Dependencies ===
+
There's a top level <code>checkpoint</code> sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.
 
 
In addition to downloading the binary above (or compiling one yourself), you need '''CRIU''' installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are '''libprotobuf-c''' and '''libnl-3'''. Here's an output of <code>ldd</code> on my system:
 
 
 
$ ldd `which criu`
 
    linux-vdso.so.1 =>  (0x00007ffc09fda000)
 
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000)
 
    libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007fd28b0b7000)
 
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd28aeb2000)
 
    libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fd28ac98000)
 
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd28a8d3000)
 
    /lib64/ld-linux-x86-64.so.2 (0x000056386bb38000)
 
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd28a5cc000)
 
 
 
=== checkpoint ===
 
  
Creating a checkpoint is a top level Docker command with this new version of Docker. Here's an example that simply logs an integer in a loop. From this point forward, commands are show using '''docker''' instead of '''docker-dev-1.10''', but if you have not installed this version globally you can use the latter.
+
Here's an example of creating a checkpoint, from a container that simply logs an integer in a loop.
  
 
First, we create container:
 
First, we create container:
Line 46: Line 30:
 
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
 
If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:
  
  $ docker checkpoint looper
+
  $ docker checkpoint create looper checkpoint1
  
 
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
 
You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.
Line 52: Line 36:
 
=== restore ===
 
=== restore ===
  
Like '''checkpoint''', '''restore''' is a top level command in this version of Docker. Continuing our example, let's restore the same container:
+
Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container '''start''' call. Here's an example:
  
  $ docker restore looper
+
  $ docker start --checkpoint checkpoint1 looper
  
 
If we then print the logs, you should see they start from where we left off and continue to increase.  
 
If we then print the logs, you should see they start from where we left off and continue to increase.  
Line 60: Line 44:
 
==== Restoring into a '''new''' container ====
 
==== Restoring into a '''new''' container ====
  
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. Right now that is done with the <code>--force</code> option, in conjunction with the <code>--image-dir</code> option. Here's a slightly revised example from before:
+
Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. This is done by providing a custom storage path with the <code>--checkpoint-dir</code> option. Here's a slightly revised example from before:
  
 
  $ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
 
  $ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
Line 66: Line 50:
 
   
 
   
 
  # wait a few seconds to give the container an opportunity to print a few lines, then
 
  # wait a few seconds to give the container an opportunity to print a few lines, then
  $ docker checkpoint --image-dir=/tmp/checkpoint1 looper2
+
  $ docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2
 
   
 
   
  $ docker create --name looper-force --security-opt seccomp:unconfined busybox \
+
  $ docker create --name looper-clone --security-opt seccomp:unconfined busybox \
 
           /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
 
           /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
 
   
 
   
  $ docker restore --force=true --image-dir=/tmp/checkpoint1 looper-force
+
  $ docker start --checkpoint-dir=/tmp --checkpoint=checkpoint2 looper-clone
 +
 
 +
 
 +
You should be able to print the logs from <code>looper-clone</code> and see that they start from wherever the logs of <code>looper</code> end.
  
 +
=== Passing additional options ===
  
You should be able to print the logs from <code>looper-force</code> and see that they start from wherever the logs of <code>looper</code> end.
+
[[Configuration files]] can be used to set additional CRIU options when performing checkpoint/restore of Docker containers. These options should be added in the file <code>/etc/criu/runc.conf</code> (in order to '''overwrite''' the ones set by runc/Docker). Note that the options stored in <code>~/.criu/default.conf</code> or <code>/etc/criu/default.conf</code> will be '''overwritten''' by the ones set via [[RPC]] by Docker.
  
=== usage ===
+
For example, in order to checkpoint and restore a container with established TCP connections CRIU requires the <code>--tcp-established</code> option to be set. However, this option is set to false by default and it is currently not possible to change this behaviour via the command-line interface of Docker. This feature can be enabled by adding <code>tcp-established</code> in the file <code>/etc/criu/runc.conf</code>. Note that for this functionality to work, the version of [[https://github.com/opencontainers/runc runc]] must be recent enough to have the commit [[https://github.com/opencontainers/runc/commit/e157963054e1be28bcd6612f15df1ea561c62571 e157963]] applied.
  
# docker checkpoint --help
+
An alternative solution is to use [https://podman.io/ Podman] which has support to specify <code>--tcp-established</code> on the command-line.
Usage: docker checkpoint [OPTIONS] CONTAINER
 
 
Checkpoint one or more running containers
 
 
    --help            Print usage
 
    --image-dir        directory for storing checkpoint image files
 
    --leave-running    leave the container running after checkpoint
 
    --work-dir        directory for storing log file
 
  
# docker restore --help
+
=== Synopsis ===
Usage: docker restore [OPTIONS] CONTAINER
 
 
Restore one or more checkpointed containers
 
 
    --force            bypass checks for current container state
 
    --help            Print usage
 
    --image-dir        directory to restore image files from
 
    --work-dir        directory for restore log
 
  
== Docker 1.12 ==
+
Checkpoint
  
More detailed instructions on running checkpoint/restore with Docker in version 1.12 will be coming in the future, but in the meantime, you must build the version of Docker available in the '''docker-checkpoint-restore''' branch of [[User:Boucher|Boucher]]'s fork of Docker, [https://github.com/boucher/docker/tree/docker-checkpoint-restore available here]. Make sure to build with the env <code>DOCKER_EXPERIMENTAL=1</code>.
+
  # docker checkpoint create --help
 +
  Usage: docker checkpoint create [OPTIONS] CONTAINER CHECKPOINT
  
The command line interface has changed from the 1.10 version. <code>docker checkpoint</code> is now an umbrella command for a few checkpoint operations. To create a checkpoint, use the <code>docker checkpoint create</code> command, which takes <code>container_id</code> and <code>checkpoint_id</code> as non-optional arguments. Example:
+
  Create a checkpoint from a running container
  
    docker checkpoint create my_container my_first_checkpoint
+
  Options:
 +
      --checkpoint-dir string  Use a custom checkpoint storage directory
 +
      --help                    Print usage
 +
      --leave-running          Leave the container running after checkpoint
  
Restoring a container is now performed just as an option to <code>docker start</code>. Although typically you may create and start a container in a single step using <code>docker run</code>, under the hood this is actually two steps: <code>docker create</code> followed by <code>docker start</code>. You can also call <code>start</code> on a container that was previously running and has since been stopped or killed. That looks something like this:
+
Restore
  
    docker start --checkpoint my_first_checkpoint my_container
+
  # docker start --help
 +
  Usage: docker start [OPTIONS] CONTAINER [CONTAINER...]
  
== Integration Status ==
+
  Start one or more stopped containers
  
CRIU has already been integrated into the lower level components that power Docker, namely '''runc''' and '''containerd'''. The final step in the process is to integrate with Docker itself. You can track the status of that process in [https://github.com/docker/docker/pull/22049 this pull request].
+
  Options:
 +
  -a, --attach                  Attach STDOUT/STDERR and forward signals
 +
      --checkpoint string      Restore from this checkpoint
 +
      --checkpoint-dir string  Use a custom checkpoint storage directory
 +
      --detach-keys string      Override the key sequence for detaching a container
 +
      --help                    Print usage
 +
  -i, --interactive            Attach container's STDIN
  
 
== Compatibility Notes ==
 
== Compatibility Notes ==
Line 120: Line 103:
 
=== TTY ===
 
=== TTY ===
  
Checkpointing an interactive container is currently not supported.  
+
Checkpointing an interactive container is supported by CRIU, runc and containerd, but not yet enabled in Docker.
 +
(See [[https://github.com/moby/moby/pull/38405 PR 38405]] for more information.)
  
 
=== Seccomp ===
 
=== Seccomp ===
Line 164: Line 148:
 
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
 
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov
  
== External Checkpoint Restore ==
+
== External checkpoint/restore ==
 
 
{{Note| External C/R was done as proof-of-concept.  Its use is highly discouraged.}}
 
  
Although it's not recommended, you can also learn more about using CRIU without integrating with docker: [[Docker_External]].
+
Although it's not recommended, you can also learn more about using CRIU without integrating with Docker. See [[Docker External]] for more info.

Latest revision as of 07:22, 25 January 2019

This article describes the status of CRIU integration with Docker, and how to use it.

Docker Experimental[edit]

Naturally, Docker wants to manage the full lifecycle of processes running inside its containers, so CRIU should be run by Docker (rather than separately). This feature is available in the experimental mode for Docker (since Docker 1.13, so every later version, like Docker 17.03, should work).

To enable experimental features (incl. CRIU), you need to do something like this:

echo "{\"experimental\": true}" >> /etc/docker/daemon.json
systemctl restart docker

In addition to having a recent version of Docker, you need CRIU 2.0 or later installed on your system (see Installation for more info).

checkpoint[edit]

There's a top level checkpoint sub-command in Docker, which lets you create a new checkpoint, and list or delete an existing checkpoint. These checkpoints are stored and managed by Docker, unless you specify a custom storage path.

Here's an example of creating a checkpoint, from a container that simply logs an integer in a loop.

First, we create container:

$ docker run -d --name looper --security-opt seccomp:unconfined busybox  \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

You can verify the container is running by printings its logs:

$ docker logs looper

If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:

$ docker checkpoint create looper checkpoint1

You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.

restore[edit]

Unlike creating a checkpoint, restoring from a checkpoint is just a flag provided to the normal container start call. Here's an example:

$ docker start --checkpoint checkpoint1 looper

If we then print the logs, you should see they start from where we left off and continue to increase.

Restoring into a new container[edit]

Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. This is done by providing a custom storage path with the --checkpoint-dir option. Here's a slightly revised example from before:

$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

# wait a few seconds to give the container an opportunity to print a few lines, then
$ docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2

$ docker create --name looper-clone --security-opt seccomp:unconfined busybox \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

$ docker start --checkpoint-dir=/tmp --checkpoint=checkpoint2 looper-clone


You should be able to print the logs from looper-clone and see that they start from wherever the logs of looper end.

Passing additional options[edit]

Configuration files can be used to set additional CRIU options when performing checkpoint/restore of Docker containers. These options should be added in the file /etc/criu/runc.conf (in order to overwrite the ones set by runc/Docker). Note that the options stored in ~/.criu/default.conf or /etc/criu/default.conf will be overwritten by the ones set via RPC by Docker.

For example, in order to checkpoint and restore a container with established TCP connections CRIU requires the --tcp-established option to be set. However, this option is set to false by default and it is currently not possible to change this behaviour via the command-line interface of Docker. This feature can be enabled by adding tcp-established in the file /etc/criu/runc.conf. Note that for this functionality to work, the version of [runc] must be recent enough to have the commit [e157963] applied.

An alternative solution is to use Podman which has support to specify --tcp-established on the command-line.

Synopsis[edit]

Checkpoint

 # docker checkpoint create --help
 Usage:	docker checkpoint create [OPTIONS] CONTAINER CHECKPOINT
 Create a checkpoint from a running container
 Options:
     --checkpoint-dir string   Use a custom checkpoint storage directory
     --help                    Print usage
     --leave-running           Leave the container running after checkpoint

Restore

  # docker start --help
  Usage:	docker start [OPTIONS] CONTAINER [CONTAINER...]
 Start one or more stopped containers
 Options:
 -a, --attach                  Attach STDOUT/STDERR and forward signals
     --checkpoint string       Restore from this checkpoint
     --checkpoint-dir string   Use a custom checkpoint storage directory
     --detach-keys string      Override the key sequence for detaching a container
     --help                    Print usage
 -i, --interactive             Attach container's STDIN

Compatibility Notes[edit]

The latest versions of the Docker integration require at least version 2.0 of CRIU in order to work correctly. Additionally, depending on the storage driver being used by Docker, and other factors, there may be other compatibility issues that will attempt to be listed here.

TTY[edit]

Checkpointing an interactive container is supported by CRIU, runc and containerd, but not yet enabled in Docker. (See [PR 38405] for more information.)

Seccomp[edit]

You'll notice that all of the above examples disable Docker's default seccomp support. In order to use seccomp, you'll need a newer version of the Kernel. **Update Needed with Exact Version**

OverlayFS[edit]

There is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. The following small kernel patches fix the mount id and symlink target path issue:

Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:

git clone  git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git
cd ubuntu-vivid
git remote add torvalds  git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git remote update

git cherry-pick 155e35d4da
git cherry-pick df1a085af1
git cherry-pick f25801ee46
git cherry-pick 4bacc9c923
git cherry-pick 9391dd00d1

cp /boot/config-$(uname -r) .config
make olddefconfig
make -j 8 bzImage modules
sudo make install modules_install
sudo reboot

Async IO[edit]

If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:

External checkpoint/restore[edit]

Although it's not recommended, you can also learn more about using CRIU without integrating with Docker. See Docker External for more info.