CRIU - User contributions [en]

Installation

2017-05-17T13:35:34Z

Boucher:

<code>criu</code> is an utility to checkpoint/restore a process tree. This page describes how to manually build and install prerequisites and the tool itself.

== Installing from packages ==

Some distributions provide ready-to-use [[packages]]. If no, or the CRIU version you want is not yet there, you will need to get CRIU sources and compile it.

== Obtaining CRIU Source ==

You can download the source code as a release tarball or sync the [https://github.com/xemul/criu git repository]. If you plan to modify CRIU sources the latter way is highly recommended.

=== Getting source tarball ===
: {{Latest release}}

=== Cloning git repository ===
git clone https://github.com/xemul/criu

== Dependencies ==

=== Compiler and C Library ===

CRIU is mostly written in C and the build system is based on Makefiles. Thus just install standard <code>gcc</code> and <code>make</code> packages (on Debian, <code>[https://packages.debian.org/build-essential build-essential]</code> will pull in both at once).

For building on x86 with compatible 32-bit applications C/R support you will need <code>libc6-dev-i386, gcc-multilib</code> instead of <code>gcc</code>.

If you are cross compiling for ARM, use distribution packages or download prebuilt toolchains from Linaro.

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
Downloading Linaro toolchains
<div class="mw-collapsible-content">
sudo apt-get install lib32stdc++6 lib32z1 # These are ia32 binaries
mkdir -p deps/`uname -m`-linux-gnu
cd deps
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
cd ..
</div>
</div>

=== Protocol Buffers ===

CRIU uses the [https://developers.google.com/protocol-buffers/ Google Protocol Buffers] to read and write [[images]] and thus requires [https://github.com/protobuf-c/protobuf-c C language bindings]. The <code>protoc</code> tool is required at build time and the <code>libprotobuf-c.so</code> shared object is required at build and run time. [[CRIT]] also uses python language bindings for protocol buffers and requires the <code>descriptor.proto</code> file typically provided by a distribution's protobuf development package.

==== Distribution Packages ====
The easiest way is to install distribution packages.

* RPM package names
** <code>group Development\ Tools</code>
** <code>protobuf</code>
** <code>protobuf-c</code>
** <code>protobuf-c-devel</code>
** <code>protobuf-compiler</code>
** <code>protobuf-devel</code>
** <code>protobuf-python</code>
** <code>libnet-devel</code>
* Debian package names
** <code>build-essential</code>
** <code>libprotobuf-dev</code>
** <code>libprotobuf-c0-dev</code>
** <code>protobuf-c-compiler</code>
** <code>protobuf-compiler</code>
** <code>python-protobuf</code>
** <code>libnet1-dev</code>
* Ubuntu
** The below will get your freshly installed Ubuntu host ready to compile criu. "--no-install-recommends" parameter is to avoid asciidoc pulling in a lot of dependencies.
** sudo apt-get install --no-install-recommends git build-essential libprotobuf-dev libprotobuf-c0-dev protobuf-c-compiler protobuf-compiler python-protobuf libnl-3-dev libpth-dev pkg-config libcap-dev asciidoc libnet-dev

==== Building Protocol Buffers From Source ====
If you would like to build from source, you can use the following commands to obtain the source code repositories, configure, and build the code. On a Debian based system, you may have to install <code>autoconf curl g++ libtool</code> packages first.

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
To build protobuf
<div class="mw-collapsible-content">
cd deps
git clone https://github.com/google/protobuf.git protobuf
cd protobuf
./autogen.sh
./configure --prefix=`pwd`/../`uname -m`-linux-gnu
make
make install
cd ../..
</div>
</div>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
To build protobuf-c
<div class="mw-collapsible-content">
cd deps
git clone https://github.com/protobuf-c/protobuf-c.git protobuf-c
cd protobuf-c
./autogen.sh
mkdir ../pbc-`uname -m`
cd ../pbc-`uname -m`
../protobuf-c/configure --prefix=`pwd`/../`uname -m`-linux-gnu \
PKG_CONFIG_PATH=`pwd`/../`uname -m`-linux-gnu/lib/pkgconfig
make
make install
cd ../..
</div>
</div>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
To cross-compile for ARM some more tricks will be required.
<div class="mw-collapsible-content">
For ARMv7

cd deps
mkdir -p pbc-arm
cd pbc-arm
../protobuf-c/configure --host=arm-linux-gnueabihf --prefix=`pwd`/../arm-linux-gnueabihf \
--disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

For ARM8

cd deps
mkdir -p pbc-aarch64
cd pbc-aarch64
../protobuf-c/configure --host=aarch64-linux-gnu --prefix=`pwd`/../aarch64-linux-gnu \
--disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..
</div>
</div>

=== Other deps ===
* <code>pkg-config</code> to check on build library dependencies.
* <code>libnl3</code> and <code>libnl3-devel</code> (RPM distros) or <code>libnl-3-dev</code> (DEB distros) for network operations.
* <code>python-ipaddr</code> is used by CRIT to pretty-print ip.
* If <code>libbsd</code> available, CRIU will be compiled with setproctitle() support. It will allow to make process titles of service workers to be more verbose.
* The iproute2 tool version 3.5.0 or higher is needed for dumping network namespaces. The latest one can be cloned from [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=summary iproute2]. It should be compiled and a path to ip written in the environment variable <code>CR_IP_TOOL</code>.
* <code>libcap-devel</code> (RPM) or <code>libcap-dev</code> (DEB)
* If you would like to use <code>make test</code> you should install <code>libaio-devel</code> (RPM) or <code>libaio-dev</code> (DEB).
* For test launcher <code>zdtm.py</code> you need <code>PyYAML</code> (RPM) or <code>python-yaml</code> (DEB).

== Linux Kernel ==

Linux kernel v3.11 or newer is required, with some specific options set. If your distribution does not provide needed kernel, you might want to compile one yourself.

=== Configuring the kernel ===

Most likely the first thing to enable is the <code>CONFIG_EXPERT=y</code> (General setup -> Configure standard kernel features (expert users)) option, which on x86_64 depends on the <code>CONFIG_EMBEDDED=y</code> (General setup -> Embedded system) one (welcome to Kconfig reverse chains hell).

The following options must be enabled for CRIU to work:

* ''General setup'' options
** <code>CONFIG_CHECKPOINT_RESTORE=y</code> (Checkpoint/restore support)
** <code>CONFIG_NAMESPACES=y</code> (Namespaces support)
** <code>CONFIG_UTS_NS=y</code> (Namespaces support -> UTS namespace)
** <code>CONFIG_IPC_NS=y</code> (Namespaces support -> IPC namespace)
** <code>CONFIG_PID_NS=y</code> (Namespaces support -> PID namespaces)
** <code>CONFIG_NET_NS=y</code> (Namespaces support -> Network namespace)
** <code>CONFIG_FHANDLE=y</code> (Open by fhandle syscalls)
** <code>CONFIG_EVENTFD=y</code> (Enable eventfd() system call)
** <code>CONFIG_EPOLL=y</code> (Enable eventpoll support)
* ''Networking support -> Networking options'' options for sock-diag subsystem
** <code>CONFIG_UNIX_DIAG=y</code> (Unix domain sockets -> UNIX: socket monitoring interface)
** <code>CONFIG_INET_DIAG=y</code> (TCP/IP networking -> INET: socket monitoring interface)
** <code>CONFIG_INET_UDP_DIAG=y</code> (TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface)
** <code>CONFIG_PACKET_DIAG=y</code> (Packet socket -> Packet: sockets monitoring interface)
** <code>CONFIG_NETLINK_DIAG=y</code> (Netlink socket -> Netlink: sockets monitoring interface)

Other options not required by CRIU, but C/R supported ([[ZDTM test suite]] may fail without them):
* <code>CONFIG_INOTIFY_USER=y</code> (File systems -> Inotify support for userspace)
* <code>CONFIG_FANOTIFY=y</code> (File systems -> Filesystem wide access notification)
* <code>CONFIG_NETFILTER_XT_MARK=y</code> (Networking support -> Networking options -> Network packet filtering framework (Netfilter) -> Core Netfilter Configuration -> Netfilter Xtables support (required for ip_tables) -> nfmark target and match support)
* <code>CONFIG_MEMCG=y</code> (General setup -> Control Group support -> Memory controller)
* <code>CONFIG_CGROUP_DEVICE=y</code> (General setup -> Control Group support -> Device controller)
* <code>CONFIG_MACVLAN=y</code> (Device Drivers -> Network device support -> Network core driver support -> MAC-VLAN support)
* <code>CONFIG_BRIDGE=y</code> (Networking support -> Networking options -> 802.1d Ethernet Bridging)
* <code>CONFIG_BINFMT_MISC=y</code> (Userspace binary formats -> Kernel support for MISC binaries)
* <code>CONFIG_IA32_EMULATION=y</code> (x86 only) (Executable file formats -> Emulations -> IA32 Emulation)

For some [[usage scenarios]] there is an ability to track memory changes and produce [[incremental dumps]]. Need to enable the <code>CONFIG_MEM_SOFT_DIRTY=y</code> (optional) (Processor type and features -> Track memory changes).

Note we also have our [[custom kernel]], which might contain some experimental CRIU related patches.

== Building CRIU From Source ==

=== Native Compilation ===
Simply run <code>make</code> in the CRIU source directory.

=== Compilation in Docker container ===

There's a ''docker-build'' target in Makefile which builds CRIU in Ubuntu Docker container. Just run <code>make docker-build</code> and that's it.

=== Non-standard compilation ===

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
Building natively, but specifying built dependencies manually
<div class="mw-collapsible-content">
cd deps
rsync -a --exclude=.git --exclude=deps .. criu-`uname -m`
cd criu-`uname -m`
make \
USERCFLAGS="-I`pwd`/../`uname -m`-linux-gnu/include -L`pwd`/../`uname -m`-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
sudo LD_LIBRARY_PATH=`pwd`/../`uname -m`-linux-gnu/lib ./criu check
cd ../..
</div>
</div>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
Cross Compilation for ARM
<div class="mw-collapsible-content">
ARMv7
cd deps
rsync -a --exclude=.git --exclude=deps .. criu-arm
cd criu-arm
make \
ARCH=arm \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/arm-linux-gnueabihf- \
USERCFLAGS="-I`pwd`/../arm-linux-gnueabihf/include -L`pwd`/../arm-linux-gnueabihf/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

ARMv8
cd deps
rsync -a --exclude=.git --exclude=deps .. criu-aarch64
cd criu-aarch64
make \
ARCH=aarch64 \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/aarch64-linux-gnu- \
USERCFLAGS="-I`pwd`/../aarch64-linux-gnu/include -L`pwd`/../aarch64-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..
</div>
</div>

=== Configuration ===

CRIU has functionality that is either optional or behaves differently depending on the kernel CRIU is running on. By default build process includes maximum of it, but this behavior can be changed.

''Main article: [[Configuring]]''

== Installation ==
CRIU works perfectly even when run from the sources directory (with the "./criu" command), but if you want to have in standard paths run <code>make install</code>.

You may need to install the following packages to generate docs in Debian-based OS's to avoid errors from install-man:
* <code>asciidoc</code>
* <code>xmlto</code>

== Checking That It Works ==

First thing to do is to run <code>criu check</code>. At the end it should say "Looks OK", if it doesn't the messages on the screen explain what functionality is missing.

Some kernel functionality is required in rare cases and may not block the dump (but sometimes may). These features can be checked by adding the <code>--extra</code> flag.

If you're using our custom kernel, then the <code>--all</code> option can be used, in this case CRIU would check for ''all'' the kernel features to work.

You can then try running the [[ZDTM Test Suite]] which sits in the <code>tests/zdtm/</code> directory.

== Further reading ==

* [[Checking the kernel]]
* [[Usage]]
* [[Advanced usage]]
* [[:Category:HOWTO]]

[[Category:HOWTO]]

Docker

2017-01-24T21:07:13Z

Boucher:

2016-07-18T18:00:34Z

Boucher: Update documentation on using CRIU with Docker

This HOWTO page describes how to checkpoint and restore a Docker container.

== Introduction ==

Docker wants to manage the full lifecycle of processes running inside one if its containers, which makes it important for CRIU and Docker to work closely together when trying to checkpoint and restore a container. This is being achieved by adding the ability to checkpoint and restore directly into Docker itself, powered under the hood by CRIU. This integration is a work in progress, and its status will be outlined below.

== Docker 1.10 ==

The easiest way to try CRIU and Docker together is to install [this pre-compiled version of Docker](https://github.com/boucher/docker/releases/tag/v1.10_2-16-16-experimental). It's based on Docker 1.10, and built with the `DOCKER_EXPERIMENTAL` build tag.

To install, download the `docker-1.10.0-dev` binary to your system. You'll need to start a docker daemon from this binary, and then you can use the same binary to communicate with that daemon. To start a docker daemon, run a command something like this:

`docker-1.10.0-dev daemon -D --graph=/var/lib/docker-dev --host unix:///var/run/docker-dev.sock`

The *graph* and *host* options will prevent colliding with an existing installation of Docker, but you can replace your existing docker if desired. In another shell, you can then connect to that daemon:

`docker-1.10.0-dev --host unix:///var/run/docker-dev.sock run -d busybox top`

=== Dependencies ===

In addition to downloading the binary above (or compiling one yourself), you need *CRIU* installed on your system, with at least version 2.0. You also need some shared libraries on your system. The most likely things you'll need to install are *libprotobuf-c* and *libnl-3*. Here's an output of `ldd` on my system:

```
# ldd `which criu`
linux-vdso.so.1 => (0x00007ffc09fda000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd28b2c7000)
libprotobuf-c.so.0 => /usr/lib/x86_64-linux-gnu/libprotobuf-c.so.0 (0x00007fd28b0b7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd28aeb2000)
libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fd28ac98000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd28a8d3000)
/lib64/ld-linux-x86-64.so.2 (0x000056386bb38000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd28a5cc000)
```

=== checkpoint ===

Creating a checkpoint is a top level Docker command with this new version of Docker. Here's an example that simply logs an integer in a loop.

First, we create container:

`docker run -d --name looper --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'`

You can verify the container is running by printings its logs:

`docker logs looper`

If you do this a few times you'll notice the integer increasing. Now, we checkpoint the container:

`docker checkpoint looper`

You should see that the process is no longer running, and if you print the logs a few times no new logs will be printed.

=== restore ===

Like `checkpoint`, `restore` is a top level command in this version of Docker. Continuing our example, let's restore the same container:

`docker restore looper`

If we then print the logs, you should see they start from where we left off and continue to increase.

==== Restoring into a *new* container ====

Beyond the straightforward case of checkpointing and restoring the same container, it's also possible to checkpoint one container, and then restore the checkpoint into a completely different container. Right now that is done with the `--force` option, in conjunction with the `--image-dir` option. Here's a slightly revised example from before:

```
$ docker run -d --name looper2 --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

# wait a few seconds to give the container an opportunity to print a few lines, then
$ docker checkpoint --image-dir=/tmp/checkpoint1 looper2

$ docker create --name looper-force --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

$ docker restore --force=true --image-dir=/tmp/checkpoint1 looper-force

```

You should be able to print the logs from `looper-force` and see that they start from wherever the logs of `looper` end.

=== usage ===

```
# docker checkpoint --help

Usage: docker checkpoint [OPTIONS] CONTAINER

Checkpoint one or more running containers

--help Print usage
--image-dir directory for storing checkpoint image files
--leave-running leave the container running after checkpoint
--work-dir directory for storing log file
```

```
# docker restore --help

Usage: docker restore [OPTIONS] CONTAINER

Restore one or more checkpointed containers

--force bypass checks for current container state
--help Print usage
--image-dir directory to restore image files from
--work-dir directory for restore log
```

== Docker 1.12 ==

More detailed instructions on running checkpoint/restore with Docker in version 1.12 will be coming in the future, but in the meantime, you must build the version of Docker available in the `docker-checkpoint-restore` branch of `@boucher`'s fork of Docker, [available here](https://github.com/boucher/docker/tree/docker-checkpoint-restore). Make sure to build with the env `DOCKER_EXPERIMENTAL=1`.

The command line interface has changed from the 1.10 version. `docker checkpoint` is now an umbrella command for a few checkpoint operations. To create a checkpoint, use the `docker checkpoint create` command, which takes `container_id` and `checkpoint_id` as non-optional arguments. Example:

`docker checkpoint create my_container my_first_checkpoint`

Restoring a container is now performed just as an option to `docker start`. Although typically you may create and start a container in a single step using `docker run`, under the hood this is actually two steps: `docker create` followed by `docker start`. You can also call `start` on a container that was previously running and has since been stopped or killed. That looks something like this:

`docker start --checkpoint my_first_checkpoint my_container`

== Integration Status ==

CRIU has already been integrated into the lower level components that power Docker, namely *runc* and *containerd*. The final step in the process is to integrate with Docker itself. You can track the status of that process in [this pull request](https://github.com/docker/docker/pull/22049).

== Compatibility Notes ==

The latest versions of the Docker integration require at least version 2.0 of CRIU in order to work correctly. Additionally, depending on the storage driver being used by Docker, and other factors, there may be other compatibility issues that will attempt to be listed here.

=== TTY ===

Checkpointing an interactive container is currently not supported.

=== Seccomp ===

You'll notice that all of the above examples disable Docker's default seccomp support. In order to use seccomp, you'll need a newer version of the Kernel. **Update Needed with Exact Version**

=== OverlayFS ===

There is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. The following small kernel patches fix the mount id and symlink target path issue:

* {{torvalds.git|155e35d4da}} by David Howells
* {{torvalds.git|df1a085af1}} by David Howells
* {{torvalds.git|f25801ee46}} by David Howells
* {{torvalds.git|4bacc9c923}} by David Howells
* {{torvalds.git|9391dd00d1}} by Al Viro

Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:

<pre>
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git
cd ubuntu-vivid
git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git remote update

git cherry-pick 155e35d4da
git cherry-pick df1a085af1
git cherry-pick f25801ee46
git cherry-pick 4bacc9c923
git cherry-pick 9391dd00d1

cp /boot/config-$(uname -r) .config
make olddefconfig
make -j 8 bzImage modules
sudo make install modules_install
sudo reboot
</pre>

=== Async IO ===

If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:

* {{torvalds.git|bd9b51e79c}} by Al Viro
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov

== External Checkpoint Restore ==

{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}

This approach is called external because it's happening external to the
Docker daemon. After checkpoint, the Docker daemon thinks that the
container has exited. After restore, the Docker daemon doesn't know that
the container is running again. Therefore, commands such as
<code>docker ps, stop, kill</code> and <code>logs</code>
will not work correctly.

Starting with CRIU 1.3, it is possible to checkpoint and restore a
process tree running inside a Docker container. However, it's
important to note that Docker needs native support for checkpoint
and restore in order to maintain its parent-child relationship and
to correctly keep track of container states. In other words, while
CRIU can C/R a process tree, the restored tree will not become a
child of Docker and, from Docker's point of view, the container's
state will remain "Exited" (even after successful restore).

It's important to re-emphasize that by checkpointing and restoring
a Docker container, we mean C/R of a process tree running inside a
container, excluding the Docker daemon itself. As CRIU currently
does not support nested PID namespaces, the C/R process tree cannot
include the Docker daemon which runs in the global PID namespace.

=== Command Line Options ===

In addition to the usual CRIU command line options used when
checkpointing and restoring a process tree, the following command
line options are needed for Docker containers.

==== <code>--root</code> ====

This option has been used in the past only for restore operations
that wanted to change the root of the mount namespace. It was not
used for checkpoint operations.

However, because Docker by default uses the AUFS graph driver and
the AUFS module in the kernel reveals branch pathnames in
<code>/proc/''pid''/map_files</code>, option <code>--root</code>
is used to specify the root of the
mount namespace. Once the kernel AUFS module is fixed, it won't
be necessary to specify this option anymore.

==== <code>--ext-mount-map</code> ====

This option is used to specify the path of the external bind mounts.
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with
source files outside the container's mount namespace. Older versions
of Docker also bind mount <code>/.dockerinit</code>.

For example, assuming the default Docker configuration, <code>/etc/hostname</code>
in the container's mount namespace is bind mounted from the source
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.

==== <code>--manage-cgroups</code> ====

When a process tree exits after a checkpoint operation, the cgroups
that Docker had created for the container are removed. This option
is needed during restore to move the process tree into its cgroups,
re-creating them if necessary.

==== <code>--evasive-devices</code> ====

Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.

==== <code>--inherit-fd</code> ====

For native C/R support, this option tells CRIU to let the restored process "inherit"
its specified file descriptor (instead of restoring from checkpoint).

=== Restore Prework for External C/R ===

Docker supports many storage drivers (AKA graph drivers) including
AUFS, Btrfs, ZFS, DeviceMapper, OverlayFS, and VFS. The user can
specify his/her desired storage driver via the <code>DOCKER_DRIVER</code>
environment variable or the <code>-s (--storage-driver)</code> command
line option.

Currently C/R can only be done on containers using either AUFS, OverlayFS, or VFS.
In the following example, we assume AUFS.

When Docker notices that the container has exited (due to CRIU dump),
it dismantles the container's filesystem. We need to set up the container's
filesystem again before attempting to restore.

=== An External C/R Example ===

Below is an example to show C/R operations for a shell script that
continuously appends a number to a file. You can use tail -f to
see the process in action.

As you will see below, after restore, the process's parent is PID
1 (init), not Docker. Also, although the process has been successfully
restored, Docker still thinks that the container has exited.

To set up the container's AUFS filesystem before restore, its branch
information should be saved before checkpointing the container.
For convenience, however, AUFS branch information is saved in the
dump.log file. So we can examine dump.log to set up the filesystem
again.

For brevity, the 64-character long container ID is replaced by the
string <container_id> in the following lines.

<pre>
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'
<container_id>
$
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds
$
$ sudo criu dump -o dump.log -v4 -t 17810 \
-D /tmp/img/<container_id> \
--root /var/lib/docker/aufs/mnt/<container_id> \
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
--ext-mount-map /etc/hosts:/etc/hosts \
--ext-mount-map /etc/hostname:/etc/hostname \
--ext-mount-map /.dockerinit:/.dockerinit \
--manage-cgroups \
--evasive-devices
$
$ sudo grep successful /tmp/img/<container_id>/dump.log
(00.020103) Dumping finished successfully
$
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago
$
$ sudo mount -t aufs -o br=\
/var/lib/docker/aufs/diff/<container_id>:\
/var/lib/docker/aufs/diff/<container_id>-init:\
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\
none /var/lib/docker/aufs/mnt/<container_id>
$
$ sudo criu restore -o restore.log -v4 -d
-D /tmp/img/<container_id> \
--root /var/lib/docker/aufs/mnt/<container_id> \
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \
--manage-cgroups \
--evasive-devices
$
$ sudo grep successful /tmp/img/<container_id>/restore.log
(00.424428) Restore finished successfully. Resuming tasks.
$
$ ps -ef | grep /bin/sh
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
$
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago
$
</pre>

=== External C/R Helper Script ===

As seen in the above examples, the CRIU command line for checkpointing and
restoring a Docker container is pretty long. For restore, there is also
an additional step to set up the root filesystem before invoking CRIU.

To automate the C/R process, there is a helper script in the contrib
subdirectory of CRIU sources, called docker_cr.sh. In addition to
invoking CRIU, this helper script sets up the root filesystem for AUFS,
UnionFS, and VFS for restore.

With docker_cr.sh, all you have to provide is the container ID.
If you don't specify a container ID, docker_cr.sh will list all running
containers and prompt you to choose one. Also, as shown in the help
output below, by setting the appropriate environment variable, it's
possible to tell docker_cr.sh which Docker and CRIU binaries to use,
where Docker's home directory is, and where CRIU should save and look
for its image files.

<pre>
# docker_cr.sh --help
Usage:
docker_cr.sh -c|-r [-hv] [<container_id>]
-c, --checkpoint checkpoint container
-h, --help print help message
-r, --restore restore container
-v, --verbose enable verbose mode

Environment:
DOCKER_HOME (default /var/lib/docker)
CRIU_IMG_DIR (default /var/lib/docker/criu_img)
DOCKER_BINARY (default docker)
CRIU_BINARY (default criu)
</pre>

Below is an example to checkpoint and restore Docker container 4397:

<pre>
# docker_cr.sh -c 4397
dump successful
# docker_cr.sh -r 4397
restore successful
</pre>

Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code>
executes. For example:

<pre>
# docker_cr.sh -c -v 40d3
docker binary: docker
criu binary: criu
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \
--manage-cgroups --evasive-devices \
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \
--ext-mount-map /etc/hosts:/etc/hosts \
--ext-mount-map /etc/hostname:/etc/hostname \
--ext-mount-map /.dockerinit:/.dockerinit \
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

dump successful
(00.020827) Dumping finished successfully

# docker_cr.sh -r -v 40d3
docker binary: docker
criu binary: criu
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

mount -t aufs -o
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
none
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf

criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \
-o restore.log --manage-cgroups --evasive-devices \
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid

restore successful
(00.408807) Restore finished successfully. Resuming tasks.

root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done
</pre>

[[Category:HOWTO]]