https://criu.org/api.php?action=feedcontributions&feedformat=atom&user=Saied+KazemiCRIU - User contributions [en]2026-05-13T17:41:38ZUser contributionsMediaWiki 1.35.6https://criu.org/index.php?title=Docker&diff=2529Docker2015-07-16T18:01:44Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. If you just want to experiment with C/R, you can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]}}<br />
<br />
{{Note| The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). However, there is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. See below for instructions on how to apply the relevant patches.}}<br />
<br />
{{Note| If your process uses async IO and your kernel is older than 3.19, you need to apply two patches. See below for instructions.}}<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''1. External C/R''' using CRIU directly on the command line as it's typically<br />
done for any process tree.<br />
<br />
This approach is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
'''2. Native C/R''' using new <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
This approach is called native because the Docker daemon is involved in both checkpoint and restore.<br />
Therefore, its notion of the container state will be corrent. All commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality.<br />
You can watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works.<br />
Source files for Docker 1.5 C/R are<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr here]'''<br />
and for Docker 1.7 C/R are<br />
'''[https://github.com/boucher/docker/tree/cr-combined here]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
<br />
== OverlayFS ==<br />
<br />
The following small kernel patches fix the mount id and symlink target path issues noted above:<br />
<br />
* {{torvalds.git|155e35d4da}} by David Howells<br />
* {{torvalds.git|df1a085af1}} by David Howells<br />
* {{torvalds.git|f25801ee46}} by David Howells<br />
* {{torvalds.git|4bacc9c923}} by David Howells<br />
* {{torvalds.git|9391dd00d1}} by Al Viro<br />
<br />
Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:<br />
<br />
<pre><br />
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git<br />
cd ubuntu-vivid<br />
git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git<br />
git remote update<br />
<br />
git cherry-pick 155e35d4da<br />
git cherry-pick df1a085af1<br />
git cherry-pick f25801ee46<br />
git cherry-pick 4bacc9c923<br />
git cherry-pick 9391dd00d1<br />
<br />
cp /boot/config-$(uname -r) .config<br />
make olddefconfig<br />
make -j 8 bzImage modules<br />
sudo make install modules_install<br />
sudo reboot<br />
</pre><br />
<br />
== Async IO (AIO) ==<br />
<br />
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:<br />
<br />
* {{torvalds.git|bd9b51e79c}} by Al Viro<br />
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov<br />
<br />
== External C/R ==<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
=== <code>--inherit-fd</code> ===<br />
<br />
For native C/R support, this option tells CRIU to let the restored process "inherit"<br />
its specified file descriptor (instead of restoring from checkpoint).<br />
<br />
== Restore Prework for External C/R ==<br />
<br />
Docker supports many storage drivers (AKA graph drivers) including<br />
AUFS, Btrfs, ZFS, DeviceMapper, OverlayFS, and VFS. The user can<br />
specify his/her desired storage driver via the <code>DOCKER_DRIVER</code><br />
environment variable or the <code>-s (--storage-driver)</code> command<br />
line option.<br />
<br />
Currently C/R can only be done on containers using either AUFS, OverlayFS, or VFS.<br />
In the following example, we assume AUFS.<br />
<br />
When Docker notices that the container has exited (due to CRIU dump),<br />
it dismantles the container's filesystem. We need to set up the container's<br />
filesystem again before attempting to restore.<br />
<br />
== An External C/R Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== External C/R Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Upstream_kernel_commits&diff=2528Upstream kernel commits2015-07-16T17:36:59Z<p>Saied Kazemi: </p>
<hr />
<div>{{Like}}<br />
<br />
== Pending patches ==<br />
<br />
{| class="wikitable"<br />
! Reference<br />
! Description<br />
! Status<br />
|-<br />
|[http://lists.openvz.org/pipermail/criu/2014-April/013655.html 013655] || tcp: allow to repair a tcp connections in closing states || ?<br />
|-<br />
|[http://lwn.net/Articles/633622/ 633622] || [RFC] kernel: add a netlink interface to get information about processes || ?<br />
|-<br />
|[https://git.kernel.org/cgit/linux/kernel/git/kees/linux.git/commit/?h=seccomp/tip&id=352871545768dcb0cb2fe7065eea4101ce026e7d seccomp/tip/35287154] || seccomp: add ptrace options for suspend/resume || In maintainer's tree<br />
|}<br />
<br />
== Merged patches ==<br />
<br />
This table lists CRIU-related kernel commits already merged to vanilla.<br />
<br />
{| class="wikitable sortable"<br />
! scope="col" class="unsortable" | Commit<br />
! scope="col" class="unsortable" | Description<br />
! scope="col" | Kernel version<br />
|-<br />
|{{torvalds.git|7773fbc541}} || procfs: make proc_get_link to use dentry instead of inode || 3.3<br />
|-<br />
|{{torvalds.git|640708a2cff}} || procfs: introduce the /proc/<pid>/map_files/ directory || 3.3<br />
|-<br />
|{{torvalds.git|067bce1a06}} || c/r: introduce CHECKPOINT_RESTORE symbol || 3.3<br />
|-<br />
|{{torvalds.git|028ee4be34}} || c/r: prctl: add PR_SET_MM codes to set up mm_struct entries || 3.3<br />
|-<br />
|{{torvalds.git|b3f7f573a2}} || c/r: procfs: add start_data, end_data, start_brk members to /proc/$pid/stat v4 || 3.3<br />
|-<br />
|{{torvalds.git|b8f566b04d}} || sysctl: add the kernel.ns_last_pid control || 3.3<br />
|-<br />
|{{net-next.git|c9da99e647}} || unix_diag: Fixup RQLEN extension report || 3.3<br />
|-<br />
|{{net-next.git|885ee74d5d}} || af_unix: Move CINQ/COUTQ code to helpers || 3.3<br />
|-<br />
|{{net-next.git|257b529876}} || unix_diag: Add the MEMINFO extension || 3.3<br />
|-<br />
|{{net-next.git|c0636faa53}} || inet_diag: Add the SKMEMINFO extension || 3.3<br />
|-<br />
|{{net-next.git|5d2e5f274f}} || sock_diag: Introduce the meminfo nla core (v2) || 3.3<br />
|-<br />
|{{net-next.git|288461e154}} || unix_diag: Include unix_diag.h into header-y target || 3.3<br />
|-<br />
|{{net-next.git|e6fe2371bd}} || sock_diag: Arrange sock_diag.h such that it is exportable to userspace || 3.3<br />
|-<br />
|{{net-next.git|e09e9d189b}} || unix: If we happen to find peer NULL when diag dumping, write zero. || 3.3<br />
|-<br />
|{{net-next.git|3b0723c12e}} || unix_diag: Fix incoming connections nla length || 3.3<br />
|-<br />
|{{net-next.git|2ea744a583}} || net: unix -- Add missing module.h inclusion || 3.3<br />
|-<br />
|{{net-next.git|5d531aaa64}} || unix_diag: Write it into kbuild || 3.3<br />
|-<br />
|{{net-next.git|cbf391958a}} || unix_diag: Receive queue lenght NLA || 3.3<br />
|-<br />
|{{net-next.git|2aac7a2cb0}} || unix_diag: Pending connections IDs NLA || 3.3<br />
|-<br />
|{{net-next.git|ac02be8d96}} || unix_diag: Unix peer inode NLA || 3.3<br />
|-<br />
|{{net-next.git|5f7b056946}} || unix_diag: Unix inode info NLA || 3.3<br />
|-<br />
|{{net-next.git|f5248b48a6}} || unix_diag: Unix socket name NLA || 3.3<br />
|-<br />
|{{net-next.git|5d3cae8bc3}} || unix_diag: Dumping exact socket core || 3.3<br />
|-<br />
|{{net-next.git|45a96b9be6}} || unix_diag: Dumping all sockets core || 3.3<br />
|-<br />
|{{net-next.git|22931d3b90}} || unix_diag: Basic module skeleton || 3.3<br />
|-<br />
|{{net-next.git|fa7ff56f75}} || af_unix: Export stuff required for diag module || 3.3<br />
|-<br />
|{{net-next.git|f65c1b534b}} || sock_diag: Generalize requests cookies managements || 3.3<br />
|-<br />
|{{net-next.git|aec8dc62f6}} || sock_diag: Fix module netlink aliases || 3.3<br />
|-<br />
|{{net-next.git|e7c466e58e}} || sock_diag: Move the SOCK_DIAG_BY_FAMILY cmd declaration || 3.3<br />
|-<br />
|{{net-next.git|86e62ad6b2}} || udp_diag: Fix the !ipv6 case || 3.3<br />
|-<br />
|{{net-next.git|b872a2371f}} || udp_diag: Make it module when ipv6 is a module || 3.3<br />
|-<br />
|{{net-next.git|507dd7961e}} || udp_diag: Wire the udp_diag module into kbuild || 3.3<br />
|-<br />
|{{net-next.git|b6d640c228}} || udp_diag: Implement the dump-all functionality || 3.3<br />
|-<br />
|{{net-next.git|a925aa00a5}} || udp_diag: Implement the get_exact dumping functionality || 3.3<br />
|-<br />
|{{net-next.git|52b7c59bc3}} || udp_diag: Basic skeleton || 3.3<br />
|-<br />
|{{net-next.git|fce823381e}} || udp: Export code sk lookup routines || 3.3<br />
|-<br />
|{{net-next.git|1942c518ca}} || inet_diag: Generalize inet_diag dump and get_exact calls || 3.3<br />
|-<br />
|{{net-next.git|3c4d05c805}} || inet_diag: Introduce the inet socket dumping routine || 3.3<br />
|-<br />
|{{net-next.git|8d07d1518a}} || inet_diag: Introduce the byte-code run on an inet socket || 3.3<br />
|-<br />
|{{net-next.git|efb3cb428d}} || inet_diag: Split inet_diag_get_exact into parts || 3.3<br />
|-<br />
|{{net-next.git|476f7dbff3}} || inet_diag: Split inet_diag_get_exact into parts || 3.3<br />
|-<br />
|{{net-next.git|b005ab4ef8}} || inet_diag: Export inet diag cookie checking routine || 3.3<br />
|-<br />
|{{net-next.git|87c22ea52e}} || inet_diag: Reduce the number of args for bytecode run routine || 3.3<br />
|-<br />
|{{net-next.git|7b35eadd7e}} || inet_diag: Remove indirect sizeof from inet diag handlers || 3.3<br />
|-<br />
|{{net-next.git|8ef874bfc7}} || sock_diag: Move the sock_ code to net/core/ || 3.3<br />
|-<br />
|{{net-next.git|a029fe26b6}} || inet_diag: Cleanup type2proto last user || 3.3<br />
|-<br />
|{{net-next.git|d23deaa07b}} || inet_diag: Introduce socket family checks || 3.3<br />
|-<br />
|{{net-next.git|25c4cd2b6d}} || inet_diag: Switch the _dump to work with new header || 3.3<br />
|-<br />
|{{net-next.git|fe50ce2846}} || inet_diag: Switch the _get_exact to work with new header || 3.3<br />
|-<br />
|{{net-next.git|126fdc3249}} || inet_diag: Introduce new inet_diag_req header || 3.3<br />
|-<br />
|{{net-next.git|d366477a52}} || sock_diag: Initial skeleton || 3.3<br />
|-<br />
|{{net-next.git|f13c95f0e2}} || inet_diag: Switch from _GETSOCK to IPPROTO_ numbers || 3.3<br />
|-<br />
|{{net-next.git|37f352b5e3}} || inet_diag: Move byte-code finding up the call-stack || 3.3<br />
|-<br />
|{{net-next.git|8d34172dfd}} || sock_diag: Introduce new message type || 3.3<br />
|-<br />
|{{torvalds.git|818411616b}} || fs, proc: Introduce /proc/<pid>/task/<tid>/children entry v9 || 3.5<br />
|-<br />
|{{torvalds.git|5b172087f9}} || c/r: procfs: add arg_start/end, env_start/end and exit_code members || 3.5<br />
|-<br />
|{{torvalds.git|fe8c7f5cbf}} || c/r: prctl: Extend PR_SET_MM to set up more mm_struct entries || 3.5<br />
|-<br />
|{{torvalds.git|d97b46a646}} || syscalls, x86: Add __NR_kcmp syscall || 3.5<br />
|-<br />
|{{torvalds.git|b32dfe3771}} || c/r: prctl: Add ability to set new mm_struct::exe_file || 3.5<br />
|-<br />
|{{torvalds.git|79f0713d40}} || prctl: Use CAP_SYS_RESOURCE for PR_SET_MM option || 3.3<br />
|-<br />
|{{torvalds.git|300f786b26}} || prctl: add ability to get clear_tid_address || 3.5<br />
|-<br />
|{{torvalds.git|4934b0329f}} || datagram: Factor out sk queue referencing || 3.4<br />
|-<br />
|{{torvalds.git|3f518bf745}} || datagram: Add offset argument to __skb_recv_datagram || 3.4<br />
|-<br />
|{{torvalds.git|da5ef6e51b}} || skb: Add skb_peek_next helper || 3.4<br />
|-<br />
|{{torvalds.git|ef64a54f6e}} || sock: Introduce the SO_PEEK_OFF sock option || 3.4<br />
|-<br />
|{{torvalds.git|f55bb7f9cb}} || unix: Support peeking offset for datagram and seqpacket sockets || 3.4<br />
|-<br />
|{{torvalds.git|fc0d753641}} || unix: Support peeking offset for stream sockets || 3.4<br />
|-<br />
|{{torvalds.git|1d151c337d}} || fcntl: Add F_GETOWNER_UIDS option v3 || 3.6<br />
|-<br />
|{{torvalds.git|370816aef0}} || tcp: Move code around || 3.5<br />
|-<br />
|{{torvalds.git|ee9952831c}} || tcp: Initial repair mode || 3.5<br />
|-<br />
|{{torvalds.git|c0e88ff0f2}} || tcp: Repair socket queues || 3.5<br />
|-<br />
|{{torvalds.git|5e6a3ce657}} || tcp: Report mss_clamp with TCP_MAXSEG option in repair mode || 3.5<br />
|-<br />
|{{torvalds.git|b139ba4e90}} || tcp: Repair connection-time negotiated parameters || 3.5<br />
|-<br />
|{{torvalds.git|de248a75c3}} || tcp repair: Fix unaligned access when repairing options (v2) || 3.5<br />
|-<br />
|{{torvalds.git|736f24d5e5}} || c/r: prctl: Drop VMA flags test on PR_SET_MM_ stack data assignment || 3.5<br />
|-<br />
|{{torvalds.git|5702c5eeab}} || c/r: prctl: Move PR_GET_TID_ADDRESS to a proper place || 3.5<br />
|-<br />
|{{torvalds.git|16fbdce62d}} || proc: report file/anon bit in /proc/pid/pagemap || 3.5<br />
|-<br />
|{{torvalds.git|bca1554373}} || proc/smaps: show amount of nonlinear ptes in vma || 3.5<br />
|-<br />
|{{torvalds.git|b14f243a42}} || net: Dont use ifindices in hash fns || 3.7<br />
|-<br />
|{{torvalds.git|9c7dafbfab}} || net: Allow to create links with given ifindex || 3.7<br />
|-<br />
|{{torvalds.git|e6f8f1a739}} || veth: Allow to create peer link with given ifindex || 3.7<br />
|-<br />
|{{torvalds.git|aa79e66eee}} || net: Make ifindex generation per-net namespace || 3.7<br />
|-<br />
|{{torvalds.git|1fb9489bf1}} || net: Loopback ifindex is constant now || 3.7<br />
|-<br />
|{{torvalds.git|faf60af17f}} || procfs: Move /proc/pid/fd[info] handling code to fd.[ch] || 3.7<br />
|-<br />
|{{torvalds.git|ddd3e0771b}} || procfs: Convert /proc/pid/fdinfo/ handling routines to seq-file || 3.7<br />
|-<br />
|{{torvalds.git|55985dd72a}} || procfs: Add ability to plug in auxiliary fdinfo providers || 3.8<br />
|-<br />
|{{torvalds.git|cbac5542d4}} || fs, eventfd: Add procfs fdinfo helper || 3.8<br />
|-<br />
|{{torvalds.git|138d22b586}} || fs, epoll: Add procfs fdinfo helper v2 || 3.8<br />
|-<br />
|{{torvalds.git|711c7bf991}} || fs, exportfs: Add export_encode_inode_fh helper || 3.8<br />
|-<br />
|{{torvalds.git|be77196b80}} || fs, notify: Add procfs fdinfo helper || 3.8<br />
|-<br />
|{{torvalds.git|e6dbcafb74}} || fs, fanotify: Add @mflags field to fanotify output || 3.8<br />
|-<br />
|{{torvalds.git|2787b04b6c}} || packet: Introduce net/packet/internal.h header || 3.7<br />
|-<br />
|{{torvalds.git|96ec632714}} || packet: Diag core and basic socket info dumping || 3.7<br />
|-<br />
|{{torvalds.git|8a360be0c5}} || packet: Report more packet sk info via diag module || 3.7<br />
|-<br />
|{{torvalds.git|eea68e2f1a}} || packet: Report socket mclist info via diag module || 3.7<br />
|-<br />
|{{torvalds.git|16f01365fa}} || packet: Report rings cfg via diag engine || 3.7<br />
|-<br />
|{{torvalds.git|fff3321d75}} || packet: Report fanout status via diag engine || 3.7<br />
|-<br />
|{{torvalds.git|0fa7fa98db}} || packet: Protect packet sk list with mutex (v2) || 3.7<br />
|-<br />
|{{torvalds.git|579035dc5d}} || kernel: limit a value of ns_last_pid to (0, max_pid) || 3.6<br />
|-<br />
|{{torvalds.git|bc26ccd8fc}} || tcp: restore rcv_wscale in a repair mode (v2) || 3.6<br />
|-<br />
|{{torvalds.git|f7b86bfe8d}} || sockopt: Make SO_BINDTODEVICE readable || 3.8<br />
|-<br />
|{{torvalds.git|e4e541a848}} || sock-diag: Report shutdown for inet and unix sockets (v2) || 3.8<br />
|-<br />
|{{torvalds.git|834f82e2aa}} || procfs: add VmFlags field in smaps output || 3.8<br />
|-<br />
|{{torvalds.git|06026d911c}} || tty: pty - Move TIOCPKT handling into pty.c || 3.8<br />
|-<br />
|{{torvalds.git|c6298038bc}} || tty, ioctls -- Add new ioctl definitions for tty flags fetching || 3.8<br />
|-<br />
|{{torvalds.git|84fd7bdf12}} || tty: Add get- ioctls to fetch tty status v3 || 3.8<br />
|-<br />
|{{torvalds.git|a8fc927780}} || sk-filter: Add ability to get socket filter program (v2) || 3.8<br />
|-<br />
|{{torvalds.git|cacb6ba0f3}} || net: inet_diag -- Return error code if protocol handler is missed || 3.7<br />
|-<br />
|{{torvalds.git|c454e6111d}} || tcp-repair: Handle zero-length data put in rcv queue || 3.7<br />
|-<br />
|{{torvalds.git|ec34232575}} || tcp: fix retransmission in repair mode || 3.7<br />
|-<br />
|{{torvalds.git|2b9164771e}} || ipv6: adapt connect for repair move || 3.8<br />
|-<br />
|{{torvalds.git|c91f6df2db}} || sockopt: Change getsockopt() of SO_BINDTODEVICE to return an interface name || 3.8<br />
|-<br />
|{{torvalds.git|3fcfe78658}} || ipc: add more comments to message copying related code || 3.8<br />
|-<br />
|{{torvalds.git|51eeacaa07}} || ipc: simplify message copying || 3.8<br />
|-<br />
|{{torvalds.git|b30efe2775}} || ipc: convert prepare_copy() from macro to function || 3.8<br />
|-<br />
|{{torvalds.git|85398aa8de}} || ipc: simplify free_copy() call || 3.8<br />
|-<br />
|{{torvalds.git|3a665531a3}} || selftests: IPC message queue copy feature test || 3.8<br />
|-<br />
|{{torvalds.git|4a674f34ba}} || ipc: introduce message queue copy feature || 3.8<br />
|-<br />
|{{torvalds.git|f9dd87f473}} || ipc: message queue receive cleanup || 3.8<br />
|-<br />
|{{torvalds.git|03f5956680}} || ipc: add sysctl to specify desired next object id || 3.8<br />
|-<br />
|{{torvalds.git|9afdacda02}} || ipc: remove forced assignment of selected message || 3.8<br />
|-<br />
|{{torvalds.git|3f7d1fe108}} || arm: Wire up kcmp syscall || 3.10<br />
|-<br />
|{{torvalds.git|1e142b29e2}} || kcmp: make it depend on CHECKPOINT_RESTORE || 3.9<br />
|-<br />
|{{torvalds.git|ceaa1fef65}} || tcp: adding a per-socket timestamp offset || 3.9<br />
|-<br />
|{{torvalds.git|93be6ce0e9}} || tcp: set and get per-socket timestamp || 3.9<br />
|-<br />
|{{torvalds.git|ee684b6f28}} || tcp: send packets with a socket timestamp || 3.9<br />
|-<br />
|{{torvalds.git|66dd34ad}} || signal: allow to send any siginfo to itself || 3.9<br />
|-<br />
|{{torvalds.git|ae5fc987}} || net: fix *_DIAG_MAX constants || 3.9<br />
|-<br />
|{{torvalds.git|0f29c768}} || net: prepare netlink code for netlink diag || 3.10<br />
|-<br />
|{{torvalds.git|eaaa3139}} || netlink: Diag core and basic socket info dumping (v2) || 3.10<br />
|-<br />
|{{torvalds.git|84c751bd}} || ptrace: add ability to retrieve signals without removing from a queue (v4) || 3.10<br />
|-<br />
|{{torvalds.git|040fa020}} || clear_refs: Sanitize accepted commands declaration || 3.11<br />
|-<br />
|{{torvalds.git|af9de7eb}} || clear_refs: Introduce private struct for mm_walk || 3.11<br />
|-<br />
|{{torvalds.git|2b0a9f01}} || pagemap-introduce-pagemap_entry_t-without-pmshift-bits || 3.11<br />
|-<br />
|{{torvalds.git|0f8975ec}} || mm: soft-dirty bits for user memory changes tracking || 3.11<br />
|-<br />
|{{torvalds.git|541c237c}} || pagemap: prepare to reuse constant bits with page-shift || 3.11<br />
|-<br />
|{{torvalds.git|57b8015e}} || posix-timers: Show sigevent info in proc file || 3.10<br />
|-<br />
|{{torvalds.git|48f6a7a5}} || posix-timers: Introduce /proc/PID/timers file || 3.10<br />
|-<br />
|{{torvalds.git|5ed67f05}} || posix timers: Allocate timer id per process (v2) || 3.10<br />
|-<br />
|{{torvalds.git|15ef0298}} || posix-timers: Show clock ID in proc file || 3.10<br />
|-<br />
|{{torvalds.git|29000cae}} || ptrace: add ability to get/set signal-blocked mask (v2) || 3.11<br />
|-<br />
|{{torvalds.git|274038f8}} || tun: Report "persist" flag to userspace || 3.11<br />
|-<br />
|{{torvalds.git|179ef71c}} || mm: Save soft-dirty bits on swapped pages || 3.11<br />
|-<br />
|{{torvalds.git|41bb3476}} || mm: Save soft-dirty bits on file pages || 3.11<br />
|-<br />
|{{torvalds.git|76975e9c}} || tun: Get skfilter layout || 3.12<br />
|-<br />
|{{torvalds.git|849c9b6f}} || tun: Allow to skip filter on attach || 3.12<br />
|-<br />
|{{torvalds.git|3d407a80}} || tun: Report whether the queue is attached or not || 3.12<br />
|-<br />
|{{torvalds.git|fb7589a1}} || tun: Add ability to create tun device with given index || 3.12<br />
|-<br />
|{{torvalds.git|e3e12028}} || tcp: don't apply tsoffset if rcv_tsecr is zero || 3.11<br />
|-<br />
|{{torvalds.git|c7781a6e}} || tcp: initialize rcv_tstamp for restored sockets || 3.11<br />
|-<br />
|{{torvalds.git|7ed5c5ae}} || tcp: set timestamps for restored skb-s || 3.11<br />
|-<br />
|{{torvalds.git|6dec97dc}} || mm: move_ptes -- Set soft dirty bit depending on pte type || 3.11<br />
|-<br />
|{{torvalds.git|c3d16e16}} || mm: migration -- Do not loose soft dirty bit if page is in migration state || 3.12<br />
|-<br />
|{{torvalds.git|e9cdd6e7}} || mm: pagemap -- Inspect _PAGE_SOFT_DIRTY only on present pages || 3.12<br />
|-<br />
|{{torvalds.git|dbde4979}} || tcp: don't update snd_nxt, when a socket is switched from repair mode || 3.13<br />
|-<br />
|{{torvalds.git|d9104d1c}} || mm: track vma changes with VM_SOFTDIRTY bit || 3.12<br />
|-<br />
|{{torvalds.git|34228d47}} || mm: Ignore VM_SOFTDIRTY on VMA merging || 3.14, 3.13-stable<br />
|-<br />
|{{torvalds.git|24f91eba1}} || mm: don't lose the SOFT_DIRTY flag on mprotect || 3.14, 3.13-stable<br />
|-<br />
|{{torvalds.git|49d063cb3}} || proc: show mnt_id in /proc/pid/fdinfo || 3.15<br />
|-<br />
|{{torvalds.git|0bf073315}} || mm: Make freshly remapped file pages being softdirty unconditionally || 3.15<br />
|-<br />
|{{torvalds.git|9aed8614a}} || mm: Don't forget to set softdirty on file mapped fault || 3.17<br />
|-<br />
|{{torvalds.git|b43790eed}} || mm: Don't forget to save file map softdiry bit on unmap || 3.15<br />
|-<br />
|{{torvalds.git|c86c97ff4}} || mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty || 3.15<br />
|-<br />
|{{torvalds.git|af9c4957c}} || timerfd: Implement show_fdinfo method || 3.17<br />
|-<br />
|{{torvalds.git|854d06d9f}} || docs: Procfs -- Document timerfd output || 3.17<br />
|-<br />
|{{torvalds.git|5442e9fbd}} || timerfd: Implement timerfd_ioctl method to restore timerfd_ctx::ticks, v3 || 3.17<br />
|-<br />
|{{torvalds.git|64e455079}} || mm: softdirty: enable write notifications on VMAs after VM_SOFTDIRTY cleared || 3.18<br />
|-<br />
|{{torvalds.git|9c5990240}} || mm: introduce check_data_rlimit helper || 3.18<br />
|-<br />
|{{torvalds.git|8764b338b}} || mm: use may_adjust_brk helper || 3.18<br />
|-<br />
|{{torvalds.git|71fe97e18}} || prctl: PR_SET_MM -- factor out mmap_sem when updating mm::exe_file || 3.18<br />
|-<br />
|{{torvalds.git|f606b77f1}} || prctl: PR_SET_MM -- introduce PR_SET_MM_MAP operation || 3.18<br />
|-<br />
|{{torvalds.git|9183df25f}} || shm: add memfd_create() syscall || 3.17<br />
|-<br />
|{{torvalds.git|e4a0d3e72}} || aio: Make it possible to remap aio ring || 3.19<br />
|-<br />
|{{torvalds.git|6c8c90319}} || proc: show locks in /proc/pid/fdinfo/X || 4.1<br />
|-<br />
|{{torvalds.git|155e35d4d}} || VFS: Introduce inode-getting helpers for layered/unioned fs environments || 4.0<br />
|-<br />
|{{torvalds.git|df1a085af}} || VFS: Add a fallthrough flag for marking virtual dentries || 4.0<br />
|-<br />
|{{torvalds.git|f25801ee4}} || overlay: Call ovl_drop_write() earlier in ovl_dentry_open() || 4.2-rc2<br />
|-<br />
|{{torvalds.git|4bacc9c92}} || overlayfs: Make f_path always point to the overlay and f_inode to the underlay || 4.2-rc2<br />
|-<br />
|{{torvalds.git|9391dd00d}} || fix a braino in ovl_d_select_inode() || 4.2-rc2<br />
|}<br />
<br />
[[Category:Development]]</div>Saied Kazemihttps://criu.org/index.php?title=Upstream_kernel_commits&diff=2527Upstream kernel commits2015-07-16T17:36:16Z<p>Saied Kazemi: </p>
<hr />
<div>{{Like}}<br />
<br />
== Pending patches ==<br />
<br />
{| class="wikitable"<br />
! Reference<br />
! Description<br />
! Status<br />
|-<br />
|[http://lists.openvz.org/pipermail/criu/2014-April/013655.html 013655] || tcp: allow to repair a tcp connections in closing states || ?<br />
|-<br />
|[http://lwn.net/Articles/633622/ 633622] || [RFC] kernel: add a netlink interface to get information about processes || ?<br />
|-<br />
|[https://git.kernel.org/cgit/linux/kernel/git/kees/linux.git/commit/?h=seccomp/tip&id=352871545768dcb0cb2fe7065eea4101ce026e7d seccomp/tip/35287154] || seccomp: add ptrace options for suspend/resume || In maintainer's tree<br />
|}<br />
<br />
== Merged patches ==<br />
<br />
This table lists CRIU-related kernel commits already merged to vanilla.<br />
<br />
{| class="wikitable sortable"<br />
! scope="col" class="unsortable" | Commit<br />
! scope="col" class="unsortable" | Description<br />
! scope="col" | Kernel version<br />
|-<br />
|{{torvalds.git|7773fbc541}} || procfs: make proc_get_link to use dentry instead of inode || 3.3<br />
|-<br />
|{{torvalds.git|640708a2cff}} || procfs: introduce the /proc/<pid>/map_files/ directory || 3.3<br />
|-<br />
|{{torvalds.git|067bce1a06}} || c/r: introduce CHECKPOINT_RESTORE symbol || 3.3<br />
|-<br />
|{{torvalds.git|028ee4be34}} || c/r: prctl: add PR_SET_MM codes to set up mm_struct entries || 3.3<br />
|-<br />
|{{torvalds.git|b3f7f573a2}} || c/r: procfs: add start_data, end_data, start_brk members to /proc/$pid/stat v4 || 3.3<br />
|-<br />
|{{torvalds.git|b8f566b04d}} || sysctl: add the kernel.ns_last_pid control || 3.3<br />
|-<br />
|{{net-next.git|c9da99e647}} || unix_diag: Fixup RQLEN extension report || 3.3<br />
|-<br />
|{{net-next.git|885ee74d5d}} || af_unix: Move CINQ/COUTQ code to helpers || 3.3<br />
|-<br />
|{{net-next.git|257b529876}} || unix_diag: Add the MEMINFO extension || 3.3<br />
|-<br />
|{{net-next.git|c0636faa53}} || inet_diag: Add the SKMEMINFO extension || 3.3<br />
|-<br />
|{{net-next.git|5d2e5f274f}} || sock_diag: Introduce the meminfo nla core (v2) || 3.3<br />
|-<br />
|{{net-next.git|288461e154}} || unix_diag: Include unix_diag.h into header-y target || 3.3<br />
|-<br />
|{{net-next.git|e6fe2371bd}} || sock_diag: Arrange sock_diag.h such that it is exportable to userspace || 3.3<br />
|-<br />
|{{net-next.git|e09e9d189b}} || unix: If we happen to find peer NULL when diag dumping, write zero. || 3.3<br />
|-<br />
|{{net-next.git|3b0723c12e}} || unix_diag: Fix incoming connections nla length || 3.3<br />
|-<br />
|{{net-next.git|2ea744a583}} || net: unix -- Add missing module.h inclusion || 3.3<br />
|-<br />
|{{net-next.git|5d531aaa64}} || unix_diag: Write it into kbuild || 3.3<br />
|-<br />
|{{net-next.git|cbf391958a}} || unix_diag: Receive queue lenght NLA || 3.3<br />
|-<br />
|{{net-next.git|2aac7a2cb0}} || unix_diag: Pending connections IDs NLA || 3.3<br />
|-<br />
|{{net-next.git|ac02be8d96}} || unix_diag: Unix peer inode NLA || 3.3<br />
|-<br />
|{{net-next.git|5f7b056946}} || unix_diag: Unix inode info NLA || 3.3<br />
|-<br />
|{{net-next.git|f5248b48a6}} || unix_diag: Unix socket name NLA || 3.3<br />
|-<br />
|{{net-next.git|5d3cae8bc3}} || unix_diag: Dumping exact socket core || 3.3<br />
|-<br />
|{{net-next.git|45a96b9be6}} || unix_diag: Dumping all sockets core || 3.3<br />
|-<br />
|{{net-next.git|22931d3b90}} || unix_diag: Basic module skeleton || 3.3<br />
|-<br />
|{{net-next.git|fa7ff56f75}} || af_unix: Export stuff required for diag module || 3.3<br />
|-<br />
|{{net-next.git|f65c1b534b}} || sock_diag: Generalize requests cookies managements || 3.3<br />
|-<br />
|{{net-next.git|aec8dc62f6}} || sock_diag: Fix module netlink aliases || 3.3<br />
|-<br />
|{{net-next.git|e7c466e58e}} || sock_diag: Move the SOCK_DIAG_BY_FAMILY cmd declaration || 3.3<br />
|-<br />
|{{net-next.git|86e62ad6b2}} || udp_diag: Fix the !ipv6 case || 3.3<br />
|-<br />
|{{net-next.git|b872a2371f}} || udp_diag: Make it module when ipv6 is a module || 3.3<br />
|-<br />
|{{net-next.git|507dd7961e}} || udp_diag: Wire the udp_diag module into kbuild || 3.3<br />
|-<br />
|{{net-next.git|b6d640c228}} || udp_diag: Implement the dump-all functionality || 3.3<br />
|-<br />
|{{net-next.git|a925aa00a5}} || udp_diag: Implement the get_exact dumping functionality || 3.3<br />
|-<br />
|{{net-next.git|52b7c59bc3}} || udp_diag: Basic skeleton || 3.3<br />
|-<br />
|{{net-next.git|fce823381e}} || udp: Export code sk lookup routines || 3.3<br />
|-<br />
|{{net-next.git|1942c518ca}} || inet_diag: Generalize inet_diag dump and get_exact calls || 3.3<br />
|-<br />
|{{net-next.git|3c4d05c805}} || inet_diag: Introduce the inet socket dumping routine || 3.3<br />
|-<br />
|{{net-next.git|8d07d1518a}} || inet_diag: Introduce the byte-code run on an inet socket || 3.3<br />
|-<br />
|{{net-next.git|efb3cb428d}} || inet_diag: Split inet_diag_get_exact into parts || 3.3<br />
|-<br />
|{{net-next.git|476f7dbff3}} || inet_diag: Split inet_diag_get_exact into parts || 3.3<br />
|-<br />
|{{net-next.git|b005ab4ef8}} || inet_diag: Export inet diag cookie checking routine || 3.3<br />
|-<br />
|{{net-next.git|87c22ea52e}} || inet_diag: Reduce the number of args for bytecode run routine || 3.3<br />
|-<br />
|{{net-next.git|7b35eadd7e}} || inet_diag: Remove indirect sizeof from inet diag handlers || 3.3<br />
|-<br />
|{{net-next.git|8ef874bfc7}} || sock_diag: Move the sock_ code to net/core/ || 3.3<br />
|-<br />
|{{net-next.git|a029fe26b6}} || inet_diag: Cleanup type2proto last user || 3.3<br />
|-<br />
|{{net-next.git|d23deaa07b}} || inet_diag: Introduce socket family checks || 3.3<br />
|-<br />
|{{net-next.git|25c4cd2b6d}} || inet_diag: Switch the _dump to work with new header || 3.3<br />
|-<br />
|{{net-next.git|fe50ce2846}} || inet_diag: Switch the _get_exact to work with new header || 3.3<br />
|-<br />
|{{net-next.git|126fdc3249}} || inet_diag: Introduce new inet_diag_req header || 3.3<br />
|-<br />
|{{net-next.git|d366477a52}} || sock_diag: Initial skeleton || 3.3<br />
|-<br />
|{{net-next.git|f13c95f0e2}} || inet_diag: Switch from _GETSOCK to IPPROTO_ numbers || 3.3<br />
|-<br />
|{{net-next.git|37f352b5e3}} || inet_diag: Move byte-code finding up the call-stack || 3.3<br />
|-<br />
|{{net-next.git|8d34172dfd}} || sock_diag: Introduce new message type || 3.3<br />
|-<br />
|{{torvalds.git|818411616b}} || fs, proc: Introduce /proc/<pid>/task/<tid>/children entry v9 || 3.5<br />
|-<br />
|{{torvalds.git|5b172087f9}} || c/r: procfs: add arg_start/end, env_start/end and exit_code members || 3.5<br />
|-<br />
|{{torvalds.git|fe8c7f5cbf}} || c/r: prctl: Extend PR_SET_MM to set up more mm_struct entries || 3.5<br />
|-<br />
|{{torvalds.git|d97b46a646}} || syscalls, x86: Add __NR_kcmp syscall || 3.5<br />
|-<br />
|{{torvalds.git|b32dfe3771}} || c/r: prctl: Add ability to set new mm_struct::exe_file || 3.5<br />
|-<br />
|{{torvalds.git|79f0713d40}} || prctl: Use CAP_SYS_RESOURCE for PR_SET_MM option || 3.3<br />
|-<br />
|{{torvalds.git|300f786b26}} || prctl: add ability to get clear_tid_address || 3.5<br />
|-<br />
|{{torvalds.git|4934b0329f}} || datagram: Factor out sk queue referencing || 3.4<br />
|-<br />
|{{torvalds.git|3f518bf745}} || datagram: Add offset argument to __skb_recv_datagram || 3.4<br />
|-<br />
|{{torvalds.git|da5ef6e51b}} || skb: Add skb_peek_next helper || 3.4<br />
|-<br />
|{{torvalds.git|ef64a54f6e}} || sock: Introduce the SO_PEEK_OFF sock option || 3.4<br />
|-<br />
|{{torvalds.git|f55bb7f9cb}} || unix: Support peeking offset for datagram and seqpacket sockets || 3.4<br />
|-<br />
|{{torvalds.git|fc0d753641}} || unix: Support peeking offset for stream sockets || 3.4<br />
|-<br />
|{{torvalds.git|1d151c337d}} || fcntl: Add F_GETOWNER_UIDS option v3 || 3.6<br />
|-<br />
|{{torvalds.git|370816aef0}} || tcp: Move code around || 3.5<br />
|-<br />
|{{torvalds.git|ee9952831c}} || tcp: Initial repair mode || 3.5<br />
|-<br />
|{{torvalds.git|c0e88ff0f2}} || tcp: Repair socket queues || 3.5<br />
|-<br />
|{{torvalds.git|5e6a3ce657}} || tcp: Report mss_clamp with TCP_MAXSEG option in repair mode || 3.5<br />
|-<br />
|{{torvalds.git|b139ba4e90}} || tcp: Repair connection-time negotiated parameters || 3.5<br />
|-<br />
|{{torvalds.git|de248a75c3}} || tcp repair: Fix unaligned access when repairing options (v2) || 3.5<br />
|-<br />
|{{torvalds.git|736f24d5e5}} || c/r: prctl: Drop VMA flags test on PR_SET_MM_ stack data assignment || 3.5<br />
|-<br />
|{{torvalds.git|5702c5eeab}} || c/r: prctl: Move PR_GET_TID_ADDRESS to a proper place || 3.5<br />
|-<br />
|{{torvalds.git|16fbdce62d}} || proc: report file/anon bit in /proc/pid/pagemap || 3.5<br />
|-<br />
|{{torvalds.git|bca1554373}} || proc/smaps: show amount of nonlinear ptes in vma || 3.5<br />
|-<br />
|{{torvalds.git|b14f243a42}} || net: Dont use ifindices in hash fns || 3.7<br />
|-<br />
|{{torvalds.git|9c7dafbfab}} || net: Allow to create links with given ifindex || 3.7<br />
|-<br />
|{{torvalds.git|e6f8f1a739}} || veth: Allow to create peer link with given ifindex || 3.7<br />
|-<br />
|{{torvalds.git|aa79e66eee}} || net: Make ifindex generation per-net namespace || 3.7<br />
|-<br />
|{{torvalds.git|1fb9489bf1}} || net: Loopback ifindex is constant now || 3.7<br />
|-<br />
|{{torvalds.git|faf60af17f}} || procfs: Move /proc/pid/fd[info] handling code to fd.[ch] || 3.7<br />
|-<br />
|{{torvalds.git|ddd3e0771b}} || procfs: Convert /proc/pid/fdinfo/ handling routines to seq-file || 3.7<br />
|-<br />
|{{torvalds.git|55985dd72a}} || procfs: Add ability to plug in auxiliary fdinfo providers || 3.8<br />
|-<br />
|{{torvalds.git|cbac5542d4}} || fs, eventfd: Add procfs fdinfo helper || 3.8<br />
|-<br />
|{{torvalds.git|138d22b586}} || fs, epoll: Add procfs fdinfo helper v2 || 3.8<br />
|-<br />
|{{torvalds.git|711c7bf991}} || fs, exportfs: Add export_encode_inode_fh helper || 3.8<br />
|-<br />
|{{torvalds.git|be77196b80}} || fs, notify: Add procfs fdinfo helper || 3.8<br />
|-<br />
|{{torvalds.git|e6dbcafb74}} || fs, fanotify: Add @mflags field to fanotify output || 3.8<br />
|-<br />
|{{torvalds.git|2787b04b6c}} || packet: Introduce net/packet/internal.h header || 3.7<br />
|-<br />
|{{torvalds.git|96ec632714}} || packet: Diag core and basic socket info dumping || 3.7<br />
|-<br />
|{{torvalds.git|8a360be0c5}} || packet: Report more packet sk info via diag module || 3.7<br />
|-<br />
|{{torvalds.git|eea68e2f1a}} || packet: Report socket mclist info via diag module || 3.7<br />
|-<br />
|{{torvalds.git|16f01365fa}} || packet: Report rings cfg via diag engine || 3.7<br />
|-<br />
|{{torvalds.git|fff3321d75}} || packet: Report fanout status via diag engine || 3.7<br />
|-<br />
|{{torvalds.git|0fa7fa98db}} || packet: Protect packet sk list with mutex (v2) || 3.7<br />
|-<br />
|{{torvalds.git|579035dc5d}} || kernel: limit a value of ns_last_pid to (0, max_pid) || 3.6<br />
|-<br />
|{{torvalds.git|bc26ccd8fc}} || tcp: restore rcv_wscale in a repair mode (v2) || 3.6<br />
|-<br />
|{{torvalds.git|f7b86bfe8d}} || sockopt: Make SO_BINDTODEVICE readable || 3.8<br />
|-<br />
|{{torvalds.git|e4e541a848}} || sock-diag: Report shutdown for inet and unix sockets (v2) || 3.8<br />
|-<br />
|{{torvalds.git|834f82e2aa}} || procfs: add VmFlags field in smaps output || 3.8<br />
|-<br />
|{{torvalds.git|06026d911c}} || tty: pty - Move TIOCPKT handling into pty.c || 3.8<br />
|-<br />
|{{torvalds.git|c6298038bc}} || tty, ioctls -- Add new ioctl definitions for tty flags fetching || 3.8<br />
|-<br />
|{{torvalds.git|84fd7bdf12}} || tty: Add get- ioctls to fetch tty status v3 || 3.8<br />
|-<br />
|{{torvalds.git|a8fc927780}} || sk-filter: Add ability to get socket filter program (v2) || 3.8<br />
|-<br />
|{{torvalds.git|cacb6ba0f3}} || net: inet_diag -- Return error code if protocol handler is missed || 3.7<br />
|-<br />
|{{torvalds.git|c454e6111d}} || tcp-repair: Handle zero-length data put in rcv queue || 3.7<br />
|-<br />
|{{torvalds.git|ec34232575}} || tcp: fix retransmission in repair mode || 3.7<br />
|-<br />
|{{torvalds.git|2b9164771e}} || ipv6: adapt connect for repair move || 3.8<br />
|-<br />
|{{torvalds.git|c91f6df2db}} || sockopt: Change getsockopt() of SO_BINDTODEVICE to return an interface name || 3.8<br />
|-<br />
|{{torvalds.git|3fcfe78658}} || ipc: add more comments to message copying related code || 3.8<br />
|-<br />
|{{torvalds.git|51eeacaa07}} || ipc: simplify message copying || 3.8<br />
|-<br />
|{{torvalds.git|b30efe2775}} || ipc: convert prepare_copy() from macro to function || 3.8<br />
|-<br />
|{{torvalds.git|85398aa8de}} || ipc: simplify free_copy() call || 3.8<br />
|-<br />
|{{torvalds.git|3a665531a3}} || selftests: IPC message queue copy feature test || 3.8<br />
|-<br />
|{{torvalds.git|4a674f34ba}} || ipc: introduce message queue copy feature || 3.8<br />
|-<br />
|{{torvalds.git|f9dd87f473}} || ipc: message queue receive cleanup || 3.8<br />
|-<br />
|{{torvalds.git|03f5956680}} || ipc: add sysctl to specify desired next object id || 3.8<br />
|-<br />
|{{torvalds.git|9afdacda02}} || ipc: remove forced assignment of selected message || 3.8<br />
|-<br />
|{{torvalds.git|3f7d1fe108}} || arm: Wire up kcmp syscall || 3.10<br />
|-<br />
|{{torvalds.git|1e142b29e2}} || kcmp: make it depend on CHECKPOINT_RESTORE || 3.9<br />
|-<br />
|{{torvalds.git|ceaa1fef65}} || tcp: adding a per-socket timestamp offset || 3.9<br />
|-<br />
|{{torvalds.git|93be6ce0e9}} || tcp: set and get per-socket timestamp || 3.9<br />
|-<br />
|{{torvalds.git|ee684b6f28}} || tcp: send packets with a socket timestamp || 3.9<br />
|-<br />
|{{torvalds.git|66dd34ad}} || signal: allow to send any siginfo to itself || 3.9<br />
|-<br />
|{{torvalds.git|ae5fc987}} || net: fix *_DIAG_MAX constants || 3.9<br />
|-<br />
|{{torvalds.git|0f29c768}} || net: prepare netlink code for netlink diag || 3.10<br />
|-<br />
|{{torvalds.git|eaaa3139}} || netlink: Diag core and basic socket info dumping (v2) || 3.10<br />
|-<br />
|{{torvalds.git|84c751bd}} || ptrace: add ability to retrieve signals without removing from a queue (v4) || 3.10<br />
|-<br />
|{{torvalds.git|040fa020}} || clear_refs: Sanitize accepted commands declaration || 3.11<br />
|-<br />
|{{torvalds.git|af9de7eb}} || clear_refs: Introduce private struct for mm_walk || 3.11<br />
|-<br />
|{{torvalds.git|2b0a9f01}} || pagemap-introduce-pagemap_entry_t-without-pmshift-bits || 3.11<br />
|-<br />
|{{torvalds.git|0f8975ec}} || mm: soft-dirty bits for user memory changes tracking || 3.11<br />
|-<br />
|{{torvalds.git|541c237c}} || pagemap: prepare to reuse constant bits with page-shift || 3.11<br />
|-<br />
|{{torvalds.git|57b8015e}} || posix-timers: Show sigevent info in proc file || 3.10<br />
|-<br />
|{{torvalds.git|48f6a7a5}} || posix-timers: Introduce /proc/PID/timers file || 3.10<br />
|-<br />
|{{torvalds.git|5ed67f05}} || posix timers: Allocate timer id per process (v2) || 3.10<br />
|-<br />
|{{torvalds.git|15ef0298}} || posix-timers: Show clock ID in proc file || 3.10<br />
|-<br />
|{{torvalds.git|29000cae}} || ptrace: add ability to get/set signal-blocked mask (v2) || 3.11<br />
|-<br />
|{{torvalds.git|274038f8}} || tun: Report "persist" flag to userspace || 3.11<br />
|-<br />
|{{torvalds.git|179ef71c}} || mm: Save soft-dirty bits on swapped pages || 3.11<br />
|-<br />
|{{torvalds.git|41bb3476}} || mm: Save soft-dirty bits on file pages || 3.11<br />
|-<br />
|{{torvalds.git|76975e9c}} || tun: Get skfilter layout || 3.12<br />
|-<br />
|{{torvalds.git|849c9b6f}} || tun: Allow to skip filter on attach || 3.12<br />
|-<br />
|{{torvalds.git|3d407a80}} || tun: Report whether the queue is attached or not || 3.12<br />
|-<br />
|{{torvalds.git|fb7589a1}} || tun: Add ability to create tun device with given index || 3.12<br />
|-<br />
|{{torvalds.git|e3e12028}} || tcp: don't apply tsoffset if rcv_tsecr is zero || 3.11<br />
|-<br />
|{{torvalds.git|c7781a6e}} || tcp: initialize rcv_tstamp for restored sockets || 3.11<br />
|-<br />
|{{torvalds.git|7ed5c5ae}} || tcp: set timestamps for restored skb-s || 3.11<br />
|-<br />
|{{torvalds.git|6dec97dc}} || mm: move_ptes -- Set soft dirty bit depending on pte type || 3.11<br />
|-<br />
|{{torvalds.git|c3d16e16}} || mm: migration -- Do not loose soft dirty bit if page is in migration state || 3.12<br />
|-<br />
|{{torvalds.git|e9cdd6e7}} || mm: pagemap -- Inspect _PAGE_SOFT_DIRTY only on present pages || 3.12<br />
|-<br />
|{{torvalds.git|dbde4979}} || tcp: don't update snd_nxt, when a socket is switched from repair mode || 3.13<br />
|-<br />
|{{torvalds.git|d9104d1c}} || mm: track vma changes with VM_SOFTDIRTY bit || 3.12<br />
|-<br />
|{{torvalds.git|34228d47}} || mm: Ignore VM_SOFTDIRTY on VMA merging || 3.14, 3.13-stable<br />
|-<br />
|{{torvalds.git|24f91eba1}} || mm: don't lose the SOFT_DIRTY flag on mprotect || 3.14, 3.13-stable<br />
|-<br />
|{{torvalds.git|49d063cb3}} || proc: show mnt_id in /proc/pid/fdinfo || 3.15<br />
|-<br />
|{{torvalds.git|0bf073315}} || mm: Make freshly remapped file pages being softdirty unconditionally || 3.15<br />
|-<br />
|{{torvalds.git|9aed8614a}} || mm: Don't forget to set softdirty on file mapped fault || 3.17<br />
|-<br />
|{{torvalds.git|b43790eed}} || mm: Don't forget to save file map softdiry bit on unmap || 3.15<br />
|-<br />
|{{torvalds.git|c86c97ff4}} || mm: Clear VM_SOFTDIRTY flag inside clear_refs_write instead of clear_soft_dirty || 3.15<br />
|-<br />
|{{torvalds.git|af9c4957c}} || timerfd: Implement show_fdinfo method || 3.17<br />
|-<br />
|{{torvalds.git|854d06d9f}} || docs: Procfs -- Document timerfd output || 3.17<br />
|-<br />
|{{torvalds.git|5442e9fbd}} || timerfd: Implement timerfd_ioctl method to restore timerfd_ctx::ticks, v3 || 3.17<br />
|-<br />
|{{torvalds.git|64e455079}} || mm: softdirty: enable write notifications on VMAs after VM_SOFTDIRTY cleared || 3.18<br />
|-<br />
|{{torvalds.git|9c5990240}} || mm: introduce check_data_rlimit helper || 3.18<br />
|-<br />
|{{torvalds.git|8764b338b}} || mm: use may_adjust_brk helper || 3.18<br />
|-<br />
|{{torvalds.git|71fe97e18}} || prctl: PR_SET_MM -- factor out mmap_sem when updating mm::exe_file || 3.18<br />
|-<br />
|{{torvalds.git|f606b77f1}} || prctl: PR_SET_MM -- introduce PR_SET_MM_MAP operation || 3.18<br />
|-<br />
|{{torvalds.git|9183df25f}} || shm: add memfd_create() syscall || 3.17<br />
|-<br />
|{{torvalds.git|e4a0d3e72}} || aio: Make it possible to remap aio ring || 3.19<br />
|-<br />
|{{torvalds.git|6c8c90319}} || proc: show locks in /proc/pid/fdinfo/X || 4.1<br />
|-<br />
|{{torvadls.git|155e35d4d}} || VFS: Introduce inode-getting helpers for layered/unioned fs environments || 4.0<br />
|-<br />
|{{torvalds.git|df1a085af}} || VFS: Add a fallthrough flag for marking virtual dentries || 4.0<br />
|-<br />
|{{torvalds.git|f25801ee4}} || overlay: Call ovl_drop_write() earlier in ovl_dentry_open() || 4.2-rc2<br />
|-<br />
|{{torvalds.git|4bacc9c92}} || overlayfs: Make f_path always point to the overlay and f_inode to the underlay || 4.2-rc2<br />
|-<br />
|{{torvalds.git|9391dd00d}} || fix a braino in ovl_d_select_inode() || 4.2-rc2<br />
|}<br />
<br />
[[Category:Development]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2526Docker2015-07-15T22:55:28Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. If you just want to experiment with C/R, you can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]}}<br />
<br />
{{Note| The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). However, there is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. See below for instructions on how to apply the relevant patches.}}<br />
<br />
{{Note| If your process uses async IO and your kernel is older than 3.19, you need to apply two patches. See below for instructions.}}<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''1. External C/R''' using CRIU directly on the command line as it's typically<br />
done for any process tree.<br />
<br />
This approach is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
'''2. Native C/R''' using new <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
This approach is called native because the Docker daemon is involved in both checkpoint and restore.<br />
Therefore, its notion of the container state will be corrent. All commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality.<br />
You can watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works.<br />
Source files for Docker 1.5 C/R are<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr here]'''<br />
and for Docker 1.7 C/R are<br />
'''[https://github.com/boucher/docker/tree/cr-combined here]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
<br />
== OverlayFS ==<br />
<br />
The following small kernel patches fix the mount id and symlink target path issues noted above:<br />
<br />
* {{torvalds.git|155e35d4da}} by David Howells<br />
* {{torvalds.git|df1a085af1}} by David Howells<br />
* {{torvalds.git|f25801ee46}} by David Howells<br />
* {{torvalds.git|4bacc9c923}} by David Howells<br />
* {{torvalds.git|9391dd00d1}} by Al Viro<br />
<br />
Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:<br />
<br />
<pre><br />
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git<br />
cd ubuntu-vivid<br />
git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git<br />
git remote update<br />
<br />
git cherry-pick 155e35d4da<br />
git cherry-pick df1a085af1<br />
git cherry-pick f25801ee46<br />
git cherry-pick 4bacc9c923<br />
git cherry-pick 9391dd00d1<br />
<br />
cp /boot/config-$(uname -r) .config<br />
make olddefconfig<br />
make -j 8 bzImage modules<br />
sudo make install modules_install<br />
sudo reboot<br />
</pre><br />
<br />
== Async IO (AIO) ==<br />
<br />
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:<br />
<br />
* {{torvalds.git|bd9b51e79c}} by Al Viro<br />
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov<br />
<br />
== External C/R ==<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
=== <code>--inherit-fd</code> ===<br />
<br />
For native C/R support, this option tells CRIU to let the restored process "inherit"<br />
its specified file descriptor (instead of restoring from checkpoint).<br />
<br />
== Restore Prework for External C/R ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An External C/R Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== External C/R Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2525Docker2015-07-15T22:53:38Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. If you just want to experiment with C/R, you can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]}}<br />
<br />
{{Note| The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). However, there is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. See below for instructions on how to apply the relevant patches.}}<br />
<br />
{{Note| If your process uses async IO and your kernel is older than 3.19, you need to apply two patches. See below for instructions.}}<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''1. External C/R''' using CRIU directly on the command line as it's typically<br />
done for any process tree.<br />
<br />
This approach is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
'''2. Native C/R''' using new <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
This approach is called native because the Docker daemon is involved in both checkpoint and restore.<br />
Therefore, its notion of the container state will be corrent. All commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality.<br />
You can watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works.<br />
Source files for Docker 1.5 C/R are<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr here]'''<br />
and for Docker 1.7 C/R are<br />
'''https://github.com/boucher/docker/tree/cr-combined here]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
<br />
== OverlayFS ==<br />
<br />
The following small kernel patches fix the mount id and symlink target path issues noted above:<br />
<br />
* {{torvalds.git|155e35d4da}} by David Howells<br />
* {{torvalds.git|df1a085af1}} by David Howells<br />
* {{torvalds.git|f25801ee46}} by David Howells<br />
* {{torvalds.git|4bacc9c923}} by David Howells<br />
* {{torvalds.git|9391dd00d1}} by Al Viro<br />
<br />
Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:<br />
<br />
<pre><br />
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git<br />
cd ubuntu-vivid<br />
git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git<br />
git remote update<br />
<br />
git cherry-pick 155e35d4da<br />
git cherry-pick df1a085af1<br />
git cherry-pick f25801ee46<br />
git cherry-pick 4bacc9c923<br />
git cherry-pick 9391dd00d1<br />
<br />
cp /boot/config-$(uname -r) .config<br />
make olddefconfig<br />
make -j 8 bzImage modules<br />
sudo make install modules_install<br />
sudo reboot<br />
</pre><br />
<br />
== Async IO (AIO) ==<br />
<br />
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:<br />
<br />
* {{torvalds.git|bd9b51e79c}} by Al Viro<br />
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov<br />
<br />
== External C/R ==<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
=== <code>--inherit-fd</code> ===<br />
<br />
For native C/R support, this option tells CRIU to let the restored process "inherit"<br />
its specified file descriptor (instead of restoring from checkpoint).<br />
<br />
== Restore Prework for External C/R ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An External C/R Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== External C/R Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2524Docker2015-07-15T22:51:48Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. If you just want to experiment with C/R, you can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]}}<br />
<br />
{{OverlayFS| The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). However, there is a bug in OverlayFS that reports the wrong mnt_id in /proc/<pid>/fdinfo/<fd> and the wrong symlink target path for /proc/<pid>/<fd>. Fortunately, these bugs have been fixed in the kernel v4.2-rc2. See below for instructions on how to apply the relevant patches.}}<br />
<br />
{{Async IO| If your process uses async IO and your kernel is older than 3.19, you need to apply two patches. See below for instructions.}}<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''1. External C/R''' using CRIU directly on the command line as it's typically<br />
done for any process tree.<br />
<br />
This approach is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
'''2. Native C/R''' using new <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
This approach is called native because the Docker daemon is involved in both checkpoint and restore.<br />
Therefore, its notion of the container state will be corrent. All commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality.<br />
You can watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works.<br />
Source files for Docker 1.5 C/R are<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr here]'''<br />
and for Docker 1.7 C/R are<br />
'''https://github.com/boucher/docker/tree/cr-combined here]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
<br />
== OverlayFS ==<br />
<br />
The following small kernel patches fix the mount id and symlink target path issues noted above:<br />
<br />
* {{torvalds.git|155e35d4da}} by David Howells<br />
* {{torvalds.git|df1a085af1}} by David Howells<br />
* {{torvalds.git|f25801ee46}} by David Howells<br />
* {{torvalds.git|4bacc9c923}} by David Howells<br />
* {{torvalds.git|9391dd00d1}} by Al Viro<br />
<br />
Assuming that you are running Ubuntu Vivid (Linux kernel 3.19), here is how you can patch your kernel:<br />
<br />
<pre><br />
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git<br />
cd ubuntu-vivid<br />
git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git<br />
git remote update<br />
<br />
git cherry-pick 155e35d4da<br />
git cherry-pick df1a085af1<br />
git cherry-pick f25801ee46<br />
git cherry-pick 4bacc9c923<br />
git cherry-pick 9391dd00d1<br />
<br />
cp /boot/config-$(uname -r) .config<br />
make olddefconfig<br />
make -j 8 bzImage modules<br />
sudo make install modules_install<br />
sudo reboot<br />
</pre><br />
<br />
== Async IO (AIO) ==<br />
<br />
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:<br />
<br />
* {{torvalds.git|bd9b51e79c}} by Al Viro<br />
* {{torvalds.git|e4a0d3e720}} by Pavel Emelyanov<br />
<br />
== External C/R ==<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
=== <code>--inherit-fd</code> ===<br />
<br />
For native C/R support, this option tells CRIU to let the restored process "inherit"<br />
its specified file descriptor (instead of restoring from checkpoint).<br />
<br />
== Restore Prework for External C/R ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An External C/R Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== External C/R Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2510Docker2015-06-18T04:24:53Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
Last update: June 17, 2015<br />
<br />
== Introduction ==<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. You can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]<br />
}}<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works. Source files for Docker 1.5 C/R are at this<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr repo]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
Work in underway to integrate C/R into the new <code>libcontainer</code>.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
== OverlayFS ==<br />
<br />
The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). For successful C/R, however, you need to apply the following kernel patch:<br />
<br />
OverlayFS patch [https://lkml.org/lkml/2015/3/20/372]<br />
<br />
== Async IO (AIO) ==<br />
<br />
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:<br />
<br />
<code>Al Viro's bd9b51e7</code><br />
<code>Pavel Emelyanov's e4a0d3e72</code><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2509Docker2015-06-18T04:23:31Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
Last update: June 17, 2015<br />
<br />
== Introduction ==<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. You can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]<br />
}}<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works. Source files for Docker 1.5 C/R are at this<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr repo]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
Work in underway to integrate C/R into the new <code>libcontainer</code>.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
== OverlayFS ==<br />
<br />
The OverlayFS filesystem was merged into the upstream Linux kernel 3.18 and is now Docker's preferred filesystem (instead of AUFS). For successful C/R, however, you need to apply the following kernel patch:<br />
<br />
OverlayFS patch [https://lkml.org/lkml/2015/3/20/372]<br />
<br />
== Async IO (AIO) ==<br />
<br />
If you are using a kernel older than 3.19 and your container uses AIO, you need the following AIO kernel patches from 3.19:<br />
<br />
==== <code>Al Viro's bd9b51e7</code> ====<br />
<br />
==== <code>Pavel Emelyanov's e4a0d3e72</code> ====<br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2508Docker2015-06-18T04:12:47Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
Last update: June 17, 2015<br />
<br />
== Introduction ==<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. You can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]<br />
}}<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
{{Note| External C/R was done as proof-of-concept. Its use is discouraged and the helper script mentioned below will be deprecated in the near future.}}<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works. Source files for Docker 1.5 C/R are at this<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr repo]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
Work in underway to integrate C/R into the new <code>libcontainer</code>.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2507Docker2015-06-18T04:09:47Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
Last update: June 17, 2015<br />
<br />
== Introduction ==<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests to add checkpoint/restore functionality to Docker have been submitted. You can use one of the following Docker versions for your C/R experiments: <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]<br />
}}<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works. Source files for Docker 1.5 C/R are at this<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr repo]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
Work in underway to integrate C/R into the new <code>libcontainer</code>.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2506Docker2015-06-18T04:07:06Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
Last update: June 17, 2015<br />
<br />
== Introduction ==<br />
<br />
{{Note| This page was originally written a few months ago. Since then, interfacing<br />
with CRIU has been added to Docker's native exec driver (libcontainer) and pull requests<br />
to add checkpoint/restore functionality to Docker have been submitted. Therefore, you<br />
can use one of the following Docker versions for your C/R experiments. <br />
<br />
Docker 1.5 [https://github.com/SaiedKazemi/docker/wiki]<br />
Docker 1.7 [https://github.com/boucher/docker/tree/cr-combined]<br />
}}<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works. Source files for Docker 1.5 C/R are at this<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr repo]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
Work in underway to integrate C/R into the new <code>libcontainer</code>.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it is possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
<code>/proc/''pid''/map_files</code>, option <code>--root</code><br />
is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up <code>/etc/{hostname,hosts,resolv.conf}</code> as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount <code>/.dockerinit</code>.<br />
<br />
For example, assuming the default Docker configuration, <code>/etc/hostname</code><br />
in the container's mount namespace is bind mounted from the source<br />
at <code>/var/lib/docker/containers/''container_id''/hostname</code>.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts <code>/dev/null</code> on <code>/dev/stdin</code> for detached containers<br />
(i.e., <code>docker run -d ...</code>). Since earlier versions of Docker used<br />
<code>/dev/null</code> in the global namespace, this option tells CRIU to treat<br />
the global <code>/dev/null</code> and the container <code>/dev/null</code> as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to CRIU dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Helper Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify <code>-v</code> to see the commands that <code>docker_cr.sh</code><br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log \<br />
--manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
-t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
-o restore.log --manage-cgroups --evasive-devices \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
-d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf \<br />
--pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Action_scripts&diff=2355Action scripts2015-04-08T16:16:48Z<p>Saied Kazemi: /* RPC */</p>
<hr />
<div>With the --action-script command line option, CRIU can call your action scripts (also known as hooks) at various stages of dumping/restoring. Hooks can be one of<br />
<br />
;network-lock<br />
: request to lock a container network (used by Docker and [[LXC]])<br />
<br />
;network-unlock<br />
: requested to unlock a container network (used by Docker and [[LXC]])<br />
<br />
;post-dump<br />
: called when CRIU finished dumping tasks before unfreezing them<br />
<br />
;post-restore<br />
: called when CRIU has finished restoring tasks before unlocking the network<br />
<br />
;setup-namespaces<br />
: called when the root task is alive and new set of namespaces is created to set them up<br />
<br />
<br />
== CLI ==<br />
These hooks are added with the --action-script shell-code-to-execute option. When called, the CRTOOLS_SCRIPT_ACTION environment is set to a value determining which type of action is performed.<br />
<br />
== RPC ==<br />
<br />
In case of RPC, action scripts are implemented as notifications -- once CRIU service wants to execute a script it sends an RPC message to caller and waits for it to respond with <code>criu_req</code> message having <code>notify_success = true</code>.<br />
<br />
== Library ==<br />
<br />
When using a library one can setup a callback using the <code>criu_set_notify</code> routine.<br />
<br />
[[Category:HOWTO]]<br />
[[Category:API]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2261Docker2015-03-19T21:33:36Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container as of March 2015.<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
'''[https://www.youtube.com/watch?v=HFt9v6yqsXo video]'''<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
'''[https://github.com/SaiedKazemi/docker/tree/cr repo]'''.<br />
The '''[https://github.com/SaiedKazemi/docker/wiki wiki]'''<br />
page provides an overview of the project history.<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2260Docker2015-03-19T21:27:41Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container as of March 2015.<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
[https://www.youtube.com/watch?v=HFt9v6yqsXo video]<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
[https://github.com/SaiedKazemi/docker/tree/cr repo].<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2259Docker2015-03-19T21:23:56Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
<code>docker ps, stop, kill</code> and <code>logs</code><br />
will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using the newly added <code>docker checkpoint</code> and<br />
<code>docker restore</code> commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
<code>docker ps, stop, kill </code> and <code>logs</code> will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
[https://www.youtube.com/watch?v=HFt9v6yqsXo video]<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
[https://github.com/SaiedKazemi/docker/tree/cr repo].<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the <code>--inherit-fd</code> command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2258Docker2015-03-19T21:21:56Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as<br />
=== <code>docker ps, stop, kill</code> === and === <code>logs</code> ===<br />
will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using the newly added === <code>docker checkpoint</code> === and<br />
=== <code>docker restore</code> === commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and all commands such as<br />
=== <code>docker ps, stop, kill </code> === and === <code>logs</code> ===<br />
will work.<br />
This is obviously the preferred method of checkpointing and restoring Docker containers.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
[https://www.youtube.com/watch?v=HFt9v6yqsXo video]<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
[https://github.com/SaiedKazemi/docker/tree/cr repo].<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
For native C/R support, additional functionality was added to CRIU.<br />
The most notable addition is the === <code>--inherit-fd</code> ===<br />
command line option.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2257Docker2015-03-19T21:12:58Z<p>Saied Kazemi: /* Introduction */</p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R''' using CRIU directly on the command line as it's typically<br />
done.<br />
<br />
This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as ''docker ps''<br />
and ''docker logs'' will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R''' using ''docker checkpoint'' and ''docker restore'' commands.<br />
<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and commands such as<br />
''docker ps'' and ''docker logs'' will work.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
[https://www.youtube.com/watch?v=HFt9v6yqsXo video]<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
[https://github.com/SaiedKazemi/docker/tree/cr repo].<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2256Docker2015-03-19T21:09:58Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
== Introduction ==<br />
<br />
There are two ways to checkpoint and restore a Docker container:<br />
<br />
'''External C/R:''' Using CRIU directly on the command line as it's typically<br />
done. This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as ''docker ps''<br />
and ''docker logs'' will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
'''Native C/R:''' Using ''docker checkpoint'' and ''docker restore'' commands.<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and commands such as<br />
''docker ps'' and ''docker logs'' will work.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
[https://www.youtube.com/watch?v=HFt9v6yqsXo video]<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
[https://github.com/SaiedKazemi/docker/tree/cr repo].<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=2255Docker2015-03-19T21:08:02Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
== Background ==<br />
<br />
1. External C/R: Using CRIU directly on the command line as it's typically<br />
done. This is called external because it's happening external to the<br />
Docker daemon. After checkpoint, the Docker daemon thinks that the<br />
container has exited. After restore, the Docker daemon doesn't know that<br />
the container is running again. Therefore, commands such as ''docker ps''<br />
and ''docker logs'' will not work correctly.<br />
<br />
External C/R was done as a proof-of-concept.<br />
<br />
2. Native C/R: Using ''docker checkpoint'' and ''docker restore'' commands.<br />
Because the Docker daemon is involved in both checkpoint and restore,<br />
its notion of the container state will be consistent and commands such as<br />
''docker ps'' and ''docker logs'' will work.<br />
<br />
Native C/R is work in progress, say pre-alpha quality. You can<br />
watch this short demo<br />
[https://www.youtube.com/watch?v=HFt9v6yqsXo video]<br />
to see how it works. Source files for Docker 1.5 C/R is at this<br />
[https://github.com/SaiedKazemi/docker/tree/cr repo].<br />
Work in underway to integrate C/R into the new libcontainer.<br />
<br />
== External C/R ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemihttps://criu.org/index.php?title=Main_Page&diff=1913Main Page2015-01-02T17:18:45Z<p>Saied Kazemi: </p>
<hr />
<div>{{Download box|right}}<br />
__NOTOC__<br />
<big>Welcome to CRIU, a project to implement checkpoint/restore functionality for Linux in userspace.<br />
<br />
Checkpoint/Restore In Userspace, or CRIU (pronounced kree-oo, IPA: /krɪʊ/, Russian: криу), is a software tool for Linux operating system. Using this tool, you can freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. You can then use the files to restore and run the application from the point it was frozen at. The distinctive feature of the CRIU project is that it is mainly implemented in user space.<br />
<br />
<div style="background-color: #f6f6f6; padding: 1em; text-align: left;" class="plainlinks"><br />
[http://criu.org http://static.openvz.org/criu_88x31.gif] is a project of [https://openvz.org http://static.openvz.org/openvz_88x31.gif] and is supported by [http://www.parallels.com http://static.openvz.org/parallels_88x31.gif] and [https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=FZ8W4VU9AS2XL your donations].</div></big><br />
{{Like}}<br />
<br clear="both"><br />
<br />
<div style="float:left; width: 33%"><br />
<br />
== Using ==<br />
<br />
<big><br />
;Getting [[packages]] for your distribution<br />
: Or try manual [[installation]] to have CRIU on your system<br />
</big><br />
<br />
;[[Usage|CLI]], [[RPC]] and [[C API]]<br />
: Three ways to start using the C/R functionality<br />
<br />
;[[Usage scenarios]]<br />
: Ideas how criu can be used (some are crazy indeed)<br />
<br />
;[[What software is supported]]<br />
: Describes TODO list in higher level terms<br />
<br />
;[[:Category:HOWTO]]<br />
: Collection of real world examples of how to use CRIU. Some are complex, some are not. HOW TO dump a [[simple loop]] might be the best one to start with.<br />
<br />
;[[:Category:API]]<br />
: Collection of pages about CRIU's API.<br />
<br />
;[[When C/R fails]]<br />
: A sort of troubleshooting guide<br />
<br />
;[[What can change after C/R]]<br />
: CRIU cannot (yet) save and restore every single bit of tasks' state. This page describes what bits visible through standard kernel API are such.<br />
<br />
;[[What cannot be checkpointed]]<br />
: What an application could do to make CRIU refuse to dump it.<br />
<br />
</div><br />
<br />
<div style="float:left; width: 33%; margin-left: 0.5%;"><br />
== Developing ==<br />
If you're interested in CRIU development, please subscribe to the criu mailing list: http://lists.openvz.org/mailman/listinfo/criu<br />
<br />
;[[Images]]<br />
: Description of image files format<br />
<br />
;[[Plugins]]<br />
: CRIU can call plugins provided by people<br />
<br />
;[[Upstream kernel commits]]<br />
: Mainline kernel commits tracker<br />
<br />
;[[Recent commits]]<br />
: CRIU tool repository commits<br />
<br />
;[[Manpages]]<br />
: Kernel's manpages commits tracker<br />
<br />
;[[ZDTM Test Suite]]<br />
: Zero downtime test suite<br />
<br />
;[[Todo|TODO]]<br />
: Current TODO list<br />
<br />
;[[UserNamespace|User namespace]]<br />
: Implementing user namespace support<br />
<br />
;[[Postulates]]<br />
: What to keep in mind when writing new code<br />
<br />
;[http://criu.org/cov/ Code coverage results]<br />
: Shows how zdtm run covers the criu code paths<br />
<br />
;[[How to submit patches]]<br />
:<br />
<br />
</div><br />
<br />
<div style="float:right; width: 33%"><br />
{{News block 2}}<br />
</div><br />
<br />
<br clear="both"><br />
<div style="float:left; width: 33%"><br />
== [[:Category:Under the hood|Under the hood]] ==<br />
* [[Sockets]]<br />
* [[Memory dumping and restoring]]<br />
* [[Checkpoint/Restore]]<br />
* [[TCP connection]]<br />
* [[Files]]<br />
* [[Pending signals]]<br />
* [[Stages of restoring]]<br />
* [[COW]]<br />
* [[Code blobs]]<br />
* [[Comparison to other CR projects]]<br />
* [[Memory changes tracking]]<br />
</div><br />
<br />
<div style="float:left; width: 33%; margin-left: 0.5%;"><br />
== External links ==<br />
{{:Articles}}<br />
</div><br />
<br />
<div style="float:right; width: 33%;"><br />
== Other ==<br />
* Project [[history]]<br />
* [[Logo]] description<br />
* [[Events]]<br />
* Join the [[CRIU acronym fun]]<br />
</div></div>Saied Kazemihttps://criu.org/index.php?title=Action_scripts&diff=1911Action scripts2015-01-02T17:14:18Z<p>Saied Kazemi: </p>
<hr />
<div>With the --action-script command line option, CRIU can call your action scripts (also known as hooks) at various stages of dumping/restoring. Hooks can be one of<br />
<br />
;network-lock<br />
: request to lock a container network (used by Docker and [[LXC]])<br />
<br />
;network-unlock<br />
: requested to unlock a container network (used by Docker and [[LXC]])<br />
<br />
;post-dump<br />
: called when CRIU finished dumping tasks before unfreezing them<br />
<br />
;post-restore<br />
: called when CRIU has finished restoring tasks before unlocking the network<br />
<br />
;setup-namespaces<br />
: called when the root task is alive and new set of namespaces is created to set them up<br />
<br />
<br />
== CLI ==<br />
These hooks are added with the --action-script shell-code-to-execute option. When called, the CRTOOLS_SCRIPT_ACTION environment is set to a value determining which type of action is performed.<br />
<br />
== RPC ==<br />
<br />
In case or RPC, action scripts are implemented as notifications -- once CRIU service wants to execute a script it send an RPC message to caller and waits for it to respond with <code>criu_req</code> message having <code>notify_success = true</code>.<br />
<br />
== Library ==<br />
<br />
When using a library one can setup a callback using the <code>criu_set_notify</code> routine.<br />
<br />
[[Category:HOWTO]]<br />
[[Category:API]]</div>Saied Kazemihttps://criu.org/index.php?title=Action_scripts&diff=1885Action scripts2014-12-30T18:33:33Z<p>Saied Kazemi: </p>
<hr />
<div>With the --action-script command line option, CRIU can call your action scripts (also known as hooks) at various stages of dumping/restoring. Hooks can be one of<br />
<br />
;network-lock<br />
: request to lock an [[LXC]] container network<br />
<br />
;network-unlock<br />
: requested to unlock an [[LXC]] container network<br />
<br />
;post-dump<br />
: called when CRIU finished dumping tasks before unfreezing them<br />
<br />
;post-restore<br />
: called when CRIU has finished restoring tasks before unlocking the network<br />
<br />
;setup-namespaces<br />
: called when the root task is alive and new set of namespaces is created to set them up<br />
<br />
<br />
== CLI ==<br />
These hooks are added with the --action-script shell-code-to-execute option. When called, the CRTOOLS_SCRIPT_ACTION environment is set to a value determining which type of action is performed.<br />
<br />
== RPC ==<br />
<br />
In case or RPC, action scripts are implemented as notifications -- once CRIU service wants to execute a script it send an RPC message to caller and waits for it to respond with <code>criu_req</code> message having <code>notify_success = true</code>.<br />
<br />
== Library ==<br />
<br />
When using a library one can setup a callback using the <code>criu_set_notify</code> routine.</div>Saied Kazemihttps://criu.org/index.php?title=Docker&diff=1697Docker2014-09-12T20:30:13Z<p>Saied Kazemi: </p>
<hr />
<div>This HOWTO page describes how to checkpoint and restore a Docker container.<br />
<br />
== Background ==<br />
<br />
Starting with CRIU 1.3, it's possible to checkpoint and restore a<br />
process tree running inside a Docker container. However, it's<br />
important to note that Docker needs native support for checkpoint<br />
and restore in order to maintain its parent-child relationship and<br />
to correctly keep track of container states. In other words, while<br />
CRIU can C/R a process tree, the restored tree will not become a<br />
child of Docker and, from Docker's point of view, the container's<br />
state will remain "Exited" (even after successful restore).<br />
<br />
Work is in progress to add native checkpoint and restore support<br />
to Docker. Once ready, specific commands (for example, "docker<br />
checkpoint" and "docker restore") will use CRIU to do the actual<br />
C/R operations while Docker continues to maintain its parent-child<br />
relationship and container states.<br />
<br />
It's important to re-emphasize that by checkpointing and restoring<br />
a Docker container, we mean C/R of a process tree running inside a<br />
container, excluding the Docker daemon itself. As CRIU currently<br />
does not support nested PID namespaces, the C/R process tree cannot<br />
include the Docker daemon which runs in the global PID namespace.<br />
<br />
== Command Line Options ==<br />
<br />
In addition to the usual CRIU command line options used when<br />
checkpointing and restoring a process tree, the following command<br />
line options are needed for Docker containers.<br />
<br />
=== <code>--root</code> ===<br />
<br />
This option has been used in the past only for restore operations<br />
that wanted to change the root of the mount namespace. It was not<br />
used for checkpoint operations.<br />
<br />
However, because Docker by default uses the AUFS graph driver and<br />
the AUFS module in the kernel reveals branch pathnames in<br />
/proc/<pid>/map_files, --root is used to specify the root of the<br />
mount namespace. Once the kernel AUFS module is fixed, it won't<br />
be necessary to specify this option anymore.<br />
<br />
=== <code>--ext-mount-map</code> ===<br />
<br />
This option is used to specify the path of the external bind mounts.<br />
Docker sets up /etc/{hostname,hosts,resolv.conf} as targets with<br />
source files outside the container's mount namespace. Older versions<br />
of Docker also bind mount /.dockerinit.<br />
<br />
For example, assuming the default Docker configuration, /etc/hostname<br />
in the container's mount namespace is bind mounted from the source<br />
at /var/lib/docker/containers/<container_id>/hostname.<br />
<br />
=== <code>--manage-cgroups</code> ===<br />
<br />
When a process tree exits after a checkpoint operation, the cgroups<br />
that Docker had created for the container are removed. This option<br />
is needed during restore to move the process tree into its cgroups,<br />
re-creating them if necessary.<br />
<br />
=== <code>--evasive-devices</code> ===<br />
<br />
Docker bind mounts /dev/null on /dev/stdin for detached containers<br />
(i.e., docker run -d ...). Since earlier versions of Docker used<br />
/dev/null in the global namespace, this option tells CRIU to treat<br />
the global /dev/null and the container /dev/null as the same device.<br />
<br />
== Restore Prework ==<br />
<br />
As mentioned earlier, by default Docker uses AUFS to set up the<br />
container's filesystem. When Docker notices that the process has<br />
exited (due to criu dump), it dismantles the filesystem. We need<br />
to set up the filesystem again before attempting to restore.<br />
<br />
== An Example ==<br />
<br />
Below is an example to show C/R operations for a shell script that<br />
continuously appends a number to a file. You can use tail -f to<br />
see the process in action.<br />
<br />
As you will see below, after restore, the process's parent is PID<br />
1 (init), not Docker. Also, although the process has been successfully<br />
restored, Docker still thinks that the container has exited.<br />
<br />
To set up the container's AUFS filesystem before restore, its branch<br />
information should be saved before checkpointing the container.<br />
For convenience, however, AUFS branch information is saved in the<br />
dump.log file. So we can examine dump.log to set up the filesystem<br />
again.<br />
<br />
For brevity, the 64-character long container ID is replaced by the<br />
string <container_id> in the following lines.<br />
<br />
<pre><br />
$ docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done'<br />
<container_id><br />
$ <br />
$ docker ps<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 seconds ago Up 4 seconds<br />
$ <br />
$ sudo criu dump -o dump.log -v4 -t 17810 \<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/etc/resolv.conf \<br />
--ext-mount-map /etc/hosts:/etc/hosts \<br />
--ext-mount-map /etc/hostname:/etc/hostname \<br />
--ext-mount-map /.dockerinit:/.dockerinit \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/dump.log<br />
(00.020103) Dumping finished successfully<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 6 minutes ago Exited (-1) 4 minutes ago<br />
$<br />
$ sudo mount -t aufs -o br=\<br />
/var/lib/docker/aufs/diff/<container_id>:\<br />
/var/lib/docker/aufs/diff/<container_id>-init:\<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721:\<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16:\<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229:\<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158:\<br />
none /var/lib/docker/aufs/mnt/<container_id><br />
$<br />
$ sudo criu restore -o restore.log -v4 -d<br />
-D /tmp/img/<container_id> \<br />
--root /var/lib/docker/aufs/mnt/<container_id> \<br />
--ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/<container_id>/resolv.conf \<br />
--ext-mount-map /etc/hosts:/var/lib/docker/containers/<container_id>/hosts \<br />
--ext-mount-map /etc/hostname:/var/lib/docker/containers/<container_id>/hostname \<br />
--ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 \<br />
--manage-cgroups \<br />
--evasive-devices<br />
$<br />
$ sudo grep successful /tmp/img/<container_id>/restore.log<br />
(00.424428) Restore finished successfully. Resuming tasks.<br />
$<br />
$ ps -ef | grep /bin/sh<br />
root 18580 1 0 12:38 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
$<br />
$ docker ps -a<br />
CONTAINER ID IMAGE COMMAND CREATED STATUS<br />
168aefb8881b busybox:latest "/bin/sh -c 'i=0; 7 minutes ago Exited (-1) 5 minutes ago<br />
$<br />
</pre><br />
<br />
== Help Script ==<br />
<br />
As seen in the above examples, the CRIU command line for checkpointing and<br />
restoring a Docker container is pretty long. For restore, there is also<br />
an additional step to set up the root filesystem before invoking CRIU.<br />
<br />
To automate the C/R process, there is a helper script in the contrib<br />
subdirectory of CRIU sources, called docker_cr.sh. In addition to<br />
invoking CRIU, this helper script sets up the root filesystem for AUFS,<br />
UnionFS, and VFS for restore.<br />
<br />
With docker_cr.sh, all you have to provide is the container ID.<br />
If you don't specify a container ID, docker_cr.sh will list all running<br />
containers and prompt you to choose one. Also, as shown in the help<br />
output below, by setting the appropriate environment variable, it's<br />
possible to tell docker_cr.sh which Docker and CRIU binaries to use,<br />
where Docker's home directory is, and where CRIU should save and look<br />
for its image files.<br />
<br />
<pre><br />
# docker_cr.sh --help<br />
Usage:<br />
docker_cr.sh -c|-r [-hv] [<container_id>]<br />
-c, --checkpoint checkpoint container<br />
-h, --help print help message<br />
-r, --restore restore container<br />
-v, --verbose enable verbose mode<br />
<br />
Environment:<br />
DOCKER_HOME (default /var/lib/docker)<br />
CRIU_IMG_DIR (default /var/lib/docker/criu_img)<br />
DOCKER_BINARY (default docker)<br />
CRIU_BINARY (default criu)<br />
</pre><br />
<br />
Below is an example to checkpoint and restore Docker container 4397:<br />
<br />
<pre><br />
# docker_cr.sh -c 4397<br />
dump successful<br />
# docker_cr.sh -r 4397<br />
restore successful<br />
</pre><br />
<br />
Optionally, you can specify -v to see the commands that docker_cr.sh<br />
executes. For example:<br />
<br />
<pre><br />
# docker_cr.sh -c -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu dump -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 5991 --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
dump successful<br />
(00.020827) Dumping finished successfully<br />
<br />
# docker_cr.sh -r -v 40d3<br />
docker binary: docker<br />
criu binary: criu<br />
image directory: /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
container root directory: /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
mount -t aufs -o<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
/var/lib/docker/aufs/diff/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf-init<br />
/var/lib/docker/aufs/diff/a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721<br />
/var/lib/docker/aufs/diff/120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16<br />
/var/lib/docker/aufs/diff/42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229<br />
/var/lib/docker/aufs/diff/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158<br />
none<br />
/var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf<br />
<br />
criu restore -v4 -D /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.0 -d --root /var/lib/docker/aufs/mnt/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf --pidfile /var/lib/docker/criu_img/40d363f564e00a2f893579fa012a200e475dcf8df47f2a22b7dd0860ffc3d7bf/restore.pid<br />
<br />
restore successful<br />
(00.408807) Restore finished successfully. Resuming tasks.<br />
<br />
root 6206 1 1 10:49 ? 00:00:00 /bin/sh -c i=0; while true; do echo $i >> /foo; i=$(expr $i + 1); sleep 3; done<br />
</pre><br />
<br />
<br />
[[Category:HOWTO]]</div>Saied Kazemi