Line 1: |
Line 1: |
− | =Prepare a Linux Container=
| + | == Requirements == |
− | ==Requirements== | |
− | * A console should be disabled (lxc.console = none)
| |
− | * udev should not run inside containers ($ mv /sbin/udevd{,.bcp})
| |
| | | |
− | == Preparing a host environment ==
| + | You should have built and installed a recent (>= 1.3.1) version of CRIU. |
| | | |
− | * Mount cgroupfs
| + | == Checkpointing and restoring a container == |
− | $ mount -t cgroup c /cgroup
| |
− | * Create a network bridge
| |
− | # cat /etc/sysconfig/network-scripts/ifcfg-br0
| |
− | DEVICE=br0
| |
− | TYPE=Bridge
| |
− | BOOTPROTO=dhcp
| |
− | ONBOOT=yes
| |
− | DELAY=5
| |
− | NM_CONTROLLED=n
| |
− | $ cat /etc/sysconfig/network-scripts/ifcfg-eth0
| |
− | DEVICE="eth0"
| |
− | NM_CONTROLLED="no"
| |
− | ONBOOT="yes"
| |
− | BRIDGE=br0
| |
− | ==Create and start a container== | |
− | * Download an OpenVZ template and extract it.
| |
− | curl http://download.openvz.org/template/precreated/centos-6-x86_64.tar.gz | tar -xz -C test-lxc
| |
| | | |
− | * Create config files
| + | LXC upstream has begun to integrate checkpoint/restore support through the lxc-checkpoint tool. This functionality has been in the recent released version of LXC---LXC 1.1.0 , you can install the LXC 1.1.0 or you can check out the development version on Ubuntu by doing: |
− | $ cat ~/test-lxc.conf
| + | <source lang="bash"> |
− | lxc.console=none
| + | sudo add-apt-repository ppa:ubuntu-lxc/daily |
− | lxc.utsname = test-lxc
| + | sudo apt-get update |
− | lxc.network.type = veth
| + | sudo apt-get install lxc |
− | lxc.network.flags = up
| + | </source> |
− | lxc.network.link = br0
| |
− | lxc.network.name = eth0
| |
− | lxc.mount = /root/test-lxc/etc/fstab
| |
− | lxc.rootfs = /root/test-lxc-root/
| |
| | | |
− | $ cat /root/test-lxc/etc/fstab
| + | Next, create a container: |
− | none /root/test-lxc-root/dev/pts devpts defaults 0 0
| |
− | none /root/test-lxc-root/proc proc defaults 0 0
| |
− | none /root/test-lxc-root/sys sysfs defaults 0 0
| |
− | none /root/test-lxc-root/dev/shm tmpfs defaults 0 0
| |
| | | |
− | * Register the container
| + | <source lang="bash"> |
− | $ lxc-create -n test-lxc -f test-lxc.conf | + | sudo lxc-create -t ubuntu -n u1 -- -r trusty -a amd64 |
| + | </source> |
| | | |
− | * Start the container
| + | And add the following lines (as above) to its config: |
− | $ mount --bind test-lxc test-lxc-root/
| |
− | $ lxc-start -n test-lxc
| |
| | | |
− | = Checkpoint and restore an LXC Container = | + | <source lang="bash"> |
− | == Preparations == | + | cat | sudo tee -a /var/lib/lxc/u1/config << EOF |
− | You only need to [[Installation | install]] the crtools.
| + | # hax for criu |
| + | lxc.console.path = none |
| + | lxc.tty.max = 0 |
| + | lxc.cgroup.devices.deny = c 5:1 rwm |
| + | # on older lxc comment the above and uncomment the below |
| + | # lxc.console = none |
| + | # lxc.tty = 0 |
| + | # lxc.cgroup.devices.deny = c 5:1 rwm |
| + | EOF |
| + | </source> |
| | | |
− | == Dump and restore ==
| + | Finally, start, and checkpoint the container: |
− | Dumping and restoring an LXC contianer means -- dumping a subtree of processes starting from container init plus all kinds of namespaces.
| |
− | Restoring is symmetrical. The way LXC container works imposes some more requirements on crtools usage.
| |
| | | |
− | * You need to use the <code>--evasive-devices</code> option to handle <code>/dev/log</code> users (there's a bug in LXC code)
| + | <source lang="bash"> |
− | * In order to properly isolate container from unwanted networking communication during checkpoint/restore you should provide a script for locking/unlocking the container network (see below)
| + | sudo lxc-start -n u1 |
− | * When restoring a container with veth device you may specify a name for the host-side veth device
| + | sleep 5s # let the container get to a more interesting state |
− | * In order to checkpoint and restore alive TCP connections you should use the <code>--tcp-established</code> option
| + | sudo lxc-checkpoint -s -D /tmp/checkpoint -n u1 |
| + | </source> |
| | | |
− | Typically a container dump command will look like
| + | At this point, the container's state is stored in /tmp/checkpoint, and the filesystem is in /var/lib/lxc/u1/rootfs. You can restore the container by doing: |
− | <pre>
| |
− | crtools dump
| |
− | --evasive-devices # handle /dev/log usage bug
| |
− | --tcp-established # allow for TCP connections dump
| |
− | -n net -n mnt -n ipc -n pid # dump all the namespaces container uses
| |
− | --action-script "net-script.sh" # use net-script.sh to lock/unlock networking
| |
− | -D dump/ -o dump.log # set images dir to dump/ and put logs into dump.log file
| |
− | -t ${init-pid} # start dumping from task ${init-pid}. It should be container's init
| |
− | </pre>
| |
− | and restore command like
| |
− | <pre>
| |
− | crtools restore
| |
− | --evasive-devices
| |
− | --tcp-established
| |
− | -n net -n mnt -n ipc -n pid
| |
− | --action-script "net-script.sh"
| |
− | --veth-pair eth0=${veth-name} # when restoring a veth link use ${veth-name} for host-side device end
| |
− | --root ${path} # path to container root. It should be a root of a (bind)mount
| |
− | -D data/ -o restore.log
| |
− | -t ${init-pid}
| |
− | </pre>
| |
| | | |
− | We also find it useful to use the <code>--restore-detached</code> option for restore to make contianer reparent to init rather than hanging on a crtools process launched from shell. Another useful option is the <code>--pidfile</code> one -- you will be able to find out the host-side pid of a container init after restore.
| + | <source lang="bash"> |
| + | sudo lxc-checkpoint -r -D /tmp/checkpoint -n u1 |
| + | </source> |
| | | |
− | == Example ==
| + | And then, get your container's IP and ssh in: |
− | We have [http://git.criu.org/?p=crtools.git;a=tree;f=test/app-emu/lxc;hb=HEAD an application test] for dumping/restoring an LXC Container. You may look at it for better understanding how to dump and restore your container with crtools.
| |
| | | |
− | This test contains two scripts:
| + | <source lang="bash"> |
− | ;[http://git.criu.org/?p=crtools.git;a=blob;f=test/app-emu/lxc/run.sh;hb=HEAD run.sh]
| + | ssh ubuntu@$(sudo lxc-info -i -H -n u1) |
− | :This is the main script, which executes ''crtools'' two times for dumping and restoring CT. It contains a working commands for dumping and restoring a container.
| + | </source> |
| | | |
− | ;[http://git.criu.org/?p=crtools.git;a=blob;f=test/app-emu/lxc/network-script.sh;hb=HEAD network-script.sh] | + | == Troubleshooting == |
− | : This one is used to lock and unlock CT's network as described above. | + | |
| + | === Error (mount.c:805): fusectl isn't empty: 8388625 === |
| + | |
| + | Dumping of fuse filesystems is currently not supported. Empty the container's <code>/sys/fs/fuse/connections</code> and try again. |
| + | |
| + | === Error (mount.c:517): Mount 58 (master_id: 12 shared_id: 0) has unreachable sharing === |
| + | |
| + | CRIU doesn't yet support shared mountpoints as LXC does; make sure your rootfs is on a non-shared mount. |
| + | |
| + | == External links == |
| + | |
| + | * [https://www.youtube.com/watch?v=a9T2gcnQg2k&feature=youtu.be&t=18m8s The New New Thing: Turning Docker Tech into a Full Speed Hypervisor] - Talk of Tycho Andersen with demo of migration LXC container with Doom inside |
| + | * [https://github.com/tych0/presentations/blob/master/ods2014.md Demo script] |
| + | |
| + | [[Category: HOWTO]] |
| + | [[Category: Live migration]] |