Difference between revisions of "LXC"

From CRIU
Jump to navigation Jump to search
(Make normal general description)
 
(27 intermediate revisions by 6 users not shown)
Line 1: Line 1:
=Prepare a Linux Container=
+
== Requirements ==
==Requirements==
 
* A console should be disabled (lxc.console = none)
 
* udev should not run inside containers ($ mv /sbin/udevd{,.bcp})
 
  
== Preparing a host environment ==
+
You should have built and installed a recent (>= 1.3.1) version of CRIU.
  
* Mount cgroupfs
+
== Checkpointing and restoring a container ==
$ mount -t cgroup c /cgroup
 
* Create a network bridge
 
# cat /etc/sysconfig/network-scripts/ifcfg-br0
 
DEVICE=br0
 
TYPE=Bridge
 
BOOTPROTO=dhcp
 
ONBOOT=yes
 
DELAY=5
 
NM_CONTROLLED=n
 
$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
 
DEVICE="eth0"
 
NM_CONTROLLED="no"
 
ONBOOT="yes"
 
BRIDGE=br0
 
==Create and start a container==  
 
* Download an OpenVZ template and extract it.
 
curl http://download.openvz.org/template/precreated/centos-6-x86_64.tar.gz | tar -xz -C test-lxc
 
  
* Create config files
+
LXC upstream has begun to integrate checkpoint/restore support through the lxc-checkpoint tool. This functionality has been in the recent released version of LXC---LXC 1.1.0 , you can install the LXC 1.1.0 or you can check out the development version on Ubuntu by doing:
$ cat ~/test-lxc.conf
+
<source lang="bash">
lxc.console=none
+
sudo add-apt-repository ppa:ubuntu-lxc/daily
lxc.utsname = test-lxc
+
sudo apt-get update
lxc.network.type = veth
+
sudo apt-get install lxc
lxc.network.flags = up
+
</source>
lxc.network.link = br0
 
lxc.network.name = eth0
 
lxc.mount = /root/test-lxc/etc/fstab
 
lxc.rootfs = /root/test-lxc-root/
 
  
$ cat /root/test-lxc/etc/fstab
+
Next, create a container:
none /root/test-lxc-root/dev/pts devpts defaults 0 0
 
none /root/test-lxc-root/proc    proc  defaults 0 0
 
none /root/test-lxc-root/sys    sysfs  defaults 0 0
 
none /root/test-lxc-root/dev/shm tmpfs  defaults 0 0
 
  
* Register the container
+
<source lang="bash">
  $ lxc-create -n test-lxc -f test-lxc.conf
+
  sudo lxc-create -t ubuntu -n u1 -- -r trusty -a amd64
 +
</source>
  
* Start the container
+
And add the following lines (as above) to its config:
$ mount --bind test-lxc test-lxc-root/
 
$ lxc-start -n test-lxc
 
  
= Checkpoint and restore an LXC Container =
+
<source lang="bash">
== Preparations ==
+
cat | sudo tee -a /var/lib/lxc/u1/config << EOF
You only need to [[Installation | install]] the crtools.
+
# hax for criu
 +
lxc.console.path = none
 +
lxc.tty.max = 0
 +
lxc.cgroup.devices.deny = c 5:1 rwm
 +
# on older lxc comment the above and uncomment the below
 +
# lxc.console = none
 +
# lxc.tty = 0
 +
# lxc.cgroup.devices.deny = c 5:1 rwm
 +
EOF
 +
</source>
  
== Dump and restore ==
+
Finally, start, and checkpoint the container:
Dumping and restoring an LXC contianer means -- dumping a subtree of processes starting from container init plus all kinds of namespaces.
 
Restoring is symmetrical. The way LXC container works imposes some more requirements on crtools usage.
 
  
* You need to use the <code>--evasive-devices</code> option to handle <code>/dev/log</code> users (there's a bug in LXC code)
+
<source lang="bash">
* In order to properly isolate container from unwanted networking communication during checkpoint/restore you should provide a script for locking/unlocking the container network (see below)
+
sudo lxc-start -n u1
* When restoring a container with veth device you may specify a name for the host-side veth device
+
sleep 5s  # let the container get to a more interesting state
* In order to checkpoint and restore alive TCP connections you should use the <code>--tcp-established</code> option
+
sudo lxc-checkpoint -s -D /tmp/checkpoint -n u1
 +
</source>
  
Typically a container dump command will look like
+
At this point, the container's state is stored in /tmp/checkpoint, and the filesystem is in /var/lib/lxc/u1/rootfs. You can restore the container by doing:
<pre>
 
crtools dump
 
    --evasive-devices                # handle /dev/log usage bug
 
    --tcp-established                # allow for TCP connections dump
 
    -n net -n mnt -n ipc -n pid      # dump all the namespaces container uses
 
    --action-script "net-script.sh"  # use net-script.sh to lock/unlock networking
 
    -D dump/ -o dump.log            # set images dir to dump/ and put logs into dump.log file
 
    -t ${init-pid}                  # start dumping from task ${init-pid}. It should be container's init
 
</pre>
 
and restore command like
 
<pre>
 
crtools restore
 
  --evasive-devices
 
  --tcp-established
 
  -n net -n mnt -n ipc -n pid
 
  --action-script "net-script.sh"
 
  --veth-pair eth0=${veth-name}    # when restoring a veth link use ${veth-name} for host-side device end
 
  --root ${path}                    # path to container root. It should be a root of a (bind)mount
 
  -D data/ -o restore.log
 
  -t ${init-pid}
 
</pre>
 
  
We also find it useful to use the <code>--restore-detached</code> option for restore to make contianer reparent to init rather than hanging on a crtools process launched from shell. Another useful option is the <code>--pidfile</code> one -- you will be able to find out the host-side pid of a container init after restore.
+
<source lang="bash">
 +
sudo lxc-checkpoint -r -D /tmp/checkpoint -n u1
 +
</source>
  
== Example ==
+
And then, get your container's IP and ssh in:
We have [http://git.criu.org/?p=crtools.git;a=tree;f=test/app-emu/lxc;hb=HEAD an application test] to test dump/restore of a Linux Container.
 
  
This test contains two scripts: [http://git.criu.org/?p=crtools.git;a=blob;f=test/app-emu/lxc/run.sh;hb=HEAD run.sh] and [http://git.criu.org/?p=crtools.git;a=blob;f=test/app-emu/lxc/network-script.sh;hb=HEAD network-script.sh].
+
<source lang="bash">
 +
ssh ubuntu@$(sudo lxc-info -i -H -n u1)
 +
</source>
  
The script run.sh is a main script, which executes ''crtools'' two times for dumping and restoring CT. This scripts contains actual options for ''crtools''.
+
== Troubleshooting ==
 +
 +
=== Error (mount.c:805): fusectl isn't empty: 8388625 ===
  
The script network-script.sh is used to lock and unlock CT's network. During dump a state of CT should not be changed, so [[CR tools]] freezes processes and executes an external script to freeze network. On restore [[CR tools]] restores states of processes and resumes the CT, which includes resume of processes and network). An external script is used to unlock CT's network.
+
Dumping of fuse filesystems is currently not supported. Empty the container's <code>/sys/fs/fuse/connections</code> and try again.
 +
 
 +
=== Error (mount.c:517): Mount 58 (master_id: 12 shared_id: 0) has unreachable sharing ===
 +
 
 +
CRIU doesn't yet support shared mountpoints as LXC does; make sure your rootfs is on a non-shared mount.
 +
 
 +
== External links ==
 +
 
 +
* [https://www.youtube.com/watch?v=a9T2gcnQg2k&feature=youtu.be&t=18m8s The New New Thing: Turning Docker Tech into a Full Speed Hypervisor] - Talk of Tycho Andersen with demo of migration LXC container with Doom inside
 +
* [https://github.com/tych0/presentations/blob/master/ods2014.md Demo script]
 +
 
 +
[[Category: HOWTO]]
 +
[[Category: Live migration]]

Latest revision as of 04:33, 1 July 2024

Requirements[edit]

You should have built and installed a recent (>= 1.3.1) version of CRIU.

Checkpointing and restoring a container[edit]

LXC upstream has begun to integrate checkpoint/restore support through the lxc-checkpoint tool. This functionality has been in the recent released version of LXC---LXC 1.1.0 , you can install the LXC 1.1.0 or you can check out the development version on Ubuntu by doing:

sudo add-apt-repository ppa:ubuntu-lxc/daily
sudo apt-get update
sudo apt-get install lxc

Next, create a container:

 sudo lxc-create -t ubuntu -n u1 -- -r trusty -a amd64

And add the following lines (as above) to its config:

cat | sudo tee -a /var/lib/lxc/u1/config << EOF
# hax for criu
lxc.console.path = none
lxc.tty.max = 0
lxc.cgroup.devices.deny = c 5:1 rwm
# on older lxc comment the above and uncomment the below
# lxc.console = none
# lxc.tty = 0
# lxc.cgroup.devices.deny = c 5:1 rwm
EOF

Finally, start, and checkpoint the container:

sudo lxc-start -n u1
sleep 5s  # let the container get to a more interesting state
sudo lxc-checkpoint -s -D /tmp/checkpoint -n u1

At this point, the container's state is stored in /tmp/checkpoint, and the filesystem is in /var/lib/lxc/u1/rootfs. You can restore the container by doing:

sudo lxc-checkpoint -r -D /tmp/checkpoint -n u1

And then, get your container's IP and ssh in:

ssh ubuntu@$(sudo lxc-info -i -H -n u1)

Troubleshooting[edit]

Error (mount.c:805): fusectl isn't empty: 8388625[edit]

Dumping of fuse filesystems is currently not supported. Empty the container's /sys/fs/fuse/connections and try again.

Error (mount.c:517): Mount 58 (master_id: 12 shared_id: 0) has unreachable sharing[edit]

CRIU doesn't yet support shared mountpoints as LXC does; make sure your rootfs is on a non-shared mount.

External links[edit]