Advanced usage
This page describes some less-than-obvious options that can be used with criu.
Evasive devices
Sometimes an application opens a device file and then somehow the path by which it was opened becomes inaccessible (e.g. overmounted or unlinked). In that case criu cannot easily dump and restore such a process. If you consider that the path doesn't really matter when dumping your apps state you can tell criu that a device file can be opened by any name, even if the original one is no longer accessible. The option for that is --evasive-devices.
External UNIX sockets
Consider an application opens a datagram UNIX socket and connects it to some address. If you will try to dump such app and the server socket for some reason will not be taken in the dumped state (e.g. -- a task holding it is not dumped) the dump will fail. You can override this behavior by allowing criu to disconnect the client after dump and re-connecting it back on restore by the server socket path using the --ext-unix-sk option.
Link unlinked files back
When an app opens a file and then unlinks it we take this file with us into image. However, we do so only if the file is really unlinked, i.e. its n_link is zero. Otherwise we cannot take file with us and should somehow open the inode. To do so we should create a temporary hardlink on it during dump, open it on restore and unlink again. The --link-remap option allows criu to create this temporary hard links on FS.
TCP connections
When dumping and restoring an application having an opened TCP connection you should use the --tcp-established option. When this option is in use criu will leave the connection(s) locked after dump and will require it(them) to be still locked before restore. Article Simple TCP pair gives an example of using this option.
Per-task logging on restore
By default criu puts all logs into one log file specified by --log-file option. If you want to split restoration logs on per-pid basis you can use the --log-pid option. The logs of a pid task will appear in a logfile.pid file.
Knowing a new PID of restored task
When you restore a process tree that lives inside a PID namespace, the pid of a new task will be generated by the system (since the pids sitting in image files are treated as virtual, i.e. seen in namespace ones). To find this pid out you can use the --pidfile file option to make criu put the new PID into file.
Restoring a veth pair of devices
When you restore a net namespace with a veth device end in it, criu will create the other end of the pair in the net namespace you launch criu from. By default it's name will be the one generated by veth kernel driver. Not to scan through all the net devices trying to fine one you can use the --veth-pair ns-dev:host-dev option. When used a device named ns-dev in a newly restored namespace will be linked with the host-dev one in the criu namespace.
Action scripts
CRIU can call your hooks on various stages of dumping/restoring. These hooks are added with the --action-script shell-code-to-execute option. When called, the CRTOOLS_SCRIPT_ACTION environment is set to a value determining which type of action is performed.
Shell jobs C/R
When you run some app directly from the shell (e.g. top), it inherits session ID from bash and uses PTY whose other end sits somewhere outside this shell session (e.g. in sshd or xterm). Strictly speaking this app cannot be checkpointed (since session leader, which is bash, is left outside the image, and so is the pty master). Nor it can be restored easily. But sometimes it might make sense to migrate such app by moving it to other session and attaching to different pty slave "on the fly". For such cases the --shell-job option should be used on both dump and restore. On dump, it will allow CRIU to ignore the "leaked" session leader and pty master. On restore, it will tell CRIU to change session, group and attach to existing pty slave.
File locks C/R
Some app may use file locks for synchronization. Generally they will use flock or posix file locks, which were achieved by flock or fcntl system calls. For dump/restore, it is hard to be handled perfectly, because we can't guarantee all potential users are dumped for a specific file lock. Right now, we assume that all file lock users are taken into dump, and we should use the --file-locks option on both dump and restore stages if our app may use any file locks. Remember that file locks dump/restore is only safe for container dumping in theory.
Leave task running after checkpoint
In some scenarios a user might want to leave program in a running state once checkpoint complete. To do that, criu has --leave-running command line option. We strongly encourage NOT to use it, until you understand what you are doing. Leaving task in a running state may lead to inconsistency between task images and external resources such as opened files, TCP connections.
Imagine a task been dumped and then continue execution, for some reason the task close and delete a file which has been checkpointed — any attempt to restore such task will simply fail because CRIU wont be able to open deleted file. Situation become more drastic if TCP connections are checkpointed — during execution the internal state of connections is changed inside kernel and restore then fail as well.
Still, leaving task running might be helpful but it is up to a user to make sure that:
- no established TCP connections are present;
- file system get snapshotted with
post-dumpscript (and then manually restored before criu restore takes place).
External bind mounts
When dumping LXC or Docker container you will likely meet those. The --ext-mount-map option should be used for such situations.