Difference between revisions of "Advanced usage"

From CRIU
Jump to navigation Jump to search
("useful when using" and "non-standard" was kinda... non-standard :))
 
(21 intermediate revisions by 4 users not shown)
Line 1: Line 1:
This page describes some less-than-obvious options that can be used with crtools.
+
This page describes some less-than-obvious options that can be used with criu.
  
 
== Evasive devices ==
 
== Evasive devices ==
  
Sometimes an application opens a device file and then somehow the path by which it was opened becomes inaccessible (e.g. overmounted or unlinked). In that case crtools cannot easily dump and restore such a process. If you consider that the path doesn't really matter when dumping your apps state you can tell crtools that a device file can be opened by '''any''' name, even if the original one is no longer accessible. The option for that is <code>--evasive-devices</code>
+
Sometimes an application opens a device file and then somehow the path by which it was opened becomes inaccessible (e.g. overmounted or unlinked). In that case criu cannot easily dump and restore such a process. If you consider that the path doesn't really matter when dumping your apps state you can tell criu that a device file can be opened by '''any''' name, even if the original one is no longer accessible. The option for that is {{Opt|--evasive-devices}}.
  
 
== External UNIX sockets ==
 
== External UNIX sockets ==
  
Consider an application opens a datagram UNIX socket and connects it to some address. If you will try to dump such app and the server socket for some reason will '''not''' be taken in the dumped state (e.g. -- a task holding it is not dumped) the dump will fail. You can override this behavior by allowing crtools to disconnect the client after dump and re-connecting it back on restore by the server socket path using the <code>--ext-unix-sk</code> option.
+
Consider an application opens a datagram UNIX socket and connects it to some address. If you will try to dump such app and the server socket for some reason will '''not''' be taken in the dumped state (e.g. -- a task holding it is not dumped) the dump will fail. You can override this behavior by allowing criu to disconnect the client after dump and re-connecting it back on restore by the server socket path using the {{Opt|--ext-unix-sk}} option.
  
 
== Link unlinked files back ==
 
== Link unlinked files back ==
  
When an app opens a file and then unlinks it we take this file with us into image. However, we do so only if the file is ''really'' unlinked, i.e. its n_link is zero. Otherwise we cannot take file with us and should somehow open the inode. To do so we should create a temporary hardlink on it during dump, open it on restore and unlink again. The <code>--link-remap</code> option allows crtools to create this temporary hard links on FS.
+
When an app opens a file and then unlinks it we take this file with us into image. However, we do so only if the file is ''really'' unlinked, i.e. its n_link is zero. Otherwise we cannot take file with us and should somehow open the inode. To do so we should create a temporary hardlink on it during dump, open it on restore and unlink again. The {{Opt|--link-remap}} option allows criu to create this temporary hard links on FS.
  
 
== TCP connections ==
 
== TCP connections ==
  
When dumping and restoring an application having an opened tcp connection you should use the <code>--tcp-established</code> option. When this option is in use crtools will leave the connection(s) locked after dump and will require it(them) to be still locked before restore.
+
When dumping and restoring an application having an opened TCP connection you should use the {{Opt|--tcp-established}} option. When this option is in use criu will leave the connection(s) locked after dump and will require it(them) to be still locked before restore. Article [[Simple TCP pair]] gives an example of using this option.
  
 
== Per-task logging on restore ==
 
== Per-task logging on restore ==
  
By default crtools puts all logs into one log file specified by <code>-o/--log-file</code> option. If you want to split restoration logs on per-pid basis you can use the <code>--log-pid</code> option. The <code>${pid}</code> task's logs will appear in a <code>${log-file}.pid</code>
+
By default criu puts all logs into one log file specified by {{Opt|--log-file|file}} option. If you want to split restoration logs on per-PID basis you can use the {{Opt|--log-pid}} option. The logs of a task having PID of N will appear in a <code>''file''.''N''</code> file.
  
== Knowing a new pid of restored task ==
+
== Knowing a new PID of restored task ==
  
When you restore a process tree with <code>-n/--namespaces pid</code> option the pid of a new task will be generated by the system (since the pids sitting in image files are treated as virtual, i.e. seen in namespace ones). To find this pid out you can use the <code>--pidfile <file></code> option to make crtools put the new pid into a pidfile.
+
When you restore a process tree that lives inside a PID namespace, the PID of a new task will be generated by the system (since the PIDs sitting in image files are treated as virtual, i.e. as seen in a PID namespace). To find out the PID, one can use the {{Opt|--pidfile|file}} option to make CRIU put the new PID into <code>''file''</code>.
  
== Restoring a veth pair of devices ==
+
== Restoring a VETH pair of devices ==
  
When you restore a net namespace with a veth device end in it, crtools will create the other end of the pair in the net namespace you launch crtools from. By default it's name will be the one generated by veth kernel driver. Not to scan through all the net devices trying to fine one you can use the <code>--veth-pair <ns-dev>:<host-dev></code> option. When used a device named <code><ns-dev></code> in a newly restored namespace will be linked with the <code><host-dev></code> one in the crtools namespace.
+
When you restore a net namespace with a VETH device end in it, CRIU will create the other end of the pair in the net namespace you launch CRIU from. By default its name will be the one generated by the <code>veth</code> kernel driver. Not to scan through all the net devices trying to find one, you can use the {{Opt|--external}} option. For more details see the article about [[VETH device]].
  
== [[Action scripts]] ==
+
== Action scripts ==
  
The '''crtools''' can call your hooks on various stages of dumping/restoring. These hooks are added with the <code>--action-sctipt <shell-code-to-execute></code> option. When called the <code>CRTOOLS_SCRIPT_ACTION</code> environment is set to a value determining which type of action is performed.
+
CRIU can call your hooks on various stages of dumping/restoring. These hooks are added with the {{Opt|--action-script|shell-code-to-execute}} option. When called, the <code>CRTOOLS_SCRIPT_ACTION</code> environment is set to a value determining which type of action is performed.
 +
 
 +
== Shell jobs C/R ==
 +
 
 +
When you run some app directly from the shell (e.g. <code>top</code>), it inherits session ID from bash and uses PTY whose other end sits somewhere outside this shell session (e.g. in sshd or xterm). Strictly speaking this app cannot be checkpointed (since session leader, which is bash, is left outside the image, and so is the pty master). Nor it can be restored easily. But sometimes it might make sense to migrate such app by moving it to other session and attaching to different pty slave "on the fly". For such cases the {{Opt|--shell-job}} option should be used on both dump and restore. On dump, it will allow CRIU to ignore the "leaked" session leader and pty master. On restore, it will tell CRIU to change session, group and attach to existing pty slave.
 +
 
 +
== File locks C/R ==
 +
 
 +
Some app may use file locks for synchronization. Generally they will use flock or posix file locks, which were achieved by flock or fcntl system calls. For dump/restore, it is hard to be handled perfectly, because we can't guarantee all potential users are dumped for a specific file lock. Right now, we assume that all file lock users are taken into dump, and we should use the {{Opt|--file-locks}} option on both dump and restore stages if our app may use any file locks. Remember that file locks dump/restore is only safe for container dumping in theory.
 +
 
 +
== Leave task running after checkpoint ==
 +
 
 +
In some scenarios a user might want to leave program in a running state once checkpoint complete. To do that, criu has {{Opt|--leave-running}} command line option. We strongly encourage NOT to use it, until you understand what you are doing. Leaving task in a running state may lead to inconsistency between task images and external resources such as opened files, TCP connections.
 +
 
 +
Imagine a task been dumped and then continue execution, for some reason the task close and delete a file which has been checkpointed — any attempt to restore such task will simply fail because CRIU wont be able to open deleted file. Situation become more drastic if TCP connections are checkpointed — during execution the internal state of connections is changed inside kernel and restore then fail as well.
 +
 
 +
Still, leaving task running might be helpful but it is up to a user to make sure that:
 +
 
 +
* no established TCP connections are present;
 +
* file system get snapshotted with <code>post-dump</code> script (and then manually restored before criu restore takes place).
 +
 
 +
== External bind mounts ==
 +
 
 +
When dumping [[LXC]] or [[Docker]] container, one will likely meet those. The {{Opt|--ext-mount-map}} option should be used for such situations.
 +
 
 +
== See also ==
 +
 
 +
* [[CLI]]
 +
 
 +
[[Category:API]]

Latest revision as of 10:47, 26 September 2016

This page describes some less-than-obvious options that can be used with criu.

Evasive devices[edit]

Sometimes an application opens a device file and then somehow the path by which it was opened becomes inaccessible (e.g. overmounted or unlinked). In that case criu cannot easily dump and restore such a process. If you consider that the path doesn't really matter when dumping your apps state you can tell criu that a device file can be opened by any name, even if the original one is no longer accessible. The option for that is --evasive-devices.

External UNIX sockets[edit]

Consider an application opens a datagram UNIX socket and connects it to some address. If you will try to dump such app and the server socket for some reason will not be taken in the dumped state (e.g. -- a task holding it is not dumped) the dump will fail. You can override this behavior by allowing criu to disconnect the client after dump and re-connecting it back on restore by the server socket path using the --ext-unix-sk option.

Link unlinked files back[edit]

When an app opens a file and then unlinks it we take this file with us into image. However, we do so only if the file is really unlinked, i.e. its n_link is zero. Otherwise we cannot take file with us and should somehow open the inode. To do so we should create a temporary hardlink on it during dump, open it on restore and unlink again. The --link-remap option allows criu to create this temporary hard links on FS.

TCP connections[edit]

When dumping and restoring an application having an opened TCP connection you should use the --tcp-established option. When this option is in use criu will leave the connection(s) locked after dump and will require it(them) to be still locked before restore. Article Simple TCP pair gives an example of using this option.

Per-task logging on restore[edit]

By default criu puts all logs into one log file specified by --log-file file option. If you want to split restoration logs on per-PID basis you can use the --log-pid option. The logs of a task having PID of N will appear in a file.N file.

Knowing a new PID of restored task[edit]

When you restore a process tree that lives inside a PID namespace, the PID of a new task will be generated by the system (since the PIDs sitting in image files are treated as virtual, i.e. as seen in a PID namespace). To find out the PID, one can use the --pidfile file option to make CRIU put the new PID into file.

Restoring a VETH pair of devices[edit]

When you restore a net namespace with a VETH device end in it, CRIU will create the other end of the pair in the net namespace you launch CRIU from. By default its name will be the one generated by the veth kernel driver. Not to scan through all the net devices trying to find one, you can use the --external option. For more details see the article about VETH device.

Action scripts[edit]

CRIU can call your hooks on various stages of dumping/restoring. These hooks are added with the --action-script shell-code-to-execute option. When called, the CRTOOLS_SCRIPT_ACTION environment is set to a value determining which type of action is performed.

Shell jobs C/R[edit]

When you run some app directly from the shell (e.g. top), it inherits session ID from bash and uses PTY whose other end sits somewhere outside this shell session (e.g. in sshd or xterm). Strictly speaking this app cannot be checkpointed (since session leader, which is bash, is left outside the image, and so is the pty master). Nor it can be restored easily. But sometimes it might make sense to migrate such app by moving it to other session and attaching to different pty slave "on the fly". For such cases the --shell-job option should be used on both dump and restore. On dump, it will allow CRIU to ignore the "leaked" session leader and pty master. On restore, it will tell CRIU to change session, group and attach to existing pty slave.

File locks C/R[edit]

Some app may use file locks for synchronization. Generally they will use flock or posix file locks, which were achieved by flock or fcntl system calls. For dump/restore, it is hard to be handled perfectly, because we can't guarantee all potential users are dumped for a specific file lock. Right now, we assume that all file lock users are taken into dump, and we should use the --file-locks option on both dump and restore stages if our app may use any file locks. Remember that file locks dump/restore is only safe for container dumping in theory.

Leave task running after checkpoint[edit]

In some scenarios a user might want to leave program in a running state once checkpoint complete. To do that, criu has --leave-running command line option. We strongly encourage NOT to use it, until you understand what you are doing. Leaving task in a running state may lead to inconsistency between task images and external resources such as opened files, TCP connections.

Imagine a task been dumped and then continue execution, for some reason the task close and delete a file which has been checkpointed — any attempt to restore such task will simply fail because CRIU wont be able to open deleted file. Situation become more drastic if TCP connections are checkpointed — during execution the internal state of connections is changed inside kernel and restore then fail as well.

Still, leaving task running might be helpful but it is up to a user to make sure that:

  • no established TCP connections are present;
  • file system get snapshotted with post-dump script (and then manually restored before criu restore takes place).

External bind mounts[edit]

When dumping LXC or Docker container, one will likely meet those. The --ext-mount-map option should be used for such situations.

See also[edit]