Inheriting FDs on restore

This article describes why and how to use the --external file[...], --external tty[...] and --inherit-fd command line options.

Background

There are cases where a process' file descriptor cannot be restored from the checkpoint images. For example, a pipe file descriptor with one end in the checkpointed process and the other end in a separate process (that was not part of the checkpointed process tree) cannot be restored because after checkpoint the pipe will be broken.

There are also cases where the user wants to use a new file during restore instead of the original file at checkpoint time. For example, the user wants to change the log file of a process from /path/to/oldlog to /path/to/newlog.

In these cases, criu's caller should set up a new file descriptor to be inherited by the restored process and specify the file descriptor with the --inherit-fd command line option.

Limitations

Please note that inherit fd support breaks applications that depend on the state of the file descriptor being inherited. For example, an application that depends on the seek offset within a file at checkpoint time will fail after restore if the file is replaced with another file at a different seek offset.

You should consider using this feature only for specific use cases that you know for sure won't break the application. In other words, use it at your own risk.

Argument Format

The argument of --inherit-fd has the format fd[%d]:%s, where %d tells criu which of its own file descriptors to use for restoring the file identified by %s. Note that the file descriptor number should be enclosed in literal square brackets (and as square brackets are handled specially by the command line shell, they might need to be properly escaped).

Example: criu restore ... --inherit-fd 'fd[7]:tmp/old' ...

Debugging aid

As a debugging aid, if the argument has the format debug[%d]:%s, criu will just write out the string after colon to the file descriptor %d. This can be used to leave a "restore marker" in the output stream of the process.

Examples

Regular files

Let's redirect the output of test.sh from the simple loop example to /tmp/old, checkpoint it, and restore it to use /tmp/new as its output file.

As you see below, we have used criu's file descriptor 7 (just as an arbitrary descriptor) with the --inherit-fd option. Note that the file path in the argument of --inherit-fd is relative to the root of the process (i.e., tmp/old).

	$ ./test.sh > /tmp/old &
	<pid>
	$ sudo criu dump -j -t <pid>
	$ sudo criu restore -d -j --inherit-fd 'fd[7]:tmp/old' 7> /tmp/new

This is an example of an application that does not depend on the state of its output file, so --inherit-fd wouldn't break it.

If your application depends on a particular state, say the seek offset, you should seek to that offset before you let criu inherit the new descriptor.

External unnamed pipes

To replace an external pipe that was connected to an external process is more involved because unnamed pipes are typically created between a parent and a child process. Therefore, the external process has to invoke criu as its child and set up a new pipe between itself and criu to be inherited by the restored process.

To demonstrate how this is done, consider a parent process that spawns a child which will write messages to its parent through a pipe. When a couple of messages has been received by parent, it invokes criu to checkpoint the child. As a result of checkpoint, the child exits and the pipe will be broken. Then, the parent sets up a new pipe between itself and criu and invokes it to restore the child using the new pipe (instead of the old one).

Note that unlike restoring, checkpointing the child doesn't have to be done by the parent (i.e., can be done manually by the user). But for simplicity, we're having the parent checkpoint the child.

The source code for pipe.c can be found in criu source tree under test/pipes. Each output line is preceded by the process that has generated it. Notice pipe:[472728] is replaced with pipe:[474324] passed through criu's fd[3].

$ cc -Wall -o pipe pipe.c
$ sudo ./pipe
[child 26371] writing hello 1 to pipe:[472728] via fd 4
[parent 26370] read hello 1 from pipe:[472728]

[child 26371] writing hello 2 to pipe:[472728] via fd 4
[parent 26370] read hello 2 from pipe:[472728]

[dump 26372] criu dump -D /tmp/criu_img -o dump.log -v4 -j -t 26371 
[parent 26370] [child 26371] exited with status 9
[parent 26370] [dump 26372] exited with status 0
[parent 26370] creating a new pipe

[restore 26378] criu restore -d -D /tmp/criu_img -o restore.log --pidfile restore.pid -v4 -j --inherit-fd fd[3]:pipe:[472728]   
[parent 26370] [restore 26378] exited with status 0

[child 26371] writing hello 3 to pipe:[474324] via fd 4
[parent 26370] read hello 3 from pipe:[474324]

[child 26371] writing hello 4 to pipe:[474324] via fd 4
[parent 26370] read hello 4 from pipe:[474324]

[parent 26370] [child 26371] exited with status 0
$

External files, FIFOs

"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via /proc/self/fd, so it doesn't matter with which flags an inherited descriptor has been opened. The format of file id is file[mnt_id:inode].

    criu dump --external file[72:a3e7]
    criu restore --inherit-fd fd[4]:file[72:a3e7]

External TTYs

The format of tty id is tty[rdev:dev]. Note that a pair of mnt_id:inode is not used here, as there can be two device files with the same rdev but different inode numbers.

Here is an example of dumping and restoring an external TTY:

    # setsid --ctty ipython
    In [1]: import os
    In [2]: st = os.stat("/proc/self/fd/0")
    In [3]: print("tty[%x:%x]" % (st.st_rdev, st.st_dev))
    tty[8802:19]
 
    # criu dump -t `pgrep ipython` --external 'tty[8802:19]'
    # criu restore --inherit-fd 'fd[1]:tty[8802:19]'