Difference between revisions of "Inheriting FDs on restore"

From CRIU
Jump to navigation Jump to search
(added to cli category)
(formatting fixes, slight rewording)
Line 1: Line 1:
This HOWTO page describes how to use the --inherit-fd command line option.
+
This article describes why and how to use the <code>--inherit-fd</code> command line option.
  
 
== Background ==
 
== Background ==
  
There are cases where a process's file descriptor cannot be restored
+
There are cases where a process' file descriptor cannot be restored
 
from the checkpoint images.  For example, a pipe file descriptor with one
 
from the checkpoint images.  For example, a pipe file descriptor with one
 
end in the checkpointed process and the other end in a separate process
 
end in the checkpointed process and the other end in a separate process
Line 16: Line 16:
 
In these cases, criu's caller should set up a new file descriptor to be
 
In these cases, criu's caller should set up a new file descriptor to be
 
inherited by the restored process and specify the file descriptor with
 
inherited by the restored process and specify the file descriptor with
the --inherit-fd command line option.
+
the <code>--inherit-fd</code> command line option.
  
== Warning ==
+
== Limitations ==
  
 
Please note that inherit fd support breaks applications that depend
 
Please note that inherit fd support breaks applications that depend
Line 26: Line 26:
 
at a different seek offset.
 
at a different seek offset.
  
You should consider inherit fd only for specific use cases that you know
+
You should consider using this feature only for specific use cases that you know
 
for sure won't break the application.  In other words, use it at your
 
for sure won't break the application.  In other words, use it at your
 
own risk.
 
own risk.
Line 32: Line 32:
 
== Argument Format ==
 
== Argument Format ==
  
The argument of --inherit-fd has the format fd[%d]:%s, where %d tells
+
The argument of <code>--inherit-fd</code> has the format <code>fd[%d]:%s</code>,
criu which of its own file descriptors to use for restoring the file
+
where <code>%d</code> tells criu which of its own file descriptors to use
identified by %s.
+
for restoring the file identified by <code>%s</code>. Note that the file descriptor
 +
number should be enclosed in literal square brackets (and as square brackets are
 +
handled specially by the command line shell, they might need to be properly escaped).
  
== Debugging Aid ==
+
Example: <code>criu restore ... --inherit-fd 'fd[7]:tmp/old' ...</code>
  
As a debugging aid, if the argument has the format debug[%d]:%s, criu
+
== Debugging aid ==
will just write out the string after colon to the file descriptor %d.
+
 
 +
As a debugging aid, if the argument has the format <code>debug[%d]:%s</code>, criu
 +
will just write out the string after colon to the file descriptor <code>%d</code>.
 
This can be used to leave a "restore marker" in the output stream of
 
This can be used to leave a "restore marker" in the output stream of
 
the process.
 
the process.
  
== Example 1 - Regular Files ==
+
== Examples ==
 +
 
 +
=== Regular files ===
  
Let's redirect the output of test.sh from [[Simple loop]]
+
Let's redirect the output of <code>test.sh</code> from [[Simple loop]]
to /tmp/old, checkpoint it, and restore it to use /tmp/new as its
+
to <code>/tmp/old</code>, checkpoint it, and restore it to use <code>/tmp/new</code> as its
 
output file.
 
output file.
  
 
As you see below, we have used criu's file descriptor 7 (just as an
 
As you see below, we have used criu's file descriptor 7 (just as an
arbitrary descriptor) with the --inherit-fd option. Notice that the path
+
arbitrary descriptor) with the <code>--inherit-fd</code> option. Note that the file path
in the argument of --inherit-fd is relative to the root of the process
+
in the argument of <code>--inherit-fd</code> is relative to the root of the process
(i.e., tmp/old).
+
(i.e., <code>tmp/old</code>).
  
 
<pre>
 
<pre>
Line 62: Line 68:
  
 
This is an example of an application that does not depend on the state
 
This is an example of an application that does not depend on the state
of its output file, so --inherit-fd wouldn't break it.
+
of its output file, so <code>--inherit-fd</code> wouldn't break it.
  
 
If your application depends on a particular state, say the seek offset,
 
If your application depends on a particular state, say the seek offset,
Line 68: Line 74:
 
descriptor.
 
descriptor.
  
== Example 2 - External Unnamed Pipes ==
+
=== External unnamed pipes ===
  
 
To replace an external pipe that was connected to an external process
 
To replace an external pipe that was connected to an external process
Line 88: Line 94:
 
simplicity, we're having the parent checkpoint the child.
 
simplicity, we're having the parent checkpoint the child.
  
The source code for pipe.c can be found in criu source tree under
+
The source code for <code>pipe.c</code> can be found in criu source tree under
test/pipes.  Each output line is preceded by the process that has
+
<code>test/pipes</code>.  Each output line is preceded by the process that has
generated it.  Notice pipe:[472728] is replaced with pipe:[474324]
+
generated it.  Notice <code>pipe:[472728]</code> is replaced with <code>pipe:[474324]</code>
passed through criu's fd[3].
+
passed through criu's <code>fd[3]</code>.
  
 
<pre>
 
<pre>
Line 120: Line 126:
 
</pre>
 
</pre>
  
== Example 3 - External files, FIFO-s ==
+
=== External files, FIFOs ===
"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via /proc/self/fd, so it doesn't metter with which flags an inherited descriptor has been opened. The format of file id is file[mnt_id:inode].
+
"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via <code>/proc/self/fd</code>, so it doesn't metter with which flags an inherited descriptor has been opened. The format of file id is <code>file[mnt_id:inode]</code>.
  
 
<pre>
 
<pre>
Line 128: Line 134:
 
</pre>
 
</pre>
  
== Example 4 - External TTY-s ==
+
=== External TTYs ===
  
The format of tty id is tty[rdev:dev]. Why can we not use a pair of mnt_id and inode here?
+
The format of tty id is <code>tty[rdev:dev]</code>. Why can we not use a pair of <code>mnt_id</code> and <code>inode</code> here?
  
 
<pre>
 
<pre>

Revision as of 21:44, 14 September 2016

This article describes why and how to use the --inherit-fd command line option.

Background

There are cases where a process' file descriptor cannot be restored from the checkpoint images. For example, a pipe file descriptor with one end in the checkpointed process and the other end in a separate process (that was not part of the checkpointed process tree) cannot be restored because after checkpoint the pipe will be broken.

There are also cases where the user wants to use a new file during restore instead of the original file at checkpoint time. For example, the user wants to change the log file of a process from /path/to/oldlog to /path/to/newlog.

In these cases, criu's caller should set up a new file descriptor to be inherited by the restored process and specify the file descriptor with the --inherit-fd command line option.

Limitations

Please note that inherit fd support breaks applications that depend on the state of the file descriptor being inherited. For example, an application that depends on the seek offset within a file at checkpoint time will fail after restore if the file is replaced with another file at a different seek offset.

You should consider using this feature only for specific use cases that you know for sure won't break the application. In other words, use it at your own risk.

Argument Format

The argument of --inherit-fd has the format fd[%d]:%s, where %d tells criu which of its own file descriptors to use for restoring the file identified by %s. Note that the file descriptor number should be enclosed in literal square brackets (and as square brackets are handled specially by the command line shell, they might need to be properly escaped).

Example: criu restore ... --inherit-fd 'fd[7]:tmp/old' ...

Debugging aid

As a debugging aid, if the argument has the format debug[%d]:%s, criu will just write out the string after colon to the file descriptor %d. This can be used to leave a "restore marker" in the output stream of the process.

Examples

Regular files

Let's redirect the output of test.sh from Simple loop to /tmp/old, checkpoint it, and restore it to use /tmp/new as its output file.

As you see below, we have used criu's file descriptor 7 (just as an arbitrary descriptor) with the --inherit-fd option. Note that the file path in the argument of --inherit-fd is relative to the root of the process (i.e., tmp/old).

	$ ./test.sh > /tmp/old &
	<pid>
	$ sudo criu dump -j -t <pid>
	$ sudo criu restore -d -j --inherit-fd 'fd[7]:tmp/old' 7> /tmp/new

This is an example of an application that does not depend on the state of its output file, so --inherit-fd wouldn't break it.

If your application depends on a particular state, say the seek offset, you should seek to that offset before you let criu inherit the new descriptor.

External unnamed pipes

To replace an external pipe that was connected to an external process is more involved because unnamed pipes are typically created between a parent and a child process. Therefore, the external process has to invoke criu as its child and set up a new pipe between itself and criu to be inherited by the restored process.

To demonstrate how this is done, consider a parent process that spawns a child which will write messages to its parent through a pipe. When a couple of messages has been received by parent, it invokes criu to checkpoint the child. As a result of checkpoint, the child exits and the pipe will be broken. Then, the parent sets up a new pipe between itself and criu and invokes it to restore the child using the new pipe (instead of the old one).

Note that unlike restoring, checkpointing the child doesn't have to be done by the parent (i.e., can be done manually by the user). But for simplicity, we're having the parent checkpoint the child.

The source code for pipe.c can be found in criu source tree under test/pipes. Each output line is preceded by the process that has generated it. Notice pipe:[472728] is replaced with pipe:[474324] passed through criu's fd[3].

$ cc -Wall -o pipe pipe.c
$ sudo ./pipe
[child 26371] writing hello 1 to pipe:[472728] via fd 4
[parent 26370] read hello 1 from pipe:[472728]

[child 26371] writing hello 2 to pipe:[472728] via fd 4
[parent 26370] read hello 2 from pipe:[472728]

[dump 26372] criu dump -D /tmp/criu_img -o dump.log -v4 -j -t 26371 
[parent 26370] [child 26371] exited with status 9
[parent 26370] [dump 26372] exited with status 0
[parent 26370] creating a new pipe

[restore 26378] criu restore -d -D /tmp/criu_img -o restore.log --pidfile restore.pid -v4 -j --inherit-fd fd[3]:pipe:[472728]   
[parent 26370] [restore 26378] exited with status 0

[child 26371] writing hello 3 to pipe:[474324] via fd 4
[parent 26370] read hello 3 from pipe:[474324]

[child 26371] writing hello 4 to pipe:[474324] via fd 4
[parent 26370] read hello 4 from pipe:[474324]

[parent 26370] [child 26371] exited with status 0
$

External files, FIFOs

"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via /proc/self/fd, so it doesn't metter with which flags an inherited descriptor has been opened. The format of file id is file[mnt_id:inode].

    criu dump --external file[72:a3e7]
    criu restore --inherit-fd fd[4]:file[72:a3e7]

External TTYs

The format of tty id is tty[rdev:dev]. Why can we not use a pair of mnt_id and inode here?

    $ ipython
    In [1]: import os
    In [2]: st = os.stat("/proc/self/fd/0")
    In [3]: print "tty[%x:%x]" % (st.st_rdev, st.st_dev)
    tty:[8800:d]
    
    $ps -C sleep
      PID TTY          TIME CMD
     4109 ?        00:00:00 sleep
    
    $ ./criu dump --external 'tty[8800:d]' -D imgs -v4 -t 4109
    $ ./criu restore --inherit-fd 'fd[1]:tty[8800:d]' -D imgs -v4