Invisible files

In Linux files may be inaccessible for open() but still be present in the system. This can happen in several ways and this page is about how this can happen and what CRIU does about it.

How a file can lose its path

This is pretty simple. A process may do this series of operations

int fd = open("/foo/bar");
unlink("/foo/bar");

After it the /foo/bar file will have its name removed from the filesystem tree (and from the on-disk data too), but since the file is still held by the process (this structure is explained in the article about dumping files), the blob with data itself is still there.

There are two different sub-cases in this scenario. First, when the number of hard links associated with the file is zero, i.e. the /foo/bar name was the last (or the only it ever had) one removed. Another situation is when the link count is not zero, which means, that some other name for this file exists (hard link). In the latter case it's important to notice, that Linux VFS layer generally does not allow to find out this other name. Sometimes it's possible, but typically it's not.

Virtual filesystems

For virtual filesystems like proc or sysfs there's another possibility for such files to appear. It's the removal of the object represented of a file on this FS. In particular, if we open some file in /proc/$pid and the respective task dies the path of the opened file would get removed, while the file itself would be still alive (though reporting ENOENT error on any attempts to read from one).

Name-less files

There's a third possibility for a file not to have a visible name. But this is another story, that has nothing to do with dumping opened files, this it's described in a another article.

Overmounted files

If a task opens a file and then a new mountpoint appears on any part of its path, the original file's path may become non existing or point to different file.

CRIU doesn't work with such files yet, aborting the dump. However, there's a way to fix this.

Although in Linux there's no way to open a file by a name looking under certain mount points, there's the openat() call which may look up a file starting from arbitrary point which, in turn, can be over-mounted too.

What CRIU does about it

Detection and dumping

First of all, CRIU should detect this situation to take place. Modulo some filesystems pecularities, this is done like this.

First, we get the files from the target process via unix socket. Then for each of them get the file's name via /proc by calling readlink on the /proc/self/fd/$fd path. It's important to note, that we readlink self FD to get the file's name we can work with. Next we fstat() the respective self file descriptor.

If the st_nlink field is zero, then the file is fully deleted from the system. Since no filesystems allow to create a name back for such files, we have no other choice other than get the file itself into images. So we generate a so called ghost file in the image directory and copy the file contents into it. Since the file is taken into the images, it would be good if the file size is not huge. By default CRIU dumps ghost files at most 1Mb in size. But with the option --ghost-limit this limit can be changed.

But what happens if the link count is not zero. Then we should check than the name we got from proc is the one with which we can see this file. So we call stat() on this name and compare st_dev and st_inode fields of it with those obtained from the fstat() call earlier. If they match the file is alive and we can just dump its name. If they don't the name we got references some other file and we fail the dump. This can be handled, but this situation is quite rare so we decided to implement support for it later.

But there's also a 3rd possibility -- the stat() could fail with ENOENT error, which means that the file has names, but the one we have it opened by is removed. In this situation we cannot just save the file name in the images, since this name is not longer alive. Neither we can dump the file as ghost, as the same file can be accessed by some other name. And, as was said, there's no way to find this other name. Fortunately, in this case filesystems allow to create a new name for a file, so CRIU calls linkat system call creating a temporary name for this file on the disk and saves this name in the image. This is called link-remap. Since this manoeuvre modifies the filesystem, CRIU requires the special option --link-remap to be passed to it allowing this behaviour. On restore the link-remap names are removed after files restore.

Please note that a file may have been opened by many removed names, and for each a link-remap name should point to the same file, so while dumping and restoring CRIU keeps track of those names to inode mappings.

Virtual filesystems

For proc CRIU uses a slightly different trick. When we see dead name in /proc we cannot link() a new name or create a ghost file. Instead, we remember the PID of the process that died, and on restore create a temporary task with the desired pid, which gets killed right after all its open()-ers are restored.