CRIU - User contributions [en]

Checkpoint/Restore

2017-05-12T11:05:50Z

OpenGridSchedulerGridEngine: /* Switch to restorer context, restore the rest and continue */ spelling

This page describes the overall design of how Checkpoint and Restore work in CRIU.

== Checkpoint ==

The checkpoint procedure relies heavily on '''/proc''' file system (it's a general place where criu takes all the information it needs).
The information gathered from /proc includes:

* Files descriptors information (via '''/proc/$pid/fd''' and '''/proc/$pid/fdinfo''').
* Pipes parameters.
* Memory maps (via '''/proc/$pid/maps''' and '''/proc/$pid/map_files/''').
* etc.

The process dumper (called a dumper below) does the following steps during the checkpoint stage.

=== Collect process tree and freeze it ===
The '''$pid''' of a process group leader is obtained from the command line (<code>--tree</code> option). By using this '''$pid''' the dumper walks though '''/proc/$pid/task/''' directory collecting threads and through the '''/proc/$pid/task/$tid/children''' to gathers children recursively. While walking tasks are stopped using the <code>ptrace</code>'s <code>PTRACE_SEIZE</code> command.

''See also: [[Freezing the tree]]''

=== Collect tasks' resources and dump them ===
At this step CRIU reads all the information (it knows) about collected tasks and writes them to dump files. The resources are obtained via
# VMAs areas are parsed from '''/proc/$pid/smaps''' and mapped files are read from '''/proc/$pid/map_files''' links
# File descriptor numbers are read via '''/proc/$pid/fd'''
# Core parameters of a task (such as registers and friends) are being dumped via ptrace interface and parsing '''/proc/$pid/stat''' entry.

Then CRIU injects a [[parasite code]] into a task via ptrace interface. This is done in two steps -- at first we inject only a few bytes for ''mmap'' syscall at CS:IP the task has at moment of seizing. Then ptrace allow us to run an injected syscall and we allocate enough memory for a parasite code chunk we need for dumping. After that the parasite code is copied into new place inside dumpee address space and CS:IP set respectively to point to our parasite code.

From parsite context CRIU does more information such as
# Credentials
# Contents of memory

=== Cleanup ===

After everything dumped (such as memory pages, which can be written out only from inside dumpee address space) we use ptrace facility again and cure dumpee by dropping out all our parasite code and restoring original code. Then CRIU detaches from tasks and they continue to operate.

== Restore ==

The restore procedure (aka restorer) is done by CRIU morphing itself into the tasks it restores. On the top-level it consists of 4 steps

=== Resolve shared resources ===

At this step CRIU reads in image files and finds out which processes share which resources. Later shared resources are restored by some one process and all the others either inherit one on the 2nd stage (like session) or obtain in some other way. The latter is, for example, shared files which are sent with SCM_CREDS messages via unix sockets, or shared memory areas that are restoring via <code>memfd</code> file descriptor.

=== Fork the process tree ===

At this step CRIU calls fork() many times to re-created the processes needed to be restored. Note, that threads are not restored here, but on the 4th step.

=== Restore basic tasks resources ===

Here CRIU restores all resources but

# memory mappings exact location
# timers
# credentials
# threads

The restoration of the above four types of resources are delayed till the last stage for the reasons described below. On this stage CRIU opens files, prepares [[namespaces]], maps (and fills with data) private memory areas, creates sockets, calls chdir() and chroot() and doing some more.

=== Switch to restorer context, restore the rest and continue ===

The reason for restorer blob is simple. Since criu morphs into the target process, it will have to unmap all its memory and put back the target one. While doing so, some code should exist in memory (the code doing the munmap and mmap). Therefore, the restorer blob is introduced. It's a small piece of code, that doesn't intersect with criu mappings AND target mappings. At the end of stage 2 criu jumps into this blob and restores the memory maps.

At the same place we restore timers not to make them fire too early, here we restore credentials to let criu do privileged operations (like fork-with-pid) and threads not to make them suffer from sudden memory layout change.

''See also: [[restorer context]], [[tree after restore]].''

[[Category:Under the hood]]

Restorer context

2017-04-17T03:15:26Z

OpenGridSchedulerGridEngine: /* What is restored there and why */ typo

This page describes what this context is and why we need one.

== What is it? ==

The restorer context is the last stage of the [[Checkpoint/Restore|restore]] process. It differs from the regular CRIU's (process) context like the [[parasite code]] does -- it doesn't have any libraries, it is PIE-compiled and can only work on fixed amount of memory. In this context CRIU restores

# memory
# timers
# credentials
# threads

== Why separate context? ==

The reasoning for this is simple -- when CRIU comes to the state when it needs to restore process' memory, it should unmap all the old mappings and map the new ones. But since CRIU process would do this operation on itself, once the old code is unmapped CRIU will seg-fault right on the exit from the <code>munmap()</code> system call. Also this code should be get over-mmaped by the mapping is restores. So we need some code that would do this. And this other code should "sit" in two address spaces simultaneously -- in the CRIU's one and in the target one.

The switch to this context is done in several steps.

First, we collect the data needed by the restorer code and puts all it into one sequential memory area. Then, knowing the data size and the restorer code size, we find the appropriate hole in the intersection of CRIU's and target mappings. Then we mmap() this region, mremap() the data into it, put the restorer blob nearby, fix the pointers (see below) and call assember's "jump" instruction to get there.

Now what "fix the pointers" mean. When we collected data for CRIU we addressed the objects in this are using pointers valid in CRIU address space. When we will jump into restorer code pointers in there should "know" where the respective objects are. So knowing where from the restorer counts pointers and the structure of the restorer data, we alter them respectively.

== What is restored there and why ==

So, memory is restored here for the reasons described above. Note, that here CRIU does two things only

# Move anonymous VMAs to proper places
# Map new file VMAs

The anonymous memory is mmaped and filled with data earlier, in restorer it's only mremap()-ed into proper addressed. The files mapping are just mmap()-ed, as data in them sits in files :)

Timers are restore here, since CRIU processes can wait for each other for some time while restoring and not to lose timer ticks there, we delay timers arming this the last moment.

Credentials are restored here to allow CRIU perform privileged operations such as fork-with-pid or chroot().

Threads are restored here for simplicity. If we restored them before, we'd have to "park" them while we change the memory layout. Instead of doing this, we first toss the memory around, then create threads.

== See also ==
* [[Code blobs]]
* [[Compel]]

[[Category:Under the hood]]

Restorer context

2017-04-16T03:32:43Z

OpenGridSchedulerGridEngine: /* What is it? */ typo

This page describes what this context is and why we need one.

== What is it? ==

The restorer context is the last stage of the [[Checkpoint/Restore|restore]] process. It differs from the regular CRIU's (process) context like the [[parasite code]] does -- it doesn't have any libraries, it is PIE-compiled and can only work on fixed amount of memory. In this context CRIU restores

# memory
# timers
# credentials
# threads

== Why separate context? ==

The reasoning for this is simple -- when CRIU comes to the state when it needs to restore process' memory, it should unmap all the old mappings and map the new ones. But since CRIU process would do this operation on itself, once the old code is unmapped CRIU will seg-fault right on the exit from the <code>munmap()</code> system call. Also this code should be get over-mmaped by the mapping is restores. So we need some code that would do this. And this other code should "sit" in two address spaces simultaneously -- in the CRIU's one and in the target one.

The switch to this context is done in several steps.

First, we collect the data needed by the restorer code and puts all it into one sequential memory area. Then, knowing the data size and the restorer code size, we find the appropriate hole in the intersection of CRIU's and target mappings. Then we mmap() this region, mremap() the data into it, put the restorer blob nearby, fix the pointers (see below) and call assember's "jump" instruction to get there.

Now what "fix the pointers" mean. When we collected data for CRIU we addressed the objects in this are using pointers valid in CRIU address space. When we will jump into restorer code pointers in there should "know" where the respective objects are. So knowing where from the restorer counts pointers and the structure of the restorer data, we alter them respectively.

== What is restored there and why ==

So, memory is restored here for the reasons described above. Note, that here CRIU does two things only

# Move anonymous VMAs to proper places
# Map new file VMAs

The anonymous memory is mmaped and filled with data earlier, in restorer it's only mremap()-ed into proper addressed. The files mapping are just mmap()-ed, as data in them sits in files :)

Timers are restore here, since CRIU processes can wait for each other for some time while restoring and not to lose timer ticks there, we delay timers arming this the last moment.

Credentials are restored here to allow CRIU perform privileged operations such as fork-with-pid or chroot().

Threads are restored here for simplicity. If we restored them before, we'd have to "park" them while we change the memory layout. Instead of doing this, we first toss the memory arond, then create threads.

== See also ==
* [[Code blobs]]
* [[Compel]]

[[Category:Under the hood]]

TCP repair TODO

2016-08-22T11:03:37Z

OpenGridSchedulerGridEngine: a API -> an API

TCP repair feature in the Linux kernel is supposed to help migrating a TCP socket. It's not yet complete and this page lists what is to be done.

; Transitional states
: Currently we support sockets in ''closed'' and ''establised'' states. However, if a socket is in e.g. ''syn-sent'' state the process of turning it into ''established'' can last long. We should teach the kernel and criu to checkpoint and restore this and other states

; Optimized restore
: Currently the whole outgoing queue is restored in a "all was sent, waiting for ACK-s" state. After this the data that was really not sent yet will be re-transmitted after a while. This will make the connection work, but will delay it for some time. Need to improve this.

; OOB data
: Nothing to say here actually. This data is just not supported currently.

; Window restore fix
: Currently what we do it send the window probe skb when repair is OFF. The other side should send us the response, but this process is not guaranteed to work. Need to fix this either by saving and restoring the window value, or by re-transmitting the probe again and again.

; Shutdown sockets repair
: Need to place checks in the inet shutdown code similar to those on connect/sendmsg paths.

; Connection tracking
: The nf_conntrack thing in the kernel is about to be live-migrated too. There's currently an API for getting the conntrack info (/proc file), but no such for restoring it by hands.

[[Category:Plans]]
[[Category:Network]]