CRIU - User contributions [en]

Todo

2021-12-06T08:33:14Z

Efiop: remove @efiop's asignees

{{note|This list is being transformed into the [https://github.com/checkpoint-restore/criu/issues github issues]}}

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| tests || automate process of measurement code coverage || easy || - || It is required to automate process of getting code coverage. We have code coverage results [http://criu.org/cov/ measured in 2012]. Would be nice to get up to date results on periodic basis and without manual actions.
|-
| crtools || Inherit resources, not restore || medium || - || Sigactions are restored for every task before it fork()-s. Then children check for the sa_action from their image matches to one it got from parent. Need to do the same for rlimits, maybe other resources too.
|-
| crtools || Implement [[restorer v2]] || hard (v2) || - ||
|-
| crtools || New images format || medium (v2) || - || See [[what's bad with V1 images]]
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || - || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard (v2) || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly — dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [https://github.com/checkpoint-restore/criu/blob/master/test/zdtm/static/session02.c session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || - || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/checkpoint-restore/criu/blob/master/criu/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| crtools || Make CRIU work on AArch32 with CONFIG_KUSER_HELPERS=n || medium || cov || CRIU currently fails on AArch32 kernels built with CONFIG_KUSER_HELPERS=n.
|-
| tests || Run many/all tests in "container" || medium || - || Currently we run zdtm tests one-by-one. It would be nice to run the all in one pseudo-container and C/R them as one big subtree.
|-
| tests || [[Fuzz testing|Trinity-like (fuzz) testing]] || hard || - || The existing suite is 99% functionality testing. Need more sophisticated testing -- take a process that has done a random set of actions, C/R one, check that all is OK. The latter is the most complicated thing.
|-
| crtools || Set checkpoint tokens without recompiling || medium || - || Sometimes you need to call checkpoint in some particular point of the code. The way to do it now is to recompile app with criu_dump() call where needed. But it is quite a bummer to recompile, repackage and redistribute an app you want to c/r. It would be great if one could set a token in app source and then let criu find that point in running task and take a snapshot. The best way to do it might be in libcriu.
|-
| crtools || Large ghost files support || medium || - || If we have a large ghost (opened unlinked) file, it's inefficient to copy it to another node via CRIU dump. Need to migrate them independently, iteratively, using memory tracking.
|}

[[Category:Development]]
[[Category:Plans]]

Criu-coredump

2016-07-12T13:43:18Z

Efiop: Fix page to describe current coredump statate

Generate valid core dump from CRIU images.
=== Description ===
Criu-coredump is a tool that takes a directory with CRIU images and produces Elf cores.

It consists of a python script(criu/coredump/criu-coredump) for cmdline use, as well as a python module(criu/coredump/criu_coredump) to manipulate generated cores on a deeper level.

=== Prerequisites ===

As the main goal of CRIU is to be able to save process tree on disk to be restored later, the amount of information its images contain is much bigger than what is typically contained in a core dump(i.e. memory + few notes). That and the fact that CRIU provides a number of convenient API's, in theory gives anyone an ability to use CRIU for debugging their applications, i.e. like the abandoned google-coredumper allows you to generate core dumps at any time. But in practice, there are no tools that have native support for criu images, making it impossible to conveniently use them for such purpose. So it would be nice if we could convert criu images into much more common core dumps to be used with a great variety of available tools.

=== Limitations ===

For now only x86_64 is supported. The matter of supporting other architectures supported by CRIU is to use proper constants in elf header and provide proper arch-dependent notes.

Some notes in coredump are not fullfilled(i.e. SIGINFO).

VVAR and VSYSCALL vmas are just filled with zeroes, as CRIU doesn't currently provide them. VVAR is a tricky one and gdb also has problems dumping it[https://lkml.org/lkml/2015/3/12/439]. VSYSCALL in theory could just be read from current process, so it should be quite easy to fix.

=== Usage ===

<pre>
criu-coredump [-h] [-i IN] [-p PID] [-o OUT]

CRIU core dump

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN directory where to get images from
-p PID, --pid PID generate coredump for specific pid(all pids py default)
-o OUT, --out OUT directory to write coredumps to
</pre>

=== Project resources ===
[https://asciinema.org/a/18936 Demo] of a previous version of criu-coredump, when it was planned to be a part of CRIT.

[[Category:Sub-projects]]

Todo

2016-05-06T19:15:40Z

Efiop: rm patch tester task

{{note|This list is being transformed into the [https://github.com/xemul/criu/issues github issues]}}

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| tests || automate process of measurement code coverage || easy || - || It is required to automate process of getting code coverage. We have code coverage results [http://criu.org/cov/ measured in 2012]. Would be nice to get up to date results on periodic basis and without manual actions.
|-
| crtools || Inherit resources, not restore || medium || - || Sigactions are restored for every task before it fork()-s. Then children check for the sa_action from their image matches to one it got from parent. Need to do the same for rlimits, maybe other resources too.
|-
| crtools || Implement [[restorer v2]] || hard (v2) || - ||
|-
| crtools || New images format || medium (v2) || - || See [[what's bad with V1 images]]
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || Kuprieiev Ruslan || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard (v2) || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly — dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [https://github.com/xemul/criu/blob/master/test/zdtm/live/static/session02.c session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || Ruslan Kuprieiev || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| crtools || Make CRIU work on AArch32 with CONFIG_KUSER_HELPERS=n || medium || cov || CRIU currently fails on AArch32 kernels built with CONFIG_KUSER_HELPERS=n.
|-
| kernel || Fix VDSO remapping on non-x86 architectures || medium || Laurent Dufour, cov || However some architectures like PowerPC and ARM are keeping a reference to the VDSO base address to build the signal return stack frame by calling the VDSO sigreturn service. So once the VDSO has been moved, this reference is no more valid and the signal frame built later are not usable.
|-
| tests || Run many/all tests in "container" || medium || - || Currently we run zdtm tests one-by-one. It would be nice to run the all in one pseudo-container and C/R them as one big subtree.
|-
| tests || [[Fuzz testing|Trinity-like (fuzz) testing]] || hard || - || The existing suite is 99% functionality testing. Need more sophisticated testing -- take a process that has done a random set of actions, C/R one, check that all is OK. The latter is the most complicated thing.
|-
| crtools || Set checkpoint tokens without recompiling || medium || - || Sometimes you need to call checkpoint in some particular point of the code. The way to do it now is to recompile app with criu_dump() call where needed. But it is quite a bummer to recompile, repackage and redistribute an app you want to c/r. It would be great if one could set a token in app source and then let criu find that point in running task and take a snapshot. The best way to do it might be in libcriu.
|-
| crtools || Large ghost files support || medium || - || If we have a large ghost (opened unlinked) file, it's inefficient to copy it to another node via CRIU dump. Need to migrate them independently, iteratively, using memory tracking.
|}

[[Category:Development]]
[[Category:Plans]]

Todo

2015-09-28T11:57:15Z

Efiop: checkpoint tokens without recompiling

This list is being transformed into the github issues @ https://github.com/xemul/criu/issues

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| tests || automate process of measurement code coverage || easy || - || It is required to automate process of getting code coverage. We have code coverage results [http://criu.org/cov/ measured in 2012]. Would be nice to get up to date results on periodic basis and without manual actions.
|-
| crtools || Inherit resources, not restore || medium || - || Sigactions are restored for every task before it fork()-s. Then children check for the sa_action from their image matches to one it got from parent. Need to do the same for rlimits, maybe other resources too.
|-
| crtools || Implement [[restorer v2]] || hard (v2) || - ||
|-
| crtools || New images format || medium (v2) || - || See [[what's bad with V1 images]]
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || Kuprieiev Ruslan || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard (v2) || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly — dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || Ruslan Kuprieiev || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| crtools || Make CRIU work on AArch32 with CONFIG_KUSER_HELPERS=n || medium || cov || CRIU currently fails on AArch32 kernels built with CONFIG_KUSER_HELPERS=n.
|-
| kernel || Fix VDSO remapping on non-x86 architectures || medium || Laurent Dufour, cov || However some architectures like PowerPC and ARM are keeping a reference to the VDSO base address to build the signal return stack frame by calling the VDSO sigreturn service. So once the VDSO has been moved, this reference is no more valid and the signal frame built later are not usable.
|-
| tests || Run many/all tests in "container" || medium || - || Currently we run zdtm tests one-by-one. It would be nice to run the all in one pseudo-container and C/R them as one big subtree.
|-
| tests || Trinity-like (fuzz) testing || hard || - || The existing suite is 99% functionality testing. Need more sophisticated testing -- take a process that has done a random set of actions, C/R one, check that all is OK. The latter is the most complicated thing.
|-
| tests/infrastructure || Run tests on patches sent to the mailing lists || medium || Ruslan Kuprieiev || It's quite typical that a set sent to the mailing list fails some tests. Need a robot that would monitor the list, check the patches and send the result back.
|-
| crtools || Set checkpoint tokens without recompiling || medium || - || Sometimes you need to call checkpoint in some particular point of the code. The way to do it now is to recompile app with criu_dump() call where needed. But it is quite a bummer to recompile, repackage and redistribute an app you want to c/r. It would be great if one could set a token in app source and then let criu find that point in running task and take a snapshot. The best way to do it might be in libcriu.
|}

[[Category:Development]]
[[Category:Plans]]

Todo

2015-09-04T00:25:24Z

Efiop: --leave-stopped for restore

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| crtools/crit || Library/API to access pagemap+page images || medium || - || The [[memory dumps]] are quite sophisticated. It would be nice to have a library or API to access the data in them using some simple API. IOW -- API-ize the page_read.c
|-
| crtools || Zombies with threads :) || easy || - || When milti-thread task's leader thread exits task turns into zombie state, but the other threads keep running. Need to support this (zdt test pthread02).
|-
| tests || automate process of measurement code coverage || easy || - || It is required to automate process of getting code coverage. We have code coverage results [http://criu.org/cov/ measured in 2012]. Would be nice to get up to date results on periodic basis and without manual actions.
|-
| crtools || Non-full mntns dump || medium || - || Systemd launches services in a new mount namespace with a single change -- /tmp is re-mounted into a private one(PrivateTmp option). Need to invent an API for dumping only a part of mntns.
|-
| kernel/crtools || Remap [[Vdso]] || medium || - || When at restore VDSO is found not in the place it was on dump we should <code>mremap()</code> one. Unfortunately not always we can do it, need to fix the kernel.
|-
| crtools || Make dump and restore work under [[selinux]] || medium || - || Selinux imposes more restrictions on the stuff we typically do.
|-
| crtools || Inherit resources, not restore || medium || - || Sigactions are restored for every task before it fork()-s. Then children check for the sa_action from their image matches to one it got from parent. Need to do the same for rlimits, maybe other resources too.
|-
| crtools || Implement [[restorer v2]] || hard (v2) || - ||
|-
| crtools || New images format || medium (v2) || - || See [[what's bad with V1 images]]
|-
| crtools || Make [[RPC]] "swrk" mode public || medium || xemul || There's an "swrk" action in CRIU [[Usage|CLI API]]. This turns CRIU into service worker accepting RPC commands. This mode is not documented. Need to standartify one and make public.
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| crtools || Deduplication for shmem [[memory dumps]] || easy || - || We have dedup action and --auto-dedup option for dump/restore which only works for pid pagemaps. Need the same for shmem.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| crtools || Show what was left in the system after dump || easy || - || When we use [[Invisible files|--link-remap]] option or [[TCP connection|--tcp-established]] one CRIU leaves some traces in the system, in particular -- temporary hard links in the former case and iptables rules in the latter. Need some way to show these to the user.
|-
| crtools || Decode flags from images into symbolic names || easy || - || When we print images contents with [[CRIT]] the flags fields are shown in decimal/hex numbers. It would be nice to print the symbolic names for known flags in some form.
|-
| crtools || Leases support || easy || - || The F_SETLEASE/F_GETLEASE API is not currently supported, but doesn't differ much from regular locks.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || Kuprieiev Ruslan || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || AIO with pending events || medium || - || When we dump AIO ring we check it not to contain events inside and abort the dump otherwise. Need to dump events too and put them back on restore.
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid. It makes sense to request CRIU to dump a set of tasks from given cgroup.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Support OFD posix locks || easy || - || These are still rarely used, but exist. Might make sense to support them in advance, it looks like kernel API allows for that.
|-
| crtools || Optimize kcmp calls || medium || - || CRIU build [[kcmp trees]] to find out IDs of such objects as MM, FDT and others. Currently we kcmp all tasks to get the ID, but we can improve that by pre-generating ID based on objects that live on MM, FS, etc. If pre-ID of two tasks matches, then we call kcmp, if not -- objects ''are'' different.
|-
| crtools || Shmem changes tracking || medium || - || Memory changes tracker works on anonymous private mappings only. Anonymous shmem is not in active use by server applications, so we don't support one currently. Supporting it should be done by tracking the changes from all the tasks that ''could'' write into the segment. For anon shared memory and sysvipc segment inside IPC namespace this works reliably.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || [[FUSE]] mount points || hard || - || When dumping mountpoints we explicitly check the filesystem mounted. The thing is -- not all filesystems can be just ignored on dump. E.g. FUSE mount involves a user-space daemon that is responsible for the files tree contents. If we just kill one on dump we might not be able to restore it. Need to special-care one.
|-
| crtools || 32-bit tasks || hard || Cyrill || For x86 we only dump and restore 64-bit tasks. Doing 32-bit should also be done, but keep in mind, that not only 64-bit tree OR 32-bit tree should be supported. There can be mixed 64-and-32-bit trees out there and CRIU should support those too.
|-
| crtools || Modify restored resources run-time in [[CRIT]] daemon || medium || - || Sometimes it might make sense to tune the objects fro images on restore. E.g. -- change the IP address of sockets from task above or fix file paths to be "chroot-ed". The best solution seems to be in launching CRIT in daemon mode, telling it what images and how to modify and teaching CRIU to "filter" the pb objects read from images through this daemon.
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard (v2) || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel?/crtools || TCP conntrack-ed connections || medium || - || When a container uses conntracks inside, we cannot just dump and restore alive TCP connection. Otherwise on restore the resurrected packets will be blocked by connection tracker as they would not be recognized as established connection. Need to check whether connection tracking is ON, dump the needed conntrack info and put the tracker back.
|-
| crtools || Bridges in container || medium || - || The bridge device state should be read, saved and restored.
|-
| crtools || VLANs in containers || medium || - || Vlan (802.1q) device state should be read, saved and restored.
|-
| crtools || [[PPP]] support || medium || - || PPP consists of several things, not just ppp devices. If container uses PPP we should take care of it, currently CRIU just aborts.
|-
| crtools/kernel || [[NFS mount points]] support || hard || - || NFS mount points from inside container cannot be easily restored. The thing is -- if we want to restore opened file we will go ahead and [[How hard is it to open a file|call]] the open system call. If the file in question resides on NFS, the latter might need to go to network to check whether the file actually exists and set up the handle. But if the networking is still not restored this operation would fail and we'll have to fail the whole restore. In order to untie this chicken-and-egg problem we may go in two directions.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly -- dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Validate .img files || easy || - || [[CRIT]] sub-task. For a given set of image files check, that they are in "restorable" shape, i.e. contain valid data and no pieces are missing.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || Ruslan Kuprieiev || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || These are SysVIPC objects created with semctl() and SEM_UNDO flag. Shame on us, we don't even detect these are created. Fortunately they are not in active use. Need to do it -- dump and restore. Requires modifications from both sides -- criu and kernel.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| kernel/criu || FS-notify queues || hard || - || We dump [[Fsnotify]] files, but when they contain events inside -- just ignore those. Need to fetch then and put back on restore. The difficulty here is that while dumping/restoring CRIU may touch files that are monitored and thus produce unwanted events into queue.
|-
| crtools || Remove hardcoded TASK_SIZE (at least for AArch32) || medium || cov || Dumping an AArch32 application using an AArch32 CRIU under an AArch64 kernel fails because TASK_SIZE is wrong. If TASK_SIZE were determined at runtime, the process would be able to proceed further. TASK_SIZE could be guessed using uname/cpuinfo and rules, probed with a series of accesses, or perhaps, following the example of PAGE_SIZE, the ELF auxiliary vector should include it.
|-
| crtools || Hardcoded PAGE_SIZE (at least for AArch64) || medium || cov || Dumping an AArch64 under an AArch64 kernel with 64K pages fails because PAGE_SIZE is wrong. Many uses of the PAGE_SIZE constant don't actually need an exact page size. Maybe split uses into PAGE_OR_LESS, EXEC_PAGESIZE (max possible for a platform), and page_size (actual value, probably pulled from auxv in memory or /proc or from smaps in /proc).
|-
| crtools || Make CRIU work on AArch32 with CONFIG_KUSER_HELPERS=n || medium || cov || CRIU currently fails on AArch32 kernels built with CONFIG_KUSER_HELPERS=n.
|-
| kernel || Fix VDSO remapping on non-x86 architectures || medium || Laurent Dufour, cov || However some architectures like PowerPC and ARM are keeping a reference to the VDSO base address to build the signal return stack frame by calling the VDSO sigreturn service. So once the VDSO has been moved, this reference is no more valid and the signal frame built later are not usable.
|-
| tests || Run many/all tests in "container" || medium || - || Currently we run zdtm tests one-by-one. It would be nice to run the all in one pseudo-container and C/R them as one big subtree.
|-
| tests || Trinity-like (fuzz) testing || hard || - || The existing suite is 99% functionality testing. Need more sophisticated testing -- take a process that has done a random set of actions, C/R one, check that all is OK. The latter is the most complicated thing.
|-
| tests || Split mountpoints.c test into pieces || easy || - || Currently this one is one big set of tests. Need more fine-grained set.
|-
| tests/infrastructure || Run tests on patches sent to the mailing lists || medium || Ruslan Kuprieiev || It's quite typical that a set sent to the mailing list fails some tests. Need a robot that would monitor the list, check the patches and send the result back.
|-
| tests || Fault injection || hard || - || Need some way to test error paths in CRIU. Right now we rely on the developers to write correct code :\ This is the most critical on dump.
|-
| crtools || Zombies with threads || medium || - || Support processes with alive threads and a dead leader
|-
| crtools || Unix sender address || medium || - || Restore sender addresses for unix socket messages
|-
| crtools || --leave-stopped for restore || easy || - || Restore task but leave it stopped
|}

[[Category:Development]]
[[Category:Plans]]

Todo

2015-06-04T21:59:28Z

Efiop: took crml patch test

Todo

2015-05-10T18:16:04Z

Efiop: Coredump task is ready.

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| crtools/crit || Library/API to access pagemap+page images || medium || - || The [[memory dumps]] are quite sophisticated. It would be nice to have a library or API to access the data in them using some simple API. IOW -- API-ize the page_read.c
|-
| crtools || Zombies with threads :) || easy || - || When milti-thread task's leader thread exits task turns into zombie state, but the other threads keep running. Need to support this (zdt test pthread02).
|-
| tests || automate process of measurement code coverage || easy || - || It is required to automate process of getting code coverage. We have code coverage results [http://criu.org/cov/ measured in 2012]. Would be nice to get up to date results on periodic basis and without manual actions.
|-
| crtools || Non-full mntns dump || medium || - || Systemd launches services in a new mount namespace with a single change -- /tmp is re-mounted into a private one(PrivateTmp option). Need to invent an API for dumping only a part of mntns.
|-
| kernel/crtools || Remap [[Vdso]] || medium || - || When at restore VDSO is found not in the place it was on dump we should <code>mremap()</code> one. Unfortunately not always we can do it, need to fix the kernel.
|-
| crtools || Make dump and restore work under [[selinux]] || medium || - || Selinux imposes more restrictions on the stuff we typically do.
|-
| crtools || Inherit resources, not restore || medium || - || Sigactions are restored for every task before it fork()-s. Then children check for the sa_action from their image matches to one it got from parent. Need to do the same for rlimits, maybe other resources too.
|-
| crtools || Implement [[restorer v2]] || hard (v2) || - ||
|-
| crtools || New images format || medium (v2) || - || See [[what's bad with V1 images]]
|-
| crtools || Make [[RPC]] "swrk" mode public || medium || xemul || There's an "swrk" action in CRIU [[Usage|CLI API]]. This turns CRIU into service worker accepting RPC commands. This mode is not documented. Need to standartify one and make public.
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| crtools || Deduplication for shmem [[memory dumps]] || easy || - || We have dedup action and --auto-dedup option for dump/restore which only works for pid pagemaps. Need the same for shmem.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| crtools || Show what was left in the system after dump || easy || - || When we use [[Invisible files|--link-remap]] option or [[TCP connection|--tcp-established]] one CRIU leaves some traces in the system, in particular -- temporary hard links in the former case and iptables rules in the latter. Need some way to show these to the user.
|-
| crtools || Decode flags from images into symbolic names || easy || - || When we print images contents with [[CRIT]] the flags fields are shown in decimal/hex numbers. It would be nice to print the symbolic names for known flags in some form.
|-
| crtools || Leases support || easy || - || The F_SETLEASE/F_GETLEASE API is not currently supported, but doesn't differ much from regular locks.
|-
| kernel/crtools || Make proper "check lock present" API in the kernel || medium || - || Currently we detect where a file lock belongs to by locking the file again with the alternative lock type and check how kernel reacts on that. This is not nice as it may tune the lock state on a file. Instead we need the "check lock on fd" call in the kernel.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || Kuprieiev Ruslan || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || AIO with pending events || medium || - || When we dump AIO ring we check it not to contain events inside and abort the dump otherwise. Need to dump events too and put them back on restore.
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid. It makes sense to request CRIU to dump a set of tasks from given cgroup.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Support OFD posix locks || easy || - || These are still rarely used, but exist. Might make sense to support them in advance, it looks like kernel API allows for that.
|-
| crtools || Optimize kcmp calls || medium || - || CRIU build [[kcmp trees]] to find out IDs of such objects as MM, FDT and others. Currently we kcmp all tasks to get the ID, but we can improve that by pre-generating ID based on objects that live on MM, FS, etc. If pre-ID of two tasks matches, then we call kcmp, if not -- objects ''are'' different.
|-
| crtools || Shmem changes tracking || medium || - || Memory changes tracker works on anonymous private mappings only. Anonymous shmem is not in active use by server applications, so we don't support one currently. Supporting it should be done by tracking the changes from all the tasks that ''could'' write into the segment. For anon shared memory and sysvipc segment inside IPC namespace this works reliably.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || [[FUSE]] mount points || hard || - || When dumping mountpoints we explicitly check the filesystem mounted. The thing is -- not all filesystems can be just ignored on dump. E.g. FUSE mount involves a user-space daemon that is responsible for the files tree contents. If we just kill one on dump we might not be able to restore it. Need to special-care one.
|-
| crtools || 32-bit tasks || hard || Cyrill || For x86 we only dump and restore 64-bit tasks. Doing 32-bit should also be done, but keep in mind, that not only 64-bit tree OR 32-bit tree should be supported. There can be mixed 64-and-32-bit trees out there and CRIU should support those too.
|-
| crtools || Modify restored resources run-time in [[CRIT]] daemon || medium || - || Sometimes it might make sense to tune the objects fro images on restore. E.g. -- change the IP address of sockets from task above or fix file paths to be "chroot-ed". The best solution seems to be in launching CRIT in daemon mode, telling it what images and how to modify and teaching CRIU to "filter" the pb objects read from images through this daemon.
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard (v2) || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel?/crtools || TCP conntrack-ed connections || medium || - || When a container uses conntracks inside, we cannot just dump and restore alive TCP connection. Otherwise on restore the resurrected packets will be blocked by connection tracker as they would not be recognized as established connection. Need to check whether connection tracking is ON, dump the needed conntrack info and put the tracker back.
|-
| crtools || Bridges in container || medium || - || The bridge device state should be read, saved and restored.
|-
| crtools || VLANs in containers || medium || - || Vlan (802.1q) device state should be read, saved and restored.
|-
| crtools || [[PPP]] support || medium || - || PPP consists of several things, not just ppp devices. If container uses PPP we should take care of it, currently CRIU just aborts.
|-
| crtools/kernel || [[NFS mount points]] support || hard || - || NFS mount points from inside container cannot be easily restored. The thing is -- if we want to restore opened file we will go ahead and [[How hard is it to open a file|call]] the open system call. If the file in question resides on NFS, the latter might need to go to network to check whether the file actually exists and set up the handle. But if the networking is still not restored this operation would fail and we'll have to fail the whole restore. In order to untie this chicken-and-egg problem we may go in two directions.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly -- dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Validate .img files || easy || - || [[CRIT]] sub-task. For a given set of image files check, that they are in "restorable" shape, i.e. contain valid data and no pieces are missing.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || Ruslan Kuprieiev || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || These are SysVIPC objects created with semctl() and SEM_UNDO flag. Shame on us, we don't even detect these are created. Fortunately they are not in active use. Need to do it -- dump and restore. Requires modifications from both sides -- criu and kernel.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| kernel/criu || FS-notify queues || hard || - || We dump [[Fsnotify]] files, but when they contain events inside -- just ignore those. Need to fetch then and put back on restore. The difficulty here is that while dumping/restoring CRIU may touch files that are monitored and thus produce unwanted events into queue.
|-
| crtools || Remove hardcoded TASK_SIZE (at least for AArch32) || medium || cov || Dumping an AArch32 application using an AArch32 CRIU under an AArch64 kernel fails because TASK_SIZE is wrong. If TASK_SIZE were determined at runtime, the process would be able to proceed further. TASK_SIZE could be guessed using uname/cpuinfo and rules, probed with a series of accesses, or perhaps, following the example of PAGE_SIZE, the ELF auxiliary vector should include it.
|-
| crtools || Hardcoded PAGE_SIZE (at least for AArch64) || medium || cov || Dumping an AArch64 under an AArch64 kernel with 64K pages fails because PAGE_SIZE is wrong. Many uses of the PAGE_SIZE constant don't actually need an exact page size. Maybe split uses into PAGE_OR_LESS, EXEC_PAGESIZE (max possible for a platform), and page_size (actual value, probably pulled from auxv in memory or /proc or from smaps in /proc).
|-
| crtools || Make CRIU work on AArch32 with CONFIG_KUSER_HELPERS=n || medium || cov || CRIU currently fails on AArch32 kernels built with CONFIG_KUSER_HELPERS=n.
|-
| kernel || Fix VDSO remapping on non-x86 architectures || medium || Laurent Dufour, cov || However some architectures like PowerPC and ARM are keeping a reference to the VDSO base address to build the signal return stack frame by calling the VDSO sigreturn service. So once the VDSO has been moved, this reference is no more valid and the signal frame built later are not usable.
|-
| tests || Run many/all tests in "container" || medium || - || Currently we run zdtm tests one-by-one. It would be nice to run the all in one pseudo-container and C/R them as one big subtree.
|-
| tests || Trinity-like (fuzz) testing || hard || - || The existing suite is 99% functionality testing. Need more sophisticated testing -- take a process that has done a random set of actions, C/R one, check that all is OK. The latter is the most complicated thing.
|-
| tests || Split mountpoints.c test into pieces || easy || - || Currently this one is one big set of tests. Need more fine-grained set.
|-
| tests/infrastructure || Run tests on patches sent to the mailing lists || medium || - || It's quite typical that a set sent to the mailing list fails some tests. Need a robot that would monitor the list, check the patches and send the result back.
|-
| tests || Fault injection || hard || - || Need some way to test error paths in CRIU. Right now we rely on the developers to write correct code :\ This is the most critical on dump.
|-
| crtools || Zombies with threads || medium || - || Support processes with alive threads and a dead leader
|}

[[Category:Development]]
[[Category:Plans]]

CRIT

2015-05-08T16:55:00Z

Efiop: criu-coredump

This page describes the CRiu Image Tool. CRIT is a feature-rich replacement for existing "criu show".
It is written completely in Python, so it is quite easy to read the code and extend its features.

== Usage ==
<pre>
usage: crit [-h] [-i IN] [-o OUT] [--pretty] {decode,encode}

CRiu Image Tool

positional arguments:
{decode,encode} decode/encode - convert criu image from/to binary type
to/from json

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default)
--pretty multi-line with indentation and some fields printed in more human-readable format
</pre>

=== Pretty output ===

By default CRIT prints JSON text in one line. This can be read by any further JSON-aware tool for parsing. For human eyes it's more convenient to read JSON multi-line with indentation. However, CRIT does a little bit more and prints some fields in even more pretty manner.

; Addresses and registers
: VM addresses and core.img register values are all printed in hex. Since JSON doesn't support this form of numbers, such fields are encoded as strings.

; Bit-fields
: Such things as flags and masks (e.g. sig-block mask) are also better understood when written in hex, so CRIT does this.

; IP addresses
: By default those a printed in decimal, but the "1.2.3.4" for v4 or "::1" for v6 can be seen in the --pretty mode.

==== TODO ====

; Symbolic names for flags
: Some known bits (e.g. MAP_PRIVATE, MAP_ANONYMOUS, etc. for vma->flags) can be shown with names.

== Functionality ==

=== Convert images to JSON and back ===

{{Status|ready}}

This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.

The output file is structured in the following way.

Without "--pretty":
<pre>
{ "magic" : "FOO", "entries" : [{"foo": "bar", "bar": "foo", "extra": "abc"}, {"foo": "bar", "bar" : "foo", "extra" : "abc"} ]}
</pre>

With "--pretty":
<pre>
{
"magic" : "FOO",
"entries" : [
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
},
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
}
]
}
</pre>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
<b>Example("crit decode -i core-5679.img --pretty")</b>
<div class="mw-collapsible-content">
<pre>
{
"magic": "CORE",
"entries": [
{
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": 24,
"sched_policy": 0,
"sched_nice": 0,
"futex_rla": 0,
"signals_p": {},
"sas": {
"ss_size": 0,
"ss_sp": 0,
"ss_flags": 2
}
},
"thread_info": {
"fpregs": {
"st_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"fop": 0,
"rdp": 0,
"twd": 0,
"mxcsr": 8064,
"swd": 0,
"rip": 0,
"xsave": {
"ymmh_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"xstate_bv": 2
},
"xmm_space": [
0,
4278190080,
4294967295,
4294967295,
1701145715,
3219568,
0,
0,
0,
0,
33,
0,
792358505,
1953460082,
1852400175,
0,
942882145,
876295483,
774519349,
1031303283,
893073459,
976565307,
1937255978,
859661936,
993344312,
3814708,
65,
0,
37049520,
0,
37049632,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
4294901760,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"cwd": 0,
"mxcsr_mask": 65535
},
"clear_tid_addr": 0,
"gpregs": {
"gs": "0x0",
"ip": "0x7f172cf1ea04",
"cx": "0xffffffffffffffff",
"cs": "0x33",
"ax": "0x38",
"orig_ax": "0x38",
"di": "0x1200011",
"es": "0x0",
"gs_base": "0x0",
"r14": "0x0",
"r15": "0x2355e00",
"r12": "0x7ffffdbf74f0",
"r13": "0x0",
"r10": "0x7f172d83d9d0",
"r11": "0x246",
"fs_base": "0x7f172d83d700",
"bp": "0x7ffffdbf7530",
"dx": "0x0",
"bx": "0x0",
"ds": "0x0",
"ss": "0x2b",
"sp": "0x7ffffdbf74f0",
"r8": "0x0",
"r9": "0x0",
"fs": "0x0",
"si": "0x0",
"flags": "0x246"
}
},
"tc": {
"timers": {
"real": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"virt": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"prof": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
}
},
"cg_set": 1,
"signals_s": {},
"blk_sigset": "0x10002",
"exit_code": 0,
"rlimits": {
"rlimits": [
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 8388608
},
{
"max": 18446744073709551615,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62844,
"cur": 62844
},
{
"max": 4096,
"cur": 1024
},
{
"max": 65536,
"cur": 65536
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62844,
"cur": 62844
},
{
"max": 819200,
"cur": 819200
},
{
"max": 0,
"cur": 0
},
{
"max": 0,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
}
]
},
"comm": "loop.sh",
"flags": 1077960704,
"task_state": 1,
"personality": 0
}
}
]
}
</pre>
</div>
</div>

=== Generate core files out of task images ===

{{Status|ready}}

Moved into separate project called [[criu-coredump]].

=== On-the-fly conversion ===

{{Status|not ready}}

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

=== Convert between different image versions ===

{{Status|not ready}}

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

=== Show images statistics ===

{{Status|not ready}}

E.g. -- total number of processes, files, memory, sockets, etc.

=== Check/validate images ===

{{Status|not ready}}

Check that
* all images are present
* the inter-images IDs are in consistent state

[[Category:Images]]
[[Category:API]]

Criu-coredump

2015-05-08T16:54:56Z

Efiop: Created page with "Generate valid core dump from CRIU images. === Description === CRIU images contain lots of information about process state. In fact, it is sufficient enough to generate a vali..."

Generate valid core dump from CRIU images.
=== Description ===
CRIU images contain lots of information about process state. In fact, it is sufficient enough to generate a
valid core dump which can be later be used by gdb, for example. criu-coredump does exactly that, it takes a
directory with CRIU images and produces Elf core dumps.

Basically, you can use CRIU to checkpoint task when you need it (i.e. when it hangs, or when an error occures)
or make your application call CRIU via one of its interfaces(CLI, RPC or C library) and then produce core dump
and inspect it via gdb.

To generate core dump of your application you don't even need to be a root, it is enough for you to be the owner
of that task. For that all you have two choices: 1) Use CRIU service running and use [[RPC]] or [[C_API]]. It is
useful if you want your app to automatically call CRIU. 2) Use CRIU binary with suid bit set.

criu-coredump project provides both cmdline tool and python module(criu_coredump), so you can use it in your
python apps.

=== Limitations ===

For now only x86_64 is supported. The matter of supporting other architectures supported by CRIU is to use proper constants in elf header and provide proper arch-dependent notes.

Some notes in coredump are not fullfilled(i.e. SIGINFO).

VVAR and VSYSCALL vmas are just filled with zeroes, as CRIU doesn't currently provide them. VVAR is a tricky one and gdb also has problems dumping it[https://lkml.org/lkml/2015/3/12/439]. VSYSCALL in theory could just be read from current process, so it should be quite easy to fix.

=== Installation ===

<pre>$ python setup.py install</pre>

=== Usage ===

<pre>
criu-coredump [-h] [-i IN] [-p PID] [-o OUT]

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN directory where to get images from.
-p PID, --pid PID generate core dump of a process identified by PID.
If not specified -- generating for each pid.
-o OUT, --out OUT directory to write core dump to.
</pre>

=== Project resources ===
Project is located at [https://github.com/efiop/criu-coredump GitHub].

Todo

2015-03-27T09:22:13Z

Efiop: PrivateTmp

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| crtools || Non-full mntns dump || medium || - || Systemd launches services in a new mount namespace with a single change -- /tmp is re-mounted into a private one(PrivateTmp option). Need to invent an API for dumping only a part of mntns.
|-
| kernel/crtools || Remap [[Vdso]] || medium || - || When at restore VDSO is found not in the place it was on dump we should <code>mremap()</code> one. Unfortunately not always we can do it, need to fix the kernel.
|-
| crtools || Make dump and restore work under [[selinux]] || medium || - || Selinux imposes more restrictions on the stuff we typically do.
|-
| crtools || Inherit resources, not restore || medium || - || Sigactions are restored for every task before it fork()-s. Then children check for the sa_action from their image matches to one it got from parent. Need to do the same for rlimits, maybe other resources too.
|-
| crtools || Implement [[restorer v2]] || hard (v2) || - ||
|-
| crtools || New images format || medium (v2) || - || See [[what's bad with V1 images]]
|-
| crtools || Make [[RPC]] "swrk" mode public || medium || xemul || There's an "swrk" action in CRIU [[Usage|CLI API]]. This turns CRIU into service worker accepting RPC commands. This mode is not documented. Need to standartify one and make public.
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| crtools || Deduplication for shmem [[memory dumps]] || easy || - || We have dedup action and --auto-dedup option for dump/restore which only works for pid pagemaps. Need the same for shmem.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| crtools || Show what was left in the system after dump || easy || - || When we use [[Invisible files|--link-remap]] option or [[TCP connection|--tcp-established]] one CRIU leaves some traces in the system, in particular -- temporary hard links in the former case and iptables rules in the latter. Need some way to show these to the user.
|-
| crtools || Decode flags from images into symbolic names || easy || - || When we print images contents with [[CRIT]] the flags fields are shown in decimal/hex numbers. It would be nice to print the symbolic names for known flags in some form.
|-
| crtools || Leases support || easy || - || The F_SETLEASE/F_GETLEASE API is not currently supported, but doesn't differ much from regular locks.
|-
| kernel/crtools || Make proper "check lock present" API in the kernel || medium || - || Currently we detect where a file lock belongs to by locking the file again with the alternative lock type and check how kernel reacts on that. This is not nice as it may tune the lock state on a file. Instead we need the "check lock on fd" call in the kernel.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || Kuprieiev Ruslan || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || AIO with pending events || medium || - || When we dump AIO ring we check it not to contain events inside and abort the dump otherwise. Need to dump events too and put them back on restore.
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid. It makes sense to request CRIU to dump a set of tasks from given cgroup.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Support OFD posix locks || easy || - || These are still rarely used, but exist. Might make sense to support them in advance, it looks like kernel API allows for that.
|-
| crtools || Optimize kcmp calls || medium || - || CRIU build [[kcmp trees]] to find out IDs of such objects as MM, FDT and others. Currently we kcmp all tasks to get the ID, but we can improve that by pre-generating ID based on objects that live on MM, FS, etc. If pre-ID of two tasks matches, then we call kcmp, if not -- objects ''are'' different.
|-
| crtools || Shmem changes tracking || medium || - || Memory changes tracker works on anonymous private mappings only. Anonymous shmem is not in active use by server applications, so we don't support one currently. Supporting it should be done by tracking the changes from all the tasks that ''could'' write into the segment. For anon shared memory and sysvipc segment inside IPC namespace this works reliably.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || [[FUSE]] mount points || hard || - || When dumping mountpoints we explicitly check the filesystem mounted. The thing is -- not all filesystems can be just ignored on dump. E.g. FUSE mount involves a user-space daemon that is responsible for the files tree contents. If we just kill one on dump we might not be able to restore it. Need to special-care one.
|-
| crtools || 32-bit tasks || hard || Cyrill || For x86 we only dump and restore 64-bit tasks. Doing 32-bit should also be done, but keep in mind, that not only 64-bit tree OR 32-bit tree should be supported. There can be mixed 64-and-32-bit trees out there and CRIU should support those too.
|-
| crtools || Generate task's core file out of images with [[CRIT]] || medium || Ruslan Kuprieiev || Nothing special -- just take core.img, mm.img and pagemap.img and produce the canonical core image out of those.
|-
| crtools || Modify restored resources run-time in [[CRIT]] daemon || medium || - || Sometimes it might make sense to tune the objects fro images on restore. E.g. -- change the IP address of sockets from task above or fix file paths to be "chroot-ed". The best solution seems to be in launching CRIT in daemon mode, telling it what images and how to modify and teaching CRIU to "filter" the pb objects read from images through this daemon.
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard (v2) || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel?/crtools || TCP conntrack-ed connections || medium || - || When a container uses conntracks inside, we cannot just dump and restore alive TCP connection. Otherwise on restore the resurrected packets will be blocked by connection tracker as they would not be recognized as established connection. Need to check whether connection tracking is ON, dump the needed conntrack info and put the tracker back.
|-
| crtools || Bridges in container || medium || - || The bridge device state should be read, saved and restored.
|-
| crtools || VLANs in containers || medium || - || Vlan (802.1q) device state should be read, saved and restored.
|-
| crtools || [[PPP]] support || medium || - || PPP consists of several things, not just ppp devices. If container uses PPP we should take care of it, currently CRIU just aborts.
|-
| crtools/kernel || [[NFS mount points]] support || hard || - || NFS mount points from inside container cannot be easily restored. The thing is -- if we want to restore opened file we will go ahead and [[How hard is it to open a file|call]] the open system call. If the file in question resides on NFS, the latter might need to go to network to check whether the file actually exists and set up the handle. But if the networking is still not restored this operation would fail and we'll have to fail the whole restore. In order to untie this chicken-and-egg problem we may go in two directions.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly -- dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Validate .img files || easy || - || [[CRIT]] sub-task. For a given set of image files check, that they are in "restorable" shape, i.e. contain valid data and no pieces are missing.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || Ruslan Kuprieiev || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || These are SysVIPC objects created with semctl() and SEM_UNDO flag. Shame on us, we don't even detect these are created. Fortunately they are not in active use. Need to do it -- dump and restore. Requires modifications from both sides -- criu and kernel.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| kernel/criu || FS-notify queues || hard || - || We dump [[Fsnotify]] files, but when they contain events inside -- just ignore those. Need to fetch then and put back on restore. The difficulty here is that while dumping/restoring CRIU may touch files that are monitored and thus produce unwanted events into queue.
|}

[[Category:Development]]
[[Category:Plans]]

Todo

2015-03-04T08:33:57Z

Efiop: took "Restore task into fresh new pid namespace"

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| crtools || Make [[RPC]] "swrk" mode public || medium || xemul || There's an "swrk" action in CRIU [[Usage|CLI API]]. This turns CRIU into service worker accepting RPC commands. This mode is not documented. Need to standartify one and make public.
|-
| kernel/crtools || Tune the start-time of tasks || medium || - || When we restore tasks their start-time goes forward (since we create the new task effectively). Need to address this somehow, most likely with the [[time namespace]].
|-
| crtools || Support chroot-ed mount namespace || medium || - || If the root task lives in another mount namespace ''and'' has its root moved (with chroot()) CRIU dump fails with errors about inability to resolve files' paths. This is because CRIU treats the mount namespace's root as the init task's root which should be "/".
|-
| crtools || Non-stop memory (first?) pre-dump || medium || - || When reading only the memory we can avoid freezing tasks and draining memory with parasite. There's a system call named "read_process_vm" which can help us accessing the other task's memory. The disadvantage of this approach is the need for additional memory. We may control this behaviour by reading memory in chunks and not allocating to much of additional buffers.
|-
| kernel/crtools || Speed up fetching info about tasks || medium || Andrey Vagin || Using proc to get info about tasks is nice but too slow. We have measured that having socket-based engine that would fetch info about tasks from the kernel speeds things up significantly. So Andrey is working on the [[Task-diag]] patchset that would implement that.
|-
| kernel || Make pipes swappable || hard || - || When [[Memory dumping and restoring|pre-dumping]] memory we pull all the task's memory into pipe with vmsplice and then send it via network splicing the pages into socket. During this period all the memory is effectively pinned as pages in pipe are not swappable.
|-
| crtools || Deduplication for shmem [[memory dumps]] || easy || - || We have dedup action and --auto-dedup option for dump/restore which only works for pid pagemaps. Need the same for shmem.
|-
| kernel/crtools || Adjust per-task/-container timers offsets || medium || - || Absolute timers differ on different nodes. When live migrating a task/container this difference may (and will) screw the timers up.
|-
| crtools || [[time namespace|Shift timers' timeouts]] according to the actual C-to-R delay || medium || - || If we pause tasks between C and R we, probably, need to adjust timers respectively. "Medium" complexity is because it's unclear ''what'' to do, not ''how''.
|-
| crtools || Show what was left in the system after dump || easy || - || When we use [[Invisible files|--link-remap]] option or [[TCP connection|--tcp-established]] one CRIU leaves some traces in the system, in particular -- temporary hard links in the former case and iptables rules in the latter. Need some way to show these to the user.
|-
| crtools || Decode flags from images into symbolic names || easy || - || When we print images contents with [[CRIT]] the flags fields are shown in decimal/hex numbers. It would be nice to print the symbolic names for known flags in some form.
|-
| crtools || Leases support || easy || - || The F_SETLEASE/F_GETLEASE API is not currently supported, but doesn't differ much from regular locks.
|-
| kernel/crtools || Make proper "check lock present" API in the kernel || medium || - || Currently we detect where a file lock belongs to by locking the file again with the alternative lock type and check how kernel reacts on that. This is not nice as it may tune the lock state on a file. Instead we need the "check lock on fd" call in the kernel.
|-
| kernel/crtools || Put call to mmap into VDSO || easy || Cyrill || To put the [[parasite code]] into target process we modify its code to call the <code>mmap()</code> system call (and the unmodify it back) and put the parasite into new area. Oleg Nesterov suggests not to patch victim, but to always have one on VDSO.
|-
| crtools || [[Integration]] with other projects || hard || - || CRIU is not working great by itself. There's alway some specific about what user wants to dump. Integrating CRIU with other projects will make CRIU work at its best.
|-
| crtools || Restore tasks into fresh new pid namespace || easy || Kuprieiev Ruslan || When we dumped processes, it can be hard to restore it back, if they didn't live in a pid namespace, due to PIDs conflict. It would be nice to have the ability to ask CRIU to create the pid namespace for those guys and restore them there. A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - || When we checkpointed process tree with -R option (let them run after checkpoint) we might want to return the tasks into checkpointed state on the same machine. Currently this can only be done by killing the processes and restoring them from scratch. If we could ask CRIU to restore the images ''into'' the ready processes that could speed things up, especially if carefully caring about [[memory changes tracking]].
|-
| crtools || AIO with pending events || medium || - || When we dump AIO ring we check it not to contain events inside and abort the dump otherwise. Need to dump events too and put them back on restore.
|-
| crtools || Restore arbitrary mountpoints tree || hard || - || Linux kernel can construct tricky knows with [[mount points]]. We don't support arbitrary configuration of such things, only those that are in active use by software. Need to fix them up.
|-
| crtools || Lazy restore using [[userfaultfd]] || medium || xemul || It might make sense to restore tasks w/o putting all the memory into respective places. Instead, the VMAs in question can be marked as "lazy" and pages will get filled into them in the background and, upon demand, in the out-of-order manner. The functionality is related to lazy migration and seamless kernel update tasks.
|-
| crtools || [[Lazy migration]] using [[userfaultfd]] || medium || xemul || Lazy migration is when we move all the tasks on another node, but leave theirs memory on the source one. Not to allow tasks read garbage from empty address space we protect all of it as inaccessible. When tasks start reading/writing the mem they got page-fault-ed. With the userfaultfd technology it can be possible to intercept the #PF, pull the page from source node and map it into expected address.
|-
| crtools || Dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid. It makes sense to request CRIU to dump a set of tasks from given cgroup.
|-
| crtools || Speed up [[logging]] || medium || Cyrill || Synchronous formatting and writes into log files slow things down. On the other hand turning logs off make it impossible to troubleshoot.
|-
| crtools || Sanitize [[logging]] messages || hard || - || Currently log messages are printed w/o any logic, it's hard to analize what has happened when CRIU fails. Need to improve that by, e.g. categorizing images and [[When C/R fails|explaining them]] in more details.
|-
| crtools || Support OFD posix locks || easy || - || These are still rarely used, but exist. Might make sense to support them in advance, it looks like kernel API allows for that.
|-
| crtools || Optimize kcmp calls || medium || - || CRIU build [[kcmp trees]] to find out IDs of such objects as MM, FDT and others. Currently we kcmp all tasks to get the ID, but we can improve that by pre-generating ID based on objects that live on MM, FS, etc. If pre-ID of two tasks matches, then we call kcmp, if not -- objects ''are'' different.
|-
| crtools || Shmem changes tracking || medium || - || Memory changes tracker works on anonymous private mappings only. Anonymous shmem is not in active use by server applications, so we don't support one currently. Supporting it should be done by tracking the changes from all the tasks that ''could'' write into the segment. For anon shared memory and sysvipc segment inside IPC namespace this works reliably.
|-
| crtools || Page transfer filters || medium || - || The page-xfer engine just splices the pages from stealing pipes into socket. Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || [[FUSE]] mount points || hard || - || When dumping mountpoints we explicitly check the filesystem mounted. The thing is -- not all filesystems can be just ignored on dump. E.g. FUSE mount involves a user-space daemon that is responsible for the files tree contents. If we just kill one on dump we might not be able to restore it. Need to special-care one.
|-
| crtools || 32-bit tasks || hard || Cyrill || For x86 we only dump and restore 64-bit tasks. Doing 32-bit should also be done, but keep in mind, that not only 64-bit tree OR 32-bit tree should be supported. There can be mixed 64-and-32-bit trees out there and CRIU should support those too.
|-
| crtools || Generate task's core file out of images with [[CRIT]] || medium || Ruslan Kuprieiev || Nothing special -- just take core.img, mm.img and pagemap.img and produce the canonical core image out of those.
|-
| crtools || Modify restored resources run-time in [[CRIT]] daemon || medium || - || Sometimes it might make sense to tune the objects fro images on restore. E.g. -- change the IP address of sockets from task above or fix file paths to be "chroot-ed". The best solution seems to be in launching CRIT in daemon mode, telling it what images and how to modify and teaching CRIU to "filter" the pb objects read from images through this daemon.
|-
| crtools || TCP socket migration with changed IP || medium || - || It might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks in that case and use CRIT images modifucation facilities.
|-
| crtools || [[Applying images]] || hard || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). Like it was described in the "rollback" feature above. Another exampl -- repopulate fdtable according to data from image. Yet another use-case -- when doing partial migration (see below) we'll need to modify one part to switch from pipes to sockets. What else? With constant replication of tree state we can do incremental dumps on source node and apply those increments on pre-created replicas on the destination node.
|-
| crtools || Partial migration || hard || - || If tasks subtree has connections to the rest of the tree (e.g. with pipes of unix sockets) we try to detect this and refuse the dump. It should be possible to take part of the tree, migrating it somewhere and recreating the mentioned links with some other appropriate IPC channel. E.g. pipes with sockets, shared memory with distributed shared memory and so on.
|-
| crtools || Shared objects (mm/fs) support || medium || - || Things created with CLONE_FOO flags are not supported now (exception -- full threads). Now we have the kcmp syscall and can do it. The shared fdtable (CLONE_FILES) is supported, the next candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Files can be overmounted. In this case CRIU will refuse the dump saying that file is [[invisible files|not alive]] but inaccessible by its name. Need a way to resolve paths to such. There are two ways: 1. Move mounts, that overlap the desired path temporarily, then open the file, then move the mountpoint back. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd.
|-
| kernel/crtools || [[TCP repair TODO|TCP repair fixes]] || hard || - || We can dump and restore live [[TCP connection]]. There are some issues with it, that should be fixed.
|-
| kernel?/crtools || TCP conntrack-ed connections || medium || - || When a container uses conntracks inside, we cannot just dump and restore alive TCP connection. Otherwise on restore the resurrected packets will be blocked by connection tracker as they would not be recognized as established connection. Need to check whether connection tracking is ON, dump the needed conntrack info and put the tracker back.
|-
| crtools || Bridges in container || medium || - || The bridge device state should be read, saved and restored.
|-
| crtools || VLANs in containers || medium || - || Vlan (802.1q) device state should be read, saved and restored.
|-
| crtools || [[PPP]] support || medium || - || PPP consists of several things, not just ppp devices. If container uses PPP we should take care of it, currently CRIU just aborts.
|-
| crtools/kernel || [[NFS mount points]] support || hard || - || NFS mount points from inside container cannot be easily restored. The thing is -- if we want to restore opened file we will go ahead and [[How hard is it to open a file|call]] the open system call. If the file in question resides on NFS, the latter might need to go to network to check whether the file actually exists and set up the handle. But if the networking is still not restored this operation would fail and we'll have to fail the whole restore. In order to untie this chicken-and-egg problem we may go in two directions.
|-
| kernel || [[Seamless kernel upgrade]] || hard || xemul || Briefly -- dump tasks (into memory), change the kernel w/ kexec, then restore tasks back. From the tasks and remote client perspective tasks has just stopped and then resumed on the newer kernel. Can be a good complement to the classic live-patching technology.
|-
| crtools || Validate .img files || easy || - || [[CRIT]] sub-task. For a given set of image files check, that they are in "restorable" shape, i.e. contain valid data and no pieces are missing.
|-
| crtools || Restore arbitrary process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]. The task of resolving the given images into operations we might need to perform seem to be NP (not proven though).
|-
| crtools || C/R [[X applications]] || hard || Ruslan Kuprieiev || Dump/restore of graphical applications (see about [[integration]]). In case of X app part of its state is stored into the X-server. Need the way to fetch this state during dump and put this state back into the server on restore. Requires fixing the X-server software too.
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || These are SysVIPC objects created with semctl() and SEM_UNDO flag. Shame on us, we don't even detect these are created. Fortunately they are not in active use. Need to do it -- dump and restore. Requires modifications from both sides -- criu and kernel.
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]). Need to extend this set as currently it's hard to understand what has happened w/o analysing CRIU log files.
|-
| kernel/criu || FS-notify queues || hard || - || We dump [[Fsnotify]] files, but when they contain events inside -- just ignore those. Need to fetch then and put back on restore. The difficulty here is that while dumping/restoring CRIU may touch files that are monitored and thus produce unwanted events into queue.
|}

[[Category:Development]]
[[Category:Plans]]

Integration

2015-02-15T15:54:19Z

Efiop: /* Wayland/Weston */ Not X-server, but weston.

CRIU is not so easy to be used as a standalone tool -- it works best integrated into other software. This page lists such software and provides details about the current status.

== OpenVZ ==
{{Status|in progress}}

Currently, vzctl supports CRIU for checkpoint/restore of upstream containers (i.e. when non-OpenVZ kernel is used). Commands <code>vzctl suspend</code> and <code>vzctl restore</code> fully work. Live migration doesn't work yet as it requires support for <code>vzctl suspend {--suspend, --dump, --kill, --restore}</code> which is not yet implemented as it requires a separate daemon to hold the state of a partially checkpointed container (or an ability from criu tool to do that).
* [https://openvz.org Project homepage]
* [http://git.openvz.org/?p=vzctl&a=search&h=HEAD&st=commit&s=CRIU%7Ccriu%7Ccrtools&sr=1 Relevant vzctl git commits]

== LXC/LXD ==
{{Status|ready}}
* [http://linuxcontainers.org Project homepage]
* The tools version 1.1.0 fully supports CRIU to C/R LXC containers
* [https://linuxcontainers.org/lxc/manpages/man1/lxc-checkpoint.1.html lxc-checkpoint man page]

== Docker ==
{{Status|in progress}}
* [https://docker.io Project homepage]
* Integration [https://github.com/docker/libcontainer/pull/204 efforts] done by Saied Kazemi from Google

== CoreOS Rocket ==
{{Status|not started}}

== OpenMPI ==
{{Status|stalled}}
* Adrian Reber [https://lisas.de/~adrian/open-mpi.git/ did] first version of patches

== Wayland/Weston ==
{{Status|under investigation}}
* Ruslan Kuprieiev plans to [http://lists.openvz.org/pipermail/criu/2015-January/018875.html patch] Weston to let CRIU C/R graphical apps

[[Category:Plans]]

Technologies

2015-02-09T15:12:58Z

Efiop: Gather useful tools that were developed for criu and could be used in other projects.

This page contains a list of tools that were developed for criu and which others might want to use in their projects.
These tools are being still maintained, unlike their alternatives. Some of them are not formatted as a separate project(still could be just used,i.e. python modules), but feel free to ask us to distinguish them.

== [https://github.com/xemul/compel Compel] ==
An utility to execute code in foreign process address space. It is based on the same technology we use in CRIU to obtain some process-private data on dump.

== [https://github.com/xemul/criu/blob/master/pycriu/images/pb2dict.py pb2dict.py] ==
A Python module to convert google protocol buffers to json and back.

Unlike other projects to convert pb to python dict or json, our pb2dict does treat optional fields with empty repeated inside properly. It includes special support for custom field options to mark those needed to be treated in a special way.
For example, you could include our [https://github.com/xemul/criu/blob/master/protobuf/opts.proto opts.proto] in your proto-file and use criu.* options
to mark your fields, i.e.:

<code>required uint64 blk_sigset = 5[(criu).hex = true];</code>

See more examples in our protobuf/*.proto files.

It is also worth noting, that we have a unique number for all kinds of custom protobuf options(see [[Images#Notes_about_protobuf]]), so you don't have to worry that it might collide with others.

CRIT

2015-02-07T12:20:10Z

Efiop: /* Convert images to JSON and back */ refresh example and use --pretty

This page describes the CRiu Image Tool. CRIT is a feature-rich replacement for existing "criu show".
It is written completely in Python, so it is quite easy to read the code and extend its features.

== Usage ==
<pre>
usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] {decode,encode}

CRiu Image Tool

positional arguments:
{decode,encode} decode/encode - convert criu image from/to binary type
to/from json

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default)
--pretty multi-line with indentation and some fields printed in more human-readable format
</pre>

=== Pretty output ===

By default CRIT prints JSON text in one line. This can be read by any further JSON-aware tool for parsing. For human eyes it's more convenient to read JSON multi-line with indentation. However, CRIT does a little bit more and prints some fields in even more pretty manner.

; Addresses and registers
: VM addresses and core.img register values are all printed in hex. Since JSON doesn't support this form of numbers, such fields are encoded as strings.

; Bit-fields
: Such things as flags and masks (e.g. sig-block mask) are also better understood when written in hex, so CRIT does this.

; IP addresses
: By default those a printed in decimal, but the "1.2.3.4" for v4 or "::1" for v6 can be seen in the --pretty mode.

== Functionality ==

=== Convert images to JSON and back ===

{{Status|ready}}

This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.

The output file is structured in the following way.

Without "--pretty":
<pre>
{ "magic" : "FOO", "entries" : [{"foo": "bar", "bar": "foo", "extra": "abc"}, {"foo": "bar", "bar" : "foo", "extra" : "abc"} ]}
</pre>

With "--pretty":
<pre>
{
"magic" : "FOO",
"entries" : [
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
},
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
}
]
}
</pre>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
<b>Example("crit decode -i core-5679.img --pretty")</b>
<div class="mw-collapsible-content">
<pre>
{
"magic": "CORE",
"entries": [
{
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": 24,
"sched_policy": 0,
"sched_nice": 0,
"futex_rla": 0,
"signals_p": {},
"sas": {
"ss_size": 0,
"ss_sp": 0,
"ss_flags": 2
}
},
"thread_info": {
"fpregs": {
"st_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"fop": 0,
"rdp": 0,
"twd": 0,
"mxcsr": 8064,
"swd": 0,
"rip": 0,
"xsave": {
"ymmh_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"xstate_bv": 2
},
"xmm_space": [
0,
4278190080,
4294967295,
4294967295,
1701145715,
3219568,
0,
0,
0,
0,
33,
0,
792358505,
1953460082,
1852400175,
0,
942882145,
876295483,
774519349,
1031303283,
893073459,
976565307,
1937255978,
859661936,
993344312,
3814708,
65,
0,
37049520,
0,
37049632,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
4294901760,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"cwd": 0,
"mxcsr_mask": 65535
},
"clear_tid_addr": 0,
"gpregs": {
"gs": "0x0",
"ip": "0x7f172cf1ea04",
"cx": "0xffffffffffffffff",
"cs": "0x33",
"ax": "0x38",
"orig_ax": "0x38",
"di": "0x1200011",
"es": "0x0",
"gs_base": "0x0",
"r14": "0x0",
"r15": "0x2355e00",
"r12": "0x7ffffdbf74f0",
"r13": "0x0",
"r10": "0x7f172d83d9d0",
"r11": "0x246",
"fs_base": "0x7f172d83d700",
"bp": "0x7ffffdbf7530",
"dx": "0x0",
"bx": "0x0",
"ds": "0x0",
"ss": "0x2b",
"sp": "0x7ffffdbf74f0",
"r8": "0x0",
"r9": "0x0",
"fs": "0x0",
"si": "0x0",
"flags": "0x246"
}
},
"tc": {
"timers": {
"real": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"virt": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"prof": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
}
},
"cg_set": 1,
"signals_s": {},
"blk_sigset": "0x10002",
"exit_code": 0,
"rlimits": {
"rlimits": [
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 8388608
},
{
"max": 18446744073709551615,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62844,
"cur": 62844
},
{
"max": 4096,
"cur": 1024
},
{
"max": 65536,
"cur": 65536
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62844,
"cur": 62844
},
{
"max": 819200,
"cur": 819200
},
{
"max": 0,
"cur": 0
},
{
"max": 0,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
}
]
},
"comm": "loop.sh",
"flags": 1077960704,
"task_state": 1,
"personality": 0
}
}
]
}
</pre>
</div>
</div>

=== Generate core files out of task images ===

{{Status|in progress}}

All the needed information is in core-$pid.img and pagemap-$pid.img

=== On-the-fly conversion ===

{{Status|not ready}}

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

=== Convert between different image versions ===

{{Status|not ready}}

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

=== Show images statistics ===

{{Status|not ready}}

E.g. -- total number of processes, files, memory, sockets, etc.

=== Check/validate images ===

{{Status|not ready}}

Check that
* all images are present
* the inter-images IDs are in consistent state

[[Category:Images]]
[[Category:API]]

CRIT

2015-02-07T12:15:15Z

Efiop: /* Generate core files out of task images */

This page describes the CRiu Image Tool. CRIT is a feature-rich replacement for existing "criu show".
It is written completely in Python, so it is quite easy to read the code and extend its features.

== Usage ==
<pre>
usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] {decode,encode}

CRiu Image Tool

positional arguments:
{decode,encode} decode/encode - convert criu image from/to binary type
to/from json

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default)
--pretty multi-line with indentation and some fields printed in more human-readable format
</pre>

=== Pretty output ===

By default CRIT prints JSON text in one line. This can be read by any further JSON-aware tool for parsing. For human eyes it's more convenient to read JSON multi-line with indentation. However, CRIT does a little bit more and prints some fields in even more pretty manner.

; Addresses and registers
: VM addresses and core.img register values are all printed in hex. Since JSON doesn't support this form of numbers, such fields are encoded as strings.

; Bit-fields
: Such things as flags and masks (e.g. sig-block mask) are also better understood when written in hex, so CRIT does this.

; IP addresses
: By default those a printed in decimal, but the "1.2.3.4" for v4 or "::1" for v6 can be seen in the --pretty mode.

== Functionality ==

=== Convert images to JSON and back ===

{{Status|ready}}

This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.

The output file is structured in the following way.

With "--format raw":
<pre>
{ "magic" : "FOO", "entries" : [{"foo": "bar", "bar": "foo", "extra": "abc"}, {"foo": "bar", "bar" : "foo", "extra" : "abc"} ]}
</pre>

With "--format nice"(default for stdout):
<pre>
{
"magic" : "FOO",
"entries" : [
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
},
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
}
]
}
</pre>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
<b>Example("crit decode -i core-20700.img")</b>
<div class="mw-collapsible-content">
<pre>
{
"magic": "CORE",
"entries": [
{
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": 24,
"sched_policy": 0,
"sched_nice": 0,
"futex_rla": 0,
"signals_p": {},
"sas": {
"ss_size": 0,
"ss_sp": 0,
"ss_flags": 2
}
},
"thread_info": {
"fpregs": {
"st_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"fop": 0,
"rdp": 0,
"twd": 0,
"mxcsr": 8064,
"swd": 0,
"rip": 0,
"xsave": {
"ymmh_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"xstate_bv": 2
},
"xmm_space": [
1953064751,
1769104175,
1702113141,
1664054387,
0,
0,
0,
0,
0,
4294901760,
255,
0,
792358505,
1953460082,
1852400175,
0,
942882145,
876295483,
774519349,
1031303283,
893073459,
976565307,
1937255978,
859661936,
993344312,
3814708,
65,
0,
21771440,
0,
21771552,
0,
0,
0,
0,
0,
4278190080,
4294967295,
4294967040,
4294967295,
0,
0,
0,
0,
0,
4278190080,
4294967040,
4294967295,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"cwd": 0,
"mxcsr_mask": 65535
},
"clear_tid_addr": 0,
"gpregs": {
"gs": 0,
"ip": 140137968588056,
"cx": 18446744073709551615,
"cs": 51,
"ax": 61,
"orig_ax": 61,
"di": 18446744073709551615,
"es": 0,
"gs_base": 0,
"r14": 0,
"r15": 4294967295,
"r12": 21789168,
"r13": 0,
"r10": 0,
"r11": 582,
"fs_base": 140137978173184,
"bp": 2560,
"dx": 0,
"bx": 0,
"ds": 0,
"ss": 43,
"sp": 140736135950616,
"r8": 21789168,
"r9": 0,
"fs": 0,
"si": 140736135950656,
"flags": 582
}
},
"tc": {
"timers": {
"real": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"virt": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"prof": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
}
},
"cg_set": 1,
"signals_s": {},
"blk_sigset": 65536,
"exit_code": 0,
"rlimits": {
"rlimits": [
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 8388608
},
{
"max": 18446744073709551615,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 4096,
"cur": 1024
},
{
"max": 65536,
"cur": 65536
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 819200,
"cur": 819200
},
{
"max": 0,
"cur": 0
},
{
"max": 0,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
}
]
},
"comm": "loop.sh",
"flags": 1077960704,
"task_state": 1,
"personality": 0
}
}
]
}
</pre>
</div>
</div>

=== Generate core files out of task images ===

{{Status|in progress}}

All the needed information is in core-$pid.img and pagemap-$pid.img

=== On-the-fly conversion ===

{{Status|not ready}}

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

=== Convert between different image versions ===

{{Status|not ready}}

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

=== Show images statistics ===

{{Status|not ready}}

E.g. -- total number of processes, files, memory, sockets, etc.

=== Check/validate images ===

{{Status|not ready}}

Check that
* all images are present
* the inter-images IDs are in consistent state

[[Category:Images]]
[[Category:API]]

CRIT

2015-02-03T18:58:08Z

Efiop: switch core dump state to "In progress"

This page describes the CRiu Image Tool. CRIT is a feature-rich replacement for existing "criu show".
It is written completely in Python, so it is quite easy to read the code and extend its features.

== Usage ==
<pre>
usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] {decode,encode}

CRiu Image Tool

positional arguments:
{decode,encode} decode/encode - convert criu image from/to binary type
to/from json

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default)
--pretty multi-line with indentation and some fields printed in more human-readable format
</pre>

=== Pretty output ===

By default CRIT prints JSON text in one line. This can be read by any further JSON-aware tool for parsing. For human eyes it's more convenient to read JSON multi-line with indentation. However, CRIT does a little bit more and prints some fields in even more pretty manner.

; Addresses and registers
: VM addresses and core.img register values are all printed in hex. Since JSON doesn't support this form of numbers, such fields are encoded as strings.

; Bit-fields
: Such things as flags and masks (e.g. sig-block mask) are also better understood when written in hex, so CRIT does this.

; IP addresses
: By default those a printed in decimal, but the "1.2.3.4" for v4 or "::1" for v6 can be seen in the --pretty mode.

== Functionality ==

=== Convert images to JSON and back ===

{{Status|ready}}

This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.

The output file is structured in the following way.

With "--format raw":
<pre>
{ "magic" : "FOO", "entries" : [{"foo": "bar", "bar": "foo", "extra": "abc"}, {"foo": "bar", "bar" : "foo", "extra" : "abc"} ]}
</pre>

With "--format nice"(default for stdout):
<pre>
{
"magic" : "FOO",
"entries" : [
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
},
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
}
]
}
</pre>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
<b>Example("crit decode -i core-20700.img")</b>
<div class="mw-collapsible-content">
<pre>
{
"magic": "CORE",
"entries": [
{
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": 24,
"sched_policy": 0,
"sched_nice": 0,
"futex_rla": 0,
"signals_p": {},
"sas": {
"ss_size": 0,
"ss_sp": 0,
"ss_flags": 2
}
},
"thread_info": {
"fpregs": {
"st_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"fop": 0,
"rdp": 0,
"twd": 0,
"mxcsr": 8064,
"swd": 0,
"rip": 0,
"xsave": {
"ymmh_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"xstate_bv": 2
},
"xmm_space": [
1953064751,
1769104175,
1702113141,
1664054387,
0,
0,
0,
0,
0,
4294901760,
255,
0,
792358505,
1953460082,
1852400175,
0,
942882145,
876295483,
774519349,
1031303283,
893073459,
976565307,
1937255978,
859661936,
993344312,
3814708,
65,
0,
21771440,
0,
21771552,
0,
0,
0,
0,
0,
4278190080,
4294967295,
4294967040,
4294967295,
0,
0,
0,
0,
0,
4278190080,
4294967040,
4294967295,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"cwd": 0,
"mxcsr_mask": 65535
},
"clear_tid_addr": 0,
"gpregs": {
"gs": 0,
"ip": 140137968588056,
"cx": 18446744073709551615,
"cs": 51,
"ax": 61,
"orig_ax": 61,
"di": 18446744073709551615,
"es": 0,
"gs_base": 0,
"r14": 0,
"r15": 4294967295,
"r12": 21789168,
"r13": 0,
"r10": 0,
"r11": 582,
"fs_base": 140137978173184,
"bp": 2560,
"dx": 0,
"bx": 0,
"ds": 0,
"ss": 43,
"sp": 140736135950616,
"r8": 21789168,
"r9": 0,
"fs": 0,
"si": 140736135950656,
"flags": 582
}
},
"tc": {
"timers": {
"real": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"virt": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"prof": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
}
},
"cg_set": 1,
"signals_s": {},
"blk_sigset": 65536,
"exit_code": 0,
"rlimits": {
"rlimits": [
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 8388608
},
{
"max": 18446744073709551615,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 4096,
"cur": 1024
},
{
"max": 65536,
"cur": 65536
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 819200,
"cur": 819200
},
{
"max": 0,
"cur": 0
},
{
"max": 0,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
}
]
},
"comm": "loop.sh",
"flags": 1077960704,
"task_state": 1,
"personality": 0
}
}
]
}
</pre>
</div>
</div>

=== Generate core files out of task images ===

{{Status|In progress}}

All the needed information is in core-$pid.img and pagemap-$pid.img

=== On-the-fly conversion ===

{{Status|not ready}}

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

=== Convert between different image versions ===

{{Status|not ready}}

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

=== Show images statistics ===

{{Status|not ready}}

E.g. -- total number of processes, files, memory, sockets, etc.

=== Check/validate images ===

{{Status|not ready}}

Check that
* all images are present
* the inter-images IDs are in consistent state

[[Category:Images]]
[[Category:API]]

Installation

2015-02-02T14:35:31Z

Efiop: python-ipaddr

<code>criu</code> is an utility to checkpoint/restore a process tree. This page describes how to manually build and install prerequisites and the tool itself.

{{Note|Most probably you don't need manual installation, but rather [[Packages]] for your distro.}}

== Obtaining CRIU Source ==

You can download the source code as a release tarball or sync the [http://git.criu.org/?p=criu.git;a=summary git repository].

{{Out|{{Latest release}}}}

git clone git://git.criu.org/criu.git
cd criu

== Dependencies ==

=== Compiler and C Library ===
For native compilation on Debian based systems, install the <code>build-essential</code> package. For cross compiling for ARM and AArch64, the Linaro prebuilt toolchains are a good choice. Installing them is described below. They are ia32 architecture binaries. On a modern Debian based x86_64 you will need to install the <code>lib32stdc++6</code> and <code>lib32z1</code> packages.

mkdir -p deps/`uname -m`-linux-gnu
cd deps
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-arm-linux-gnueabihf-4.9-2014.06_linux.tar.xz
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-aarch64-linux-gnu-4.9-2014.06-02_linux.tar.xz
cd ..

=== Protocol Buffers with C Bindings ===

CRIU uses the [https://github.com/protobuf-c/protobuf-c C language bindings] of [https://developers.google.com/protocol-buffers/ Google Protocol Buffers] for serialization. The <code>protoc</code> tool is required at build time and <code>libprotobuf-c.so</code> is required at build time and at run time, assuming dynamic linking.

[[CRIT]] also uses python language bindings of Google Protocol Buffers and requires descriptor.proto from developer files that could be found in protobuf-devel package.

==== Distribution Packages ====
The easiest approach for most would be to install distribution packages. RPM package names: <code>protobuf-c-compiler</code>, <code>protobuf-c-devel</code>, <code>protobuf-devel</code>, <code>protobuf-python</code>. Debian package names: <code>protobuf-c-compiler</code>, <code>libprotobuf-c0-dev</code>, <code>protobuf-compiler</code>, <code>protobuf-python</code>.

==== Building Protocol Buffers From Source ====
If you would like to build from source, you can use the following commands to obtain the source code repositories, configure, and build the code. On a Debian based system, you may have to install the following packages first: <code>autoconf curl g++ libtool</code>.

===== Native protobuf =====
cd deps
git clone https://github.com/google/protobuf.git protobuf
cd protobuf
./autogen.sh
./configure --prefix=`pwd`/../`uname -m`-linux-gnu
make
make install
cd ../..

===== Native protobuf-c =====

cd deps
git clone https://github.com/protobuf-c/protobuf-c.git protobuf-c
cd protobuf-c
./autogen.sh
mkdir ../pbc-`uname -m`
cd ../pbc-`uname -m`
../protobuf-c/configure --prefix=`pwd`/../`uname -m`-linux-gnu \
PKG_CONFIG_PATH=`pwd`/../`uname -m`-linux-gnu/lib/pkgconfig
make
make install
cd ../..

===== Cross Compiling for ARMv7 =====
If you would like to cross-compile for armv7:

cd deps
mkdir -p pbc-arm
cd pbc-arm
../protobuf-c/configure --host=arm-linux-gnueabihf --prefix=`pwd`/../arm-linux-gnueabihf --disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

===== Cross Compiling for ARMv8 =====
If you would like to cross-compile for armv8:

cd deps
mkdir -p pbc-aarch64
cd pbc-aarch64
../protobuf-c/configure --host=aarch64-linux-gnu --prefix=`pwd`/../aarch64-linux-gnu --disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

=== Other deps ===
==== python-ipaddr ====
Used in CRIT to pretty-print ip.
=== Some minor, but useful dependencies ===
==== libbsd ====
If libbsd is available, CRIU will be compiled with setproctitle() support. It allows to make process titles of service workers to be more verbose.

== Building CRIU From Source ==

=== Native Compilation ===
With the CRIU source obtained in the first step and dependencies satisfied in the second step, we are now compile CRIU. For native compilation with the dependencies met using distribution packages, simply run <code>make</code> in the CRIU source directory.

Here is an example of building natively specifying manually built dependencies.

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-`uname -m`
cd criu-`uname -m`
make \
USERCFLAGS="-I`pwd`/../`uname -m`-linux-gnu/include -L`pwd`/../`uname -m`-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
sudo LD_LIBRARY_PATH=`pwd`/../`uname -m`-linux-gnu/lib ./criu check
cd ../..

=== Cross Compilation for ARMv7 ===

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-arm
cd criu-arm
make \
ARCH=arm \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/arm-linux-gnueabihf- \
USERCFLAGS="-I`pwd`/../arm-linux-gnueabihf/include -L`pwd`/../arm-linux-gnueabihf/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

=== Cross Compilation for ARMv8 ===

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-aarch64
cd criu-aarch64
make \
ARCH=aarch64 \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/aarch64-linux-gnu- \
USERCFLAGS="-I`pwd`/../aarch64-linux-gnu/include -L`pwd`/../aarch64-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

=== Linux Kernel ===

Linux kernel v3.11 or newer is required, with some specific options set. If your distribution does not provide needed kernel, you might want to compile one yourself. Note we also have our [[custom kernel]], which might contain some experimental CRIU related patches.

Note you might have to enable
; <code>CONFIG_EXPERT</code>
: General setup -> Configure standard kernel features (expert users)
option, which depends on
; <code>CONFIG_EMBEDDED</code>
: General setup -> Embedded system
(welcome to Kconfig reverse chains hell).

The following options must be enabled for CRIU to work:

; <code>CONFIG_CHECKPOINT_RESTORE</code>
: General setup -> Checkpoint/restore support

; <code>CONFIG_NAMESPACES</code>
: General setup -> Namespaces support

; <code>CONFIG_UTS_NS</code>
: General setup -> Namespaces support -> UTS namespace

; <code>CONFIG_IPC_NS</code>
: General setup -> Namespaces support -> IPC namespace

; <code>CONFIG_PID_NS</code>
: General setup -> Namespaces support -> PID namespaces

; <code>CONFIG_NET_NS</code>
: General setup -> Namespaces support -> Network namespace

; <code>CONFIG_FHANDLE</code>
: General setup -> open by fhandle syscalls

; <code>CONFIG_EVENTFD</code>
: General setup -> Enable eventfd() system call

; <code>CONFIG_EPOLL</code>
: General setup -> Enable eventpoll support

; <code>CONFIG_INOTIFY_USER</code>
: File systems -> Inotify support for userspace

; <code>CONFIG_IA32_EMULATION</code>
: Executable file formats -> Emulations -> IA32 Emulation

; <code>CONFIG_UNIX_DIAG</code>
: Networking support -> Networking options -> Unix domain sockets -> UNIX: socket monitoring interface

; <code>CONFIG_INET_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface

; <code>CONFIG_INET_UDP_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface

; <code>CONFIG_PACKET_DIAG</code>
: Networking support -> Networking options -> Packet socket -> Packet: sockets monitoring interface

; <code>CONFIG_NETLINK_DIAG</code>
: Networking support -> Networking options -> Netlink socket -> Netlink: sockets monitoring interface

For some [[usage scenarios]] there is an ability to track memory changes and produce [[incremental dumps]]. Need to enable
; <code>CONFIG_MEM_SOFT_DIRTY</code>
: Processor type and features -> Track memory changes

At the moment it's known that CRIU will '''NOT''' work if packet generator module is loaded. Thus make sure
that either module is unloaded or not compiled at all.
; <code>CONFIG_NET_PKTGEN</code>
: Networking support -> Networking options -> Network testing -> Packet generator

=== iproute2 ===
The iproute2 tool version 3.5.0 or higher is needed for dumping network namespaces.
The latest one can be cloned from [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=summary iproute2]. It should be compiled and a path to ip written in the environment variable <code>CR_IP_TOOL</code>.

== Installation ==
<pre>
# make install
</pre>

== Checking That It Works ==

First thing to do is to run

<pre>
# criu check --ms
</pre>

At the end it should say "Looks OK", if it doesn't the messages on the screen explain what functionality is missing.
If you're using our custom kernel, then the <code>--ms</code> option should not be used, in this case CRIU would
check for ''all'' the kernel features to work.

You can then try running the [[ZDTM Test Suite]] which sits in the <code>tests/zdtm/</code> directory.

{{Out|There's a known issue with BTRFS spoiling dev_t values for files and sockets! Not all tests will work on it.}}

== Using CR tools ==

Please see [[Usage]] and [[Advanced usage]], as well as [[:Category:HOWTO]].

[[Category:HOWTO]]

CRIT

2015-02-01T18:33:43Z

Efiop: add a bit more verbose CRIT description and its purpose

This page describes the CRiu Image Tool. CRIT is a feature-rich replacement for existing "criu show".
It is written completely in Python, so it is quite easy to read the code and extend its features.

== Usage ==
<pre>
usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] {decode,encode}

CRiu Image Tool

positional arguments:
{decode,encode} decode/encode - convert criu image from/to binary type
to/from json

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default)
-f {raw,nice}, --format {raw,nice}
well-formated output (by default: raw for files and
nice for stdout)
</pre>

== Functionality ==

=== Convert images to JSON and back ===

{{Status|ready}}

This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.

The output file is structured in the following way.

With "--format raw":
<pre>
{ "magic" : "FOO", "entries" : [{"foo": "bar", "bar": "foo", "extra": "abc"}, {"foo": "bar", "bar" : "foo", "extra" : "abc"} ]}
</pre>

With "--format nice"(default for stdout):
<pre>
{
"magic" : "FOO",
"entries" : [
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
},
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
}
]
}
</pre>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
<b>Example("crit decode -i core-20700.img")</b>
<div class="mw-collapsible-content">
<pre>
{
"magic": "CORE",
"entries": [
{
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": 24,
"sched_policy": 0,
"sched_nice": 0,
"futex_rla": 0,
"signals_p": {},
"sas": {
"ss_size": 0,
"ss_sp": 0,
"ss_flags": 2
}
},
"thread_info": {
"fpregs": {
"st_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"fop": 0,
"rdp": 0,
"twd": 0,
"mxcsr": 8064,
"swd": 0,
"rip": 0,
"xsave": {
"ymmh_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"xstate_bv": 2
},
"xmm_space": [
1953064751,
1769104175,
1702113141,
1664054387,
0,
0,
0,
0,
0,
4294901760,
255,
0,
792358505,
1953460082,
1852400175,
0,
942882145,
876295483,
774519349,
1031303283,
893073459,
976565307,
1937255978,
859661936,
993344312,
3814708,
65,
0,
21771440,
0,
21771552,
0,
0,
0,
0,
0,
4278190080,
4294967295,
4294967040,
4294967295,
0,
0,
0,
0,
0,
4278190080,
4294967040,
4294967295,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"cwd": 0,
"mxcsr_mask": 65535
},
"clear_tid_addr": 0,
"gpregs": {
"gs": 0,
"ip": 140137968588056,
"cx": 18446744073709551615,
"cs": 51,
"ax": 61,
"orig_ax": 61,
"di": 18446744073709551615,
"es": 0,
"gs_base": 0,
"r14": 0,
"r15": 4294967295,
"r12": 21789168,
"r13": 0,
"r10": 0,
"r11": 582,
"fs_base": 140137978173184,
"bp": 2560,
"dx": 0,
"bx": 0,
"ds": 0,
"ss": 43,
"sp": 140736135950616,
"r8": 21789168,
"r9": 0,
"fs": 0,
"si": 140736135950656,
"flags": 582
}
},
"tc": {
"timers": {
"real": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"virt": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"prof": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
}
},
"cg_set": 1,
"signals_s": {},
"blk_sigset": 65536,
"exit_code": 0,
"rlimits": {
"rlimits": [
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 8388608
},
{
"max": 18446744073709551615,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 4096,
"cur": 1024
},
{
"max": 65536,
"cur": 65536
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 819200,
"cur": 819200
},
{
"max": 0,
"cur": 0
},
{
"max": 0,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
}
]
},
"comm": "loop.sh",
"flags": 1077960704,
"task_state": 1,
"personality": 0
}
}
]
}
</pre>
</div>
</div>

=== Generate core files out of task images ===

{{Status|not ready}}

All the needed information is in core-$pid.img and pagemap-$pid.img

=== On-the-fly conversion ===

{{Status|not ready}}

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

=== Convert between different image versions ===

{{Status|not ready}}

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

=== Show images statistics ===

{{Status|not ready}}

E.g. -- total number of processes, files, memory, sockets, etc.

=== Check/validate images ===

{{Status|not ready}}

Check that
* all images are present
* the inter-images IDs are in consistent state

[[Category:Images]]
[[Category:API]]

Todo

2015-02-01T17:33:44Z

Efiop: sign for core dump

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| crtools || Restore tasks into fresh new pid namespace || easy || - || A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - ||
|-
| crtools || AIO with pending events || medium || - ||
|-
| crtools || restore mountpoints || hard || - || -
|-
| crtools || lazy migration using userfaultd || medium || xemul || -
|-
| crtools || dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid
|-
| crtools || speed up logging || medium || Cyrill || Synchronous formatting and writes to log files slow things down ([[Logging]])
|-
| crtools || sanitize logging messages || hard || - || Currently log messages are placed w/o any logic. Need to improve that ... somehow.
|-
| crtools || support OFD posix locks || easy || - ||
|-
| crtools || optimize kcmp calls || medium || - || For fs/fdt/vm/etc we can pre-generate ID based on objects that live on them. If pre-ID matches, then call kcmp.
|-
| crtools || Shmem changes tracking || medium || - ||
|-
| crtools || Page transfer filters || medium || Ruslan Kuprieiev || Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || FUSE || hard || - ||
|-
| crtools || 32-bit tasks || hard || Cyrill ||
|-
| crtools || Support one of MPI implementations || hard || Adrian Reber || worth starting with OpenMPI
|-
| crtools || Generate task's core file out of images || medium || Ruslan Kuprieiev || [[CRIT]]
|-
| crtools || Modify restored resources run-time || medium || - || [[CRIT]] Need (probably) some way to alter what is being restored. Usage example -- change the IP address of sockets from task above.
|-
| crtools || TCP socket migration with changed IP || medium || - || [[CRIT]] it might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks, but this might make sense.
|-
| crtools || [[Applying images]] || hard || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). E.g. -- repopulate fdtable according to data from image. Another use-case -- when doing partial migration we'll need to modify one part to switch from pipes to sockets
|-
| crtools || Partial migration || hard || - || migrate some tasks while proxying IPC to existing others (pipes->sockets, etc.)
|-
| crtools || Shared objects (mm/fs) support || medium || - || Now we have the kcmp syscall and can do it. The first candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Need a way to resolve paths to overmounted files. There are two ways: 1. Move mounts, that overlap the desired path temprarily. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd
|-
| kernel/crtools || TCP repair fixes || hard || - || [[TCP repair TODO]]
|-
| crtools || Bridges in container || medium || - ||
|-
| crtools || VLANs in containers || medium || - ||
|-
| crtools || PPP support || medium || - ||
|-
| crtools/kernel || DelayFS for NFS/Fuse migration || hard || - ||
|-
| kernel || Seamless kernel upgrade || hard || xemul ||
|-
| crtools/kernel || NFS || hard || - || Probably depends on DelayFS
|-
| crtools || Validate .img files || easy || - || [[CRIT]] For a given set of image files check, that they are in "restorable" shape
|-
| crtools || Restore a process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]
|-
| crtools || X app || hard || - || Dump/restore of graphical applications(Maybe worth switching to Wayland?)
|-
| crtools || Wayland app || hard || - || Dump/restore of a graphical application. Require mods from compositors(worth starting with Weston) to get\set state. [http://people.freedesktop.org/~cbrill/dri-log/?channel=wayland&highlight_names=&date=2015-01-30 Here] are some talks on #wayland.
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || Require mods from both sides -- criu and kernel
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h])
|-
| kernel/criu || fsnotify queues || hard || - || [[Fsnotify]]
|}

[[Category:Development]]
[[Category:Plans]]

Todo

2015-02-01T17:31:28Z

Efiop: wayland

{| class="wikitable sortable"
|-
! component
! task
! complexity
! potential/willing assignee
! comments
|-
| crtools || Restore tasks into fresh new pid namespace || easy || - || A thing to worry about is this new namespace's init task.
|-
| crtools || Rollback tree state || medium || - ||
|-
| crtools || AIO with pending events || medium || - ||
|-
| crtools || restore mountpoints || hard || - || -
|-
| crtools || lazy migration using userfaultd || medium || xemul || -
|-
| crtools || dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid
|-
| crtools || speed up logging || medium || Cyrill || Synchronous formatting and writes to log files slow things down ([[Logging]])
|-
| crtools || sanitize logging messages || hard || - || Currently log messages are placed w/o any logic. Need to improve that ... somehow.
|-
| crtools || support OFD posix locks || easy || - ||
|-
| crtools || optimize kcmp calls || medium || - || For fs/fdt/vm/etc we can pre-generate ID based on objects that live on them. If pre-ID matches, then call kcmp.
|-
| crtools || Shmem changes tracking || medium || - ||
|-
| crtools || Page transfer filters || medium || Ruslan Kuprieiev || Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || FUSE || hard || - ||
|-
| crtools || 32-bit tasks || hard || Cyrill ||
|-
| crtools || Support one of MPI implementations || hard || Adrian Reber || worth starting with OpenMPI
|-
| crtools || Generate task's core file out of images || medium || - || [[CRIT]]
|-
| crtools || Modify restored resources run-time || medium || - || [[CRIT]] Need (probably) some way to alter what is being restored. Usage example -- change the IP address of sockets from task above.
|-
| crtools || TCP socket migration with changed IP || medium || - || [[CRIT]] it might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks, but this might make sense.
|-
| crtools || [[Applying images]] || hard || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). E.g. -- repopulate fdtable according to data from image. Another use-case -- when doing partial migration we'll need to modify one part to switch from pipes to sockets
|-
| crtools || Partial migration || hard || - || migrate some tasks while proxying IPC to existing others (pipes->sockets, etc.)
|-
| crtools || Shared objects (mm/fs) support || medium || - || Now we have the kcmp syscall and can do it. The first candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Need a way to resolve paths to overmounted files. There are two ways: 1. Move mounts, that overlap the desired path temprarily. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd
|-
| kernel/crtools || TCP repair fixes || hard || - || [[TCP repair TODO]]
|-
| crtools || Bridges in container || medium || - ||
|-
| crtools || VLANs in containers || medium || - ||
|-
| crtools || PPP support || medium || - ||
|-
| crtools/kernel || DelayFS for NFS/Fuse migration || hard || - ||
|-
| kernel || Seamless kernel upgrade || hard || xemul ||
|-
| crtools/kernel || NFS || hard || - || Probably depends on DelayFS
|-
| crtools || Validate .img files || easy || - || [[CRIT]] For a given set of image files check, that they are in "restorable" shape
|-
| crtools || Restore a process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]
|-
| crtools || X app || hard || - || Dump/restore of graphical applications(Maybe worth switching to Wayland?)
|-
| crtools || Wayland app || hard || - || Dump/restore of a graphical application. Require mods from compositors(worth starting with Weston) to get\set state. [http://people.freedesktop.org/~cbrill/dri-log/?channel=wayland&highlight_names=&date=2015-01-30 Here] are some talks on #wayland.
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || Require mods from both sides -- criu and kernel
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h])
|-
| kernel/criu || fsnotify queues || hard || - || [[Fsnotify]]
|}

[[Category:Development]]
[[Category:Plans]]

Asciinema

2015-01-31T15:17:32Z

Efiop: Self-dump of a python app using CRIU RPC

There's a nice service called [https://asciinema.org asciinema] out there. We plan to create a set of tutorials and this page lists the plans and already finished ones.

* [https://asciinema.org/a/15649 Getting started]
* [https://asciinema.org/a/15651 Single ZDTM test run] ([[ZDTM Test Suite]])
* [https://asciinema.org/a/15652 Exploring the images] ([[CRIT]])
* [https://asciinema.org/a/15653 Running the "criu dump" command]
* [https://asciinema.org/a/15655 Preparing the patch for submission] ([[How to submit patches]])
* [https://asciinema.org/a/15668 Adding a new ZDTM test case] ([[ZDTM API]])
* [https://asciinema.org/a/15832 C/R of running "top"] ([[Advanced usage]])
* [https://asciinema.org/a/15833 Live TCP connection C/R] ([[TCP connection]])
* [https://asciinema.org/a/15842 Playing with action scripts] ([[Action scripts]])
* [https://asciinema.org/a/15844 Stackable images basics] ([[Memory changes tracking]])
* [https://asciinema.org/a/15847 Using page-server] ([[Disk-less migration]])
* [https://asciinema.org/a/15857 Using the pre-dump command] ([[Iterative migration]])
* [https://asciinema.org/a/15920 Self-dump of a python app using CRIU RPC] ([[RPC]])

[[Category:HOWTO]]

Images

2015-01-28T18:00:50Z

Efiop: mention undocumented tcp-stream image structure

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

{{Note| You might also want to checkout our image tool called [[CRIT]]}}

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

Or, you can visualize it like

{| class="wikitable"
|-
! Type !! Size(bytes)
|-
| Magic || 4
|-
| Size0 || 4
|-
| Message0 || Size0
|-
| ... || ...
|-
| SizeN || 4
|-
| MessageN || SizeN
|}

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || <code>entry.inq_len</code> bytes of in-queue data followed by <code>entry.outq_len</code> bytes of out-queue data || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
{{Note| Even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry. }}

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== Notes about protobuf ==
We have a registered field number(1018) for [https://developers.google.com/protocol-buffers/docs/proto#options custom options] of all kinds. See protobuf/opts.proto for more info.

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]
[[Category:Images]]

C API

2015-01-19T19:44:54Z

Efiop: fix

'''libcriu''' is a C API for CRIU, which is a simple wrapper around our [[RPC]]. Although you can use [[RPC]] directly, libcriu is a wrapper providing the interface that is much easier to use from C code. Note that [[RPC]] is supported in the first place, and if you want the most recent set of features you should probably use [[RPC]] directly.

== Introduction ==

libcriu functions are defined in <code>lib/criu.h</code>.

To create a library <code>lib/libcriu.so</code>, run <code>make</code> in the main directory.

The library is not thread- and fork- safe.

== Preparation ==

=== init options ===

Call <code>int criu_init_opts(void)</code> to initialize request options.

Note: it should be called before using any other functions from libcriu, except <code>criu_check()</code>. Also you should use it to reinitialize options. It returns 0 on success and -1 on fail.

=== set service address ===

Use <code>void criu_set_service_address(char *address)</code> to specify path to a CRIU service socket. Call it with NULL to make libcriu use the default address (currently <code>/var/run/criu_service.socket</code>).

=== set dump/restore options ===
Use <code>criu_set_*</code> functions to setup dump/restore options.

<pre>
void criu_set_pid(int pid);
void criu_set_images_dir_fd(int fd); /* must be set for dump/restore */
void criu_set_leave_running(bool leave_running);
void criu_set_ext_unix_sk(bool ext_unix_sk);
void criu_set_tcp_established(bool tcp_established);
void criu_set_evasive_devices(bool evasive_devices);
void criu_set_shell_job(bool shell_job);
void criu_set_file_locks(bool file_locks);
void criu_set_log_level(int log_level);
void criu_set_log_file(char *log_file);
</pre>

If no pid is set on dump, CRIU will dump the calling process itself.
{{Note|If a calling process is not run as root, the whole process tree to be dumped must have the same uid, otherwise CRIU refuses to dump. See [[Usage#Security]].}}

{{Note|<code>images_dir_fd</code> is '''required''' at dump/restore, all other options might be left unset.
The client must open directory for/with images by itself and set <code>images_dir_fd</code> to the opened directory fd.
CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>, so it will work even if the client is in another namespace.}}

The logic of setting request is the same as when setting options in console. Here is an example:

#criu restore -D /path/to/imgs_dir -v4 -o restore.log

is equal to:
<pre>
criu_init_opts();
criu_set_service_address("/path/to/criu/service/socket");

int fd = open("/path/to/imgs_dir", O_DIRECTORY);
criu_set_images_dir_fd(fd);

criu_set_log_file("restore.log");
criu_set_log_level(4);

criu_restore();
</pre>

== Operations ==

Use the following functions to perform CRIU actions.

=== check ===

int criu_check(void);

=== dump ===

int criu_dump(void);

=== restore ===

int criu_restore(void);

=== restore as child ===
This one is special. It will fork and exec criu swrk(Service WoRKer) to allow restoring process as a caller child. Calling "exec" implies some restrictions, as, for example, one should make sure, that criu binary is present in PATH and has suid bit set(see [[Usage#Security]]).

int criu_restore_child(void);

=== Return values ===

Here is a table of return and errno values of the above functions:
{| class="wikitable"
|-
! Return value
! errno
! Description

|-
| 0
| undefined
| Success

|-
| >0
| undefined
| Success (<code>criu_restore()</code> only)

|-
| -EBADE
| RPC error (if provided(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]), 0 otherwise)
| RPC has returned fail.

|-
| -ECONNREFUSED
| errno
| Unable to connect to CRIU.

|-
| -ECOMM
| errno
| Unable to send/recv msg to/from CRIU.

|-
| -EINVAL
| undefined
| CRIU doesn't support this type of request. You should probably update CRIU.

|-
| -EBADMSG
| undefined
| Unexpected response from CRIU. You should probably update CRIU.
|}

== Examples ==

You could find example of using libcriu at [http://git.criu.org/?p=criu.git;a=tree;f=test/libcriu test/libcriu].

[[Category: API]]

C API

2015-01-19T19:43:03Z

Efiop: add criu_restore_child()

'''libcriu''' is a C API for CRIU, which is a simple wrapper around our [[RPC]]. Although you can use [[RPC]] directly, libcriu is a wrapper providing the interface that is much easier to use from C code. Note that [[RPC]] is supported in the first place, and if you want the most recent set of features you should probably use [[RPC]] directly.

== Introduction ==

libcriu functions are defined in <code>lib/criu.h</code>.

To create a library <code>lib/libcriu.so</code>, run <code>make</code> in the main directory.

The library is not thread- and fork- safe.

== Preparation ==

=== init options ===

Call <code>int criu_init_opts(void)</code> to initialize request options.

Note: it should be called before using any other functions from libcriu, except <code>criu_check()</code>. Also you should use it to reinitialize options. It returns 0 on success and -1 on fail.

=== set service address ===

Use <code>void criu_set_service_address(char *address)</code> to specify path to a CRIU service socket. Call it with NULL to make libcriu use the default address (currently <code>/var/run/criu_service.socket</code>).

=== set dump/restore options ===
Use <code>criu_set_*</code> functions to setup dump/restore options.

<pre>
void criu_set_pid(int pid);
void criu_set_images_dir_fd(int fd); /* must be set for dump/restore */
void criu_set_leave_running(bool leave_running);
void criu_set_ext_unix_sk(bool ext_unix_sk);
void criu_set_tcp_established(bool tcp_established);
void criu_set_evasive_devices(bool evasive_devices);
void criu_set_shell_job(bool shell_job);
void criu_set_file_locks(bool file_locks);
void criu_set_log_level(int log_level);
void criu_set_log_file(char *log_file);
</pre>

If no pid is set on dump, CRIU will dump the calling process itself.
{{Note|If a calling process is not run as root, the whole process tree to be dumped must have the same uid, otherwise CRIU refuses to dump. See [[Usage#Security]].}}

{{Note|<code>images_dir_fd</code> is '''required''' at dump/restore, all other options might be left unset.
The client must open directory for/with images by itself and set <code>images_dir_fd</code> to the opened directory fd.
CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>, so it will work even if the client is in another namespace.}}

The logic of setting request is the same as when setting options in console. Here is an example:

#criu restore -D /path/to/imgs_dir -v4 -o restore.log

is equal to:
<pre>
criu_init_opts();
criu_set_service_address("/path/to/criu/service/socket");

int fd = open("/path/to/imgs_dir", O_DIRECTORY);
criu_set_images_dir_fd(fd);

criu_set_log_file("restore.log");
criu_set_log_level(4);

criu_restore();
</pre>

== Operations ==

Use the following functions to perform CRIU actions.

=== check ===

int criu_check(void);

=== dump ===

int criu_dump(void);

=== restore ===

int criu_restore(void);

=== restore as child ===
This one is special. It will fork and exec criu swrk(Service WoRKer) to allow restoring process as a caller child. Calling "exec" implies some restrictions, as, for example, one should make sure, that criu binary is present in PATH and has [[Security suid bit]] set.

int criu_restore_child(void);

=== Return values ===

Here is a table of return and errno values of the above functions:
{| class="wikitable"
|-
! Return value
! errno
! Description

|-
| 0
| undefined
| Success

|-
| >0
| undefined
| Success (<code>criu_restore()</code> only)

|-
| -EBADE
| RPC error (if provided(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]), 0 otherwise)
| RPC has returned fail.

|-
| -ECONNREFUSED
| errno
| Unable to connect to CRIU.

|-
| -ECOMM
| errno
| Unable to send/recv msg to/from CRIU.

|-
| -EINVAL
| undefined
| CRIU doesn't support this type of request. You should probably update CRIU.

|-
| -EBADMSG
| undefined
| Unexpected response from CRIU. You should probably update CRIU.
|}

== Examples ==

You could find example of using libcriu at [http://git.criu.org/?p=criu.git;a=tree;f=test/libcriu test/libcriu].

[[Category: API]]

Installation

2015-01-19T12:50:27Z

Efiop: protobuf-devel

<code>criu</code> is an utility to checkpoint/restore a process tree. This page describes how to manually build and install prerequisites and the tool itself.

{{Note|Most probably you don't need manual installation, but rather [[Packages]] for your distro.}}

== Obtaining CRIU Source ==

You can download the source code as a release tarball or sync the [http://git.criu.org/?p=criu.git;a=summary git repository].

{{Out|{{Latest release}}}}

git clone git://git.criu.org/criu.git
cd criu

== Dependencies ==

=== Compiler and C Library ===
For native compilation on Debian based systems, install the <code>build-essential</code> package. For cross compiling for ARM and AArch64, the Linaro prebuilt toolchains are a good choice. Installing them is described below. They are ia32 architecture binaries. On a modern Debian based x86_64 you will need to install the <code>lib32stdc++6</code> and <code>lib32z1</code> packages.

mkdir -p deps/`uname -m`-linux-gnu
cd deps
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-arm-linux-gnueabihf-4.9-2014.06_linux.tar.xz
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-aarch64-linux-gnu-4.9-2014.06-02_linux.tar.xz
cd ..

=== Protocol Buffers with C Bindings ===

CRIU uses the [https://github.com/protobuf-c/protobuf-c C language bindings] of [https://developers.google.com/protocol-buffers/ Google Protocol Buffers] for serialization. The <code>protoc</code> tool is required at build time and <code>libprotobuf-c.so</code> is required at build time and at run time, assuming dynamic linking.

[[CRIT]] also uses python language bindings of Google Protocol Buffers and requires descriptor.proto from developer files that could be found in protobuf-devel package.

==== Distribution Packages ====
The easiest approach for most would be to install distribution packages. RPM package names: <code>protobuf-c-compiler</code>, <code>protobuf-c-devel</code>, <code>protobuf-devel</code>, <code>protobuf-python</code>. Debian package names: <code>protobuf-c-compiler</code>, <code>libprotobuf-c0-dev</code>, <code>protobuf-compiler</code>, <code>protobuf-python</code>.

==== Building Protocol Buffers From Source ====
If you would like to build from source, you can use the following commands to obtain the source code repositories, configure, and build the code. On a Debian based system, you may have to install the following packages first: <code>autoconf curl g++ libtool</code>.

===== Native protobuf =====
cd deps
git clone https://github.com/google/protobuf.git protobuf
cd protobuf
./autogen.sh
./configure --prefix=`pwd`/../`uname -m`-linux-gnu
make
make install
cd ../..

===== Native protobuf-c =====

cd deps
git clone https://github.com/protobuf-c/protobuf-c.git protobuf-c
cd protobuf-c
./autogen.sh
mkdir ../pbc-`uname -m`
cd ../pbc-`uname -m`
../protobuf-c/configure --prefix=`pwd`/../`uname -m`-linux-gnu \
PKG_CONFIG_PATH=`pwd`/../`uname -m`-linux-gnu/lib/pkgconfig
make
make install
cd ../..

===== Cross Compiling for ARMv7 =====
If you would like to cross-compile for armv7:

cd deps
mkdir -p pbc-arm
cd pbc-arm
../protobuf-c/configure --host=arm-linux-gnueabihf --prefix=`pwd`/../arm-linux-gnueabihf --disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

===== Cross Compiling for ARMv8 =====
If you would like to cross-compile for armv8:

cd deps
mkdir -p pbc-aarch64
cd pbc-aarch64
../protobuf-c/configure --host=aarch64-linux-gnu --prefix=`pwd`/../aarch64-linux-gnu --disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

=== Some minor, but useful dependencies ===
==== libbsd ====
If libbsd is available, CRIU will be compiled with setproctitle() support. It allows to make process titles of service workers to be more verbose.

== Building CRIU From Source ==

=== Native Compilation ===
With the CRIU source obtained in the first step and dependencies satisfied in the second step, we are now compile CRIU. For native compilation with the dependencies met using distribution packages, simply run <code>make</code> in the CRIU source directory.

Here is an example of building natively specifying manually built dependencies.

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-`uname -m`
cd criu-`uname -m`
make \
USERCFLAGS="-I`pwd`/../`uname -m`-linux-gnu/include -L`pwd`/../`uname -m`-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
sudo LD_LIBRARY_PATH=`pwd`/../`uname -m`-linux-gnu/lib ./criu check
cd ../..

=== Cross Compilation for ARMv7 ===

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-arm
cd criu-arm
make \
ARCH=arm \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/arm-linux-gnueabihf- \
USERCFLAGS="-I`pwd`/../arm-linux-gnueabihf/include -L`pwd`/../arm-linux-gnueabihf/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

=== Cross Compilation for ARMv8 ===

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-aarch64
cd criu-aarch64
make \
ARCH=aarch64 \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/aarch64-linux-gnu- \
USERCFLAGS="-I`pwd`/../aarch64-linux-gnu/include -L`pwd`/../aarch64-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

=== Linux Kernel ===

Linux kernel v3.11 or newer is required, with some specific options set. If your distribution does not provide needed kernel, you might want to compile one yourself. Note we also have our [[custom kernel]], which might contain some experimental CRIU related patches.

Note you might have to enable
; <code>CONFIG_EXPERT</code>
: General setup -> Configure standard kernel features (expert users)
option, which depends on
; <code>CONFIG_EMBEDDED</code>
: General setup -> Embedded system
(welcome to Kconfig reverse chains hell).

The following options must be enabled for CRIU to work:

; <code>CONFIG_CHECKPOINT_RESTORE</code>
: General setup -> Checkpoint/restore support

; <code>CONFIG_NAMESPACES</code>
: General setup -> Namespaces support

; <code>CONFIG_UTS_NS</code>
: General setup -> Namespaces support -> UTS namespace

; <code>CONFIG_IPC_NS</code>
: General setup -> Namespaces support -> IPC namespace

; <code>CONFIG_PID_NS</code>
: General setup -> Namespaces support -> PID namespaces

; <code>CONFIG_NET_NS</code>
: General setup -> Namespaces support -> Network namespace

; <code>CONFIG_FHANDLE</code>
: General setup -> open by fhandle syscalls

; <code>CONFIG_EVENTFD</code>
: General setup -> Enable eventfd() system call

; <code>CONFIG_EPOLL</code>
: General setup -> Enable eventpoll support

; <code>CONFIG_INOTIFY_USER</code>
: File systems -> Inotify support for userspace

; <code>CONFIG_IA32_EMULATION</code>
: Executable file formats -> Emulations -> IA32 Emulation

; <code>CONFIG_UNIX_DIAG</code>
: Networking support -> Networking options -> Unix domain sockets -> UNIX: socket monitoring interface

; <code>CONFIG_INET_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface

; <code>CONFIG_INET_UDP_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface

; <code>CONFIG_PACKET_DIAG</code>
: Networking support -> Networking options -> Packet socket -> Packet: sockets monitoring interface

; <code>CONFIG_NETLINK_DIAG</code>
: Networking support -> Networking options -> Netlink socket -> Netlink: sockets monitoring interface

For some [[usage scenarios]] there is an ability to track memory changes and produce [[incremental dumps]]. Need to enable
; <code>CONFIG_MEM_SOFT_DIRTY</code>
: Processor type and features -> Track memory changes

At the moment it's known that CRIU will '''NOT''' work if packet generator module is loaded. Thus make sure
that either module is unloaded or not compiled at all.
; <code>CONFIG_NET_PKTGEN</code>
: Networking support -> Networking options -> Network testing -> Packet generator

=== iproute2 ===
The iproute2 tool version 3.5.0 or higher is needed for dumping network namespaces.
The latest one can be cloned from [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=summary iproute2]. It should be compiled and a path to ip written in the environment variable <code>CR_IP_TOOL</code>.

== Installation ==
<pre>
# make install
</pre>

== Checking That It Works ==

First thing to do is to run

<pre>
# criu check --ms
</pre>

At the end it should say "Looks OK", if it doesn't the messages on the screen explain what functionality is missing.
If you're using our custom kernel, then the <code>--ms</code> option should not be used, in this case CRIU would
check for ''all'' the kernel features to work.

You can then try running the [[ZDTM Test Suite]] which sits in the <code>tests/zdtm/</code> directory.

{{Out|There's a known issue with BTRFS spoiling dev_t values for files and sockets! Not all tests will work on it.}}

== Using CR tools ==

Please see [[Usage]] and [[Advanced usage]], as well as [[:Category:HOWTO]].

[[Category:HOWTO]]

Images

2015-01-19T10:38:02Z

Efiop:

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

{{Note| You might also want to checkout our image tool called [[CRIT]]}}

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

Or, you can visualize it like

{| class="wikitable"
|-
! Type !! Size(bytes)
|-
| Magic || 4
|-
| Size0 || 4
|-
| Message0 || Size0
|-
| ... || ...
|-
| SizeN || 4
|-
| MessageN || SizeN
|}

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || - || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
{{Note| Even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry. }}

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== Notes about protobuf ==
We have a registered field number(1018) for [https://developers.google.com/protocol-buffers/docs/proto#options custom options] of all kinds. See protobuf/opts.proto for more info.

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]
[[Category:Images]]

Images

2015-01-19T10:37:03Z

Efiop: protobuf custom options field number

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

{{Note| You might also want to checkout our image tool called [[CRIT]]}}

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

Or, you can visualize it like

{| class="wikitable"
|-
! Type !! Size(bytes)
|-
| Magic || 4
|-
| Size0 || 4
|-
| Message0 || Size0
|-
| ... || ...
|-
| SizeN || 4
|-
| MessageN || SizeN
|}

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || - || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
{{Note| Even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry. }}

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== Notes about protobuf ==
;We have a registered field number(1018) for [https://developers.google.com/protocol-buffers/docs/proto#options custom options] of all kinds. See protobuf/opts.proto for more info.

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]
[[Category:Images]]

Comparison to other CR projects

2015-01-16T15:39:51Z

Efiop: say "manipulate" instead of "dump"

This pages tries to explain differences between CRIU and other C/R solutions.

== [http://dmtcp.sourceforge.net DMTCP] ==

{{:DMTCP}}

== [https://ftg.lbl.gov/projects/CheckpointRestart BLCR] ==

Berkeley Lab Checkpoint/Restart (BLCR) is a part of the Scalable Systems Software Suite ,
developed by the Future Technologies Group at Lawrence Berkeley National Lab under SciDAC
funding from the United States Department of Energy. It is an Open Source, system-level
checkpointer designed with High Performance Computing (HPC) applications in mind: in particular
CPU and memory intensive batch-scheduled MPI jobs. BLCR is implemented as a GPL-licensed
loadable kernel module for Linux 2.4.x and 2.6.x kernels on the x86, x86_64, PPC/PPC64, ARM architectures, and a
small LGPL-licensed library.

== [http://criu.org CRIU], [http://dmtcp.sourceforge.net DMTCP], [https://ftg.lbl.gov/projects/CheckpointRestart BLCR], [https://openvz.org OpenVZ] ==

“looks\seems like yes/no” - i found only unproved message(s) saying “yes”/“no”

“not yet” - it is officially planned or i found no reasons, why it can’t be done.

{| class="wikitable sortable"
|-
!
! CRIU
! DMTCP
! BLCR
! OpenVZ

|-
| Arch
| x86_64, ARM
| x86, x86_64, ARM
| x86, x86_64, PPC/PPC64, ARM
| x86, x86_64

|-
| OS
| Linux
| Linux
| Linux
| Linux

|-
| Need in modified kernel
| No, provided it's 3.11 and above
| No
| No, just need to load module. May be problems with installation on new kernels
| Yes

|-
| App need to pre-load special libraries
| No
| Yes
| Yes
| No

|-
| Requires root privileges
| No, but user can only manipulate tasks belonging to him
| No
| No
| Yes

|-
| Need to modify programs to C/R
| No
| No
| Yes. There are some difficulties with statically linked applications, and with LinuxThreads (it does not support them at all)
| No

|-
| Need to prepare tasks
| No
| Yes. It preloadsthe DMTCP library. That library runs before the routinemain(). It creates a second thread. Thecheckpoint thread then creates a socket to the DMTCP coordinator andregisters itself. The checkpoint thread also creates a signal handler.
| Yes. CR shall notify processes when a checkpoint is to occur (before the kernel takes a checkpoint) to allow the processes to prepare itself accordingly.
| No

|-
| Does it change behavior of the c/r-ed programs?
| No
| Yes, because of wrappers on system calls
| Yes, because of wrappers on system calls
| No

|-
| Live migration
| Yes, even if kernel, libs, etc are newer. Can use Memory Changes Tracking to decrease freeze time
| Yes, if both kernels are recent
| Yes, but if all components are the same. Even if prelinked addresses are different,it will not restore, but it can save the whole used libs and localization files to restore program on the different machine
| Yes

|-
| Containers
| Yes, LXC and OpenVZ containers
| No. It doesn't support namespaces, so it probably can’t dump containers
| Looks like no
| Yes

|-
| Parallel/distributed computations libraries
| No (in plans)
| Yes. OpenMPI, MPICH2, OpenMP, Cilk are alredy supported and Infiniband is in progress
| Yes. Cray MPI, Intel MPI, LAM/MPI, MPICH-V, MPICH2, MVAPICH, Open MPI, SGI MPT
| Yes

|-
| C/R gdb with debugging app
| No, because they are using the same interface
| Yes
| No
| Yes

|-
| X-Windows apps (KDE, GNOME, etc)
| Yes, by using vnc
| Yes, by using vnc
| Looks like no
| Yes, by using vnc

|-
| Solutions for invocation in the custom software
| Yes. [[RPC]] and [[C API]]
| Yes. Plugins and API
| Not yet
| Yes. via ioctl calls

|-
| colspan="4" |

|-
| Unix sockets
| Yes
| Yes
| No
| Yes

|-
| UDP sockets
| Yes, both ipv4 and ipv6
| Not yet. Developers of dmtcp had no request for this
| Not yet
| Yes

|-
| TCP sockets
| Yes
| Yes
| Not yet
| Yes

|-
| Established tcp connection
| Yes
| No, but you can write a simple DMTCP plugin that tells DMTCP how you want to reconnect on restart
| No
| Yes

|-
| Infiniband
| No
| Not yet, developing is on the half-way
| No
| No

|-
| Multithread support
| Yes
| Yes
| Yes
| Yes

|-
| Multiprocess
| Yes
| Yes
| Yes
| Yes

|-
| Process groups and sessions
| Yes
| Yes
| Not yet
| Yes

|-
| Zombies
| Yes
| No
| No
| Yes

|-
| Namespaces
| Yes
| No
| No
| Yes

|-
| Ptraced programs
| No
| Yes
| No
| Yes

|-
| System V IPC
| Yes
| Yes
| No
| Yes

|-
| Memory mappings
| Yes, all kinds
| Yes
| Yes, partially
| Yes

|-
| Pipes
| Yes
| Yes
| Not yet
| Yes

|-
| Terminals
| Yes, but only Unix98 PTYs
| Yes
| Yes
| Yes

|-
| Non-posix files (inotify, signalfd, eventfd, etc)
| Yes, inotify, fanotify, epoll, signalfd, eventfd
| Yes, epoll, eventfd, signalfd are already supported and inotify will be supported in future
| Looks like no
| Yes

|-
| Timers
| Yes
| No. Any counter or timer active since the beginning of a process will consider the restarted process to be a new process.
| Yes
| Yes

|-
| Shared resources (files, mm, etc.)
| Yes. SysVIPC, files, fd table and memory
| Yes. System V shared memory(shmget, etc.), mmap-based shared memory, shared sockets, pipes, file descriptors
| No, but it is planned to suppord shared mmap regions
| Yes

|-
| Block devices
| No
| Looks like yes
| No
| No

|-
| Character devices
| Yes, only /dev/null, /dev/zero, etc. are supported
| Yes, looks like null and zero are supported
| Yes, /dev/null and /dev/zero
| Yes

|-
| Capture the contents of open files
| Yes, if file is unlinked
| Looks like no
| Not yet
| Yes

|}

== Sources ==
DMTCP:
*http://dmtcp.sourceforge.net/
*http://dmtcp.sourceforge.net/papers/dmtcp.pdf
*http://www.ccs.neu.edu/home/gene/papers/ccgrid06.pdf
*http://research.cs.wisc.edu/htcondor/CondorWeek2010/condor-presentations/cooperman-dmtcp.pdf
*http://dmtcp.sourceforge.net/papers/mtcp.pdf

BLCR:
*https://upc-bugs.lbl.gov/blcr/doc/html/
*https://ftg.lbl.gov/assets/projects/CheckpointRestart/Pubs/LBNL-49659.pdf
*https://ftg.lbl.gov/assets/projects/CheckpointRestart/Pubs/blcr.pdf
*https://ftg.lbl.gov/assets/projects/CheckpointRestart/Pubs/checkpointSurvey-020724b.pdf
*https://ftg.lbl.gov/assets/projects/CheckpointRestart/Pubs/lacsi-2003.pdf
*https://ftg.lbl.gov/assets/projects/CheckpointRestart/Pubs/LBNL-60520.pdf

== External links ==

* [http://dmtcp.sourceforge.net/FAQ.html#Internals How does DMTCP work?]

[[Category:Under the hood]]

CRIT

2015-01-15T16:29:40Z

Efiop: Add example

This page describes the CRiu Image Tool.

== Usage ==
<pre>
usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] {decode,encode}

CRiu Image Tool

positional arguments:
{decode,encode} decode/encode - convert criu image from/to binary type
to/from json

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default)
-f {raw,nice}, --format {raw,nice}
well-formated output (by default: raw for files and
nice for stdout)
</pre>

== Functionality ==

=== Convert images to JSON and back ===

{{Status|ready}}

This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.

The output file is structured in the following way.

With "--format raw":
<pre>
{ "magic" : "FOO", "entries" : [{"foo": "bar", "bar": "foo", "extra": "abc"}, {"foo": "bar", "bar" : "foo", "extra" : "abc"} ]}
</pre>

With "--format nice"(default for stdout):
<pre>
{
"magic" : "FOO",
"entries" : [
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
},
{
"foo" : "bar",
"bar" : "foo",
"extra" : "abc"
}
]
}
</pre>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
<b>Example("crit decode -i core-20700.img")</b>
<div class="mw-collapsible-content">
<pre>
{
"magic": "CORE",
"entries": [
{
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": 24,
"sched_policy": 0,
"sched_nice": 0,
"futex_rla": 0,
"signals_p": {},
"sas": {
"ss_size": 0,
"ss_sp": 0,
"ss_flags": 2
}
},
"thread_info": {
"fpregs": {
"st_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"fop": 0,
"rdp": 0,
"twd": 0,
"mxcsr": 8064,
"swd": 0,
"rip": 0,
"xsave": {
"ymmh_space": [
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"xstate_bv": 2
},
"xmm_space": [
1953064751,
1769104175,
1702113141,
1664054387,
0,
0,
0,
0,
0,
4294901760,
255,
0,
792358505,
1953460082,
1852400175,
0,
942882145,
876295483,
774519349,
1031303283,
893073459,
976565307,
1937255978,
859661936,
993344312,
3814708,
65,
0,
21771440,
0,
21771552,
0,
0,
0,
0,
0,
4278190080,
4294967295,
4294967040,
4294967295,
0,
0,
0,
0,
0,
4278190080,
4294967040,
4294967295,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"cwd": 0,
"mxcsr_mask": 65535
},
"clear_tid_addr": 0,
"gpregs": {
"gs": 0,
"ip": 140137968588056,
"cx": 18446744073709551615,
"cs": 51,
"ax": 61,
"orig_ax": 61,
"di": 18446744073709551615,
"es": 0,
"gs_base": 0,
"r14": 0,
"r15": 4294967295,
"r12": 21789168,
"r13": 0,
"r10": 0,
"r11": 582,
"fs_base": 140137978173184,
"bp": 2560,
"dx": 0,
"bx": 0,
"ds": 0,
"ss": 43,
"sp": 140736135950616,
"r8": 21789168,
"r9": 0,
"fs": 0,
"si": 140736135950656,
"flags": 582
}
},
"tc": {
"timers": {
"real": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"virt": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
},
"prof": {
"isec": 0,
"vusec": 0,
"iusec": 0,
"vsec": 0
}
},
"cg_set": 1,
"signals_s": {},
"blk_sigset": 65536,
"exit_code": 0,
"rlimits": {
"rlimits": [
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 8388608
},
{
"max": 18446744073709551615,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 4096,
"cur": 1024
},
{
"max": 65536,
"cur": 65536
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
},
{
"max": 62845,
"cur": 62845
},
{
"max": 819200,
"cur": 819200
},
{
"max": 0,
"cur": 0
},
{
"max": 0,
"cur": 0
},
{
"max": 18446744073709551615,
"cur": 18446744073709551615
}
]
},
"comm": "loop.sh",
"flags": 1077960704,
"task_state": 1,
"personality": 0
}
}
]
}
</pre>
</div>
</div>

=== Generate core files out of task images ===

{{Status|not ready}}

All the needed information is in core-$pid.img and pagemap-$pid.img

=== On-the-fly conversion ===

{{Status|not ready}}

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

=== Convert between different image versions ===

{{Status|not ready}}

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

=== Show images statistics ===

{{Status|not ready}}

E.g. -- total number of processes, files, memory, sockets, etc.

=== Check/validate images ===

{{Status|not ready}}

Check that
* all images are present
* the inter-images IDs are in consistent state

[[Category:Images]]
[[Category:API]]

CRIT

2015-01-14T19:16:06Z

Efiop: fix examples

C API

2015-01-04T21:27:49Z

Efiop: /* Return values */

'''libcriu''' is a C API for CRIU, which is a simple wrapper around our [[RPC]]. Although you can use [[RPC]] directly, libcriu is a wrapper providing the interface that is much easier to use from C code. Note that [[RPC]] is supported in the first place, and if you want the most recent set of features you should probably use [[RPC]] directly.

== Introduction ==

libcriu functions are defined in <code>lib/criu.h</code>.

To create a library <code>lib/libcriu.so</code>, run <code>make</code> in the main directory.

The library is not thread- and fork- safe.

== Preparation ==

=== init options ===

Call <code>int criu_init_opts(void)</code> to initialize request options.

Note: it should be called before using any other functions from libcriu, except <code>criu_check()</code>. Also you should use it to reinitialize options. It returns 0 on success and -1 on fail.

=== set service address ===

Use <code>void criu_set_service_address(char *address)</code> to specify path to a CRIU service socket. Call it with NULL to make libcriu use the default address (currently <code>/var/run/criu_service.socket</code>).

=== set dump/restore options ===
Use <code>criu_set_*</code> functions to setup dump/restore options.

<pre>
void criu_set_pid(int pid);
void criu_set_images_dir_fd(int fd); /* must be set for dump/restore */
void criu_set_leave_running(bool leave_running);
void criu_set_ext_unix_sk(bool ext_unix_sk);
void criu_set_tcp_established(bool tcp_established);
void criu_set_evasive_devices(bool evasive_devices);
void criu_set_shell_job(bool shell_job);
void criu_set_file_locks(bool file_locks);
void criu_set_log_level(int log_level);
void criu_set_log_file(char *log_file);
</pre>

If no pid is set on dump, CRIU will dump the calling process itself.
{{Note|If a calling process is not run as root, the whole process tree to be dumped must have the same uid, otherwise CRIU refuses to dump. See [[Usage#Security]].}}

{{Note|<code>images_dir_fd</code> is '''required''' at dump/restore, all other options might be left unset.
The client must open directory for/with images by itself and set <code>images_dir_fd</code> to the opened directory fd.
CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>, so it will work even if the client is in another namespace.}}

The logic of setting request is the same as when setting options in console. Here is an example:

#criu restore -D /path/to/imgs_dir -v4 -o restore.log

is equal to:
<pre>
criu_init_opts();
criu_set_service_address("/path/to/criu/service/socket");

int fd = open("/path/to/imgs_dir", O_DIRECTORY);
criu_set_images_dir_fd(fd);

criu_set_log_file("restore.log");
criu_set_log_level(4);

criu_restore();
</pre>

== Operations ==

Use the following functions to perform CRIU actions.

=== check ===

int criu_check(void);

=== dump ===

int criu_dump(void);

=== restore ===

int criu_restore(void);

=== Return values ===

Here is a table of return and errno values of the above functions:
{| class="wikitable"
|-
! Return value
! errno
! Description

|-
| 0
| undefined
| Success

|-
| >0
| undefined
| Success (<code>criu_restore()</code> only)

|-
| -EBADE
| RPC error (if provided(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]), 0 otherwise)
| RPC has returned fail.

|-
| -ECONNREFUSED
| errno
| Unable to connect to CRIU.

|-
| -ECOMM
| errno
| Unable to send/recv msg to/from CRIU.

|-
| -EINVAL
| undefined
| CRIU doesn't support this type of request. You should probably update CRIU.

|-
| -EBADMSG
| undefined
| Unexpected response from CRIU. You should probably update CRIU.
|}

== Examples ==

You could find example of using libcriu at [http://git.criu.org/?p=criu.git;a=tree;f=test/libcriu test/libcriu].

[[Category: API]]

RPC

2015-01-04T21:26:19Z

Efiop: Mention cr_errno field.

CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU service launched with <code>criu service</code> command. It uses a SEQPACKET Unix domain socket listening at <code>/var/run/criu-service.socket</code> as a transport.

The criu_req/criu_resp are two main messages for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them ''must'' be set accordingly to type of request/response that is stored. Types of request/response are defined in enum criu_req_type. See the [[API compliance]] page for information what each option might mean.

== Protocol ==

The protocol is simple -- client connect to service, send a ''criu_req'' message to server, server responds back with ''criu_resp'' and closes the socket. There are three exceptions from this rule, see below.

== Request ==

This is the header of the request. It defines the operation requested and options.

<pre>
message criu_req {
required criu_req_type type = 1;
optional criu_opts opts = 2;
optional notify_success = 3; /* see Notifications below */
optional keep_open = 4; /* for multi-req, below */
}
</pre>

There are 8 request/response types for now.

<pre>enum criu_req_type {
EMPTY = 0;
DUMP = 1; /* criu dump */
RESTORE = 2; /* criu restore */
CHECK = 3; /* criu check */
PRE_DUMP = 4; /* criu pre-dump */
PAGE_SERVER = 5; /* criu page-server */
NOTIFY = 6; /* see Notifications below */
CPUINFO_DUMP = 7; /* criu cpuinfo dump */
CPUINFO_CHECK = 8; /* criu cpuinfo check */
}
</pre>

<pre>message criu_opts {
required int32 images_dir_fd = 1;
optional int32 pid = 2; /* if not set on dump, will dump requesting process */

optional bool leave_running = 3;
optional bool ext_unix_sk = 4;
optional bool tcp_established = 5;
optional bool evasive_devices = 6;
optional bool shell_job = 7;
optional bool file_locks = 8;
optional int32 log_level = 9 [default = 2];
optional string log_file = 10; /* No subdirs are allowed. Consider using work-dir */

optional criu_page_server_info ps = 11;

optional bool notify_scripts = 12;

optional string root = 13;
optional string parent_img = 14;
optional bool track_mem = 15;
optional bool auto_dedup = 16;

optional int32 work_dir_fd = 17;
optional bool link_remap = 18;
repeated criu_veth_pair veths = 19;

optional uint32 cpu_cap = 20 [default = 0xffffffff];
optional bool force_irmap = 21;
repeated string exec_cmd = 22;

repeated ext_mount_map ext_mnt = 23;
optional bool manage_cgroups = 24;
repeated cgroup_root cg_root = 25;

optional bool rst_sibling = 26; /* swrk only */
}
</pre>

=== Comments and examples ===

* If no pid is set and type is DUMP, CRIU will dump client process by default.
* All processes in the subtree starting with <pid> must have the same uid, as a client, or client's uid must be root (uid == 0), otherwise CRIU will return error.
* Only the images_dir_fd is required, all other fields are optional. Client must open directory for/with images by himself and set images_dir_fd to it's fd. CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>.

The logic of setting request is the same as when setting options in console. Here is an example:
<pre>#criu restore -D /path/to/imgs_dir -v4 -o restore.log</pre>
is equal to:
<pre>
request.type = RESTORE;

request.opts.imgs_dir_fd = open("/path/to/imgs_dir")
request.opts.log_level = 4
request.opts.log_file = "restore.log"
</pre>

=== Sub-messages for options ===
; Info about page-server.

<pre>
message criu_page_server_info {
optional string address = 1; /* bind address -- if not set 0.0.0.0 is used */
optional int32 port = 2; /* bind port -- if not set on request, autobind is used and port is returned in response */
optional int32 pid = 3; /* page-server pid -- returned in response */
optional int32 fd = 4; /* could be used to inherit fd by page-server */
}
</pre>

* If ''port'' is 0 server will auto-bind and the port will be returned back in responce.

<pre>
message criu_veth_pair {
required string if_in = 1; /* inside veth device name */
required string if_out = 2; /* outside veth device name */
};
</pre>

; Info about veth mappings (--ext-mount-map analogue)
<pre>
message ext_mount_map {
required string key = 1;
required string val = 2;
};
</pre>

; Specifying where cgroup root should be (--cgroup-root analogue)
<pre>
message cgroup_root {
optional string ctrl = 1;
required string path = 2;
};
</pre>

== Response ==

This message is sent after (un)successful execution of the request.

<pre>
message criu_resp {
required criu_req_type type = 1;
required bool success = 2;

optional criu_dump_resp dump = 3;
optional criu_restore_resp restore = 4;
optional criu_notify notify = 5;
optional criu_page_server_info ps = 6;

optional int32 cr_errno = 7;
}
</pre>
Field "success" reports result of processing request, while criu_***_resp store some request-specific information. The response type is set to the corresponding request type or to <code>EMPTY</code> to report a "generic" error. If success == false one should check cr_errno field to get a more detailed error code(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]).

;The criu_dump_resp is used to store response from DUMP request.
<pre>
message criu_dump_resp {
optional bool restored = 1;
}
</pre>

This message can be sent twice -- one for the process that calls DUMP and the other for the same process again if it requested a self-dump. In the latter case the ''restored'' field would be true.

;The response on RESTORE request.
<pre>
message criu_restore_resp {
required int32 pid = 1;
}
</pre>
Field "pid" is set to the PID of the newly restored process.

;Info about page server
The ''criu_page_server_info'' from requests will be sent back on PAGE_SERVER request. The ''port'' field will contain the port to which server is bound.

=== Notifications ===
If the ''opts.notify_scripts'' in the request is set to TRUE CRIU would report back resp messages with type set to NOTIFY and this field present. The notifications are the way [[action scripts]] work for RPC mode.

<pre>
message criu_notify {
optional string script = 1;
optional int32 pid = 2;
}
</pre>

After handling the notification the client must response with the request again with the type set to NOTIFY and the ''notify_success'' set to the successfulness of the notification. In case of successful notification acknowledge the server doesn't close the socket and continues to work.

== Pre-dumps ==

Before issuing a DUMP request client may send one or more PRE_DUMP requests. Once the PRE_DUMP is sent and response is received, client may send one more PRE_DUMP or DUMP request. The server would only close the socket after the DUMP one.

== Multi-request mode ==

If the ''req.keep_open'' flag is set to true server will not close the socket after response, but will wait for more requests. This mode is supported only for the following request types:

* PRE_DUMP (automatically)
* PAGE_SERVER
* CPUINFO_DUMP and CPUINFO_CHECK

== Run ==
=== Server ===
On a server side, CRIU creates SOCK_SEQPACKET Unix socket and listens for connections on it. After receiving criu_req, CRIU processes it, do what is requested and sends criu_resp with set request-specific criu_***_resp field back.
If CRIU gets unknown type of request, it will return criu_resp with type == EMPTY and success == false.

To launch service server, run:

<pre>#criu service [options]</pre>

Options accepted by service are

; --address <path>
: is where to put listening socket

; --pid-file <path>
: is where to write pid of service process

; --daemon
: tells service to daemonize

; -o <file>
: says where to write logs

; -v[N]
: sets the log-level

====systemd====
If you are running systemd you can make service start and operate automatically. First do
<pre>
make install
</pre>
to make files <code>criu.service</code> and <code>criu.socket</code> appear in systemd configs (<code>/lib/systemd/system/</code>). Then

<pre>
systemctl start criu.socket
</pre>

to get /var/run/criu-service.socket, and
<pre>
systemctl enable criu.socket
</pre>

to make /var/run/criu-service.socket available at boot.

=== Client ===
Client, in its turn, must connect to service socket, send criu_req with request in it, and wait for a criu_resp with response.
You can find examples of client programs in C and Python in test/rpc/.

With RPC facilities one can perform a [[self dump]].

There's a [[C_API|library]] that implements simple wrappers on top of RPC.

[[Category: API]]

Todo

2015-01-01T09:35:22Z

Efiop: note that we have cr-errno with 3 typical errors and we need to expand that list

{| class="wikitable sortable"
|-
! component
! task
! complexity
! assignee
! comments
|-
| crtools || Rollback tree state || medium || - ||
|-
| crtools || AIO with pending events || medium || - ||
|-
| crtools || restore mountpoints || hard || - || -
|-
| crtools || lazy migration using userfaultd || medium || - || -
|-
| crtools || dump tasks from cgroup || easy || - || Currently criu dumps a subtree from given pid
|-
| crtools || speed up logging || medium || - || Synchronous writes to log files slow things down
|-
| crtools || sanitize logging messages || hard || - || Currently log messages are placed w/o any logic. Need to improve that ... somehow.
|-
| crtools || support OFD posix locks || easy || - ||
|-
| crtools || optimize kcmp calls || medium || - || For fs/fdt/vm/etc we can pre-generate ID based on objects that live on them. If pre-ID matches, then call kcmp.
|-
| crtools || Shmem changes tracking || medium || - ||
|-
| crtools || Page transfer filters || medium || Ruslan Kuprieiev || Packing or encrypting the data would be nice. Maybe it's purely for [[P.Haul]]?
|-
| crtools || FUSE || hard || - ||
|-
| crtools || 32-bit tasks || hard || Cyrill ||
|-
| crtools || Support one of MPI implementations || hard || Adrian Reber || worth starting with OpenMPI
|-
| crtools || Generate task's core file out of images || medium || - || [[CRIT]]
|-
| crtools || Convert images to JSON and back || easy || - || [[CRIT]]
|-
| crtools || Modify restored resources run-time || medium || - || [[CRIT]] Need (probably) some way to alter what is being restored. Usage example -- change the IP address of sockets from task above.
|-
| crtools || TCP socket migration with changed IP || medium || - || [[CRIT]] it might make sense to migrate a tcp connection on a box with changed IP address _if_ both boxes are NAT-ed to the destination. We will then have to go to NAT box and fix the conntracks, but this might make sense.
|-
| crtools || [[Applying images]] || hard || xemul@ w/ students || Think about ability to take images and apply them to a living task(s). E.g. -- repopulate fdtable according to data from image. Another use-case -- when doing partial migration we'll need to modify one part to switch from pipes to sockets
|-
| crtools || Partial migration || hard || - || migrate some tasks while proxying IPC to existing others (pipes->sockets, etc.)
|-
| crtools || Shared objects (mm/fs) support || medium || - || Now we have the kcmp syscall and can do it. The first candidate is mm sharing, as we do know, that MySQL does so sometimes.
|-
| crtools || Smart paths resolution || hard || - || Need a way to resolve paths to overmounted files. There are two ways: 1. Move mounts, that overlap the desired path temprarily. 2. When creating a new mount pre-open an fd keeping the mountpoint. Later, do accurate path resolve and call openat() on proper mountpoint fd
|-
| kernel/crtools || TCP repair fixes || hard || - || [[TCP repair TODO]]
|-
| crtools || Bridges in container || medium || - ||
|-
| crtools || VLANs in containers || medium || - ||
|-
| crtools || PPP support || medium || - ||
|-
| crtools/kernel || DelayFS for NFS/Fuse migration || hard || - ||
|-
| kernel || Seamless kernel upgrade || hard || Vladimir Davydov ||
|-
| crtools/kernel || NFS || hard || - || Probably depends on DelayFS
|-
| crtools || Validate .img files || easy || - || [[CRIT]] For a given set of image files check, that they are in "restorable" shape
|-
| crtools || Restore a process tree || hard || - || Need to restore any process tree, which could be created with help PR_SET_CHILD_SUBREAPER and CLONE_PARENT. Processes can share other resources [http://man7.org/linux/man-pages/man2/clone2.2.html clone(2)]. Look at [http://git.criu.org/?p=crtools.git;a=blob;f=test/zdtm/live/static/session02.c;hb=HEAD session02]
|-
| crtools || X app || hard || - || Dump/restore of graphical applications
|-
| crtools/kernel || Undo semaphores || medium || Cyrill Gorcunov || Require mods from both sides -- criu and kernel
|-
| crtools || More detailed RPC fail codes || easy || - || Currently only 3 typical errors are reported(see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h])
|-
| kernel/criu || fsnotify queues || hard || - || [[Fsnotify]]
|}

[[Category:Development]]
[[Category:Plans]]

Images

2014-12-24T11:04:32Z

Efiop:

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

{{Note| You might also want to checkout our image tool called [[CRIT]]}}

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || - || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
{{Note| Even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry. }}

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]
[[Category:Images]]

What's bad with V1 images

2014-12-18T17:44:25Z

Efiop:

Current [[images]] format is great, but sometimes we find problems with it. This page collects all we don't like about them and would like to change some time in the future (once we change this, image format version will be changed as well).

* vma.proto vma_entry->fd is unused
* ipc-msg.proto ipc_msg/ipc_msg_entry names mess
* sk-packet.proto vs packet-sock.proto names mess (all sockets .proto files are sk-<something>)
* core.proto blk_sigset for master thread is on task_core_entry, but should be in respective thread_core_entry
* pstree.proto: sid and pgid should NOT be there, but in core image
* core.proto -- padding for fpu is not used
* what to do with absent image files? Backward compatibility requires we still restore, common sense -- abort. Flooding inventory with every single new image doesn't sound that great.
* core.proto -- unused (though optional) ids field
* clear_thread_info in per-arch members
* it would be great to unify inotify and fanotify images into one fsnotify file
* currently we write checkpoint files with '''*.img''' extension, which might be confusing since this extension is a way overloaded by meaning and quite widespreaded, thus it might be worth to change extension to '''*.criu'''
* mnt_entry::fstype (in mnt.proto) should have enum fstype type
* all IDs in image should be in uint64 format (not uint32, like in reg_file_entry), this allows us to use unified ID generator over all images
* IPC mqueue sysctls are required, while in kernel they can be missing
* In proto files for registers (and more) it's better to use fixed- types, rather than plain
* use fixed32 and fixed64 where approiate instead of uint32 and uint64 https://developers.google.com/protocol-buffers/docs/proto
* flock's flag and type fields is actually too much for what is stored there
* remap image uses older bit od ID to distinguish one remap type from another, while it should be a typed field
* evenpoll's tfd ID field is always zero
* Images structure is not natural. Some contain pb entries of a same type, some have a header of one pb type and then an array of another type or some raw data.

[[Category:Development]]
[[Category:Images]]

Packages

2014-12-14T15:45:45Z

Efiop: add link for arch people to vote for enabling CONFIG_CHECKPOINT_RESTORE

This page summarizes our knowledge about CRIU packages in various Linux distributions.
<big>
* '''Debian''': http://packages.qa.debian.org/c/criu.html

* '''Fedora''': https://apps.fedoraproject.org/packages/criu

* '''Ubuntu''': https://bugs.launchpad.net/ubuntu/+source/crtools/+bug/1256675 (currently just a request for updating)

* '''Arch''': https://aur.archlinux.org/packages/crtools-git/ ([https://lists.archlinux.org/pipermail/arch-dev-public/2014-April/026102.html CONFIG_CHECKPOINT_RESTORE is disabled since v3.14], vote [https://bugs.archlinux.org/task/43088 here] for enabling it again)

* '''OpenSUSE''': https://build.opensuse.org/package/show/openSUSE:Factory/criu#

* '''Gentoo''': http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-process/criu/

* '''CoreOS''': https://github.com/coreos/coreos-overlay/tree/master/sys-process/criu

* '''AltLinux''': http://packages.altlinux.org/crtools
</big>

RPC

2014-12-01T14:01:25Z

Efiop:

CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU service launched with <code>criu service</code> command. It uses a SEQPACKET Unix domain socket listening at <code>/var/run/criu-service.socket</code> as a transport.

== Protobuf messages ==
criu_req/criu_resp -- wrappers for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them _must_ be set accordingly to type of request/response that is stored. Types of request/response are defined in enum criu_req_type.

=== Request ===
==== criu_req ====
<pre>
message criu_req {
required criu_req_type type = 1;

optional criu_opts opts = 2;
}
</pre>

==== criu_req_type ====
There are 8 request/response types for now. Comments show cmdline analogs.
<pre>enum criu_req_type {
EMPTY = 0;
DUMP = 1; /* criu dump */
RESTORE = 2; /* criu restore */
CHECK = 3; /* criu check */
PRE_DUMP = 4; /* criu pre-dump */
PAGE_SERVER = 5; /* criu page-server */

NOTIFY = 6;

CPUINFO_DUMP = 7; /* criu cpuinfo dump */
CPUINFO_CHECK = 8; /* criu cpuinfo check */
}
</pre>

==== criu_opts ====
It is used to store options.
<pre>message criu_opts {
required int32 images_dir_fd = 1;
optional int32 pid = 2; /* if not set on dump, will dump requesting process */

optional bool leave_running = 3;
optional bool ext_unix_sk = 4;
optional bool tcp_established = 5;
optional bool evasive_devices = 6;
optional bool shell_job = 7;
optional bool file_locks = 8;
optional int32 log_level = 9 [default = 2];
optional string log_file = 10; /* No subdirs are allowed. Consider using work-dir */

optional criu_page_server_info ps = 11;

optional bool notify_scripts = 12;

optional string root = 13;
optional string parent_img = 14;
optional bool track_mem = 15;
optional bool auto_dedup = 16;

optional int32 work_dir_fd = 17;
optional bool link_remap = 18;
repeated criu_veth_pair veths = 19;

optional uint32 cpu_cap = 20 [default = 0xffffffff];
optional bool force_irmap = 21;
repeated string exec_cmd = 22;

repeated ext_mount_map ext_mnt = 23;
optional bool manage_cgroups = 24;
repeated cgroup_root cg_root = 25;

optional bool rst_sibling = 26; /* swrk only */
}</pre>

If no pid is set and type is DUMP, CRIU will dump client process by default.
Note: Whole tree <pid> must have the same uid, as a client, or client's uid must be == 0, otherwise CRIU won't dump nothing at all.

Only images_dir_fd is required, all other fields may not be set.
Client must open directory for/with images by himself and set images_dir_fd to it's fd.
CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>, so it will work, if client is in another namespace.

The logic of setting request is the same as when setting options in console. Here is an example:
<pre>#criu restore -D /path/to/imgs_dir -v4 -o restore.log</pre>
is equal to:
<pre>
request.type = RESTORE;

request.opts.imgs_dir_fd = open("/path/to/imgs_dir")
request.opts.log_level = 4
request.opts.log_file = "restore.log"
</pre>

===== criu_page_server_info =====
Stores info about page-server.
<pre>
message criu_page_server_info {
optional string address = 1; /* bind address -- if not set 0.0.0.0 is used */
optional int32 port = 2; /* bind port -- if not set on request, autobind is used and port is returned in response */
optional int32 pid = 3; /* page-server pid -- returned in response */
optional int32 fd = 4; /* could be used to inherit fd by page-server */
}
</pre>
===== criu_veth_pair =====
--veth-pair analog.
<pre>
message criu_veth_pair {
required string if_in = 1; /* inside veth device name */
required string if_out = 2; /* outside veth device name */
};
</pre>
===== ext_mount_map =====
--ext-mount-map analog.
<pre>
message ext_mount_map {
required string key = 1;
required string val = 2;
};
</pre>
===== cgroup_root =====
--cgroup-root analog.
<pre>
message cgroup_root {
optional string ctrl = 1;
required string path = 2;
};
</pre>
=== Response ===
==== criu_resp ====
<pre>message criu_resp {
required criu_req_type type = 1;
required bool success = 2;

optional criu_dump_resp dump = 3;
optional criu_restore_resp restore = 4;
optional criu_notify notify = 5;
optional criu_page_server_info ps = 6;
}
</pre>
Field "success" reports result of processing request, while criu_***_resp store some request-specific information. The response type is set to the corresponding request type or to <code>EMPTY</code> to report a "generic" error.
==== criu_dump_resp ====
criu_dump_resp is used to store dump response from CRIU.
<pre>
message criu_dump_resp {
optional bool restored = 1;
}
</pre>

Field "restored" is set to "true" if process was restored.

==== criu_restore_resp ====
<pre>
message criu_restore_resp {
required int32 pid = 1;
}
</pre>
Field "pid" is set to the PID of the restored process.

==== criu_notify ====
--action-script analog.
<pre>
message criu_notify {
optional string script = 1;
optional int32 pid = 2;
}
</pre>
== Run ==
=== Server ===
On a server side, CRIU creates SOCK_SEQPACKET Unix socket and listens for connections on it. After receiving criu_req, CRIU processes it, do what is requested and sends criu_resp with set request-specific criu_***_resp field back.
If CRIU gets unknown type of request, it will return criu_resp with type == EMPTY and success == false.

To launch service server, run:

<pre>#criu service [options]</pre>

Options accepted by service are

; --address <path>
: is where to put listening socket

; --pid-file <path>
: is where to write pid of service process

; --daemon
: tells service to daemonize

; -o <file>
: says where to write logs

; -v[N]
: sets the log-level

====systemd====
If you are running systemd you can make service start and operate automatically. First do
<pre>
make install
</pre>
to make files <code>criu.service</code> and <code>criu.socket</code> appear in systemd configs (<code>/lib/systemd/system/</code>). Then

<pre>
systemctl start criu.socket
</pre>

to get /var/run/criu-service.socket, and
<pre>
systemctl enable criu.socket
</pre>

to make /var/run/criu-service.socket available at boot.

=== Client ===
Client, in its turn, must connect to service socket, send criu_req with request in it, and wait for a criu_resp with response.
You can find examples of client programs in C and Python in test/rpc/.

With RPC facilities one can perform a [[self dump]].

There's a [[C_API|library]] that implements simple wrappers on top of RPC.

== Security ==

See [[Security]].

RPC

2014-12-01T12:17:03Z

Efiop: Add cmdline analogs.

CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU service launched with <code>criu service</code> command. It uses a SEQPACKET Unix domain socket listening at <code>/var/run/criu-service.socket</code> as a transport.

== Protobuf messages ==
criu_req/criu_resp -- wrappers for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them _must_ be set accordingly to type of request/response that is stored. Types of request/response are defined in enum criu_req_type.

=== Request ===
==== criu_req ====
<pre>
message criu_req {
required criu_req_type type = 1;

optional criu_opts opts = 2;
}
</pre>

==== criu_req_type ====
There are 8 request/response types for now. Comments show cmdline analogs.
<pre>enum criu_req_type {
EMPTY = 0;
DUMP = 1; /* criu dump */
RESTORE = 2; /* criu restore */
CHECK = 3; /* criu check */
PRE_DUMP = 4; /* criu pre-dump */
PAGE_SERVER = 5; /* criu page-server */

NOTIFY = 6;

CPUINFO_DUMP = 7; /* criu cpuinfo dump */
CPUINFO_CHECK = 8; /* criu cpuinfo check */
}
</pre>

==== criu_opts ====
It is used to store options.
<pre>message criu_opts {
required int32 images_dir_fd = 1;
optional int32 pid = 2; /* if not set on dump, will dump requesting process */

optional bool leave_running = 3;
optional bool ext_unix_sk = 4;
optional bool tcp_established = 5;
optional bool evasive_devices = 6;
optional bool shell_job = 7;
optional bool file_locks = 8;
optional int32 log_level = 9 [default = 2];
optional string log_file = 10; /* No subdirs are allowed. Consider using work-dir */

optional criu_page_server_info ps = 11;

optional bool notify_scripts = 12;

optional string root = 13;
optional string parent_img = 14;
optional bool track_mem = 15;
optional bool auto_dedup = 16;

optional int32 work_dir_fd = 17;
optional bool link_remap = 18;
repeated criu_veth_pair veths = 19;

optional uint32 cpu_cap = 20 [default = 0xffffffff];
optional bool force_irmap = 21;
repeated string exec_cmd = 22;

repeated ext_mount_map ext_mnt = 23;
optional bool manage_cgroups = 24;
repeated cgroup_root cg_root = 25;

optional bool rst_sibling = 26; /* swrk only */
}</pre>

If no pid is set and type is DUMP, CRIU will dump client process by default.
Note: Whole tree <pid> must have the same uid, as a client, or client's uid must be == 0, otherwise CRIU won't dump nothing at all.

Only images_dir_fd is required, all other fields may not be set.
Client must open directory for/with images by himself and set images_dir_fd to it's fd.
CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>, so it will work, if client is in another namespace.

The logic of setting request is the same as when setting options in console. Here is an example:
<pre>#criu restore -D /path/to/imgs_dir -v4 -o restore.log</pre>
is equal to:
<pre>
request.type = RESTORE;

request.opts.imgs_dir_fd = open("/path/to/imgs_dir")
request.opts.log_level = 4
request.opts.log_file = "restore.log"
</pre>

===== criu_page_server_info =====
Stores info about page-server.
<pre>
message criu_page_server_info {
optional string address = 1; /* bind address -- if not set 0.0.0.0 is used */
optional int32 port = 2; /* bind port -- if not set on request, autobind is used and port is returned in response */
optional int32 pid = 3; /* page-server pid -- returned in response */
optional int32 fd = 4; /* could be used to inherit fd by page-server */
}
</pre>
===== criu_veth_pair =====
--veth-pair analog.
<pre>
message criu_veth_pair {
required string if_in = 1; /* inside veth device name */
required string if_out = 2; /* outside veth device name */
};
</pre>
===== ext_mount_map =====
--ext-mount-map analog.
<pre>
message ext_mount_map {
required string key = 1;
required string val = 2;
};
</pre>
===== cgroup_root =====
--cgroup-root analog.
<pre>
message cgroup_root {
optional string ctrl = 1;
required string path = 2;
};
</pre>
=== Response ===
==== criu_resp ====
<pre>message criu_resp {
required criu_req_type type = 1;
required bool success = 2;

optional criu_dump_resp dump = 3;
optional criu_restore_resp restore = 4;
optional criu_notify notify = 5;
optional criu_page_server_info ps = 6;
}
</pre>
Field "success" reports result of processing request, while criu_***_resp store some request-specific information. The response type is set to the corresponding request type or to <code>EMPTY</code> to report a "generic" error.
==== criu_dump_resp ====
criu_dump_resp is used to store dump response from CRIU.
<pre>
message criu_dump_resp {
optional bool restored = 1;
}
</pre>

Field "restored" is set to "true" if process was restored.

==== criu_restore_resp ====
<pre>
message criu_restore_resp {
required int32 pid = 1;
}
</pre>
Field "pid" is set to the PID of the restored process.

==== criu_notify ====
TODO
<pre>
message criu_notify {
optional string script = 1;
optional int32 pid = 2;
}
</pre>
== Run ==
=== Server ===
On a server side, CRIU creates SOCK_SEQPACKET Unix socket and listens for connections on it. After receiving criu_req, CRIU processes it, do what is requested and sends criu_resp with set request-specific criu_***_resp field back.
If CRIU gets unknown type of request, it will return criu_resp with type == EMPTY and success == false.

To launch service server, run:

<pre>#criu service [options]</pre>

Options accepted by service are

; --address <path>
: is where to put listening socket

; --pid-file <path>
: is where to write pid of service process

; --daemon
: tells service to daemonize

; -o <file>
: says where to write logs

; -v[N]
: sets the log-level

====systemd====
If you are running systemd you can make service start and operate automatically. First do
<pre>
make install
</pre>
to make files <code>criu.service</code> and <code>criu.socket</code> appear in systemd configs (<code>/lib/systemd/system/</code>). Then

<pre>
systemctl start criu.socket
</pre>

to get /var/run/criu-service.socket, and
<pre>
systemctl enable criu.socket
</pre>

to make /var/run/criu-service.socket available at boot.

=== Client ===
Client, in its turn, must connect to service socket, send criu_req with request in it, and wait for a criu_resp with response.
You can find examples of client programs in C and Python in test/rpc/.

With RPC facilities one can perform a [[self dump]].

There's a [[C_API|library]] that implements simple wrappers on top of RPC.

== Security ==

See [[Security]].

RPC

2014-12-01T12:00:22Z

Efiop: Refresh info.

CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU service launched with <code>criu service</code> command. It uses a SEQPACKET Unix domain socket listening at <code>/var/run/criu-service.socket</code> as a transport.

== Protobuf messages ==
criu_req/criu_resp -- wrappers for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them _must_ be set accordingly to type of request/response that is stored. Types of request/response are defined in enum criu_req_type.

=== Request ===
==== criu_req ====
<pre>
message criu_req {
required criu_req_type type = 1;

optional criu_opts opts = 2;
}
</pre>

==== criu_req_type ====
There are 8 request/response types for now. Comments show cmdline analogs.
<pre>enum criu_req_type {
EMPTY = 0;
DUMP = 1; /* criu dump */
RESTORE = 2; /* criu restore */
CHECK = 3; /* criu check */
PRE_DUMP = 4; /* criu pre-dump */
PAGE_SERVER = 5; /* criu page-server */

NOTIFY = 6;

CPUINFO_DUMP = 7; /* criu cpuinfo dump */
CPUINFO_CHECK = 8; /* criu cpuinfo check */
}
</pre>

==== criu_opts ====
It is used to store options.
<pre>message criu_opts {
required int32 images_dir_fd = 1;
optional int32 pid = 2; /* if not set on dump, will dump requesting process */

optional bool leave_running = 3;
optional bool ext_unix_sk = 4;
optional bool tcp_established = 5;
optional bool evasive_devices = 6;
optional bool shell_job = 7;
optional bool file_locks = 8;
optional int32 log_level = 9 [default = 2];
optional string log_file = 10; /* No subdirs are allowed. Consider using work-dir */

optional criu_page_server_info ps = 11;

optional bool notify_scripts = 12;

optional string root = 13;
optional string parent_img = 14;
optional bool track_mem = 15;
optional bool auto_dedup = 16;

optional int32 work_dir_fd = 17;
optional bool link_remap = 18;
repeated criu_veth_pair veths = 19;

optional uint32 cpu_cap = 20 [default = 0xffffffff];
optional bool force_irmap = 21;
repeated string exec_cmd = 22;

repeated ext_mount_map ext_mnt = 23;
optional bool manage_cgroups = 24;
repeated cgroup_root cg_root = 25;

optional bool rst_sibling = 26; /* swrk only */
}</pre>

If no pid is set and type is DUMP, CRIU will dump client process by default.
Note: Whole tree <pid> must have the same uid, as a client, or client's uid must be == 0, otherwise CRIU won't dump nothing at all.

Only images_dir_fd is required, all other fields may not be set.
Client must open directory for/with images by himself and set images_dir_fd to it's fd.
CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>, so it will work, if client is in another namespace.

The logic of setting request is the same as when setting options in console. Here is an example:
<pre>#criu restore -D /path/to/imgs_dir -v4 -o restore.log</pre>
is equal to:
<pre>
request.type = RESTORE;

request.opts.imgs_dir_fd = open("/path/to/imgs_dir")
request.opts.log_level = 4
request.opts.log_file = "restore.log"
</pre>

===== criu_page_server_info =====
Stores info about page-server.
<pre>
message criu_page_server_info {
optional string address = 1; /* bind address -- if not set 0.0.0.0 is used */
optional int32 port = 2; /* bind port -- if not set on request, autobind is used and port is returned in response */
optional int32 pid = 3; /* page-server pid -- returned in response */
optional int32 fd = 4; /* could be used to inherit fd by page-server */
}
</pre>
===== criu_veth_pair =====
TODO
<pre>
message criu_veth_pair {
required string if_in = 1;
required string if_out = 2;
};
</pre>
===== ext_mount_map =====
TODO
<pre>
message ext_mount_map {
required string key = 1;
required string val = 2;
};
</pre>
===== cgroup_root =====
TODO
<pre>
message cgroup_root {
optional string ctrl = 1;
required string path = 2;
};
</pre>
=== Response ===
==== criu_resp ====
<pre>message criu_resp {
required criu_req_type type = 1;
required bool success = 2;

optional criu_dump_resp dump = 3;
optional criu_restore_resp restore = 4;
optional criu_notify notify = 5;
optional criu_page_server_info ps = 6;
}
</pre>
Field "success" reports result of processing request, while criu_***_resp store some request-specific information. The response type is set to the corresponding request type or to <code>EMPTY</code> to report a "generic" error.
==== criu_dump_resp ====
criu_dump_resp is used to store dump response from CRIU.
<pre>
message criu_dump_resp {
optional bool restored = 1;
}
</pre>

Field "restored" is set to "true" if process was restored.

==== criu_restore_resp ====
<pre>
message criu_restore_resp {
required int32 pid = 1;
}
</pre>
Field "pid" is set to the PID of the restored process.

==== criu_notify ====
TODO
<pre>
message criu_notify {
optional string script = 1;
optional int32 pid = 2;
}
</pre>
== Run ==
=== Server ===
On a server side, CRIU creates SOCK_SEQPACKET Unix socket and listens for connections on it. After receiving criu_req, CRIU processes it, do what is requested and sends criu_resp with set request-specific criu_***_resp field back.
If CRIU gets unknown type of request, it will return criu_resp with type == EMPTY and success == false.

To launch service server, run:

<pre>#criu service [options]</pre>

Options accepted by service are

; --address <path>
: is where to put listening socket

; --pid-file <path>
: is where to write pid of service process

; --daemon
: tells service to daemonize

; -o <file>
: says where to write logs

; -v[N]
: sets the log-level

====systemd====
If you are running systemd you can make service start and operate automatically. First do
<pre>
make install
</pre>
to make files <code>criu.service</code> and <code>criu.socket</code> appear in systemd configs (<code>/lib/systemd/system/</code>). Then

<pre>
systemctl start criu.socket
</pre>

to get /var/run/criu-service.socket, and
<pre>
systemctl enable criu.socket
</pre>

to make /var/run/criu-service.socket available at boot.

=== Client ===
Client, in its turn, must connect to service socket, send criu_req with request in it, and wait for a criu_resp with response.
You can find examples of client programs in C and Python in test/rpc/.

With RPC facilities one can perform a [[self dump]].

There's a [[C_API|library]] that implements simple wrappers on top of RPC.

== Security ==

See [[Security]].

Images

2014-11-26T19:37:57Z

Efiop: Mention CRIT

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

{{Note| You might also want to checkout our image tool called [[CRIT]]}}

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

Or, you can visualize it like

{| class="wikitable"
|-
! Type !! Size(bytes)
|-
| Magic || 4
|-
| Size0 || 4
|-
| Message0 || Size0
|-
| ... || ...
|-
| SizeN || 4
|-
| MessageN || SizeN
|}

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || - || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
{{Note| Even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry. }}

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]

CRIT

2014-11-24T19:27:10Z

Efiop: typo

This page describes the CRiu Image Tool.
{{Note|As for now(24 Nov 2014) it is not added to the official repo, but you could checkout it [https://github.com/efiop/criu/tree/crit-test here].}}

== Usage ==
<pre>Usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] [-t {json}] {convert}

CRiu Image Tool

positional arguments:
{convert} use "covert" to convert CRIU image to/from human-
readable format

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default
-f {raw,nice}, --format {raw,nice}
well-formated output (by default: raw for files and
nice for stdout)
-t {json}, --type {json}
Type of output file (by default img is converted to
txt and vice versa)
</pre>

== Functionality ==

=== Convert images to human-readable and back ===
This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

'''Status: Ready'''<br>
It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.
Output file is structured like:
With "--format raw":
<pre>
#<magic value>
#0
<pb entry in text_format>
....
#N
<pb entry in text_format>
</pre>
With "--format nice":
<pre>
#<magic name>
#0 <pb entry name>
<pb entry in text_format with tabs>
....
#N <pb entry name>
<pb entry in text_format with tabs>
</pre>

So, here is how core file will look like with --format nice:
<pre>
#CORE
#0 core_entry
mtype: X86_64
thread_info {
clear_tid_addr: 0
gpregs {
r15: 0
r14: 19884608
r13: 1
r12: 0
bp: 19879232
bx: 0
r11: 582
r10: 0
r9: 0
r8: 19879232
ax: 61
cx: 18446744073709551615
dx: 0
si: 140733241187376
di: 18446744073709551615
orig_ax: 61
ip: 213251769082
cs: 51
flags: 582
sp: 140733241187336
ss: 43
fs_base: 140140334651200
gs_base: 0
ds: 0
es: 0
fs: 0
gs: 0
}
fpregs {
cwd: 895
swd: 0
twd: 0
fop: 0
rip: 0
rdp: 0
mxcsr: 8064
mxcsr_mask: 65535
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 16777215
xmm_space: 16843009
xmm_space: 16843009
xmm_space: 16843009
xmm_space: 4478800
xmm_space: 661857632
xmm_space: 1869488186
xmm_space: 543236212
xmm_space: 1970037624
xmm_space: 1680696676
xmm_space: 775778921
xmm_space: 544500071
xmm_space: 2019896621
xmm_space: 1685417059
xmm_space: 1768172901
xmm_space: 1747860850
xmm_space: 809328999
xmm_space: 909327153
xmm_space: 1932405306
xmm_space: 809334896
xmm_space: 909327153
xmm_space: 2016291386
xmm_space: 1030123635
xmm_space: 859517232
xmm_space: 14902
xmm_space: 0
xmm_space: 65
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 4294967040
xmm_space: 4294967295
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
}
}
tc {
task_state: 1
exit_code: 0
personality: 0
flags: 1077960704
blk_sigset: 65536
comm: "loop.sh"
timers {
real {
isec: 0
iusec: 0
vsec: 0
vusec: 0
}
virt {
isec: 0
iusec: 0
vsec: 0
vusec: 0
}
prof {
isec: 0
iusec: 0
vsec: 0
vusec: 0
}
}
rlimits {
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 8388608
max: 18446744073709551615
}
rlimits {
cur: 0
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 3898
max: 3898
}
rlimits {
cur: 1024
max: 4096
}
rlimits {
cur: 65536
max: 65536
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 3898
max: 3898
}
rlimits {
cur: 819200
max: 819200
}
rlimits {
cur: 0
max: 0
}
rlimits {
cur: 0
max: 0
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
}
cg_set: 1
signals_s {
}
}
thread_core {
futex_rla: 0
futex_rla_len: 24
sched_nice: 0
sched_policy: 0
sas {
ss_sp: 0
ss_size: 0
ss_flags: 2
}
signals_p {
}
}
</pre>

=== Convert to json (and back) ===

Same as before, but the human-readable format is JSON

'''Status: Ready'''<br>
For now, it does it in a dumb way, by just wrapping text_format in brackets and adding commas where needed.
Example:
<pre>
{
"magic": "CORE",
"0": {
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": "24",
"sched_policy": "0",
"sched_nice": "0",
"futex_rla": "0",
"signals_p": {},
"sas": {
"ss_size": "0",
"ss_sp": "0",
"ss_flags": "2"
}
},
"pbtype": "core_entry",
"thread_info": {
"fpregs": {
"st_space": [
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0"
],
"fop": "0",
"rdp": "0",
"twd": "0",
"mxcsr": "8064",
"swd": "0",
"rip": "0",
"xmm_space": [
"0",
"0",
"4294967295",
"4294967295",
"16777215",
"16843009",
"16843009",
"16843009",
"4478800",
"661857632",
"1869488186",
"543236212",
"1970037624",
"1680696676",
"775778921",
"544500071",
"2019896621",
"1685417059",
"1768172901",
"1747860850",
"809328999",
"909327153",
"1932405306",
"809334896",
"909327153",
"2016291386",
"1030123635",
"859517232",
"14902",
"0",
"65",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"4294967295",
"4294967295",
"4294967295",
"4294967295",
"4294967040",
"4294967295",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0"
],
"cwd": "895",
"mxcsr_mask": "65535"
},
"clear_tid_addr": "0",
"gpregs": {
"gs": "0",
"ip": "213251769082",
"cx": "18446744073709551615",
"cs": "51",
"ax": "61",
"orig_ax": "61",
"di": "18446744073709551615",
"es": "0",
"gs_base": "0",
"r14": "19884608",
"r15": "0",
"r12": "0",
"r13": "1",
"r10": "0",
"r11": "582",
"fs_base": "140140334651200",
"bp": "19879232",
"dx": "0",
"bx": "0",
"ds": "0",
"ss": "43",
"sp": "140733241187336",
"r8": "19879232",
"r9": "0",
"fs": "0",
"si": "140733241187376",
"flags": "582"
}
},
"tc": {
"timers": {
"real": {
"isec": "0",
"vusec": "0",
"iusec": "0",
"vsec": "0"
},
"virt": {
"isec": "0",
"vusec": "0",
"iusec": "0",
"vsec": "0"
},
"prof": {
"isec": "0",
"vusec": "0",
"iusec": "0",
"vsec": "0"
}
},
"cg_set": "1",
"signals_s": {},
"blk_sigset": "65536",
"exit_code": "0",
"rlimits": {
"rlimits": [
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "8388608"
},
{
"max": "18446744073709551615",
"cur": "0"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "3898",
"cur": "3898"
},
{
"max": "4096",
"cur": "1024"
},
{
"max": "65536",
"cur": "65536"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "3898",
"cur": "3898"
},
{
"max": "819200",
"cur": "819200"
},
{
"max": "0",
"cur": "0"
},
{
"max": "0",
"cur": "0"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
}
]
},
"comm": "\"loop.sh\"",
"flags": "1077960704",
"task_state": "1",
"personality": "0"
}
}
}
</pre>
=== Generate core files out of task images ===

All the needed information is in core-$pid.img and pagemap-$pid.img

'''Status: Not ready'''

=== On-the-fly conversion ===

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

'''Status: Not ready'''

=== Convert between different image versions ===

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

'''Status: Not ready'''

=== Show images statistics ===

E.g. -- total number of processes, files, memory, sockets, etc.

Status: Not ready

=== Check/validate images ===

Check that
* all images are present
* the inter-images IDs are in consistent state

'''Status: Not ready'''

CRIT

2014-11-24T19:26:13Z

Efiop: Show what we have for now

This page describes the CRiu Image Tool.
{{Note|As for now(24 Nov 2014) it is not added to the official repo, but you could checkout it [https://github.com/efiop/criu/tree/crit-test here].}}

== Usage ==
<pre>Usage: crit [-h] [-i IN] [-o OUT] [-f {raw,nice}] [-t {json}] {convert}

CRiu Image Tool

positional arguments:
{convert} use "covert" to convert CRIU image to/from human-
readable format

optional arguments:
-h, --help show this help message and exit
-i IN, --in IN input file (stdin by default)
-o OUT, --out OUT output file (stdout by default
-f {raw,nice}, --format {raw,nice}
well-formated output (by default: raw for files and
nice for stdout)
-t {json}, --type {json}
Type of output file (by default img is converted to
txt and vice versa)
</pre>

== Functionality ==

=== Convert images to human-readable and back ===
This is the replacement for (rather nasty) criu show code. Also this is the way to edit the images before restoring from them.

'''Status: Ready'''<br>
It uses [https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module text_format] as a human-readable format for protobuf messages.
Output file is structured like:
With "--format raw":
<pre>
#<magic value>
#0
<pb entry in text_format>
....
#N
<pb entry in text_format>
</pre>
With "--format nice":
<pre>
#<magic name>
#0 <pb entry name>
<pb entry in text_format with tabs>
....
#N <pb entry name>
<pb entry in text_format with tabs>
</pre>

So, here is how core file will look like with --format nice:
<pre>
#CORE
#0 core_entry
mtype: X86_64
thread_info {
clear_tid_addr: 0
gpregs {
r15: 0
r14: 19884608
r13: 1
r12: 0
bp: 19879232
bx: 0
r11: 582
r10: 0
r9: 0
r8: 19879232
ax: 61
cx: 18446744073709551615
dx: 0
si: 140733241187376
di: 18446744073709551615
orig_ax: 61
ip: 213251769082
cs: 51
flags: 582
sp: 140733241187336
ss: 43
fs_base: 140140334651200
gs_base: 0
ds: 0
es: 0
fs: 0
gs: 0
}
fpregs {
cwd: 895
swd: 0
twd: 0
fop: 0
rip: 0
rdp: 0
mxcsr: 8064
mxcsr_mask: 65535
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
st_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 16777215
xmm_space: 16843009
xmm_space: 16843009
xmm_space: 16843009
xmm_space: 4478800
xmm_space: 661857632
xmm_space: 1869488186
xmm_space: 543236212
xmm_space: 1970037624
xmm_space: 1680696676
xmm_space: 775778921
xmm_space: 544500071
xmm_space: 2019896621
xmm_space: 1685417059
xmm_space: 1768172901
xmm_space: 1747860850
xmm_space: 809328999
xmm_space: 909327153
xmm_space: 1932405306
xmm_space: 809334896
xmm_space: 909327153
xmm_space: 2016291386
xmm_space: 1030123635
xmm_space: 859517232
xmm_space: 14902
xmm_space: 0
xmm_space: 65
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 4294967295
xmm_space: 4294967040
xmm_space: 4294967295
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
xmm_space: 0
}
}
tc {
task_state: 1
exit_code: 0
personality: 0
flags: 1077960704
blk_sigset: 65536
comm: "loop.sh"
timers {
real {
isec: 0
iusec: 0
vsec: 0
vusec: 0
}
virt {
isec: 0
iusec: 0
vsec: 0
vusec: 0
}
prof {
isec: 0
iusec: 0
vsec: 0
vusec: 0
}
}
rlimits {
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 8388608
max: 18446744073709551615
}
rlimits {
cur: 0
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 3898
max: 3898
}
rlimits {
cur: 1024
max: 4096
}
rlimits {
cur: 65536
max: 65536
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
rlimits {
cur: 3898
max: 3898
}
rlimits {
cur: 819200
max: 819200
}
rlimits {
cur: 0
max: 0
}
rlimits {
cur: 0
max: 0
}
rlimits {
cur: 18446744073709551615
max: 18446744073709551615
}
}
cg_set: 1
signals_s {
}
}
thread_core {
futex_rla: 0
futex_rla_len: 24
sched_nice: 0
sched_policy: 0
sas {
ss_sp: 0
ss_size: 0
ss_flags: 2
}
signals_p {
}
}
</pre>

=== Convert to json (and back) ===

Same as before, but the human-readable format is JSON

'''Status: Ready'''<br>
For now, it does it in a dumb way, by just wrapping text_format in brackets and adding ";" where needed.
Example:
<pre>
{
"magic": "CORE",
"0": {
"mtype": "X86_64",
"thread_core": {
"futex_rla_len": "24",
"sched_policy": "0",
"sched_nice": "0",
"futex_rla": "0",
"signals_p": {},
"sas": {
"ss_size": "0",
"ss_sp": "0",
"ss_flags": "2"
}
},
"pbtype": "core_entry",
"thread_info": {
"fpregs": {
"st_space": [
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0"
],
"fop": "0",
"rdp": "0",
"twd": "0",
"mxcsr": "8064",
"swd": "0",
"rip": "0",
"xmm_space": [
"0",
"0",
"4294967295",
"4294967295",
"16777215",
"16843009",
"16843009",
"16843009",
"4478800",
"661857632",
"1869488186",
"543236212",
"1970037624",
"1680696676",
"775778921",
"544500071",
"2019896621",
"1685417059",
"1768172901",
"1747860850",
"809328999",
"909327153",
"1932405306",
"809334896",
"909327153",
"2016291386",
"1030123635",
"859517232",
"14902",
"0",
"65",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"4294967295",
"4294967295",
"4294967295",
"4294967295",
"4294967040",
"4294967295",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0",
"0"
],
"cwd": "895",
"mxcsr_mask": "65535"
},
"clear_tid_addr": "0",
"gpregs": {
"gs": "0",
"ip": "213251769082",
"cx": "18446744073709551615",
"cs": "51",
"ax": "61",
"orig_ax": "61",
"di": "18446744073709551615",
"es": "0",
"gs_base": "0",
"r14": "19884608",
"r15": "0",
"r12": "0",
"r13": "1",
"r10": "0",
"r11": "582",
"fs_base": "140140334651200",
"bp": "19879232",
"dx": "0",
"bx": "0",
"ds": "0",
"ss": "43",
"sp": "140733241187336",
"r8": "19879232",
"r9": "0",
"fs": "0",
"si": "140733241187376",
"flags": "582"
}
},
"tc": {
"timers": {
"real": {
"isec": "0",
"vusec": "0",
"iusec": "0",
"vsec": "0"
},
"virt": {
"isec": "0",
"vusec": "0",
"iusec": "0",
"vsec": "0"
},
"prof": {
"isec": "0",
"vusec": "0",
"iusec": "0",
"vsec": "0"
}
},
"cg_set": "1",
"signals_s": {},
"blk_sigset": "65536",
"exit_code": "0",
"rlimits": {
"rlimits": [
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "8388608"
},
{
"max": "18446744073709551615",
"cur": "0"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "3898",
"cur": "3898"
},
{
"max": "4096",
"cur": "1024"
},
{
"max": "65536",
"cur": "65536"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
},
{
"max": "3898",
"cur": "3898"
},
{
"max": "819200",
"cur": "819200"
},
{
"max": "0",
"cur": "0"
},
{
"max": "0",
"cur": "0"
},
{
"max": "18446744073709551615",
"cur": "18446744073709551615"
}
]
},
"comm": "\"loop.sh\"",
"flags": "1077960704",
"task_state": "1",
"personality": "0"
}
}
}
</pre>
=== Generate core files out of task images ===

All the needed information is in core-$pid.img and pagemap-$pid.img

'''Status: Not ready'''

=== On-the-fly conversion ===

There's an idea to make CRIU spawn CRIT and read images "through" it, to allow for at-the-restore-time modifications

'''Status: Not ready'''

=== Convert between different image versions ===

Right now we store the images version in inventory.img and collect info about [[what's bad with V1 images]]. If some day we have v2, CRIT will convert from v1. And for backward compatibility we'll use on-the-fly conversion when restoring from old images.

'''Status: Not ready'''

=== Show images statistics ===

E.g. -- total number of processes, files, memory, sockets, etc.

Status: Not ready

=== Check/validate images ===

Check that
* all images are present
* the inter-images IDs are in consistent state

'''Status: Not ready'''

Images

2014-11-24T18:35:34Z

Efiop: fix format

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

Or, you can visualize it like

{| class="wikitable"
|-
! Type !! Size(bytes)
|-
| Magic || 4
|-
| Size0 || 4
|-
| Message0 || Size0
|-
| ... || ...
|-
| SizeN || 4
|-
| MessageN || SizeN
|}

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || - || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
{{Note| Even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry. }}

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]

Images

2014-11-24T18:34:00Z

Efiop: Explain and ilustrate image structure a bit more clear

The criu utility dumps the state of processes/containers into a set of image files. This article describes the format of them.

== Types of image files ==

CRIU images can be in one of formats

* criu specific images in google protocol buffer format (PB format)
* criu specific images with binary data in it
* image files in 3rd party format (a.k.a. raw images)

== Images in criu-specific format ==

All criu-specific image files begin with 32-bit magic cookie. Images in PB format are followed by zero or more entries of the same type (not size!), each entry is preceded with 32-bit entry size value (not including this 32-bit value itself). Optionally each entry may be followed by extra payload which depends on the entry type.

IOW protocol-buffers image files look like

<pre>
IMAGE_FILE ::= MAGIC { ENTRY }
ENTRY ::= SIZE PAYLOAD [ EXTRA ]
PAYLOAD ::= "message encoded in ProtocolBuffer format"
EXTRA ::= "arbitrary blob, depends on the PAYLOAD contents"

MAGIC ::= "32 bit integer"
SIZE ::= "32 bit integer, equals the PAYLOAD length"
</pre>

Or, you can visualize it like

{| class="wikitable"
|-
! Type !! Size(bytes)
|-
| Magic || 4
|-
| Size0 || 4
|-
| Message0 || Size0
|-
| ... || ...
|-
| SizeN || 4
|-
| MessageN || SizeN
|}

The amount of entries in a image file depends on the type of file.

=== Images with PB data ===

Such images can be one of

; Array image files
: In these files the amount of entries can be any. You should read the image file up to the EOF to find out the exact number.
; Single-entry image files
: In these files exactly one entry is stored.

A file type can be guessed by the magic. The description of the entries in ProtocolBuffers language are in respective .proto files which reside in <code>protobuf/</code> directory in the source tree.

{|class="wikitable sortable"
|-
! name
! type
! description
! extra payload
! describing proto file
|-
| inventory || single-entry || Top level description of images || - || inventory.proto
|-
| fdinfo || array || Open file descriptors || - || fdinfo.proto
|-
| reg-files || array || Paths to files opened with <code>open(2)</code> syscall || - || regfile.proto
|-
| eventfd || array || Eventfd file information || - || eventfd.proto
|-
| eventpoll || array || Eventpoll file information || - || eventpoll.proto
|-
| eventpoll-tfd || array || Target file descriptors of eventpoll fds || - || eventpoll.proto
|-
| inotify || array || Inotify file information || - || intotify.proto
|-
| inotify-wd || array || Watch descriptors of inotify fds || - || inotify.proto
|-
| signalfd || array || signalfd info || - || signalfd.proto
|-
| core || single-entry || Arch-dependent information (registers, etc.) || - || core.proto
|-
| mm || single-entry || Address space generic information (segments, exe file, etc.) || - || mm.proto
|-
| vmas || array || Virtual mappings (<code>mmap(2)</code>) || - || vma.proto
|-
| pipes || array || Pipes information || - || pipe.proto
|-
| pipes-data || array || Contents of pipes || <code>entry.bytes</code> bytes of data sitting in a pipe || pipe-data.proto
|-
| fifo || array || FIFO information || - || fifo.proto
|-
| fifo-data || array || Contents of FIFOs || same as in pipes-data || pipe-data.proto
|-
| pstree || array || Process tree linkage and IDs || - || pstree.proto
|-
| sigacts || array || Signal handling map || - || sa.proto
|-
| unixsk || array || Unix domain sockets || - || sk-unix.proto
|-
| inetsk || array || PF_INET sockets, both IPv4 and IPv6 || - || sk-inet.proto
|-
| sk-queues || array || Contents of socket queues || <code>entry.length</code> bytes of data, one entry per packet || sk-packet.proto
|-
| itimers || array || Interval timers state || - || itimer.proto
|-
| creds || single-entry || Task credentials: uids, gids, caps, etc. || - || creds.proto
|-
| fs || single-entry || Chroot and chdir information || - || fs.proto
|-
| remap-fpath || array || File paths remaps || - || remap-file-path.proto
|-
| ghost-file || single-entry || Ghost files (those, not visible from FS, but still used by tasks) || Right after the entry up to the EOF goes the contents of the file || ghost-file.proto
|-
| tcp-stream || single-entry || TCP connection state (including data in queues) || - || tcp-stream.proto
|-
| mountpoints || array || Mountpoints information || - || mnt.proto
|-
| utsns || single-entry || Uname nodename and domainname of a UTS namespace || - || utsns.proto
|-
| tty || array || Information about opened tty-s || - || tty.proto
|-
| tty-info || array || Termios and similar stuff about tty-s || - || tty.proto
|-
| packetsk || array || Info about PF_PACKET sockets || - || packet-sock.proto
|-
| netdev || array || Info about network devices || - || netdev.proto
|}

=== Images with memory dumps ===

Memory info is stored in two types of images

; Pagemap files
: These files contain info about which virtual regions are populated with data. The file is a set of protobuf messages.
: Note that, even though, pagemap is an array kind of image(and can be included to the previous type), first pb message is of type pagemap_head and all the following ones are of type pagemap_entry.

; Pages files
: These contain 4k pages that are to be put into the memory according to the pagemap.

== Raw images ==

These images contain data that were collected by criu with the help of some external tools. These are

{|class="wikitable sortable"
|-
! name
! tool that understand this format
! description
|-
| ifaddr || iproute2's ip || Contains info about IP addresses on network devices
|-
| route || iproute2's ip || Contains routing tables
|-
| tmpfs || tar + gzip || Contains contents of tmpfs filesystem
|}

== See also ==

* [[What's bad with V1 images]]
* [[ImageFieldMerging | Image field merging]]

[[Category:Development]]

Installation

2014-10-20T13:46:38Z

Efiop:

<code>criu</code> is an utility to checkpoint/restore a process tree. This page describes how to manually build and install prerequisites and the tool itself.

{{Note|Most probably you don't need manual installation, but rather [[Packages]] for your distro.}}

== Obtaining CRIU Source ==

You can download the source code as a release tarball or sync the [http://git.criu.org/?p=criu.git;a=summary git repository].

{{Out|{{Latest release}}}}

git clone git://git.criu.org/criu.git
cd criu

== Dependencies ==

=== Compiler and C Library ===
For native compilation on Debian based systems, install the <code>build-essential</code> package. For cross compiling for ARM and AArch64, the Linaro prebuilt toolchains are a good choice. Installing them is described below. They are ia32 architecture binaries. On a modern Debian based x86_64 you will need to install the <code>lib32stdc++6</code> and <code>lib32z1</code> packages.

mkdir -p deps/`uname -m`-linux-gnu
cd deps
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-arm-linux-gnueabihf-4.9-2014.06_linux.tar.xz
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-aarch64-linux-gnu-4.9-2014.06-02_linux.tar.xz
cd ..

=== Protocol Buffers with C Bindings ===

CRIU uses the [https://github.com/protobuf-c/protobuf-c C language bindings] of [https://developers.google.com/protocol-buffers/ Google Protocol Buffers] for serialization. The <code>protoc</code> tool is required at build time and <code>libprotobuf-c.so</code> is required at build time and at run time, assuming dynamic linking.

==== Distribution Packages ====
The easiest approach for most would be to install distribution packages. RPM package names: <code>protobuf-c-compiler</code>, <code>protobuf-c-devel</code>. Debian package names: <code>protobuf-c-compiler</code>, <code>libprotobuf-c0-dev</code>.

==== Building Protocol Buffers From Source ====
If you would like to build from source, you can use the following commands to obtain the source code repositories, configure, and build the code. On a Debian based system, you may have to install the following packages first: <code>autoconf curl g++ libtool</code>.

===== Native protobuf =====
cd deps
git clone https://github.com/google/protobuf.git protobuf
cd protobuf
./autogen.sh
./configure --prefix=`pwd`/../`uname -m`-linux-gnu
make
make install
cd ../..

===== Native protobuf-c =====

cd deps
git clone https://github.com/protobuf-c/protobuf-c.git protobuf-c
cd protobuf-c
./autogen.sh
mkdir ../pbc-`uname -m`
cd ../pbc-`uname -m`
../protobuf-c/configure --prefix=`pwd`/../`uname -m`-linux-gnu \
PKG_CONFIG_PATH=`pwd`/../`uname -m`-linux-gnu/lib/pkgconfig
make
make install
cd ../..

===== Cross Compiling for ARMv7 =====
If you would like to cross-compile for armv7:

cd deps
mkdir -p pbc-arm
cd pbc-arm
../protobuf-c/configure --host=arm-linux-gnueabihf --prefix=`pwd`/../arm-linux-gnueabihf --disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

===== Cross Compiling for ARMv8 =====
If you would like to cross-compile for armv8:

cd deps
mkdir -p pbc-aarch64
cd pbc-aarch64
../protobuf-c/configure --host=aarch64-linux-gnu --prefix=`pwd`/../aarch64-linux-gnu --disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

=== Some minor, but useful dependencies ===
==== libbsd ====
If libbsd is available, CRIU will be compiled with setproctitle() support. It allows to make process titles of service workers to be more verbose.

== Building CRIU From Source ==

=== Native Compilation ===
With the CRIU source obtained in the first step and dependencies satisfied in the second step, we are now compile CRIU. For native compilation with the dependencies met using distribution packages, simply run <code>make</code> in the CRIU source directory.

Here is an example of building natively specifying manually built dependencies.

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-`uname -m`
cd criu-`uname -m`
make \
USERCFLAGS="-I`pwd`/../`uname -m`-linux-gnu/include -L`pwd`/../`uname -m`-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
sudo LD_LIBRARY_PATH=`pwd`/../`uname -m`-linux-gnu/lib ./criu check
cd ../..

=== Cross Compilation for ARMv7 ===

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-arm
cd criu-arm
make \
ARCH=arm \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/arm-linux-gnueabihf- \
USERCFLAGS="-I`pwd`/../arm-linux-gnueabihf/include -L`pwd`/../arm-linux-gnueabihf/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

=== Cross Compilation for ARMv8 ===

cd deps
rsync -a --exclude=.git --exclude=deps .. criu-aarch64
cd criu-aarch64
make \
ARCH=aarch64 \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/aarch64-linux-gnu- \
USERCFLAGS="-I`pwd`/../aarch64-linux-gnu/include -L`pwd`/../aarch64-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

=== Linux Kernel ===

Linux kernel v3.11 or newer is required, with some specific options set. If your distribution does not provide needed kernel, you might want to compile one yourself. Note we also have our [[custom kernel]], which might contain some experimental CRIU related patches.

Note you might have to enable
; <code>CONFIG_EXPERT</code>
: General setup -> Configure standard kernel features (expert users)
option, which depends on
; <code>CONFIG_EMBEDDED</code>
: General setup -> Embedded system
(welcome to Kconfig reverse chains hell).

The following options must be enabled for CRIU to work:

; <code>CONFIG_CHECKPOINT_RESTORE</code>
: General setup -> Checkpoint/restore support

; <code>CONFIG_NAMESPACES</code>
: General setup -> Namespaces support

; <code>CONFIG_UTS_NS</code>
: General setup -> Namespaces support -> UTS namespace

; <code>CONFIG_IPC_NS</code>
: General setup -> Namespaces support -> IPC namespace

; <code>CONFIG_PID_NS</code>
: General setup -> Namespaces support -> PID namespaces

; <code>CONFIG_NET_NS</code>
: General setup -> Namespaces support -> Network namespace

; <code>CONFIG_FHANDLE</code>
: General setup -> open by fhandle syscalls

; <code>CONFIG_EVENTFD</code>
: General setup -> Enable eventfd() system call

; <code>CONFIG_EPOLL</code>
: General setup -> Enable eventpoll support

; <code>CONFIG_INOTIFY_USER</code>
: File systems -> Inotify support for userspace

; <code>CONFIG_IA32_EMULATION</code>
: Executable file formats -> Emulations -> IA32 Emulation

; <code>CONFIG_UNIX_DIAG</code>
: Networking support -> Networking options -> Unix domain sockets -> UNIX: socket monitoring interface

; <code>CONFIG_INET_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface

; <code>CONFIG_INET_UDP_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface

; <code>CONFIG_PACKET_DIAG</code>
: Networking support -> Networking options -> Packet socket -> Packet: sockets monitoring interface

; <code>CONFIG_NETLINK_DIAG</code>
: Networking support -> Networking options -> Netlink socket -> Netlink: sockets monitoring interface

For some [[usage scenarios]] there is an ability to track memory changes and produce [[incremental dumps]]. Need to enable
; <code>CONFIG_MEM_SOFT_DIRTY</code>
: Processor type and features -> Track memory changes

At the moment it's known that CRIU will '''NOT''' work if packet generator module is loaded. Thus make sure
that either module is unloaded or not compiled at all.
; <code>CONFIG_NET_PKTGEN</code>
: Networking support -> Networking options -> Network testing -> Packet generator

=== iproute2 ===
The iproute2 tool version 3.5.0 or higher is needed for dumping network namespaces.
The latest one can be cloned from [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=summary iproute2]. It should be compiled and a path to ip written in the environment variable <code>CR_IP_TOOL</code>.

== Installation ==
<pre>
# make install
</pre>

== Checking That It Works ==

First thing to do is to run

<pre>
# criu check --ms
</pre>

At the end it should say "Looks OK", if it doesn't the messages on the screen explain what functionality is missing.
If you're using our custom kernel, then the <code>--ms</code> option should not be used, in this case CRIU would
check for ''all'' the kernel features to work.

You can then try running the [[ZDTM Test Suite]] which sits in the <code>tests/zdtm/</code> directory.

{{Out|There's a known issue with BTRFS spoiling dev_t values for files and sockets! Not all tests will work on it.}}

== Using CR tools ==

Please see [[Usage]] and [[Advanced usage]], as well as [[:Category:HOWTO]].

Installation

2014-09-11T15:02:23Z

Efiop: mention setproctitle from libbsd

<code>criu</code> is an utility to checkpoint/restore a process tree. This page describes how to manually build and install prerequisites and the tool itself.

{{Note|Most probably you don't need manual installation, but rather [[Packages]] for your distro.}}

== Obtaining CRIU Source ==

You can download the source code as a release tarball or sync the [http://git.criu.org/?p=criu.git;a=summary git repository].

{{Out|{{Latest release}}}}

git clone git://git.criu.org/criu.git
cd criu

== Dependencies ==

=== Compiler and C Library ===
For native compilation on Debian based systems, install the <code>build-essential</code> package. For cross compiling for ARM and AArch64, the Linaro prebuilt toolchains are a good choice. Installing them is described below. They are ia32 architecture binaries. On a modern Debian based x86_64 you will need to install the <code>lib32stdc++6</code> package.

mkdir deps
cd deps
wget http://releases.linaro.org/14.06/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.06_linux.tar.xz
tar --strip=1 -xf gcc-linaro-arm-linux-gnueabihf-4.9-2014.06_linux.tar.xz
wget http://releases.linaro.org/14.06/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.9-2014.06-02_linux.tar.xz
tar --strip=1 -xf gcc-linaro-aarch64-linux-gnu-4.9-2014.06-02_linux.tar.xz
cd ..

=== Protocol Buffers with C Bindings ===

CRIU uses the [http://code.google.com/p/protobuf-c/ C language bindings] of [http://code.google.com/p/protobuf/ Google Protocol Buffers] for serialization. The <code>protoc</code> tool is required at build time and <code>libprotobuf-c.so</code> is required at build time and at run time, assuming dynamic linking.

==== Distribution Packages ====
The easiest approach for most would be to install distribution packages. RPM package names: <code>protobuf-c-compiler</code>, <code>protobuf-c-devel</code>. Debian package names: <code>protobuf-c-compiler</code>, <code>libprotobuf-c0-dev</code>.

==== Building Protocol Buffers From Source ====
If you would like to build from source, you can use the following commands to obtain the source code repositories, configure, and build the code. On a Debian based system, you may have to install the following packages first: <code>autoconf curl g++ libtool</code>.

===== Native protobuf =====
cd deps
git svn clone http://protobuf.googlecode.com/svn/trunk protobuf
cd protobuf
./autogen.sh
./configure --prefix=`pwd`/../`uname -m`-linux-gnu
make
make install
cd ../..

===== Native protobuf-c =====

cd deps
git svn clone http://protobuf-c.googlecode.com/svn/trunk protobuf-c
cd protobuf-c
./autogen.sh
mkdir ../pbc-`uname -m`
cd ../pbc-`uname -m`
../protobuf-c/configure --prefix=`pwd`/../`uname -m`-linux-gnu \
PKG_CONFIG_PATH=`pwd`/../`uname -m`-linux-gnu/lib/pkgconfig
make
make install
cd ../..

===== Cross Compiling for ARMv7 =====
If you would like to cross-compile for armv7:

cd deps
mkdir -p pbc-arm
cd pbc-arm
../protobuf-c/configure --host=arm-linux-gnueabihf --prefix=`pwd`/../arm-linux-gnueabihf --disable-protoc CC=`pwd`/../bin/arm-linux-gnueabihf-gcc
make
make install
cd ../..

===== Cross Compiling for ARMv8 =====
If you would like to cross-compile for armv8:

cd deps
mkdir -p pbc-aarch64
cd pbc-aarch64
../protobuf-c/configure --host=aarch64-linux-gnu --prefix=`pwd`/../aarch64-linux-gnu --disable-protoc CC=`pwd`/../bin/aarch64-linux-gnu-gcc
make
make install
cd ../..

=== Some minor, but useful dependencies ===
==== libbsd ====
If libbsd is available, CRIU will be compiled with setproctitle() support. It allows to make process titles of service workers to be more verbose.

== Building CRIU From Source ==

=== Native Compilation ===
With the CRIU source obtained in the first step and dependencies satisfied in the second step, we are now compile CRIU. For native compilation with the dependencies met using distribution packages, simply run <code>make</code> in the CRIU source directory.

Here is an example of building natively specifying manually built dependencies.

make \
USERCFLAGS="-I`pwd`/deps/`uname -m`-linux-gnu/include -L`pwd`/deps/`uname -m`-linux-gnu/lib" \
PATH="`pwd`/deps/`uname -m`-linux-gnu/bin:$PATH"

=== Cross Compilation for ARMv7 ===

make \
ARCH=arm \
CROSS_COMPILE=`pwd`/deps/bin/arm-linux-gnueabihf- \
USERCFLAGS="-I`pwd`/deps/arm-linux-gnueabihf/include -L`pwd`/deps/arm-linux-gnueabihf/lib" \
PATH="`pwd`/deps/`uname -m`-linux-gnu/bin:$PATH"

=== Cross Compilation for ARMv8 ===

make \
ARCH=aarch64 \
CROSS_COMPILE=`pwd`/deps/bin/aarch64-linux-gnu- \
USERCFLAGS="-I`pwd`/deps/aarch64-linux-gnu/include -L`pwd`/deps/aarch64-linux-gnu/lib" \
PATH="`pwd`/deps/`uname -m`-linux-gnu/bin:$PATH"

=== Linux Kernel ===

Linux kernel v3.11 or newer is required, with some specific options set. If your distribution does not provide needed kernel, you might want to compile one yourself. Note we also have our [[custom kernel]], which might contain some experimental CRIU related patches.

Note you might have to enable
; <code>CONFIG_EXPERT</code>
: General setup -> Configure standard kernel features (expert users)
option, which depends on
; <code>CONFIG_EMBEDDED</code>
: General setup -> Embedded system
(welcome to Kconfig reverse chains hell).

The following options must be enabled for CRIU to work:

; <code>CONFIG_CHECKPOINT_RESTORE</code>
: General setup -> Checkpoint/restore support

; <code>CONFIG_NAMESPACES</code>
: General setup -> Namespaces support

; <code>CONFIG_UTS_NS</code>
: General setup -> Namespaces support -> UTS namespace

; <code>CONFIG_IPC_NS</code>
: General setup -> Namespaces support -> IPC namespace

; <code>CONFIG_PID_NS</code>
: General setup -> Namespaces support -> PID namespaces

; <code>CONFIG_NET_NS</code>
: General setup -> Namespaces support -> Network namespace

; <code>CONFIG_FHANDLE</code>
: General setup -> open by fhandle syscalls

; <code>CONFIG_EVENTFD</code>
: General setup -> Enable eventfd() system call

; <code>CONFIG_EPOLL</code>
: General setup -> Enable eventpoll support

; <code>CONFIG_INOTIFY_USER</code>
: File systems -> Inotify support for userspace

; <code>CONFIG_IA32_EMULATION</code>
: Executable file formats -> Emulations -> IA32 Emulation

; <code>CONFIG_UNIX_DIAG</code>
: Networking support -> Networking options -> Unix domain sockets -> UNIX: socket monitoring interface

; <code>CONFIG_INET_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface

; <code>CONFIG_INET_UDP_DIAG</code>
: Networking support -> Networking options -> TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface

; <code>CONFIG_PACKET_DIAG</code>
: Networking support -> Networking options -> Packet socket -> Packet: sockets monitoring interface

; <code>CONFIG_NETLINK_DIAG</code>
: Networking support -> Networking options -> Netlink socket -> Netlink: sockets monitoring interface

For some [[usage scenarios]] there is an ability to track memory changes and produce [[incremental dumps]]. Need to enable
; <code>CONFIG_MEM_SOFT_DIRTY</code>
: Processor type and features -> Track memory changes

At the moment it's known that CRIU will '''NOT''' work if packet generator module is loaded. Thus make sure
that either module is unloaded or not compiled at all.
; <code>CONFIG_NET_PKTGEN</code>
: Networking support -> Networking options -> Network testing -> Packet generator

=== iproute2 ===
The iproute2 tool version 3.5.0 or higher is needed for dumping network namespaces.
The latest one can be cloned from [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=summary iproute2]. It should be compiled and a path to ip written in the environment variable <code>CR_IP_TOOL</code>.

== Checking That It Works ==

First thing to do is to run

<pre>
# criu check --ms
</pre>

At the end it should say "Looks OK", if it doesn't the messages on the screen explain what functionality is missing.
If you're using our custom kernel, then the <code>--ms</code> option should not be used, in this case CRIU would
check for ''all'' the kernel features to work.

You can then try running the [[ZDTM Test Suite]] which sits in the <code>tests/zdtm/</code> directory.

{{Out|There's a known issue with BTRFS spoiling dev_t values for files and sockets! Not all tests will work on it.}}

== Using CR tools ==

Please see [[Usage]] and [[Advanced usage]], as well as [[:Category:HOWTO]].

ZDTM test suite

2014-09-11T14:54:38Z

Efiop: delete wrong note

== Description ==

ZDTM stands for zero-down-time-migration. It's test suite developed for testing how OpenVZ live migration works. We use this test suite for checking how criu do their job. The suit consists of many small atomic tests -- each puts a process into some "state" (opens a file, maps a memory segment, puts data in a pipe, etc.), then asks to be checkpointed and restored. The in checks that the "state" was preserved (file is still open, memory is still mapped, pipe contains what was put into it).

== Running ==
=== All zdtm tests ===
You can run all the tests by:
make test
or by:
test/zdtm.sh

=== Certain test ===
You can also run the tests manually by issuing a
make <testname>.pid
command. After you've done c/r-ing it you should run
make <testname>.out
and check for the <testname>.out file contents.

If you don't want to mess with this you can use the <code>zdtm.sh</code> script. When launched without arguments runs all the tests one-by-one. The exact test can be specified by the command line argument. The <code>-l</code> option lists the tests it can run.

[[Category:Development]]

How to submit patches

2014-09-11T14:51:27Z

Efiop: fix link

== Setting up working enviroment ==
Although criu could be run as non-root(see [[Security]]), development is better to be done as root. For example, some tests require root. So, it would be a good idea to set up some recent Linux distro on a virtual machine.

=== Obtaining the source code ===

The CRIU sources are tracked by Git SCM at http://git.criu.org
repository. You either could download packed sources or use git
tool itself.

For example to clone criu one need to type

git clone git://git.criu.org/criu.git

=== Compiling the source code ===
{{Note|Dependencies are listed in the [Installation] article.}}
Run:
cd criu
make
This will produce ./criu executable.
As you are going to work with sources, it would come in handy to get tags by running:
make tags
{{Note|It requires ctags to be installed.}}

== Changing the source code ==

When you change the source code keep in mind - we prefer tabs and
indentations to be 8 characters width. Consider reading [https://www.kernel.org/doc/Documentation/CodingStyle Linux kernel coding style].

Other "rules" could be learned from the source code - just make your code
to look similar.

== Producing a patch ==

There are at least two ways to make it right.

1) git format-patch

You might need to read documentation on Git SCM how to prepare patch
for mail submission. Take a look on http://book.git-scm.com/ and/or
http://git-scm.com/documentation for details. It should not be hard
at all.

2) Use "diff -up"

Use <code>diff -up</code> or <code>diff -uprN</code> to create patches.

== Signing your work ==

To improve tracking of who did what we've introduced a "sign-off" procedure
on patches that are being emailed around.

The sign-off is a simple line at the end of the explanation for the
patch, which certifies that you wrote it or otherwise have the right to
pass it on as a open-source patch. The rules are pretty simple: if you
can certify the below:

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

then you just add a line saying

Signed-off-by: Random J Developer <random at developer.example.org>

using your real name (please, no pseudonyms or anonymous contributions if
it possible)

== An example of patch message ==

From: Random J Developer <random at developer.example.org>
Subject: [PATCH] Short patch description

Long patch description (could be skipped if patch
is trivial enough)

Signed-off-by: Random J Developer <random at developer.example.org>
---
Patch body here

== Checking patches for errors ==
Basic style error checking is provided by scripts/checkpatch.pl script
from linux kernel source tree. You might have headers or complete sources
of linux kernel at /usr/src/. Run
checkpatch.pl my_patch.patch (consider --no-tree option for headers-only)
It will produce detailed report on style problems in your patch.
Make sure to fix all _errors_(some warnings may be ignored).

To check whether your patches don't harm anything run
make test
It will launch [[ZDTM_Test_Suite]]. Check for error messages produced by it.
You may also want to run other tests(rpc, libcriu, security etc) from test/ dir.

== Mailing patches ==

The patches should be sent to CRIU development mailing list
which is located at https://openvz.org/mailman/listinfo/criu

Please make sure the email client you're using doesn't screw
your patch (line wrapping and so on).

== Wait for response ==

Be patient. Most CRIU developers are pretty busy people so if
there is no immediate response on your patch - don't be surprised,
sometimes a patch may fly around a week(s) before it get reviewed.
But definitely the patches will not go to <code>/dev/null</code>.

[[Category:Development]]

How to submit patches

2014-09-05T09:20:43Z

Efiop: add some hints about criu development

== Setting up working enviroment ==
Although criu could be run as non-root(see [Security]), development is better to be done as root. For example, some tests require root. So, it would be a good idea to set up some recent Linux distro on a virtual machine.

=== Obtaining the source code ===

The CRIU sources are tracked by Git SCM at http://git.criu.org
repository. You either could download packed sources or use git
tool itself.

For example to clone criu one need to type

git clone git://git.criu.org/criu.git

=== Compiling the source code ===
{{Note|Dependencies are listed in the [Installation] article.}}
Run:
cd criu
make
This will produce ./criu executable.
As you are going to work with sources, it would come in handy to get tags by running:
make tags
{{Note|It requires ctags to be installed.}}

== Changing the source code ==

When you change the source code keep in mind - we prefer tabs and
indentations to be 8 characters width. Consider reading [https://www.kernel.org/doc/Documentation/CodingStyle Linux kernel coding style].

Other "rules" could be learned from the source code - just make your code
to look similar.

== Producing a patch ==

There are at least two ways to make it right.

1) git format-patch

You might need to read documentation on Git SCM how to prepare patch
for mail submission. Take a look on http://book.git-scm.com/ and/or
http://git-scm.com/documentation for details. It should not be hard
at all.

2) Use "diff -up"

Use <code>diff -up</code> or <code>diff -uprN</code> to create patches.

== Signing your work ==

To improve tracking of who did what we've introduced a "sign-off" procedure
on patches that are being emailed around.

The sign-off is a simple line at the end of the explanation for the
patch, which certifies that you wrote it or otherwise have the right to
pass it on as a open-source patch. The rules are pretty simple: if you
can certify the below:

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

then you just add a line saying

Signed-off-by: Random J Developer <random at developer.example.org>

using your real name (please, no pseudonyms or anonymous contributions if
it possible)

== An example of patch message ==

From: Random J Developer <random at developer.example.org>
Subject: [PATCH] Short patch description

Long patch description (could be skipped if patch
is trivial enough)

Signed-off-by: Random J Developer <random at developer.example.org>
---
Patch body here

== Checking patches for errors ==
Basic style error checking is provided by scripts/checkpatch.pl script
from linux kernel source tree. You might have headers or complete sources
of linux kernel at /usr/src/. Run
checkpatch.pl my_patch.patch (consider --no-tree option for headers-only)
It will produce detailed report on style problems in your patch.
Make sure to fix all _errors_(some warnings may be ignored).

To check whether your patches don't harm anything run
make test
It will launch [[ZDTM_Test_Suite]]. Check for error messages produced by it.
You may also want to run other tests(rpc, libcriu, security etc) from test/ dir.

== Mailing patches ==

The patches should be sent to CRIU development mailing list
which is located at https://openvz.org/mailman/listinfo/criu

Please make sure the email client you're using doesn't screw
your patch (line wrapping and so on).

== Wait for response ==

Be patient. Most CRIU developers are pretty busy people so if
there is no immediate response on your patch - don't be surprised,
sometimes a patch may fly around a week(s) before it get reviewed.
But definitely the patches will not go to <code>/dev/null</code>.

[[Category:Development]]

ZDTM test suite

2014-09-02T12:23:23Z

Efiop: add a bit more information

== Description ==

ZDTM stands for zero-down-time-migration. It's test suite developed for testing how OpenVZ live migration works. We use this test suite for checking how criu do their job. The suit consists of many small atomic tests -- each puts a process into some "state" (opens a file, maps a memory segment, puts data in a pipe, etc.), then asks to be checkpointed and restored. The in checks that the "state" was preserved (file is still open, memory is still mapped, pipe contains what was put into it).

== Running ==
=== All zdtm tests ===
You can run all the tests by:
make test
or by:
test/zdtm.sh
Note, that if one test fails, ztdm will continue to run others. So, be sure to check its output for error messages.

=== Certain test ===
You can also run the tests manually by issuing a
make <testname>.pid
command. After you've done c/r-ing it you should run
make <testname>.out
and check for the <testname>.out file contents.

If you don't want to mess with this you can use the <code>zdtm.sh</code> script. When launched without arguments runs all the tests one-by-one. The exact test can be specified by the command line argument. The <code>-l</code> option lists the tests it can run.

[[Category:Development]]