Memory changes tracking

2014-12-10T23:29:57Z

Paschalis Mpeis:

CRIU can detect what memory pages a task (or tasks) has changed since some moment of time. This page describes why this thing is required, how it works and how to use it.

== Why do we need to track memory changed ==

There are several scenarios when detecting what parts of memory has changed is required:

; [[Incremental dumps]]
: When you take a series of dumps from a process tree, it is a very good optimization not to dump ''all'' the memory every time, but get only those memory pages that has changed since previous dump

; Smaller freeze time for big Applications
: When a task uses a LOT of memory, dumping it may take time and during all this time this task should be frozen. To reduce the freeze time we can
* get memory from task and start writing it in images
* freeze task and get only changed memory from it

; [[Live migration]]
: When doing live migration a lot of time takes the procedure of copying tasks' memory on remote host. And yet again, during this time process is frozen. Acting like in the previous example also reduces the freeze time, i.e. -- the live migration becomes really Live.

== How we track memory changes ==

In order to find out what pages in memory has changed, we [http://lwn.net/Articles/546966/ patched] the kernel. Tracking the memory changes consists of two steps:

* Tell the kernel that we want it to keep track of memory changes by particular task by writing 4 into <core>/proc/$pid/clear_refs</code> file.

and after a while

* Get what pages were modified by task by reading its <code>/proc/$pid/pagemap</code> file and looking at so called ''soft-dirty'' bit in the pagemap entries.

On the first step kernel will re-map all the tasks' mapping in read-only manner. If a task will try to write into one of its pages later, a page fault would occur and the kernel would note which page was written to. Reading the <code>pagemap</code> file reveals this information.

== How to use this with CRIU ==

First of all, the

#criu check

command (without the <code>--ms</code> option) should sat, that everything is OK. The thing is that the soft-dirty tracker is not yet merged upstream, thus you should make sure, that you're using the properly patched kernel.

There are several options how this functionality can be used.

;<code>--prev-images-dir</code> option
:This option says where the images from previous <code>dump</code> or <code>pre-dump</code> (see below) action reside. If possible, CRIU will not dump memory pages that hasn't changed since that time.

;<code>--track-mem</code> option
:This option makes CRIU to reset memory changes tracker. If done, the next dump <code>--prev-images-dir</code> will have chances to successfully find not changed pages.

;<code>pre-dump</code> action
:This action dumps only part of information about processes and does this keeping tasks frozen for minimally possible time. The images generated by pre-dump cannot and should not be used for restore. After this action the proper <code>dump</code> should be performed with properly configured <code>--prev-images-dir</code> path.

[[Category:Under the hood]]

Remote syscall execution

2014-12-10T15:32:33Z

Paschalis Mpeis: /* Syntax */

This page describes how the <code>exec</code> command in criu works.

== Description ==

The <code>exec</code> command executes arbitrary system call from another task's context. It does so by exploiting the parasite code execution feature, which is also used to dump various process' information.

== Syntax ==

The command syntax is

<pre>
criu exec -t <pid> <syscall-name> <syscall-arguments>
</pre>

The <code>syscall-name</code> is just a name of a system call. Since typically each syscall is wrapped with the respective glibc function, you can find out information about the interesting system call in the respective man page.

The rest of the command line is treated as <code>syscall-arguments</code>. Each command line argument in this list is converted into the system call argument by the following rules:

* If an argument starts with '&', the rest of it is copied to the target task's address space (it's allocated by remotely calling the <code>mmap</code> syscall) and the pointer to this area is passed as the system call argument

* If an argument starts with '@', the rest of it is considered to be a size of a memory buffer, pointer to which is passed into a syscall and which contents is printed on the screen after the syscall returns. Note, that before pushing the argument into the syscall, the memory is ''not'' initialized.

* Otherwise, the argument is converted into an <code>unsigned long</code> number with <code>strtol</code> and passed to system call directly

* Not specified arguments (if required by system call) are set to 0

In order to execute a system call for remote task, you don't have to be root -- you should only have rights to do debugging (strace) on it.
If you want to make task perform some action, that consists of several syscalls, you should first stop it with <code>SIG_STOP</code>.

== Examples ==

=== Re-opening stdout ===

You can close the 1st fd from a task and re-open it into some other file like this

<pre>
# criu exec -t <pid> close 1
# criu exec -t <pid> open '&<path-to-file>' 2
</pre>

In the 2nd string 2 means the <code>O_RDWR</code> opening mode.

=== Adding madvise-s to a mapping ===

You can tune the task's address space with <code>madvise</code> bits like this

<pre>
# criu exec -t <pid> madvise <start> <lenght> <madvise-bits-value>
</pre>

CRIU - User contributions [en]

Memory changes tracking

Remote syscall execution