Memory changes tracking

CRIU can detect what memory pages a task (or tasks) has changed sine some moment of time. This page describes why this thing is required, how it works and how to use it.

Why do we need to track memory changed

There are several scenarios when detecting what parts of memory has changed is required:

Incremental dumps: When you take a series of dumps from a process tree, it is a very good optimization not to dump all the memory every time, but get only those memory pages that has changed since previous dump

Smaller freeze time for big Applications: When a task uses a LOT of memory, dumping it may take time and during all this time this task should be frozen. To reduce the freeze time we can

get memory from task and start writing it in images
freeze task and get only changed memory from it

Live migration: When doing live migration a lot of time takes the procedure of copying tasks' memory on remote host. And yet again, during this time process is frozen. Acting like in the previous example also reduces the freeze time, i.e. -- the live migration becomes really Live.

How we track memory changes

In order to find out what pages in memory has changed, we patched the kernel. Tracking the memory changes consists of two steps:

Tell the kernel that we want it to keep track of memory changes by particular task by writing 4 into <core>/proc/$pid/clear_refs file.

and after a while

Get what pages were modified by task by reading its /proc/$pid/pagemap file and looking at so called soft-dirty bit in the pagemap entries.

On the first step kernel will re-map all the tasks' mapping in read-only manner. If a task will try to write into one of its pages later, a page fault would occur and the kernel would note which page was written to. Reading the pagemap file reveals this information.

How to use this with CRIU

First of all, the

#criu check

command (without the --ms option) should sat, that everything is OK. The thing is that the soft-dirty tracker is not yet merged upstream, thus you should make sure, that you're using the properly patched kernel.

There are several options how this functionality can be used.

--prev-images-dir option: This option says where the images from previous dump or pre-dump (see below) action reside. If possible, CRIU will not dump memory pages that hasn't changed since that time.

--track-mem option: This option makes CRIU to reset memory changes tracker. If done, the next dump --prev-images-dir will have chances to successfully find not changed pages.

pre-dump action: This action dumps only part of information about processes and does this keeping tasks frozen for minimally possible time. The images generated by pre-dump cannot and should not be used for restore. After this action the proper dump should be performed with properly configured --prev-images-dir path.