Memory images deduplication - Revision history

Xemul: /* On the fly */

2016-08-31T15:12:09Z

On the fly

Xemul: /* Shared memory deduplication */

2016-08-31T15:11:35Z

Shared memory deduplication

Xemul: /* Shared memory deduplication */

2016-08-31T11:19:45Z

Shared memory deduplication

Xemul: /* Shared memory deduplication */

2016-08-31T11:19:34Z

Shared memory deduplication

Kir: Add 'impementation notes', 'auto-dedup' description for criu restore

2016-08-31T06:13:19Z

Add 'impementation notes', 'auto-dedup' description for criu restore

Kir: add "memory dumps" to see also, api and under the hood to categories

2016-08-31T05:50:09Z

add "memory dumps" to see also, api and under the hood to categories

Kir: more on punching holes

2016-08-31T04:34:23Z

Kir: created!

2016-08-31T04:30:34Z

created!

New page

When performing [[incremental dumps]] or [[iterative migration]], a layered stack of memory images is created. In that stack, some data is duplicated (i.e. same memory page is present in multiple images). This article describes ways to deduplicate such data by punching holes in image files (using <code>fallocate()</code> syscall with <code>FALLOC_FL_PUNCH_HOLE</code> flag).

Two ways to deduplicate memory images are available.

== Deduplication action ==

The <code>criu dedup</code> command opens the image directory and punches holes in the ''parent'' images where ''child'' images would replace them.

== On the fly deduplication ==

The <code>--auto-dedup</code> option can be used for <code>criu dump</code> and <code>criu page-server</code>. It causes every write to images with process' pages to punch holes in the respective parent images.

The option is extremely useful in [[disk-less migration]] scenario.

== See also ==

* [[Iterative migration]]
* [[Incremental dumps]]
* [[Disk-less migration]]
* [[Page server]]

[[Category: Memory]]
[[Category: Images]]

@@ Line 11: / Line 11: @@
 === On the fly ===
-The <code>--auto-dedup</code> option can be used for <code>criu dump</code> and <code>criu page-server</code>. It causes every write to images with process' pages to punch holes in the respective parent images, which is extremely useful in [[disk-less migration]] scenario.
+The <code>--auto-dedup</code> option can be used for <code>criu dump</code>, <code>criu pre-dump</code> and <code>criu page-server</code>. It causes every write to images with process' pages to punch holes in the respective parent images, which is extremely useful in [[disk-less migration]] scenario.
 The <code>--auto-dedup</code> option can also be used for <code>criu restore</code>. This makes CRIU to punch holes in images as memory is being restored. This should be used if images are stored on tmpfs (i.e. in RAM, see [[disk-less migration]]), as this way RAM usage is not growing.

@@ Line 18: / Line 18: @@
 ''Main article: [[Shared memory]]''
-An experimental shared memory deduplication is currently (August 2016) available in CRIU development branch.
+Deduplication only makes sense for incremental memory dumps. For now CRIU can only track changes, create incremental checkpoints and do dedup for anonymous memory. Changes tracking, increments and deduplication for shared memory is currently (August 2016) available in CRIU development branch.
 == Implementation notes ==

@@ Line 16: / Line 16: @@
 == Shared memory deduplication ==
-''Main article: [[Shared memory]]
+''Main article: [[Shared memory]]''
 An experimental shared memory deduplication is currently (August 2016) available in CRIU development branch.

@@ Line 16: / Line 16: @@
 == Shared memory deduplication ==
+''Main article: [[Shared memory]]
 An experimental shared memory deduplication is currently (August 2016) available in CRIU development branch.

@@ Line 1: / Line 1: @@
 When performing [[incremental dumps]] or [[iterative migration]], a layered stack of memory images is created. In that stack, some data is duplicated (i.e. same memory page is present in multiple images). This article describes ways to deduplicate such data by punching holes in image files (using <code>fallocate()</code> syscall with <code>FALLOC_FL_PUNCH_HOLE</code> flag), effectively freeing used disk space.
 Two ways to deduplicate memory images are available.
-== Deduplication action ==
+=== Offline ===
 The <code>criu dedup</code> command opens the image directory and punches holes in the ''parent'' images where ''child'' images would replace them.
-== On the fly deduplication ==
+=== On the fly ===
-The <code>--auto-dedup</code> option can be used for <code>criu dump</code> and <code>criu page-server</code>. It causes every write to images with process' pages to punch holes in the respective parent images.
+Memory images are stored into two files: ''pagemap'' and ''pages'' (see [[memory dumps]] for details). Note that the deduplication process does not change ''pagemap'' in any way, it only punches holes in ''pages'' image files.
-The option is extremely useful in [[disk-less migration]] scenario.
+Note that having a hole in an image file have totally different meaning that is in no way similar to the one of '''in_parent''' flag in ''pagemap'' entry (described in [[memory dumps]]).
 == See also ==

@@ Line 15: / Line 15: @@
 == See also ==
 * [[Iterative migration]]
 * [[Incremental dumps]]
@@ Line 26: / Line 27: @@
 [[Category: Memory]]
 [[Category: Images]]