Difference between revisions of "Memory images deduplication"
(created!) |
|||
| (6 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | When performing [[incremental dumps]] or [[iterative migration]], a layered stack of memory images is created. In that stack, some data is duplicated (i.e. same memory page is present in multiple images). This article describes ways to deduplicate such data by punching holes in image files (using <code>fallocate()</code> syscall with <code>FALLOC_FL_PUNCH_HOLE</code> flag). | + | When performing [[incremental dumps]] or [[iterative migration]], a layered stack of memory images is created. In that stack, some data is duplicated (i.e. same memory page is present in multiple images). This article describes ways to deduplicate such data by punching holes in image files (using <code>fallocate()</code> syscall with <code>FALLOC_FL_PUNCH_HOLE</code> flag), effectively freeing used disk space. |
| + | |||
| + | == Deduplication mode == | ||
Two ways to deduplicate memory images are available. | Two ways to deduplicate memory images are available. | ||
| − | == | + | === Offline === |
The <code>criu dedup</code> command opens the image directory and punches holes in the ''parent'' images where ''child'' images would replace them. | The <code>criu dedup</code> command opens the image directory and punches holes in the ''parent'' images where ''child'' images would replace them. | ||
| − | == On the fly deduplication == | + | === On the fly === |
| + | |||
| + | The <code>--auto-dedup</code> option can be used for <code>criu dump</code>, <code>criu pre-dump</code> and <code>criu page-server</code>. It causes every write to images with process' pages to punch holes in the respective parent images, which is extremely useful in [[disk-less migration]] scenario. | ||
| + | |||
| + | The <code>--auto-dedup</code> option can also be used for <code>criu restore</code>. This makes CRIU to punch holes in images as memory is being restored. This should be used if images are stored on tmpfs (i.e. in RAM, see [[disk-less migration]]), as this way RAM usage is not growing. | ||
| + | |||
| + | == Shared memory deduplication == | ||
| + | ''Main article: [[Shared memory]]'' | ||
| + | |||
| + | Deduplication only makes sense for incremental memory dumps. For now CRIU can only track changes, create incremental checkpoints and do dedup for anonymous memory. Changes tracking, increments and deduplication for shared memory is currently (August 2016) available in CRIU development branch. | ||
| + | |||
| + | == Implementation notes == | ||
| − | + | Memory images are stored into two files: ''pagemap'' and ''pages'' (see [[memory dumps]] for details). Note that the deduplication process does not change ''pagemap'' in any way, it only punches holes in ''pages'' image files. | |
| − | + | Note that having a hole in an image file have totally different meaning that is in no way similar to the one of '''in_parent''' flag in ''pagemap'' entry (described in [[memory dumps]]). | |
== See also == | == See also == | ||
| + | * [[Memory dumps]] | ||
* [[Iterative migration]] | * [[Iterative migration]] | ||
* [[Incremental dumps]] | * [[Incremental dumps]] | ||
* [[Disk-less migration]] | * [[Disk-less migration]] | ||
* [[Page server]] | * [[Page server]] | ||
| + | |||
| + | == External links == | ||
| + | |||
| + | * http://man7.org/linux/man-pages/man2/fallocate.2.html | ||
[[Category: Memory]] | [[Category: Memory]] | ||
[[Category: Images]] | [[Category: Images]] | ||
| + | [[Category: API]] | ||
| + | [[Category: Under the hood]] | ||
Latest revision as of 15:12, 31 August 2016
When performing incremental dumps or iterative migration, a layered stack of memory images is created. In that stack, some data is duplicated (i.e. same memory page is present in multiple images). This article describes ways to deduplicate such data by punching holes in image files (using fallocate() syscall with FALLOC_FL_PUNCH_HOLE flag), effectively freeing used disk space.
Deduplication mode[edit]
Two ways to deduplicate memory images are available.
Offline[edit]
The criu dedup command opens the image directory and punches holes in the parent images where child images would replace them.
On the fly[edit]
The --auto-dedup option can be used for criu dump, criu pre-dump and criu page-server. It causes every write to images with process' pages to punch holes in the respective parent images, which is extremely useful in disk-less migration scenario.
The --auto-dedup option can also be used for criu restore. This makes CRIU to punch holes in images as memory is being restored. This should be used if images are stored on tmpfs (i.e. in RAM, see disk-less migration), as this way RAM usage is not growing.
[edit]
Main article: Shared memory
Deduplication only makes sense for incremental memory dumps. For now CRIU can only track changes, create incremental checkpoints and do dedup for anonymous memory. Changes tracking, increments and deduplication for shared memory is currently (August 2016) available in CRIU development branch.
Implementation notes[edit]
Memory images are stored into two files: pagemap and pages (see memory dumps for details). Note that the deduplication process does not change pagemap in any way, it only punches holes in pages image files.
Note that having a hole in an image file have totally different meaning that is in no way similar to the one of in_parent flag in pagemap entry (described in memory dumps).