Difference between revisions of "Memory changes tracking"
Jump to navigation
Jump to search
| (12 intermediate revisions by 4 users not shown) | |||
| Line 1: | Line 1: | ||
| − | CRIU can detect what memory pages a task (or tasks) has changed since some moment of time. This page describes why this | + | CRIU can detect what memory pages a task (or tasks) has changed since some moment of time. This page describes why this is required, how it works and how to use it. |
== Why do we need to track memory changed == | == Why do we need to track memory changed == | ||
| − | There are several scenarios | + | There are several scenarios where detecting what parts of memory has changed is required: |
; [[Incremental dumps]] | ; [[Incremental dumps]] | ||
: When you take a series of dumps from a process tree, it is a very good optimization not to dump ''all'' the memory every time, but get only those memory pages that has changed since previous dump | : When you take a series of dumps from a process tree, it is a very good optimization not to dump ''all'' the memory every time, but get only those memory pages that has changed since previous dump | ||
| − | ; Smaller freeze time for big | + | ; Smaller freeze time for big applications |
: When a task uses a LOT of memory, dumping it may take time and during all this time this task should be frozen. To reduce the freeze time we can | : When a task uses a LOT of memory, dumping it may take time and during all this time this task should be frozen. To reduce the freeze time we can | ||
| − | * get memory from task and start writing it in images | + | :* get memory from task and start writing it in images |
| − | * freeze task and get only changed memory from it | + | :* freeze task and get only changed memory from it |
; [[Live migration]] | ; [[Live migration]] | ||
| − | : When doing live migration a lot of time | + | : When doing live migration, a lot of time is used by the procedure of copying tasks' memory to the destination host. Note that the processes are frozen during that time. Acting like in the previous example also reduces the freeze time, i.e. the live migration becomes more live. |
== How we track memory changes == | == How we track memory changes == | ||
| − | In order to find out | + | In order to find out which memory pages have changed, we [http://lwn.net/Articles/546966/ patched] the kernel. Tracking the memory changes consists of two steps: |
| − | * | + | * ask the kernel to keep track of memory changes (by writing 4 into <code>/proc/$pid/clear_refs</code> file for each $pid we are interested in). |
| − | and after a while | + | and, after a while, |
| − | * | + | * get the list of modified pages of a process by reading its <code>/proc/$pid/pagemap</code> file and looking at so called ''soft-dirty'' bit in the pagemap entries. |
| − | + | During the first step, kernel will re-map all the tasks' mapping in read-only manner. If a task then tries to write into any of its pages, a page fault will occur, and the kernel will note which page is being written to. Reading the <code>pagemap</code> file reveals this information. | |
| − | |||
== How to use this with CRIU == | == How to use this with CRIU == | ||
| Line 33: | Line 32: | ||
First of all, the | First of all, the | ||
| − | #criu check | + | # criu check --feature mem_dirty_track |
| − | command | + | command should say the feature is supported. The memory changes tracking was initially merged into Linux kernel v3.11, and was further polished until v3.18 (see [[Upstream kernel commits]] for details). |
| − | There are several options | + | There are several command line options to use the functionality: |
;<code>--prev-images-dir</code> option | ;<code>--prev-images-dir</code> option | ||
| − | :This option | + | :This option is used to provide the path where images from a previous <code>dump</code> or <code>pre-dump</code> (see below) action reside. If possible, CRIU will dump only the memory pages that have been modified since that time. |
;<code>--track-mem</code> option | ;<code>--track-mem</code> option | ||
| Line 46: | Line 45: | ||
;<code>pre-dump</code> action | ;<code>pre-dump</code> action | ||
| − | :This action dumps only part of information about processes and does | + | :This action dumps only part of the information about processes and does that by keeping tasks frozen for the shortest possible time. The images generated by pre-dump cannot and should not be used for restore. After this action the proper <code>dump</code> should be performed with properly configured <code>--prev-images-dir</code> path. |
| + | |||
| + | == See also == | ||
| + | * [[Live migration]] | ||
| + | * [[Incremental dumps]] | ||
| + | * [[Memory dumping and restoring#Advanced C/R]] | ||
| + | * [[Directories]] | ||
| + | * [[Memory pre dump]] | ||
| + | |||
| + | == External links == | ||
| + | * http://lwn.net/Articles/546966/ | ||
| − | [[Category:Under the hood]] | + | [[Category: Under the hood]] |
| + | [[Category: Memory]] | ||
| + | [[Category: Live migration]] | ||
Latest revision as of 10:14, 27 March 2019
CRIU can detect what memory pages a task (or tasks) has changed since some moment of time. This page describes why this is required, how it works and how to use it.
Why do we need to track memory changed[edit]
There are several scenarios where detecting what parts of memory has changed is required:
- Incremental dumps
- When you take a series of dumps from a process tree, it is a very good optimization not to dump all the memory every time, but get only those memory pages that has changed since previous dump
- Smaller freeze time for big applications
- When a task uses a LOT of memory, dumping it may take time and during all this time this task should be frozen. To reduce the freeze time we can
- get memory from task and start writing it in images
- freeze task and get only changed memory from it
- Live migration
- When doing live migration, a lot of time is used by the procedure of copying tasks' memory to the destination host. Note that the processes are frozen during that time. Acting like in the previous example also reduces the freeze time, i.e. the live migration becomes more live.
How we track memory changes[edit]
In order to find out which memory pages have changed, we patched the kernel. Tracking the memory changes consists of two steps:
- ask the kernel to keep track of memory changes (by writing 4 into
/proc/$pid/clear_refsfile for each $pid we are interested in).
and, after a while,
- get the list of modified pages of a process by reading its
/proc/$pid/pagemapfile and looking at so called soft-dirty bit in the pagemap entries.
During the first step, kernel will re-map all the tasks' mapping in read-only manner. If a task then tries to write into any of its pages, a page fault will occur, and the kernel will note which page is being written to. Reading the pagemap file reveals this information.
How to use this with CRIU[edit]
First of all, the
# criu check --feature mem_dirty_track
command should say the feature is supported. The memory changes tracking was initially merged into Linux kernel v3.11, and was further polished until v3.18 (see Upstream kernel commits for details).
There are several command line options to use the functionality:
--prev-images-diroption- This option is used to provide the path where images from a previous
dumporpre-dump(see below) action reside. If possible, CRIU will dump only the memory pages that have been modified since that time.
--track-memoption- This option makes CRIU to reset memory changes tracker. If done, the next dump
--prev-images-dirwill have chances to successfully find not changed pages.
pre-dumpaction- This action dumps only part of the information about processes and does that by keeping tasks frozen for the shortest possible time. The images generated by pre-dump cannot and should not be used for restore. After this action the proper
dumpshould be performed with properly configured--prev-images-dirpath.
See also[edit]
- Live migration
- Incremental dumps
- Memory dumping and restoring#Advanced C/R
- Directories
- Memory pre dump