Difference between revisions of "Userfaultfd"

Revision as of 07:48, 14 May 2018

This article describes usage of userfaultfd for lazy restore and lazy migration in CRIU.

Background

The userfaultfd mechanism is designed to allow user-space paging. Its initial implementation merged in Linux 4.3 was designed for KVM/QEMU use-case and lacked some functionality necessary for CRIU. In Linux 4.11 the userfaultfd was extended with so-called "non-cooperative" mode, that allows, at least in theory, lazy (or post-copy) restore in CRIU.

Concepts

The restore action accepts yet another API switch: option --lazy-pages. In this mode, restore skips injection of lazy pages into the processes address space, but rather registers lazy memory areas with userfaultfd.
The lazy pages are completely handled by dedicated lazy-pages daemon. The daemon receives userfault file descriptors from restore via UNIX socket. The userfault file descriptors allow reception of page-fault and other events and resolution of these events by the daemon.
For the migration case, the dump action also accepts API switch: option --lazy-pages. When this option is used, the dump keeps the memory pages and allows the lazy-pages daemon to request these pages via TCP connection.

Daemon

Tasks after restore have lazy VMAs registered with userfaultfd, the fd itself is sent before resume to lazy-pages daemon and closed. The daemon monitors the UFFD events and repopulates the tasks address space. The lazy-pages daemon can get pages either from images (both local and remote) or directly from the remote side dump.

When the restored task accesses a missing memory page, it causes a page fault. The lazy-pages daemon receives the page fault notification and resolves it by populating the faulting task memory. If there were no page faults for some time, the daemon copies the task's remaining memory pages in the background.

Local images

The daemon uses local page-read engine to read pages from images.

Remote images

The page server is run on the remote side with --lazy-pages option.
The lazy-pages daemon connects to the remote page server with --page-server option. The --address and --port options allow setting of IP address and port of the listening page server.
Current protocol allows the lazy-pages daemon to request several continuous pages.

Migration

The dump collects the pages into pipes and starts the page server in a mode that allows lazy-pages daemon to connect to it and request the memory pages
When the restored task accesses a missing memory page, the lazy-pages daemon request the page from the page server running on the dump side
After the page is received, the lazy-pages daemon injects it into the task's address space using userfautlfd

Limitations

Currently only MAP_PRIVATE | MAP_ANONYMOUS is supported. Newer kernels (4.11+) allow userfaultfd for hugetlbfs and shared memory, yet to be implemented in CRIU.
Userfault is known not to map one page into two places. Thus -- COW-ed pages will get COW-ed.
The Lazy migration use-case might be racy because there is no means to synchronize between pending forks, remote pages transfers and page faults.

@@ Line 1: / Line 1: @@
-Collected on this page is the design notes about supporting userfaultfd in CRIU
+This article describes usage of userfaultfd for lazy restore and lazy migration in CRIU.
+== Background ==
+The [http://man7.org/linux/man-pages/man2/userfaultfd.2.html userfaultfd] mechanism is designed to allow user-space paging. Its initial implementation merged in Linux 4.3 was designed for KVM/QEMU use-case and lacked some functionality necessary for CRIU. In Linux 4.11 the userfaultfd was extended with so-called "non-cooperative" mode, that allows, at least in theory, lazy (or post-copy) restore in CRIU.
 == Concepts ==
-* Only MAP_PRIVATE | MAP_ANONYMOUS will be supported in the 1st version due to kernel constraints.
+* The <code>restore</code> action accepts yet another API switch: option <code>--lazy-pages</code>. In this mode, <code>restore</code> skips injection of lazy pages into the processes address space, but rather registers lazy memory areas with userfaultfd.
-* The <code>restore</code> action should accept yet another [[CLI]] option <code>--lazy-pages</code>.
+* The lazy pages are completely handled by dedicated <code>lazy-pages</code> daemon. The daemon receives userfault file descriptors from <code>restore</code> via UNIX socket. The userfault file descriptors allow reception of page-fault and other events and resolution of these events by the daemon.
+* For the migration case, the <code>dump</code> action also accepts API switch: option <code>--lazy-pages</code>. When this option is used, the <code>dump</code> keeps the memory pages and allows the <code>lazy-pages</code> daemon to request these pages via TCP connection.
+[[File:Criu-memory-wflow.png]]
 === Daemon ===
-Tasks after restore should have lazy VMAs being backed by userfaultfd, the fd itself should be sent before resume to CRIU daemon and closed. This is that daemon who will monitor the UFFD events and repopulate the tasks address space. The daemon should be able to get pages from both -- remote and local images.
+Tasks after restore have lazy VMAs registered with userfaultfd, the fd itself is sent before resume to <code>lazy-pages</code> daemon and closed. The daemon monitors the UFFD events and repopulates the tasks address space. The <code>lazy-pages</code> daemon can get pages either from images (both local and remote) or directly from the remote side <code>dump</code>.
+When the restored task accesses a missing memory page, it causes a page fault. The <code>lazy-pages</code> daemon receives the page fault notification and resolves it by populating the faulting task memory. If there were no page faults for some time, the daemon copies the task's remaining memory pages in the background.
+==== Local images ====
+The daemon uses local page-read engine to read pages from images.
+==== Remote images ====
+* The [[page server]] is run on the remote side with <code>--lazy-pages</code> option.
+* The lazy-pages daemon connects to the remote [[page server]] with <code>--page-server</code> option. The <code>--address</code> and <code>--port</code> options allow setting of IP address and port of the listening [[page server]].
+* Current protocol allows the lazy-pages daemon to request several continuous pages.
+==== Migration ====
+* The <code>dump</code> collects the pages into pipes and starts the [[page server]] in a mode that allows <code>lazy-pages</code> daemon to connect to it and request the memory pages
+* When the restored task accesses a missing memory page, the <code>lazy-pages</code> daemon request the page from the [[page server]] running on the dump side
+* After the page is received, the <code>lazy-pages</code> daemon injects it into the task's address space using userfautlfd
-=== Local images ===
+== Limitations ==
-The daemon should just use local page-read engine and read pages from images
+* Currently only MAP_PRIVATE | MAP_ANONYMOUS is supported. Newer kernels (4.11+) allow userfaultfd for hugetlbfs and shared memory, yet to be implemented in CRIU.
+* Userfault is known not to map one page into two places. Thus -- COW-ed pages will get COW-ed.
+* The [[Lazy migration]] use-case might be racy because there is no means to synchronize between pending forks, remote pages transfers and page faults.
-=== Remote images ===
+== See also ==
-* The page-read engine should be patched to learn how to talk to the [[page server]] on the other end
+* [[Disk-less migration]]
-* The page server should get pages from tasks dumped and send them out on the destination node.
+* [[Lazy migration]]
-* Protocol should include out-of-order pages and background pages pushing (sending them before demand from the process)
-* The now dump-only <code>--page-server</code> option should be used for restore
 [[Category:Memory]]
-[[Category:Plans]]
+[[Category:New features]]
-[[Category:Development]]
+[[Category:Under the hood]]
-[[Category:Empty articles]]

Difference between revisions of "Userfaultfd"

Revision as of 07:48, 14 May 2018

Contents