Difference between revisions of "LPC2015 Checkpoint Restore notes"

From CRIU
Jump to navigation Jump to search
(Created page with "== Checkpoint/Restore Microconference Notes == Welcome to Linux Plumbers Conference 2015 The structure will be short introductions to an issue or topic followed by a discuss...")
 
 
(4 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Checkpoint/Restore Microconference Notes ==
+
Welcome to Linux Plumbers Conference 2015
  
Welcome to Linux Plumbers Conference 2015
+
[[Image:LPC-microconf-2015.jpg|right|thumb|300px]]
  
 
The structure will be short introductions to an issue or topic followed by a discussion with the audience.
 
The structure will be short introductions to an issue or topic followed by a discussion with the audience.
Line 14: Line 14:
 
SCHEDULE
 
SCHEDULE
  
==== Checkpoint and Restore of processes within kernel security mechanisms ====
+
== Checkpoint and Restore of processes within kernel security mechanisms ==
 
Tycho Andersen of Canonical, CRIU contributor
 
Tycho Andersen of Canonical, CRIU contributor
  
Line 24: Line 24:
 
Answer: yes, there is a tool for it.
 
Answer: yes, there is a tool for it.
  
 
+
== CRIU in HPC ==
==== CRIU in HPC ====
+
Adrian Reber, an old-time CRIU contributor, Red Hat
Adrian Reber, an old-time CRIU contributor, RedHat
 
 
* research project in a university involving CRIU, started in 2009
 
* research project in a university involving CRIU, started in 2009
 
* usage scenario: computer simulation for product develpment
 
* usage scenario: computer simulation for product develpment
Line 37: Line 36:
 
* summary: transparent migration works, OpenMPI supports CRIU (but keeps breaking as OpenMPI moves forward very fast), Parallel migration has WIP status
 
* summary: transparent migration works, OpenMPI supports CRIU (but keeps breaking as OpenMPI moves forward very fast), Parallel migration has WIP status
  
 
+
== CRIU on PowerPC ==
==== CRIU on PowerPC ====
 
 
Laurent Dufour, ported CRIU to PowerPC architecture, IBM
 
Laurent Dufour, ported CRIU to PowerPC architecture, IBM
 
CRIU 1.6 ported on Power this summer (What did you do this summer? I ported CRIU to PowerPC).
 
CRIU 1.6 ported on Power this summer (What did you do this summer? I ported CRIU to PowerPC).
Line 50: Line 48:
 
* A hindrance to getting docker up and running in CRIU/ppc is that the images it downloads at build/startup are x86_64 images. Help requested.
 
* A hindrance to getting docker up and running in CRIU/ppc is that the images it downloads at build/startup are x86_64 images. Help requested.
  
 
+
== CRIU Support in Docker for Native Checkpoint and Restore ==
==== CRIU Support in Docker for Native Checkpoint and Restore ====
 
 
Saied Kazemi, working on container migration at Google
 
Saied Kazemi, working on container migration at Google
 
* short history of Docker C/R
 
* short history of Docker C/R
Line 68: Line 65:
 
Slides from Docker Meetup 2014 https://speakerdeck.com/saied/experimental-docker-checkpoint-and-restore-with-criu
 
Slides from Docker Meetup 2014 https://speakerdeck.com/saied/experimental-docker-checkpoint-and-restore-with-criu
  
==== Break (2:45 - 3:00) ====
+
== P.Haul: live migrating with CRIU ==
 
 
==== P.Haul: live migrating with CRIU ====
 
 
Pavel Emelyanov, man in a red T-shirt with microphone
 
Pavel Emelyanov, man in a red T-shirt with microphone
 
* Process-Haul (like U-Haul, but about processes) See http://criu.org/P.Haul
 
* Process-Haul (like U-Haul, but about processes) See http://criu.org/P.Haul
Line 76: Line 71:
 
* CRIU part of live migration: perform nodes compat checks, pre-dumps in a loop, copy fs if not shared, don't forget container config
 
* CRIU part of live migration: perform nodes compat checks, pre-dumps in a loop, copy fs if not shared, don't forget container config
 
* P.Haul migrate container is two steps
 
* P.Haul migrate container is two steps
* More information about P.Haul: written in Python, source code https://github.com/xemul/p.haul/, can migrate OpenVZ container (How-to https://github.com/xemul/p.haul/blob/master/test/mtouch/HOWTO), call rsync on chroot-ed container
+
* More information about P.Haul: written in Python, source code https://github.com/checkpoint-restore/p.haul/, can migrate OpenVZ container (How-to https://github.com/checkpoint-restore/p.haul/blob/master/test/mtouch/HOWTO), call rsync on chroot-ed container
 
* P.Haul future: Docker/LXC support, smarter interations, shared FS autodetection, post-copy live migration, send images over the wire (CRIU)
 
* P.Haul future: Docker/LXC support, smarter interations, shared FS autodetection, post-copy live migration, send images over the wire (CRIU)
 
* Discussion of P.Haul started
 
* Discussion of P.Haul started
Line 82: Line 77:
 
Predump allows process to continue running, vs full dump which is a freeze
 
Predump allows process to continue running, vs full dump which is a freeze
  
 
+
== Rebootless Kernel Update and its verification ==
==== Rebootless Kernel Update and its verification ====
 
 
Sanidhya Kashyap, PhD student (Georgia Institute of Technology) and CRIU contributor http://www.cc.gatech.edu/~skashyap/
 
Sanidhya Kashyap, PhD student (Georgia Institute of Technology) and CRIU contributor http://www.cc.gatech.edu/~skashyap/
 
* types of OS update with low downtime: dynamic kernel patching (KSplice, Kpatch and Kgraft)
 
* types of OS update with low downtime: dynamic kernel patching (KSplice, Kpatch and Kgraft)
Line 98: Line 92:
 
* Conclusion: implement native support for page cache, hugepages; on-demand restore with usefaultd; need to overhaul kexec
 
* Conclusion: implement native support for page cache, hugepages; on-demand restore with usefaultd; need to overhaul kexec
  
 
+
== Using CRIU for Computer Architecture and Software Studies ==
==== Using CRIU for Computer Architecture and Software Studies ====
+
Christopher Covington, spec in performance analysis and simulation at Qualcomm,  http://covlibre.net/
Christopher Covington, in performance analysis and simulation at Qualcomm,  http://covlibre.net/
 
 
* speaker starts from terminology, it seems talk will be quite technical :P zOMG a technical talk? Isn't it a sales conference?? (it was a joke :) ) (I'll buy two) Take three and get a 25% discount!
 
* speaker starts from terminology, it seems talk will be quite technical :P zOMG a technical talk? Isn't it a sales conference?? (it was a joke :) ) (I'll buy two) Take three and get a 25% discount!
 
* Counting instructions seems preferable to counting time in order to gain determinancy of elapsed work performed.
 
* Counting instructions seems preferable to counting time in order to gain determinancy of elapsed work performed.

Latest revision as of 08:46, 28 September 2017

Welcome to Linux Plumbers Conference 2015

LPC-microconf-2015.jpg

The structure will be short introductions to an issue or topic followed by a discussion with the audience. A limit of 3 slides per presentation is enforced to ensure focus and allocate enough time for discussions.

Please use this etherpad to take notes. Microconf leaders will be giving a TWO MINUTE summary of their microconference during the Friday afternoon closing session.

Please remember there is no video this year, so your notes are the only record of your microconference.

Miniconf leaders: Please remember to take note of the approximate number of attendees in your session(s).

SCHEDULE

Checkpoint and Restore of processes within kernel security mechanisms[edit]

Tycho Andersen of Canonical, CRIU contributor

  • CRIU port to ARM
  • support for AppArmor, other LSMs need help but framework exists
  • integration of CRIU with seccomp
  • cgroup port to usernsd and unified heirarchy: come help!
  • Question: is there a mechanism/plugin/whatever to migrate a container that is tied to a FPGA or a GPU, so they should be migrated together?

Answer: yes, there is a tool for it.

CRIU in HPC[edit]

Adrian Reber, an old-time CRIU contributor, Red Hat

  • research project in a university involving CRIU, started in 2009
  • usage scenario: computer simulation for product develpment
  • many nodes, many fails, need for a fail-tolerant HPC
  • hypervisor is not the best virtualization type for HPC due to CPU & I/O overhead
  • looked into BCLR (Berkeley Lab Checkpoint/Restart)
  • requirements to C/R: upstream integration
  • there were different attempts to implement C/R: see also: http://criu.org/Comparison_to_other_CR_projects
  • key features of CRIU project: upstream inclusion, transparent solution wo pre-loading or re-compilation, most functionality in userspace, using standard interfaces as much as possible
  • summary: transparent migration works, OpenMPI supports CRIU (but keeps breaking as OpenMPI moves forward very fast), Parallel migration has WIP status

CRIU on PowerPC[edit]

Laurent Dufour, ported CRIU to PowerPC architecture, IBM CRIU 1.6 ported on Power this summer (What did you do this summer? I ported CRIU to PowerPC).

  • Linux kernel was ported to PowerPC arch in 1994 (See http://gate.crashing.org/doc/ppc/doc003.htm)
  • Big Endian/Little Endian. PowerPC supports both.
  • CRIU support only LE mode
  • Laurent tell about mess in Linux kernel ABI: two ABI currently in kernel
  • impacts: enabling kcmp() syscall, VDSO remapping tracking (both are available since 4.2)
  • There is a general problem with checkpointing / restoring whilst in vDSO. It is possible to end up with two (old and new) entry points and there is currently no way to detect this.
  • The vector register save/restore can get confused in some rare signal call stack cases. Fixing this is in the todo list.
  • A hindrance to getting docker up and running in CRIU/ppc is that the images it downloads at build/startup are x86_64 images. Help requested.

CRIU Support in Docker for Native Checkpoint and Restore[edit]

Saied Kazemi, working on container migration at Google

  • short history of Docker C/R
  • C/R benefits: CRIU allows to save work inside container, shorten startup time, debugging using checkpoint "iage"
  • new web based REPL for node.js - Tonic (https://www.npmjs.com/package/tonic). Checkpoints each step to fix mistakes by restoring from the most recent change.
  • speaker show scheme with Docker client, server and Containers and explain how it works
  • external C/R for Docker: four steps to C/R container in Docker
  • live demo with C/R of Docker Container - looks impressive
  • Manually C/R'ing with this methods leaves 'docker ps' not being able to show the container
  • TODO list is available on Docker's wiki under CRIU
  • broken things after C/R: logging won't work, container control is lost (docker stop, docker kill etc)
  • native C/R in Docker 1.5 (current version is 1.8 - https://blog.docker.com/2015/08/docker-1-8-content-trust-toolbox-registry-orchestration/)
  • C/R support already in libcontainer (runC) - live demo with Quake http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-dockercon-work.html
  • useful links: CRIU PPA; GitHub: 1.5 (https://github.com/SaiedKazemi/docker) Saied Kazemi, 1.8 - Ross Bucher, current - Michael Crosby docker git branch

Slides from Docker Meetup 2014 https://speakerdeck.com/saied/experimental-docker-checkpoint-and-restore-with-criu

P.Haul: live migrating with CRIU[edit]

Pavel Emelyanov, man in a red T-shirt with microphone

  • Process-Haul (like U-Haul, but about processes) See http://criu.org/P.Haul
  • CRIU is a part of solution for live migration of containers
  • CRIU part of live migration: perform nodes compat checks, pre-dumps in a loop, copy fs if not shared, don't forget container config
  • P.Haul migrate container is two steps
  • More information about P.Haul: written in Python, source code https://github.com/checkpoint-restore/p.haul/, can migrate OpenVZ container (How-to https://github.com/checkpoint-restore/p.haul/blob/master/test/mtouch/HOWTO), call rsync on chroot-ed container
  • P.Haul future: Docker/LXC support, smarter interations, shared FS autodetection, post-copy live migration, send images over the wire (CRIU)
  • Discussion of P.Haul started

LXC has a similar concept, but slightly different. Consensus seems to be easy to achieve as to how to merge (it just needs coding effort). Predump allows process to continue running, vs full dump which is a freeze

Rebootless Kernel Update and its verification[edit]

Sanidhya Kashyap, PhD student (Georgia Institute of Technology) and CRIU contributor http://www.cc.gatech.edu/~skashyap/

  • types of OS update with low downtime: dynamic kernel patching (KSplice, Kpatch and Kgraft)
  • scheme with typical OS update
  • KUP: update with C/R of applications
  • CRIU benefits to compare similar projects
  • opportunities in each phase of KUP: checkpoint -> incrementally, restore -> on-demand, C/R -> new abstraction
  • reusing memory of application across reboot
  • memory reuse: PPP (Persistent Physical Pages)
  • evaluation of each techniques performance
  • evaluation: KUP vs kpatch
  • evaluation: end-to-end performance of memcached
  • evaluation: microbenchmark
  • Conclusion: implement native support for page cache, hugepages; on-demand restore with usefaultd; need to overhaul kexec

Using CRIU for Computer Architecture and Software Studies[edit]

Christopher Covington, spec in performance analysis and simulation at Qualcomm, http://covlibre.net/

  • speaker starts from terminology, it seems talk will be quite technical :P zOMG a technical talk? Isn't it a sales conference?? (it was a joke :) ) (I'll buy two) Take three and get a 25% discount!
  • Counting instructions seems preferable to counting time in order to gain determinancy of elapsed work performed.
  • Useful trick of dumping after zero instructions to gather an image complete with file descriptors, environment, etc, as startup conditions

CRIU project wiki: http://criu.org/ || Google+ page: https://plus.google.com/+CriuOrg || Twitter: @__criu__ https://twitter.com/__criu__

>>> anyone can write to etherpad, please help us filling in the notes <<<


                      `       ,           
      .              `       ,+.          
       +         .+'        '+'+          
      +;+;       `         ++++:+         
     :.++++     :   +     ;+++++          
      ++++++    ;   +    `++++++'         
     +++++++'   ,        ++++++++         
      :++++++'   +      +++++++,          
      :'+++++++  +;    ++++++++:          
      ++++++++++ `+:  `++++++':.          
       +;+'+++++  ++, '+++++'+.           
        +;+;+++++  ++.+++++ +             
          ,+:;++++,+++++ :                
           .   :++++++'                   
                 ,.+++                    
                  ++;                     
                 +,                       
               .`   ;+                    
             .++  .+++ ++'`++ ++ ,++  '+' 
         +++++++; ++   ++. +; ++ :++  ++, 
         :++++++  ++   ++'+.  ++ '+:  ++` 
       .;.+++++.  +'   ++     ++ ++   ++  
     .'`..:., +`  +'   ++`++  ++ ++   ++  
    `;. .,.  `:   ++ . ++  +; ++ ++   ++  
    ., .,,.  ..   +++' ++  ++ ++ ++,,,++  
    .. .;:...;`   ``          +    ,+,    
    `,...;.;,`                      +     
     ,...+........,:,...     ,      +     
     ..'+';'++':.......,+:.  .            
    ..'...+.......... `.++'.              
    ....,...;++++++.`.;+'.              
    .+. ...........':: ....               
    .,`  ,:................               
   ....`  .......```, ..                
   ..+'.     ...',..`   .                 
   ..++.       ..++..                     
   ..+'.`     `..+++.                     
    .....      ..+++.                     
               `.'+'.                     
                 .'                       
                 .`                       
                 `