LPC2015 Checkpoint Restore notes

Welcome to Linux Plumbers Conference 2015

The structure will be short introductions to an issue or topic followed by a discussion with the audience. A limit of 3 slides per presentation is enforced to ensure focus and allocate enough time for discussions.

Please use this etherpad to take notes. Microconf leaders will be giving a TWO MINUTE summary of their microconference during the Friday afternoon closing session.

Please remember there is no video this year, so your notes are the only record of your microconference.

Miniconf leaders: Please remember to take note of the approximate number of attendees in your session(s).

SCHEDULE

Checkpoint and Restore of processes within kernel security mechanisms

Tycho Andersen of Canonical, CRIU contributor

• CRIU port to ARM
• support for AppArmor, other LSMs need help but framework exists
• integration of CRIU with seccomp
• cgroup port to usernsd and unified heirarchy: come help!
• Question: is there a mechanism/plugin/whatever to migrate a container that is tied to a FPGA or a GPU, so they should be migrated together?

Answer: yes, there is a tool for it.

CRIU in HPC

Adrian Reber, an old-time CRIU contributor, Red Hat

• research project in a university involving CRIU, started in 2009
• usage scenario: computer simulation for product develpment
• many nodes, many fails, need for a fail-tolerant HPC
• hypervisor is not the best virtualization type for HPC due to CPU & I/O overhead
• looked into BCLR (Berkeley Lab Checkpoint/Restart)
• requirements to C/R: upstream integration
• there were different attempts to implement C/R: see also: http://criu.org/Comparison_to_other_CR_projects
• key features of CRIU project: upstream inclusion, transparent solution wo pre-loading or re-compilation, most functionality in userspace, using standard interfaces as much as possible
• summary: transparent migration works, OpenMPI supports CRIU (but keeps breaking as OpenMPI moves forward very fast), Parallel migration has WIP status

CRIU on PowerPC

Laurent Dufour, ported CRIU to PowerPC architecture, IBM CRIU 1.6 ported on Power this summer (What did you do this summer? I ported CRIU to PowerPC).

• Linux kernel was ported to PowerPC arch in 1994 (See http://gate.crashing.org/doc/ppc/doc003.htm)
• Big Endian/Little Endian. PowerPC supports both.
• CRIU support only LE mode
• Laurent tell about mess in Linux kernel ABI: two ABI currently in kernel
• impacts: enabling kcmp() syscall, VDSO remapping tracking (both are available since 4.2)
• There is a general problem with checkpointing / restoring whilst in vDSO. It is possible to end up with two (old and new) entry points and there is currently no way to detect this.
• The vector register save/restore can get confused in some rare signal call stack cases. Fixing this is in the todo list.
• A hindrance to getting docker up and running in CRIU/ppc is that the images it downloads at build/startup are x86_64 images. Help requested.

CRIU Support in Docker for Native Checkpoint and Restore

Saied Kazemi, working on container migration at Google

• short history of Docker C/R
• C/R benefits: CRIU allows to save work inside container, shorten startup time, debugging using checkpoint "iage"
• new web based REPL for node.js - Tonic (https://www.npmjs.com/package/tonic). Checkpoints each step to fix mistakes by restoring from the most recent change.
• speaker show scheme with Docker client, server and Containers and explain how it works
• external C/R for Docker: four steps to C/R container in Docker
• live demo with C/R of Docker Container - looks impressive
• Manually C/R'ing with this methods leaves 'docker ps' not being able to show the container
• TODO list is available on Docker's wiki under CRIU
• broken things after C/R: logging won't work, container control is lost (docker stop, docker kill etc)
• native C/R in Docker 1.5 (current version is 1.8 - https://blog.docker.com/2015/08/docker-1-8-content-trust-toolbox-registry-orchestration/)
• C/R support already in libcontainer (runC) - live demo with Quake http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-dockercon-work.html
• useful links: CRIU PPA; GitHub: 1.5 (https://github.com/SaiedKazemi/docker) Saied Kazemi, 1.8 - Ross Bucher, current - Michael Crosby docker git branch

Slides from Docker Meetup 2014 https://speakerdeck.com/saied/experimental-docker-checkpoint-and-restore-with-criu

P.Haul: live migrating with CRIU

Pavel Emelyanov, man in a red T-shirt with microphone

• Process-Haul (like U-Haul, but about processes) See http://criu.org/P.Haul
• CRIU is a part of solution for live migration of containers
• CRIU part of live migration: perform nodes compat checks, pre-dumps in a loop, copy fs if not shared, don't forget container config
• P.Haul migrate container is two steps
• More information about P.Haul: written in Python, source code https://github.com/xemul/p.haul/, can migrate OpenVZ container (How-to https://github.com/xemul/p.haul/blob/master/test/mtouch/HOWTO), call rsync on chroot-ed container
• P.Haul future: Docker/LXC support, smarter interations, shared FS autodetection, post-copy live migration, send images over the wire (CRIU)
• Discussion of P.Haul started

LXC has a similar concept, but slightly different. Consensus seems to be easy to achieve as to how to merge (it just needs coding effort). Predump allows process to continue running, vs full dump which is a freeze

Rebootless Kernel Update and its verification

Sanidhya Kashyap, PhD student (Georgia Institute of Technology) and CRIU contributor http://www.cc.gatech.edu/~skashyap/

• types of OS update with low downtime: dynamic kernel patching (KSplice, Kpatch and Kgraft)
• scheme with typical OS update
• KUP: update with C/R of applications
• CRIU benefits to compare similar projects
• opportunities in each phase of KUP: checkpoint -> incrementally, restore -> on-demand, C/R -> new abstraction
• reusing memory of application across reboot
• memory reuse: PPP (Persistent Physical Pages)
• evaluation of each techniques performance
• evaluation: KUP vs kpatch
• evaluation: end-to-end performance of memcached
• evaluation: microbenchmark
• Conclusion: implement native support for page cache, hugepages; on-demand restore with usefaultd; need to overhaul kexec

Using CRIU for Computer Architecture and Software Studies

Christopher Covington, spec in performance analysis and simulation at Qualcomm, http://covlibre.net/

• speaker starts from terminology, it seems talk will be quite technical :P zOMG a technical talk? Isn't it a sales conference?? (it was a joke :) ) (I'll buy two) Take three and get a 25% discount!
• Counting instructions seems preferable to counting time in order to gain determinancy of elapsed work performed.
• Useful trick of dumping after zero instructions to gather an image complete with file descriptors, environment, etc, as startup conditions

CRIU project wiki: http://criu.org/ || Google+ page: https://plus.google.com/+CriuOrg || Twitter: @__criu__ https://twitter.com/__criu__

>>> anyone can write to etherpad, please help us filling in the notes <<<

                      `       ,           
      .              `       ,+.          
       +         .+'        '+'+          
      +;+;       `         ++++:+         
     :.++++     :   +     ;+++++          
      ++++++    ;   +    `++++++'         
     +++++++'   ,        ++++++++         
      :++++++'   +      +++++++,          
      :'+++++++  +;    ++++++++:          
      ++++++++++ `+:  `++++++':.          
       +;+'+++++  ++, '+++++'+.           
        +;+;+++++  ++.+++++ +             
          ,+:;++++,+++++ :                
           .   :++++++'                   
                 ,.+++                    
                  ++;                     
                 +,                       
               .`   ;+                    
             .++  .+++ ++'`++ ++ ,++  '+' 
         +++++++; ++   ++. +; ++ :++  ++, 
         :++++++  ++   ++'+.  ++ '+:  ++` 
       .;.+++++.  +'   ++     ++ ++   ++  
     .'`..:., +`  +'   ++`++  ++ ++   ++  
    `;. .,.  `:   ++ . ++  +; ++ ++   ++  
    ., .,,.  ..   +++' ++  ++ ++ ++,,,++  
    .. .;:...;`   ``          +    ,+,    
    `,...;.;,`                      +     
     ,...+........,:,...     ,      +     
     ..'+';'++':.......,+:.  .            
    ..'...+.......... `.++'.              
    ....,...;++++++.`.;+'.              
    .+. ...........':: ....               
    .,`  ,:................               
   ....`  .......```, ..                
   ..+'.     ...',..`   .                 
   ..++.       ..++..                     
   ..+'.`     `..+++.                     
    .....      ..+++.                     
               `.'+'.                     
                 .'                       
                 .`                       
                 `