Difference between revisions of "LPC2015 Checkpoint Restore notes"

Revision as of 21:01, 21 August 2015

Welcome to Linux Plumbers Conference 2015

The structure will be short introductions to an issue or topic followed by a discussion with the audience. A limit of 3 slides per presentation is enforced to ensure focus and allocate enough time for discussions.

Please use this etherpad to take notes. Microconf leaders will be giving a TWO MINUTE summary of their microconference during the Friday afternoon closing session.

Please remember there is no video this year, so your notes are the only record of your microconference.

Miniconf leaders: Please remember to take note of the approximate number of attendees in your session(s).

SCHEDULE

Checkpoint and Restore of processes within kernel security mechanisms

Tycho Andersen of Canonical, CRIU contributor

CRIU port to ARM
support for AppArmor, other LSMs need help but framework exists
integration of CRIU with seccomp
cgroup port to usernsd and unified heirarchy: come help!
Question: is there a mechanism/plugin/whatever to migrate a container that is tied to a FPGA or a GPU, so they should be migrated together?

Answer: yes, there is a tool for it.

CRIU in HPC

Adrian Reber, an old-time CRIU contributor, Red Hat

research project in a university involving CRIU, started in 2009
usage scenario: computer simulation for product develpment
many nodes, many fails, need for a fail-tolerant HPC
hypervisor is not the best virtualization type for HPC due to CPU & I/O overhead
looked into BCLR (Berkeley Lab Checkpoint/Restart)
requirements to C/R: upstream integration
there were different attempts to implement C/R: see also: http://criu.org/Comparison_to_other_CR_projects
key features of CRIU project: upstream inclusion, transparent solution wo pre-loading or re-compilation, most functionality in userspace, using standard interfaces as much as possible
summary: transparent migration works, OpenMPI supports CRIU (but keeps breaking as OpenMPI moves forward very fast), Parallel migration has WIP status

CRIU on PowerPC

Laurent Dufour, ported CRIU to PowerPC architecture, IBM CRIU 1.6 ported on Power this summer (What did you do this summer? I ported CRIU to PowerPC).

Linux kernel was ported to PowerPC arch in 1994 (See http://gate.crashing.org/doc/ppc/doc003.htm)
Big Endian/Little Endian. PowerPC supports both.
CRIU support only LE mode
Laurent tell about mess in Linux kernel ABI: two ABI currently in kernel
impacts: enabling kcmp() syscall, VDSO remapping tracking (both are available since 4.2)
There is a general problem with checkpointing / restoring whilst in vDSO. It is possible to end up with two (old and new) entry points and there is currently no way to detect this.
The vector register save/restore can get confused in some rare signal call stack cases. Fixing this is in the todo list.
A hindrance to getting docker up and running in CRIU/ppc is that the images it downloads at build/startup are x86_64 images. Help requested.

CRIU Support in Docker for Native Checkpoint and Restore

Saied Kazemi, working on container migration at Google

short history of Docker C/R
C/R benefits: CRIU allows to save work inside container, shorten startup time, debugging using checkpoint "iage"
new web based REPL for node.js - Tonic (https://www.npmjs.com/package/tonic). Checkpoints each step to fix mistakes by restoring from the most recent change.
speaker show scheme with Docker client, server and Containers and explain how it works
external C/R for Docker: four steps to C/R container in Docker
live demo with C/R of Docker Container - looks impressive
Manually C/R'ing with this methods leaves 'docker ps' not being able to show the container
TODO list is available on Docker's wiki under CRIU
broken things after C/R: logging won't work, container control is lost (docker stop, docker kill etc)
native C/R in Docker 1.5 (current version is 1.8 - https://blog.docker.com/2015/08/docker-1-8-content-trust-toolbox-registry-orchestration/)
C/R support already in libcontainer (runC) - live demo with Quake http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-dockercon-work.html
useful links: CRIU PPA; GitHub: 1.5 (https://github.com/SaiedKazemi/docker) Saied Kazemi, 1.8 - Ross Bucher, current - Michael Crosby docker git branch

Slides from Docker Meetup 2014 https://speakerdeck.com/saied/experimental-docker-checkpoint-and-restore-with-criu

P.Haul: live migrating with CRIU

Pavel Emelyanov, man in a red T-shirt with microphone

Process-Haul (like U-Haul, but about processes) See http://criu.org/P.Haul
CRIU is a part of solution for live migration of containers
CRIU part of live migration: perform nodes compat checks, pre-dumps in a loop, copy fs if not shared, don't forget container config
P.Haul migrate container is two steps
More information about P.Haul: written in Python, source code https://github.com/xemul/p.haul/, can migrate OpenVZ container (How-to https://github.com/xemul/p.haul/blob/master/test/mtouch/HOWTO), call rsync on chroot-ed container
P.Haul future: Docker/LXC support, smarter interations, shared FS autodetection, post-copy live migration, send images over the wire (CRIU)
Discussion of P.Haul started

LXC has a similar concept, but slightly different. Consensus seems to be easy to achieve as to how to merge (it just needs coding effort). Predump allows process to continue running, vs full dump which is a freeze

Rebootless Kernel Update and its verification

Sanidhya Kashyap, PhD student (Georgia Institute of Technology) and CRIU contributor http://www.cc.gatech.edu/~skashyap/

types of OS update with low downtime: dynamic kernel patching (KSplice, Kpatch and Kgraft)
scheme with typical OS update
KUP: update with C/R of applications
CRIU benefits to compare similar projects
opportunities in each phase of KUP: checkpoint -> incrementally, restore -> on-demand, C/R -> new abstraction
reusing memory of application across reboot
memory reuse: PPP (Persistent Physical Pages)
evaluation of each techniques performance
evaluation: KUP vs kpatch
evaluation: end-to-end performance of memcached
evaluation: microbenchmark
Conclusion: implement native support for page cache, hugepages; on-demand restore with usefaultd; need to overhaul kexec

Using CRIU for Computer Architecture and Software Studies

Christopher Covington, spec in performance analysis and simulation at Qualcomm, http://covlibre.net/

speaker starts from terminology, it seems talk will be quite technical :P zOMG a technical talk? Isn't it a sales conference?? (it was a joke :) ) (I'll buy two) Take three and get a 25% discount!
Counting instructions seems preferable to counting time in order to gain determinancy of elapsed work performed.
Useful trick of dumping after zero instructions to gather an image complete with file descriptors, environment, etc, as startup conditions

CRIU project wiki: http://criu.org/ || Google+ page: https://plus.google.com/+CriuOrg || Twitter: @__criu__ https://twitter.com/__criu__

>>> anyone can write to etherpad, please help us filling in the notes <<<

                      `       ,           
      .              `       ,+.          
       +         .+'        '+'+          
      +;+;       `         ++++:+         
     :.++++     :   +     ;+++++          
      ++++++    ;   +    `++++++'         
     +++++++'   ,        ++++++++         
      :++++++'   +      +++++++,          
      :'+++++++  +;    ++++++++:          
      ++++++++++ `+:  `++++++':.          
       +;+'+++++  ++, '+++++'+.           
        +;+;+++++  ++.+++++ +             
          ,+:;++++,+++++ :                
           .   :++++++'                   
                 ,.+++                    
                  ++;                     
                 +,                       
               .`   ;+                    
             .++  .+++ ++'`++ ++ ,++  '+' 
         +++++++; ++   ++. +; ++ :++  ++, 
         :++++++  ++   ++'+.  ++ '+:  ++` 
       .;.+++++.  +'   ++     ++ ++   ++  
     .'`..:., +`  +'   ++`++  ++ ++   ++  
    `;. .,.  `:   ++ . ++  +; ++ ++   ++  
    ., .,,.  ..   +++' ++  ++ ++ ++,,,++  
    .. .;:...;`   ``          +    ,+,    
    `,...;.;,`                      +     
     ,...+........,:,...     ,      +     
     ..'+';'++':.......,+:.  .            
    ..'...+.......... `.++'.              
    ....,...;++++++.`.;+'.              
    .+. ...........':: ....               
    .,`  ,:................               
   ....`  .......```, ..                
   ..+'.     ...',..`   .                 
   ..++.       ..++..                     
   ..+'.`     `..+++.                     
    .....      ..+++.                     
               `.'+'.                     
                 .'                       
                 .`                       
                 `