Schedulev. 1.5
New features
- CRIT tool
- Ability to request CPU compatibility on instructions level only
- C/R of empty AIO rings
- More detailed errno report via RPC
- Per-feature "criu check"
- Inheriting FDs on restore
- Ability to automatically move veth device to host-side bridge on netns restore
- VT terminals support
- More user namespaces C/R stuff
Optimizations
- TCP send queue is restored in the maximal portions allowed by the kernel
- Pre-loading sock-diag modules now happens in a more elegant way
Fixes
- Multi-threaded tasks on 64bit ARM could segfault upon restore
- When doing "check" CRIU could leave un-killed piggie task
- The --cpu-cap option argument was parsed with errors
- Incorrect handling of --cpu-cap fpu compatibility mode on restore
- Criu ignored trailing CLI arguments that resulted in usage confusions
- Irmap hints didn't include common "/" path
- When run per user request, CRIU left log and pid files belonging to root
- Mappings on AUFS could be looked up on wrong mount point
- Fixed compilation on Centos6.5
- Wrong /proc was used when reading the list of FDs to close on restore
- Race in restoring TCP established and listening sockets results in failed bind() on the latter
- Legacy ttys errorneously treated as unix98
- TTY pairs slavery setup could pick wrong peer
- For user-dump the log and pid files still belonged to root
- Task could die while being frozen thus causing dump to fail or save wrong task state
- Failures in mount points validation and sharing resolving didn't abort the dump (error arose on restore)
v. 1.4
New features
- Dump and check cpuinfo. Needed to make sure CPU is capable to run the images after restore, e.g. during live migration
- Initial support for user namespaces
- Use memfd to restore shared memory segments
- New (slightly faster) API for mm stuff restore via prctl
- [UG]ID-s are dumped from parasite, not from /proc files
- The docker_cr.sh script to show how Docker container C/R should (will) look like
- New API for writing plugins (old one is still possible)
- Service workers change their title to better look in ps output
- Ability to feed socket for pre-dump and page-server in swrk mode
- Page-server can auto-bind its port
- Ability to perform several actions during one connection to RPC service
- C/R of opened /proc/$pid/foo files of dead tasks
- C/R of /dev/console
- C/R of virtualized devtmpfs (openvz and future upstream kernels)
- C/R of empty mqueue fs (posix message queues)
- C/R of shared bind-mounts
Optimizations
- BFD engine
- Faster that glibc's FILE * buffered read from /proc files
- Buffered image files IO
- Faster parasite/restorer unload
- Use HW breakpoints
- Less ptrace GETREGS calls sometimes
- Wake pie after sending the FINI command to socket
- Merged some pairs of images into one
- eventpoll and -tfd
- inotify and -wd
- fsnotify and -mark
- Less setns()-s on dump is much faster on older kernels
- Faster access to /proc/self files -- cached fd of /proc/self and openat(this_cache)
Fixes
- Sibling restore mode didn't set up CRIU signals properly
- Unpredictable sibling/child root task restore. Fixed with explicit CLI option
- Validation for leaf mount points was skipped
- Mount options were corrupted on dump, which resulted in errors bind mounts detection
- Uninitialized properties of some cgroups prevented moving tasks into them (e.g. empty cpuset masks and low memcg limit)
- File locks could belong to task with different pid (inherited on fork) blocked the dump
- Bogus error printed in logs about SIGCHLD catch (was caused by thread dump using traps)
- Irmap engine accessed freed root_task on pre-dump
- Restore of net namespace could always fail (pid mismatch on fork) if kernel thread was created on netns setup
- Cgroups service descriptor was closed too early and failed restore
- Auto-loaded *diag modules caused audit netlink socket to contain data on dump (dump fails in this case)
- The "(deleted)" prefix accumulated in unlinked files while doing C/R
- The devpts filesystem and ptmx file were only dumped when found on /dev/pts and /dev respectively
- Data in netlink socket and fanotify was lost after C/R (now dump is aborted if data found in it)
- Fanotify mark was restore in different mount namespace
- Images were writable by group. Not secure when user-dump was requested
- Rootfs has parent id equal to self. CRIU didn't expect this and failed the dump
- Shared mount of the --root path failed the restore
- Absence (e.g. not compiled in) of any namespace in the kernel failed the dump
- Page-server incremental dump didn't detect new tasks properly and failed the stage
- Big TCP queues sometimes failed to get restored
- Incremental pre-dump could lose track of memory changes by task
v. 1.3
New features
- TimerFD support
- VVAR area (newer kernels' part of VDSO) support
- CGroups hierarchies support
- AUFS support (for Docker)
- PDeathSig support
- Check for opened file's size on dump and restore is the same
- Ability to restore tasks as children using libcriu (
criu_restore_child)
- Add pkgconfig file for libcriu
- CRTOOLS_IMAGE_DIR variable available in action scripts
Optimizations
- Merged images with pending signal into core
- Per-task images with file locks are merged into one big image
- Smaller tasks orchestration memory area on restore
- Sigactions are inherited on restore when possible, not overwritten
- ZDTM suite now executes tests in parallel
Fixes
- Dump failed if robust lists were off
- Link remaps on tmpfs mounts were not dumped
- Non root tasks with custom groups couldn't dump its peers (Security)
- Opened and unlinked FIFOs, dirs and devices were restored as regular files
- Files opened from alien mount namespace were restored in the local one
- Link remap name sometimes was generated with error
- Opened and removed cwd couldn't be restored
- Sysctl kernel.msgmni was overwritten by subsequent auto_msgmni
- Library and RPC APIs didn't match the CLI one
- Some external mounts were constantly "postponed" and never got mounted
- The self.mm_dumpable prlctl value of 2 caused restore to fail
- Errors when writing sysctls with tail \n
- The
criu show printed nested repeated fields corrupted
- Dump stats were initialized with garbage
- Restore sometimes stuck on waiting for inet socket port bind
- Spurious SIGHUP when restoring slave ttys
- Restore wasn't aborted if sub-task failed early
v. 1.3.1
Fixes
- Sibling restore mode didn't set up CRIU signals properly
- Unpredictable sibling/child root task restore. Fixed with explicit CLI option
- Validation for leaf mount points was skipped
- Mount options were corrupted on dump, which resulted in errors bind mounts detection
v. 1.3-rc2
New features
- Native (w/o plugins) c/r of external bind mounts
- C/R of the info in which cgroups tasks live
- C/R of task's dumpable flag
- Dump pstore, securityfs, fusectl and debugfs mountpoints
Fixes
- VDSO was searched on stack's guard page
- Mount namespace w/o /proc mount blocked the restore
- Several misses in searching for COW VMA resulted in sub-optimal pages sharing on restore
- FIFO-s path was restored in wrong mount namespace
- Mountpoint fsnotify could be restored on a bind-mount
- One tmpfs mounted several times was dumped several times
- Bind-mount's root path of the top mount was calculated with error
- Fix device number calculation out of major:minor on some distros
- Devpts mount options got lost on dump
- Page-pipes grew endlessly resulting in dump failures on big VMAs
- IO and PF mappings were tried to be dumped
- Two merged MAP_GROWSDOWN VMAs got dumped with overlapping guard page
- Too small shared area was used to fetch tasks mappings that resulted in failed dump of huge mappings
- Many fixes in build system
- Zdtm's COW test sometimes ignored COW failures
v. 1.3-rc1
New features
- AArch64
- Multiple mount namespaces
- FPU state restore control
- Restore old FPU state on newer CPUs
- Ability to ignore FPU restoration
- Support stopped multi-threaded tasks
- CRIU now can execv() other binary right after restore is complete
- Inode-reverse mapping can be enforced to allow live-migration with FS copying
- Gold linker can now be used to compile CRIU
- "Berserker" test to check CRIU scalability
- Punch pages from mem images on restore (optimizes live-migration)
Optimizations
- Batched deduplication of memory images
- Packed rlimits into core image
- Packed timers into core image
Fixes
- Bad checks for
kcmp() ret codes resulted in errors in file sharing detection
- Multiple mmaps of same files with different flags blocked the restore
- Integer overflow in huge mapping restore caused restoration failure
- devpts's
newinstance option was lost during dump
- Subsequent dump could try to find old mem dump for newly forked task
- Bad detection of overmounted mountpoints on fsnotify restore
- Page-server could read partial message and failed
- Errors in dumping of two subsequent anon VMAs in some cases
- Irmap mis-compared devices for disk FSs
- TMPFS handles always change during dump/restore
- Pre-dump sometimes hangs on FIFOs
- Post-restore script fails too late (if does it)
v. 1.2
New features
- Performance improvements
- Shared entries in reg-files image
- Less accesses to
/proc/$pid/map_files links
- Cache for
/proc/$pid/pagemap reads
- VDSO page is seeked only in anonymous mappings
- Task's auxv is read in one call
- Merged mm and vma image files for better packing
- NFS inodes' path resolution (for fsnotify) cache
- One
readlink() call when checking anon inodes
- Don't dump kernel's zero-page
- Parse fast
/proc/self/maps when searching for hole for restorer
- A bit faster write into image files with
writev()
- Library versioning
- RPC API got closer to CLI
- New "post-restore" call in action scripts
- Logrotate rules file
- Default log file for service when starting via systemd
Fixes
- A lot for ARM cross-compile
- Fsnotifies dumping didn't work on NFS
- Images auto-deduplication only worked one level up
- Packet socket ID was treated as file-descriptor and close()-d
- Badly counted pages stats on restore
- Linked remap name conflict when dump and restore on NFS
- Sporadic failures in memory draining due to huge pipes used
- Broken
criu show of repeated fields
- Failure to open mountpoint in foreign pid namespace
- Unlinked bound unix socket dump error
- Small memory leak when writing to incremental image(s)
- Restoring fsnotify for links results in ELOOP
- Host's PATH is not suitable when execv-ing tar/ip/iptable to restore namespace (workaround, proper fix will be in 1.3)
- Using subdirs in log file name via RPC breaks security
v. 1.1
Fixes
- Errors from memory dumping are not handled resulting in corrupted dumps
- EOF detection in stacked images is done with error
- Stacked images don't work on non-shared FS (missing pagemap-s)
v. 1.1-rc2
Fixes
- Crash in
criu check
- RPC check always fail on 3.11 kernel
- Failed fork() didn't abort restore
- Dump fail not reported via RPC
- RPC client disconnect wasn't handled
- Page server could connect to self for writing images
- Hang on pre-dumping task livig in net-namespace
- VDSO page mis-handle on pre-dump
- FPU state loss on pre-dump
- Memory tracking turns ON w/o request
- Various fixes (and improvements) in build system
v. 1.1-rc1
New features
- libcriu.so -- wrapper library for RPC clients
- Plugins
- External unix sockets
- External bind mounts
- External net devices
- Unknown file types
- Images deduplication in incremental dumps
- Integration with systemd
- Filtering of
criu show output
|
Note: The API defined in the first two items above may change after -rc1
|
Fixes
- Errors in unlinked files/sockets detection on BTRFS
- NFS silly-rename files are not treated as unlinked
- Freezer fail to seize quickly forking/pthread_create-ing tasks
- Extra stop signal queued for stopped tasks after pre-dump
- Wrong dying task state detection
- Lost RPC dump response
- Crash when reporting restore error via RPC
- Negative return code into shell
- Tasks left in wrong states after failed dump
- A little bit more verbose check action
- Coverity checks fail here and there
v. 1.0
Fixes
- After --leave-running linked remaps were not cleaned
- TCP was left locked after --leave-running
- Weak criteria in memory COW detection
- Private mapping's premmapped address overwrote file ID
- Restorer memory could overlap with timers/signals arrays
- RPC worker reused options from service task
- Suboptimal memory utilization by restorer arguments
- TCP unsent/unacked data boundary was lost
- Wrong dev_t decoding on 64 bit
- Unpredictable daemons (service and page-service) working dir
- Parasite stack could be corrupted by its arguments
- Error from exe link restore was ignored
- Artificial small limit on the number of shared memory segments to restore
- Bug in ARM VFP restore
- VDSO proxy was unmapped at the very end of restore
New features
- -W option to specify working dir
- CHECK request in RPC
- Optimized headers
- More info in logs about undumpable files
- More comments about tricky dump/restore places
- Generic memory allocation for restorer
v. 0.8
New features
- RPC service
- Ability to work from non-root user (via +s bit)
- Handle stopped tasks
- Restore tasks' root path
- Dump and restore net ns iptables configuration (w/o conntracks)
- Support for external net devices in netns (e.g. openvz venet)
- Support CORK and NODELAY TCP options
- SEQPACKET unix sockets support
Fixes and improvements
- Unload restorer blob after restore
- Fixes and enhancements in
criu show
- Fix in unsorted inotify wd restoring
- Fixed trimmed messages in parasite transport
- Fine-grained pgrps restore
- Fix in large TCP buffers restore
- Fixed buffer overflow in IPC ns dumping
- Fix in early page server connection close on pre-dump
- Fixed race in handling aborted parasite blob
- Fixed lost unmapped criu vmas in restore
- Fixes in parsing devices in /proc/pid/maps and /proc/locks
- Fixed snd/rcv buf sockoptions restore
- Enhanced logging in parasite
v. 0.7
New features
- TUN devices support (requires patched kernel)
- Alternative stack (sigaltstack) C/R
- Shared and master/slave mounts support
- Restore statistics
- Net device address C/R
- Safer and simpler asynchronous parasite
- Fixes in
- Stack guard page dump/restore
- Page server communications
- COW mappings restore
- Zombies restore
- Aborting failed restore
- TCP connection restore (some require patched kernel)
- Semi-closed Unix sockets with data dump/restore
- etc.
v. 0.6
New features
- Cross-compiling support
- Ready to accept images from OpenVZ RHEL6-based kernel
- Posix CPU timers C/R
- Asynchronous parasite
- Self-heal dumpee in case of crtools crash
- Memory changes tracking (requires patched kernel)
- Incremental backups
- Pre-dump task command for smaller freeze time
- Dump statistics
- Return-arg for remote syscalls execution
- Improved "show" output
- Opened /proc/PID/ns/* files C/R
- Daemon mode for page-server
- Build-time features test (makes it possible to build on different distros)
v. 0.5
New features
- C/R netlink sockets
- C/R pending signals
- New format for memory dumps
- Incremental dumps (preliminary)
- Ability to directly send memory dumps on remote host (for faster live migration)
- Images showing enhancements
- Kernel features checking enhancements
- Lots of BUGs fixed
- Renamed
crtools to criu (both binary and source package names)
v. 0.4
crtools-0.4.tar.bz2
Released: 20 Feb 2013
GIT tag: v0.4
New features
- ARM port
- Remote syscall execution
- C/R of
- FPU state
- File locks (basic support)
- Rlimits
- FANotify descriptors
- Shared fdtable (table of file descriptors)
- Tasks' umask
- Pipe buffer size
- Unix sockets' credentials
- TCP time-stamp offset (allows to migrate a TCP socket, requires custom kernel)
- Automatic namespaces detection (
--namespace option is deprecated)
Internal
- Build system rework
- Dumping/restoring memory pointers standardized (while doing ARM port)
- Ability to collect coverage (gcov, reported here)
- 32/64-bit problems mostly resolved (while doing ARM port)
v. 0.3
crtools-0.3.tar.bz2
Released: 11 Dec 2012
GIT tag: v0.3
New features
- C/R of shell jobs
- Handle files, that are opened and unlinked, but some other hard link exists
- More SOL_SOCKET socket options. Two most interesting are:
- Binding socket to device (
SO_BINDTODEVICE)
- Per-socket packet filter (
SO_ATTACH_FILTER)
- Shutdown state of a socket
- Task scheduler parameters (nice, policy and prio)
- Properly handle COW pages
- Memory mapped packet sockets (with SOL- options this is enough for tcpdump tool support)
- Complementary groups (
getgroups(2))
- Various mapping features, configured with
madvise(2)
- Tune TCP sockets support to work with IPv6 sockets
- Closed (or not yet connected/bound) unix sockets
Bug fixes and improvements
- Print timestamps in log files
- Restructure source tree a little
- PIE stuff in separate dir
- Arch-specific code in arch/<Arch>/ directory
- Nicer
show -c output for pages/pipes/etc contents
- Fixes in help text
- Save and restore blocked signals mask for threads
- Don't dump (and warn about it) corked UDP sockets
- Fixed livelocks when restoring too big socket queues
- Refresh info about sockets, that might have changed between diag dump and actual process freeze
- Lost file params and socket options for some unix sockets
- Fix inotify on opened and unlinked files
- Fix shared anonymous memory detection
- Properly close all files before restoring them
- Fixes in running external tools (ip/tar)
- Several fixes in control tty restoring code
- RT signals handlers dumped and restored
v. 0.2v. 0.1 