Xsave

Summary

XSAVE stands for similar x86 instruction xsave which places extended processor state into a memory area. The saving can be initiated by any userspace application at any moment and size of the memory frame depends on processor features and may vary between different models. Thus if checkpoint and restore are done on different processors the next call to xsave may corrupt memory if sizes mismatch.

Helpers

There are several helpers we will refer on in this page

static inline void native_cpuid(unsigned int *eax, unsigned int *ebx,
				unsigned int *ecx, unsigned int *edx)
{
	/* ecx is often an input as well as an output. */
	asm volatile("cpuid"
	    : "=a" (*eax),
	      "=b" (*ebx),
	      "=c" (*ecx),
	      "=d" (*edx)
	    : "0" (*eax), "2" (*ecx)
	    : "memory");
}

static inline void cpuid(unsigned int op,
			 unsigned int *eax, unsigned int *ebx,
			 unsigned int *ecx, unsigned int *edx)
{
	*eax = op;
	*ecx = 0;
	native_cpuid(eax, ebx, ecx, edx);
}

static inline void cpuid_count(unsigned int op, int count,
			       unsigned int *eax, unsigned int *ebx,
			       unsigned int *ecx, unsigned int *edx)
{
	*eax = op;
	*ecx = count;
	native_cpuid(eax, ebx, ecx, edx);
}

Frame size

Run cpuid(0x1, &eax, &ebx, &ecx, &edx) and bits 26 and 27 are both set in ecx if xsave is supported (strictly speaking bit 27 is reserved for operating system which can clear it to indicate that instruction is disabled).

After that we can fetch maximal frame size which applications may use via cpuid_count(0xd, 0, &eax, &ebx, &ecx, &edx), in result ebx will contain the size to keep currently enabled components of the frame and ecx will keep the value of maximal frame size. The maximal here means the size needed when all components are enabled (OS may disable some of components).

Enumerating frame components

To enumerate which components of the frame are enabled execute cpuid_count(0xd, 0, &eax, &ebx, &ecx, &edx). Each component will have bit set to 1 in 64 bit mask eax + ((uint64_t)edx << 32) if enabled.

Current list of known components is the following (numbers are the bit position):

• 0: x87 floating point registers
• 1: SSE registers
• 2: AVX registers
• 3: MPX bounds registers
• 4: MPX CSR
• 5: AVX-512 opmask
• 6: AVX-512 Hi256
• 7: AVX-512 ZMM_Hi256
• 8: Processor Trace
• 9: Protection Keys User registers
• 10: Hardware Duty Cycling

Once the bit mask is obtained we have to walk over each bit set and call cpuid_count(0xd, component, &eax, &ebx, &ecx, &edx), where component is the bit position we are interested in. In other words it should be from 0 to 10. The result of this call is sitting in ebx which represent offset of the component from the frame base address and eax which shows component size. Note that some of components are supervisor components and if (ecx & 1) == 0 from the cpuid_count call above then its offset should not be considered while size is still valid.

Potential memory corruption

When processes are dumped and restored on different cpu, the application may have remembered frame size on its own somewhere inside own code and in worst scenario it may allocate memory with size less than needed on different cpu, so the next call to xsave silently overwrite memory leading to sigsegv in best case.

Current criu implementation check for cpuinfo images to be compatible and size and features required to match. In turn some OS may mask some of the features with cpuid faulting engine but still all cpus in the pool should report same maximal size of the frame.