Rseq

"Restartable sequences" (rseq) are small segments of user-space code designed to access per-CPU data structures without the need for heavyweight locking. rseq is supported since Linux kernel 4.18 [1]

I strongly suggest reading the article Linux restartable sequences before this one.

Linux kernel interface

The Linux kernel interface for rseq is fairly simple. It's just rseq syscall: sys_rseq(struct rseq *rseq, uint32_t rseq_len, int flags, uint32_t sig)

enum rseq_cs_flags {
	RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT = (1U << RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT_BIT),
	RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL = (1U << RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT),
	RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE = (1U << RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT),
};

struct rseq_cs {
	__u32 version; /* always 0 at this moment */
	enum rseq_cs_flags flags;
	void *start_ip;
	/* Offset from start_ip. */
	intptr_t post_commit_offset;
	void *abort_ip;
}

struct rseq {
	__u32 cpu_id_start;
	__u32 cpu_id;
	struct rseq_cs *rseq_cs;
	enum rseq_cs_flags flags;
}

From the userspace side, we need to keep struct rseq somewhere and register it on the kernel side using the rseq syscall. Then, once we want to execute some code as a rseq critical section (rseq cs or just CS) we need to allocate and fill with the data struct rseq_cs. We have to specify the start address of our CS, and the address of the abort handler (called when CS was interrupted by a preemption, migration or signal). Then we need to put an pointer to struct rseq_cs to the (struct rseq).rseq_cs field.

What about flags?

You may have noticed that both struct rseq and struct rseq_cs have flags field. It may took values from enum rseq_cs_flags.

First of all, a user may specify flags in any place they will be combined on the kernel side:

static int rseq_need_restart(struct task_struct *t, u32 cs_flags)
{
	u32 flags, event_mask;
	int ret;

	/* Get thread flags. */
	ret = get_user(flags, &t->rseq->flags);
	if (ret)
		return ret;

	/* Take critical section flags into account. */
	flags |= cs_flags; // <<<<<<<< here we have flags combined from struct rseq + struct rseq_cs

The most common flags value is zero. In this case, the rseq CS will be interrupted (and IP will be fixed up to the abort handler) if preemption, migration, or signal occurs. But there are situations when users may want not to abort section once one of these events happen.

It's worth mentioning that RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL can be used only in combination with RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT and RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE:

	/*
	 * Restart on signal can only be inhibited when restart on
	 * preempt and restart on migrate are inhibited too. Otherwise,
	 * a preempted signal handler could fail to restart the prior
	 * execution context on sigreturn.
	 */
	if (unlikely((flags & RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL) &&
		     (flags & RSEQ_CS_PREEMPT_MIGRATE_FLAGS) !=
		     RSEQ_CS_PREEMPT_MIGRATE_FLAGS))
		return -EINVAL;

How CRIU handles rseq

CRIU handles the rseq differently depending on the particular case. Let's classify and cover all of them.

1. the process is not inside the rseq critical section
2. the process is inside the rseq CS
   1. flags is 0 or RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT or RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE
   2. flags is RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL

the process is not inside the rseq critical section

Simplest case. Process just have struct rseq registered in the kernel but currently instruction pointer (IP) not inside CS.

Dump

We need only to determine where the struct rseq is and dump its address length and signature. To achieve that we use special ptrace handle PTRACE_GET_RSEQ_CONFIGURATION (refer to the dump_thread_rseq function).

Restore

We need to take data about the struct rseq from the image (see images/rseq.proto) and register it from the parasite context using the rseq syscall (take a look on restore_rseq in criu/pie/restorer.c)

inside CS: flags is 0 or RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT or RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE

The process was caught with IP inside CS. Can we act as before? So, dump struct rseq address, restore it, and so on. No, we can't. The reason is that CRIU saves IP as it was during the dump. But the rseq semantic is to jump to abort handler if CS execution was interrupted. In this particular case we have flags equal to 0 or RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT or RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE it means that if CS will be interrupted by the preeption, migration (0) or migration (RSEQ_CS_FLAG_NO_RESTART_ON_PREEMPT) or preemption (RSEQ_CS_FLAG_NO_RESTART_ON_MIGRATE) the kernel will fixup IP of the process to the abort handler address.

When we dump the process using CRIU it will just save IP as it was and restore it. That's a serious problem and this may break the user application (even cause crash!).

Lets see fixup_thread_rseq function:

	if (task_in_rseq(rseq_cs, TI_IP(core))) {
		struct pid *tid = &item->threads[i];

...

		pr_warn("The %d task is in rseq critical section. IP will be set to rseq abort handler addr\n",
				tid->real);

...

		if (!(rseq_cs->flags & RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL)) {
			pr_warn("The %d task is in rseq critical section. IP will be set to rseq abort handler addr\n",
				tid->real);

			TI_IP(core) = rseq_cs->abort_ip;

			if (item->pid->real == tid->real) {
				compel_set_leader_ip(dmpi(item)->parasite_ctl, rseq_cs->abort_ip);
			} else {
				compel_set_thread_ip(dmpi(item)->thread_ctls[i], rseq_cs->abort_ip);
			}
		}
	}

It checks that process IP inside CS and fixes it up to the abort handler IP as the kernel does.

Dump

We need to determine where the struct rseq is and dump its address length and signature. To achieve that we use special ptrace handle PTRACE_GET_RSEQ_CONFIGURATION (refer to the dump_thread_rseq function).

We have to fix up IP to the abort handler.

Restore

We need to take data about the struct rseq from the image (see images/rseq.proto) and register it from the parasite context using the rseq syscall (take a look on restore_rseq in criu/pie/restorer.c)

No additional actions here. The process will be restored and will continue execution from the abort handler (not within the rseq CS!).

inside CS: flags is RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL

Rare case, but we support it too. If the rseq CS has RSEQ_CS_FLAG_NO_RESTART_ON_SIGNAL flag it means that its technically non-abortable. So, from the first glance, it seems like we can just not do anything special: save rseq structure address, not fix up IP. This is incorrect.

The kernel will clean up (struct rseq).rseq_cs pointer once we jump into the parasite on the dump:

static int rseq_ip_fixup(struct pt_regs *regs)
{
...

	/*
	 * Handle potentially not being within a critical section.
	 * If not nested over a rseq critical section, restart is useless.
	 * Clear the rseq_cs pointer and return.
	 */
	if (!in_rseq_cs(ip, &rseq_cs))
		return clear_rseq_cs(t);

and after the restore process will continue the rseq CS execution from the same place (it's okay) but from the kernel point of view, the process will continue this execution as not being within the rseq CS (that's bad!). Because the kernel determines execution context from the (struct rseq).rseq_cs field.

Dump

We need to determine where the struct rseq is and dump its address length and signature. To achieve that we use special ptrace handle PTRACE_GET_RSEQ_CONFIGURATION (refer to the dump_thread_rseq function).

We save IP as it was (not doing fixup), but we have to save (struct rseq).rseq_cs field into the CRIU image.

Restore

We need to take data about the struct rseq from the image (see images/rseq.proto) and register it from the parasite context using the rseq syscall (take a look on restore_rseq in criu/pie/restorer.c)

We need to restore (struct rseq).rseq_cs memory externaly using ptrace POKEAREA (see restore_rseq_cs).

TODO

• tests for all architectures (right now we have ZDTM tests only for x86_64)
• improvement support of built-in rseq for non-Glibc libraries
• pre-dump tests (?)
• leave-running tests (?)
• crfail test
• threaded test

Useful links

• [1] https://github.com/torvalds/linux/blob/b2d229d4ddb17db541098b83524d901257e93845/kernel/rseq.c#L1
• [2] https://www.efficios.com/blog/2019/02/08/linux-restartable-sequences/
• [3] https://lwn.net/Articles/883104/