Difference between revisions of "RPC"

From CRIU
Jump to navigation Jump to search
m (→‎Client: update RPC examples path)
 
(9 intermediate revisions by 4 users not shown)
Line 1: Line 1:
CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU service launched with <code>criu service</code> command. It uses a SEQPACKET Unix domain socket listening at <code>/var/run/criu-service.socket</code> as a transport.
+
CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU when either launched in so called "swrk" mode or by a service started with the <code>criu service</code> command. It uses a <code>SEQPACKET</code> Unix domain socket for transport. In case of a standalone service it listens at <code>/var/run/criu-service.socket</code> as a transport.
  
The criu_req/criu_resp are two main messages for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them ''must'' be set accordingly to type of request/response that is stored. Types of request/response are defined in enum criu_req_type. See the [[API compliance]] page for information what each option might mean.
+
The <code>criu_req</code>/<code>criu_resp</code> are two main messages for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them ''must'' be set accordingly to type of request/response that is stored. Types of request/response are defined in <code>enum criu_req_type</code>. See the [[API compliance]] page for information what each option might mean.
  
 
== Protocol ==
 
== Protocol ==
  
The protocol is simple -- client connect to service, send a ''criu_req'' message to server, server responds back with ''criu_resp'' and closes the socket. There are three exceptions from this rule, see below.
+
The protocol is simple: client sends a <code>criu_req</code> message to server, server responds back with <code>criu_resp</code>. In most of the cases the socket gets closed, but there are three exceptions from this rule, see below.
  
 
== Request ==
 
== Request ==
Line 11: Line 11:
 
This is the header of the request. It defines the operation requested and options.
 
This is the header of the request. It defines the operation requested and options.
  
<pre>
+
<source lang="c">
 
message criu_req {
 
message criu_req {
 
required criu_req_type type = 1;
 
required criu_req_type type = 1;
Line 18: Line 18:
 
optional keep_open              = 4; /* for multi-req, below */
 
optional keep_open              = 4; /* for multi-req, below */
 
}
 
}
</pre>
+
</source>
  
There are 8 request/response types for now.
+
Currently, there are a few request/response types:
  
<pre>enum criu_req_type {
+
<source lang="c">
 +
enum criu_req_type {
 
EMPTY = 0;
 
EMPTY = 0;
 
DUMP = 1; /* criu dump */
 
DUMP = 1; /* criu dump */
Line 33: Line 34:
 
CPUINFO_CHECK = 8; /* criu cpuinfo check */
 
CPUINFO_CHECK = 8; /* criu cpuinfo check */
 
}
 
}
</pre>
+
</source>
  
<pre>message criu_opts {
+
The following options are available:
 +
 
 +
<source lang="c">
 +
message criu_opts {
 
required int32 images_dir_fd = 1;
 
required int32 images_dir_fd = 1;
 
optional int32 pid = 2; /* if not set on dump, will dump requesting process */
 
optional int32 pid = 2; /* if not set on dump, will dump requesting process */
Line 71: Line 75:
 
optional bool rst_sibling = 26; /* swrk only */
 
optional bool rst_sibling = 26; /* swrk only */
 
}
 
}
</pre>
+
</source>
  
 
=== Comments and examples ===
 
=== Comments and examples ===
  
* If no pid is set and type is DUMP, CRIU will dump client process by default.
+
* If no <code>pid</code> is set and type is <code>DUMP</code>, CRIU will dump client process by default.
* All processes in the subtree starting with <pid> must have the same uid, as a client, or client's uid must be root (uid == 0), otherwise CRIU will return error.
+
* All processes in the subtree starting with <code>''pid''</code> must have the same uid, as a client, or client's uid must be root (uid == 0), otherwise CRIU will return an error.
* Only the images_dir_fd is required, all other fields are optional. Client must open directory for/with images by himself and set images_dir_fd to it's fd. CRIU will open /proc/<client's_pid>/fd/<images_dir_fd>.
+
* Only the <code>images_dir_fd</code> is required, all other fields are optional. Client must open directory for/with images by itself and set <code>images_dir_fd</code> to the opened <code>fd</code>. CRIU will open <code>/proc/''client_pid''/fd/''images_dir_fd''</code>.
 +
 
 +
The logic of setting request is the same as when setting options in console.
 +
 
 +
Here is an example:
 +
 
 +
# criu restore -D /path/to/imgs_dir -v4 -o restore.log
  
The logic of setting request is the same as when setting options in console. Here is an example:
+
This is equal to:
<pre>#criu restore -D /path/to/imgs_dir -v4 -o restore.log</pre>
+
 
is equal to:
+
<source lang="c">
<pre>
 
 
request.type = RESTORE;
 
request.type = RESTORE;
  
Line 88: Line 97:
 
request.opts.log_level = 4
 
request.opts.log_level = 4
 
request.opts.log_file = "restore.log"
 
request.opts.log_file = "restore.log"
</pre>
+
</source>
  
 
=== Sub-messages for options ===
 
=== Sub-messages for options ===
; Info about page-server.
 
  
<pre>
+
==== Info about page-server ====
 +
 
 +
<source lang="c">
 
message criu_page_server_info {
 
message criu_page_server_info {
 
optional string address = 1; /* bind address -- if not set 0.0.0.0 is used */
 
optional string address = 1; /* bind address -- if not set 0.0.0.0 is used */
Line 100: Line 110:
 
optional int32 fd = 4; /* could be used to inherit fd by page-server */
 
optional int32 fd = 4; /* could be used to inherit fd by page-server */
 
}
 
}
</pre>
+
</source>
 
 
* If ''port'' is 0 server will auto-bind and the port will be returned back in responce.
 
  
<pre>
+
<source lang="c">
 
message criu_veth_pair {
 
message criu_veth_pair {
 
required string if_in = 1; /* inside veth device name */
 
required string if_in = 1; /* inside veth device name */
 
required string if_out = 2; /* outside veth device name */
 
required string if_out = 2; /* outside veth device name */
 
};
 
};
</pre>
+
</source>
  
; Info about veth mappings (--ext-mount-map analogue)
+
==== Info about veth mappings (<code>--ext-mount-map</code> analogue) ====
<pre>
+
<source lang="c">
 
message ext_mount_map {
 
message ext_mount_map {
 
required string key = 1;
 
required string key = 1;
 
required string val = 2;
 
required string val = 2;
 
};
 
};
</pre>
+
</source>
  
; Specifying where cgroup root should be (--cgroup-root analogue)
+
==== Specifying where cgroup root should be (<code>--cgroup-root</code> analogue) ====
<pre>
+
<source lang="c">
 
message cgroup_root {
 
message cgroup_root {
 
optional string ctrl = 1;
 
optional string ctrl = 1;
 
required string path = 2;
 
required string path = 2;
 
};
 
};
</pre>
+
</source>
  
 
== Response ==
 
== Response ==
Line 131: Line 139:
 
This message is sent after (un)successful execution of the request.
 
This message is sent after (un)successful execution of the request.
  
<pre>
+
<source lang="c">
 
message criu_resp {
 
message criu_resp {
 
required criu_req_type type = 1;
 
required criu_req_type type = 1;
Line 140: Line 148:
 
optional criu_notify notify = 5;
 
optional criu_notify notify = 5;
 
optional criu_page_server_info ps = 6;
 
optional criu_page_server_info ps = 6;
 +
 +
        optional int32 cr_errno = 7;
 
}
 
}
</pre>
+
</source>
Field "success" reports result of processing request, while criu_***_resp store some request-specific information. The response type is set to the corresponding request type or to <code>EMPTY</code> to report a "generic" error.
+
 
 +
The field <code>success</code> reports result of processing request, while <code>criu_***_resp</code> store some request-specific information. The response type is set to the corresponding request type or to <code>EMPTY</code> to report a "generic" error. If <code>success == false</code>, one should check <code>cr_errno</code> field to get a more detailed error code (see [https://github.com/xemul/criu/blob/master/include/cr-errno.h#L8 include/cr-errno.h]).
 +
 
 +
==== The criu_dump_resp is used to store response from DUMP request ====
  
;The criu_dump_resp is used to store response from DUMP request.
+
<source lang="c">
<pre>
 
 
message criu_dump_resp {
 
message criu_dump_resp {
 
optional bool restored = 1;
 
optional bool restored = 1;
 
}
 
}
</pre>
+
</source>
  
This message can be sent twice -- one for the process that calls DUMP and the other for the same process again if it requested a self-dump. In the latter case the ''restored'' field would be true.
+
This message can be sent twice one time for the process that calls DUMP, and another time for the same process again, in case it requested a self-dump. In the latter case the ''restored'' field would be true.
  
;The response on RESTORE request.
+
==== The response on RESTORE request ====
<pre>
+
 
 +
<source lang="c">
 
message criu_restore_resp {
 
message criu_restore_resp {
 
required int32 pid = 1;
 
required int32 pid = 1;
 
}
 
}
</pre>
+
</source>
Field "pid" is set to the PID of the newly restored process.
+
 
 +
The <code>pid</code> field is set to the PID of the newly restored process.
 +
 
 +
==== Info about page server ====
  
;Info about page server
+
The <code>criu_page_server_info</code> from requests will be sent back on <code>PAGE_SERVER</code> request. The <code>port</code> field will contain the port to which the server is bound.
The ''criu_page_server_info'' from requests will be sent back on PAGE_SERVER request. The ''port'' field will contain the port to which server is bound.
 
  
 
=== Notifications ===
 
=== Notifications ===
If the ''opts.notify_scripts'' in the request is set to TRUE CRIU would report back resp messages with type set to NOTIFY and this field present. The notifications are the way [[action scripts]] work for RPC mode.
 
  
<pre>
+
If the <code>opts.notify_scripts</code> in the request is set to <code>TRUE</code>, CRIU would report back resp messages with type set to <code>NOTIFY</code> and this field present. The notifications are the way [[action scripts]] work for RPC mode.
 +
 
 +
<source lang="c">
 
message criu_notify {
 
message criu_notify {
 
optional string script = 1;
 
optional string script = 1;
 
optional int32 pid = 2;
 
optional int32 pid = 2;
 
}
 
}
</pre>
+
</source>
  
After handling the notification the client must response with the request again with the type set to NOTIFY and the ''notify_success'' set to the successfulness of the notification. In case of successful notification acknowledge the server doesn't close the socket and continues to work.
+
After handling the notification the client must response with the request again with the type set to <code>NOTIFY</code> and the <code>notify_success</code> set to the whether the notification was successful. In case of successful notification acknowledge the server doesn't close the socket and continues to work.
  
 
== Pre-dumps ==
 
== Pre-dumps ==
  
Before issuing a DUMP request client may send one or more PRE_DUMP requests. Once the PRE_DUMP is sent and response is received, client may send one more PRE_DUMP or DUMP request. The server would only close the socket after the DUMP one.
+
Before issuing a <code>DUMP</code> request client may send one or more <code>PRE_DUMP</code> requests. Once the <code>PRE_DUMP</code> is sent and response is received, client may send one more <code>PRE_DUMP</code> or <code>DUMP</code> request. The server would only close the socket after the <code>DUMP</code> one.
  
 
== Multi-request mode ==
 
== Multi-request mode ==
  
If the ''req.keep_open'' flag is set to true server will not close the socket after response, but will wait for more requests. This mode is supported only for the following request types:
+
If the <code>req.keep_open</code> flag is set to true server will not close the socket after response, but will wait for more requests. This mode is supported only for the following request types:
  
* PRE_DUMP (automatically)
+
* <code>PRE_DUMP</code> (automatically)
* PAGE_SERVER
+
* <code>PAGE_SERVER</code>
* CPUINFO_DUMP and CPUINFO_CHECK
+
* <code>CPUINFO_DUMP</code> and <code>CPUINFO_CHECK</code>
  
 
== Run ==
 
== Run ==
 +
 +
=== SWRK mode ===
 +
 +
This mode turns on when one <code>fork() + exec()</code> CRIU with the <code>swrk</code> action and one more argument specifying the number of descriptor with <code>SOCK_SEQPACKET</code> Unix socket. With this CRIU works as service worker task accepting standard RPC requests via the mentioned socket and using one to do action scripts notifications and result reporting.
 +
 
=== Server ===
 
=== Server ===
On a server side, CRIU creates SOCK_SEQPACKET Unix socket and listens for connections on it. After receiving criu_req, CRIU processes it, do what is requested and sends criu_resp with set request-specific criu_***_resp field back.
 
If CRIU gets unknown type of request, it will return criu_resp with type == EMPTY and success == false.
 
  
To launch service server, run:
+
On a server side, CRIU creates <code>SOCK_SEQPACKET</code> Unix socket and listens for connections on it. After receiving <code>criu_req</code>, CRIU processes it, does what was requested and sends <code>criu_resp</code> with set request-specific <code>criu_***_resp</code> field back.
 +
If CRIU gets unknown type of request, it will return <code>criu_resp</code> with <code>type == EMPTY</code> and <code>success == false</code>.
 +
 
 +
To launch the service, run:
  
<pre>#criu service [options]</pre>
+
# criu service [options]
  
 
Options accepted by service are
 
Options accepted by service are
  
 
; --address <path>
 
; --address <path>
: is where to put listening socket
+
: where to put listening socket
  
 
; --pid-file <path>
 
; --pid-file <path>
: is where to write pid of service process
+
: where to write pid of service process
  
 
; --daemon
 
; --daemon
Line 212: Line 234:
  
 
; -v[N]
 
; -v[N]
: sets the log-level
+
: sets the log level
  
====systemd====
+
=== Client ===
If you are running systemd you can make service start and operate automatically. First do
 
<pre>
 
make install
 
</pre>
 
to make files <code>criu.service</code> and <code>criu.socket</code> appear in systemd configs (<code>/lib/systemd/system/</code>). Then
 
  
<pre>
+
Client, in its turn, must connect to service socket, send <code>criu_req</code> with request in it, and wait for a <code>criu_resp</code> with response.
systemctl start criu.socket
+
You can find examples of client programs in C and Python in test/others/rpc/.
</pre>
 
  
to get /var/run/criu-service.socket, and
+
With RPC facilities one can perform a [[self dump]].
<pre>
 
systemctl enable criu.socket
 
</pre>
 
 
 
to make /var/run/criu-service.socket available at boot.
 
 
 
=== Client ===
 
Client, in its turn, must connect to service socket, send criu_req with request in it, and wait for a criu_resp with response.
 
You can find examples of client programs in C and Python in test/rpc/.
 
  
With RPC facilities one can perform a [[self dump]].
+
There's a [[C API|library]] that implements simple wrappers on top of RPC.
  
There's a [[C_API|library]] that implements simple wrappers on top of RPC.
+
== See also ==
 +
* [[CLI]]
 +
* [[C API]]
  
 
[[Category: API]]
 
[[Category: API]]

Latest revision as of 22:09, 10 April 2020

CRIU-RPC is a remote procedure call (RPC) protocol which uses Google Protocol Buffers to encode its calls. The requests are served by CRIU when either launched in so called "swrk" mode or by a service started with the criu service command. It uses a SEQPACKET Unix domain socket for transport. In case of a standalone service it listens at /var/run/criu-service.socket as a transport.

The criu_req/criu_resp are two main messages for requests/responses. They are to be used for transferring messages and needed to provide compatibility with an older versions of rpc. Field type in them must be set accordingly to type of request/response that is stored. Types of request/response are defined in enum criu_req_type. See the API compliance page for information what each option might mean.

Protocol[edit]

The protocol is simple: client sends a criu_req message to server, server responds back with criu_resp. In most of the cases the socket gets closed, but there are three exceptions from this rule, see below.

Request[edit]

This is the header of the request. It defines the operation requested and options.

message criu_req {
	required criu_req_type type	= 1;
	optional criu_opts opts		= 2;
	optional notify_success         = 3; /* see Notifications below */
	optional keep_open              = 4; /* for multi-req, below */
}

Currently, there are a few request/response types:

enum criu_req_type {
	EMPTY		= 0;
	DUMP		= 1; /* criu dump */
	RESTORE		= 2; /* criu restore */
	CHECK		= 3; /* criu check */
	PRE_DUMP	= 4; /* criu pre-dump */
	PAGE_SERVER	= 5; /* criu page-server */
	NOTIFY          = 6; /* see Notifications below */
	CPUINFO_DUMP	= 7; /* criu cpuinfo dump */
	CPUINFO_CHECK	= 8; /* criu cpuinfo check */
}

The following options are available:

message criu_opts {
	required int32			images_dir_fd	= 1;
	optional int32			pid		= 2; /* if not set on dump, will dump requesting process */

	optional bool			leave_running	= 3;
	optional bool			ext_unix_sk	= 4;
	optional bool			tcp_established	= 5;
	optional bool			evasive_devices	= 6;
	optional bool			shell_job	= 7;
	optional bool			file_locks	= 8;
	optional int32			log_level	= 9 [default = 2];
	optional string			log_file	= 10; /* No subdirs are allowed. Consider using work-dir */

	optional criu_page_server_info	ps		= 11;

	optional bool			notify_scripts	= 12;

	optional string			root		= 13;
	optional string			parent_img	= 14;
	optional bool			track_mem	= 15;
	optional bool			auto_dedup	= 16;

	optional int32			work_dir_fd	= 17;
	optional bool			link_remap	= 18;
	repeated criu_veth_pair		veths		= 19;

	optional uint32			cpu_cap		= 20 [default = 0xffffffff];
	optional bool			force_irmap	= 21;
	repeated string			exec_cmd	= 22;

	repeated ext_mount_map		ext_mnt		= 23;
	optional bool			manage_cgroups	= 24;
	repeated cgroup_root		cg_root		= 25;

	optional bool			rst_sibling	= 26; /* swrk only */
}

Comments and examples[edit]

  • If no pid is set and type is DUMP, CRIU will dump client process by default.
  • All processes in the subtree starting with pid must have the same uid, as a client, or client's uid must be root (uid == 0), otherwise CRIU will return an error.
  • Only the images_dir_fd is required, all other fields are optional. Client must open directory for/with images by itself and set images_dir_fd to the opened fd. CRIU will open /proc/client_pid/fd/images_dir_fd.

The logic of setting request is the same as when setting options in console.

Here is an example:

# criu restore -D /path/to/imgs_dir -v4 -o restore.log

This is equal to:

request.type = RESTORE;

request.opts.imgs_dir_fd	= open("/path/to/imgs_dir")
request.opts.log_level		= 4
request.opts.log_file		= "restore.log"

Sub-messages for options[edit]

Info about page-server[edit]

message criu_page_server_info {
	optional string		address	= 1; /* bind address -- if not set 0.0.0.0 is used */
	optional int32		port	= 2; /* bind port -- if not set on request, autobind is used and port is returned in response */
	optional int32		pid	= 3; /* page-server pid -- returned in response */
	optional int32		fd	= 4; /* could be used to inherit fd by page-server */
}
message criu_veth_pair {
	required string		if_in	= 1; /* inside veth device name */
	required string		if_out	= 2; /* outside veth device name */
};

Info about veth mappings (--ext-mount-map analogue)[edit]

message ext_mount_map {
	required string		key	= 1;
	required string		val	= 2;
};

Specifying where cgroup root should be (--cgroup-root analogue)[edit]

message cgroup_root {
	optional string		ctrl	= 1;
	required string		path	= 2;
};

Response[edit]

This message is sent after (un)successful execution of the request.

message criu_resp {
	required criu_req_type		type		= 1;
	required bool			success		= 2;

	optional criu_dump_resp		dump		= 3;
	optional criu_restore_resp	restore		= 4;
	optional criu_notify		notify		= 5;
	optional criu_page_server_info	ps		= 6;

        optional int32			cr_errno	= 7;
}

The field success reports result of processing request, while criu_***_resp store some request-specific information. The response type is set to the corresponding request type or to EMPTY to report a "generic" error. If success == false, one should check cr_errno field to get a more detailed error code (see include/cr-errno.h).

The criu_dump_resp is used to store response from DUMP request[edit]

message criu_dump_resp {
	optional bool restored		= 1;
}

This message can be sent twice — one time for the process that calls DUMP, and another time for the same process again, in case it requested a self-dump. In the latter case the restored field would be true.

The response on RESTORE request[edit]

message criu_restore_resp {
	required int32 pid		= 1;
}

The pid field is set to the PID of the newly restored process.

Info about page server[edit]

The criu_page_server_info from requests will be sent back on PAGE_SERVER request. The port field will contain the port to which the server is bound.

Notifications[edit]

If the opts.notify_scripts in the request is set to TRUE, CRIU would report back resp messages with type set to NOTIFY and this field present. The notifications are the way action scripts work for RPC mode.

message criu_notify {
	optional string script		= 1;
	optional int32	pid		= 2;
}

After handling the notification the client must response with the request again with the type set to NOTIFY and the notify_success set to the whether the notification was successful. In case of successful notification acknowledge the server doesn't close the socket and continues to work.

Pre-dumps[edit]

Before issuing a DUMP request client may send one or more PRE_DUMP requests. Once the PRE_DUMP is sent and response is received, client may send one more PRE_DUMP or DUMP request. The server would only close the socket after the DUMP one.

Multi-request mode[edit]

If the req.keep_open flag is set to true server will not close the socket after response, but will wait for more requests. This mode is supported only for the following request types:

  • PRE_DUMP (automatically)
  • PAGE_SERVER
  • CPUINFO_DUMP and CPUINFO_CHECK

Run[edit]

SWRK mode[edit]

This mode turns on when one fork() + exec() CRIU with the swrk action and one more argument specifying the number of descriptor with SOCK_SEQPACKET Unix socket. With this CRIU works as service worker task accepting standard RPC requests via the mentioned socket and using one to do action scripts notifications and result reporting.

Server[edit]

On a server side, CRIU creates SOCK_SEQPACKET Unix socket and listens for connections on it. After receiving criu_req, CRIU processes it, does what was requested and sends criu_resp with set request-specific criu_***_resp field back. If CRIU gets unknown type of request, it will return criu_resp with type == EMPTY and success == false.

To launch the service, run:

# criu service [options]

Options accepted by service are

--address <path>
where to put listening socket
--pid-file <path>
where to write pid of service process
--daemon
tells service to daemonize
-o <file>
says where to write logs
-v[N]
sets the log level

Client[edit]

Client, in its turn, must connect to service socket, send criu_req with request in it, and wait for a criu_resp with response. You can find examples of client programs in C and Python in test/others/rpc/.

With RPC facilities one can perform a self dump.

There's a library that implements simple wrappers on top of RPC.

See also[edit]