Validate files on restore - Revision history

Ymanton at 14:27, 19 October 2023

2023-10-19T14:27:59Z

Ptikhomirov at 05:33, 28 August 2023

2023-08-28T05:33:39Z

AnorexicAtticusFinch at 13:35, 25 August 2020

2020-08-25T13:35:52Z

AnorexicAtticusFinch: The article has been updated to include the performance impact of the changes made as part of GSoC 2020.

2020-08-25T13:31:19Z

The article has been updated to include the performance impact of the changes made as part of GSoC 2020.

AnorexicAtticusFinch at 16:05, 10 August 2020

2020-08-10T16:05:51Z

AnorexicAtticusFinch at 05:57, 10 August 2020

2020-08-10T05:57:14Z

AnorexicAtticusFinch: The article has been updated to include the changes made as part of GSoC 2020.

2020-08-10T05:55:54Z

The article has been updated to include the changes made as part of GSoC 2020.

Xemul at 15:56, 10 March 2020

2020-03-10T15:56:42Z

Xemul: Created page with "This page describes what CRIU does to make sure it restores on the correct set of files. Category:Files"

2020-03-10T15:52:53Z

Created page with "This page describes what CRIU does to make sure it restores on the correct set of files. Category:Files"

New page

This page describes what CRIU does to make sure it restores on the correct set of files.

[[Category:Files]]

← Older revision		Revision as of 14:27, 19 October 2023
Line 92:		Line 92:
	* Calculating the checksum can be made faster by using a lookup table.		* Calculating the checksum can be made faster by using a lookup table.

		+	[[Category:Files]]
	[[Category:Under the hood]]		[[Category:Under the hood]]

← Older revision		Revision as of 05:33, 28 August 2023
Line 1:		Line 1:
	This article describes what CRIU does to make sure it restores the correct set of files and how this file validation is implemented in CRIU. This project was completed under the [https://summerofcode.withgoogle.com/projects/#5773537320632320 GSoC 2020 program].		This article describes what CRIU does to make sure it restores the correct set of files and how this file validation is implemented in CRIU. This project was completed under the [https://summerofcode.withgoogle.com/projects/#5773537320632320 GSoC 2020 program].
		+
		+	Note: This is NOT merged to CRIU, see https://github.com/checkpoint-restore/criu/pull/1148

	== The previous implementation ==		== The previous implementation ==

@@ Line 49: / Line 49: @@
 For reference, these tests were run on tmpfs (To remove any disk latency) and on an undervolted i5 4800H.
-{| class="wikitable"
+{| class="wikitable" style="text-align: center;
 |+zdtm/transition/shmem:
 |-
@@ Line 68: / Line 68: @@
 |}
-{| class="wikitable"
+{| class="wikitable" style="text-align: center;
 |+zdtm/static/maps04:
 |-

← Older revision		Revision as of 13:31, 25 August 2020
Line 41:		Line 41:

	To explicitly use only the file size check all the time, the following command-line option can be used: <code>--file-validation filesize</code> (This is the fastest and least intensive check).		To explicitly use only the file size check all the time, the following command-line option can be used: <code>--file-validation filesize</code> (This is the fastest and least intensive check).
		+
		+	== Performance impact ==
		+	The values shown below are the average times it took to finish the ZDTM tests over multiple runs, and are only to indicate the impact each method has in general.
		+
		+	Each test has 3 files (Of sizes: 0.09 MB, 2 MB and 0.17 MB approximately) and each test is run 3 times (In Host, Namespace and User Namespace). For each file the checksum/build-ID is obtained twice (During dump and restore) therefore the function to find checksum/build-ID is called 18 times overall per test.
		+
		+	For reference, these tests were run on tmpfs (To remove any disk latency) and on an undervolted i5 4800H.
		+
		+	{\| class="wikitable"
		+	\|+zdtm/transition/shmem:
		+	\|-
		+	\|File Size
		+	\|3.782s
		+	\|-
		+	\|Build-ID
		+	\|4.153s (~9% increase)
		+	\|-
		+	\|Checksum (First 1024 bytes)
		+	\|4.465s (~18% increase)
		+	\|-
		+	\|Checksum (Entire File)
		+	\|4.722s (~24% increase)
		+	\|-
		+	\|Checksum (Every 1024th byte)
		+	\|4.498s (~19% increase)
		+	\|}
		+
		+	{\| class="wikitable"
		+	\|+zdtm/static/maps04:
		+	\|-
		+	\|File Size
		+	\|35.317s
		+	\|-
		+	\|Build-ID
		+	\|35.720s (~1% increase)
		+	\|-
		+	\|Checksum (First 1024 bytes)
		+	\|35.919s (~2% increase)
		+	\|-
		+	\|Checksum (Entire File)
		+	\|36.679s (~4% increase)
		+	\|-
		+	\|Checksum (Every 1024th byte)
		+	\|36.476s (~3% increase)
		+	\|}

	== Scope for improvement and future work ==		== Scope for improvement and future work ==

@@ Line 5: / Line 5: @@
 == The current implementation ==
-The file size method should be a preliminary method, if it fails there’s no need to do any of the more intensive checks and immediately give out an error and stop restoring. Stronger checks are used only if it passes.
+The file size method is used as a preliminary method, if it fails there’s no need to do any of the more intensive checks, instead it will immediately give out an error and stop restoring. Stronger checks are used only if it passes.
 The simplest and strongest check is to calculate the checksum for the entire file but this would be very intensive for large files and therefore not always feasible. A reasonable compromise would be to calculate the checksum only for certain parts of the file. This is the checksum method and it is one of the two methods that have been implemented in CRIU.

@@ Line 23: / Line 23: @@
 Adding a new configuration is quite simple and only requires the iterator to be moved to the necessary bytes (0 refers to the first byte of the file and 1 refers to the second byte and so on):
 # Input handling for the new configuration
-# The <code>calculate_checksum_iterator_init</code> function in ''"criu/files-reg.c”'' sets the initial iterator position (The first byte to calculate the checksum)
+# The <code>checksum_iterator_init</code> function in ''"criu/files-reg.c”'' sets the initial iterator position (The first byte to calculate the checksum)
-# The <code>calculate_checksum_iterator_next</code> function in ''“criu/files-reg.c”'' moves the iterator to the next position (The next byte to calculate the checksum)
+# The <code>checksum_iterator_next</code> function in ''“criu/files-reg.c”'' moves the iterator to the next position (The next byte to calculate the checksum)
-# The <code>calculate_checksum_iterator_stop_condition</code> function in ''“criu/files-reg.c”'' returns true when the iterator has reached its final position
+# The <code>checksum_iterator_stop</code> function in ''“criu/files-reg.c”'' returns true when the iterator has reached its final position
 There is a separate check in the <code>calculate_checksum</code> function to make sure the iterator refers to a valid byte (Not negative and smaller than the total number of bytes in the file). If the iterator is outside the mapped region but still valid, the required region of the file will be mapped.

@@ Line 1: / Line 1: @@
-This page describes what CRIU does to make sure it restores on the correct set of files.
+This article describes what CRIU does to make sure it restores the correct set of files and how this file validation is implemented in CRIU. This project was completed under the [https://summerofcode.withgoogle.com/projects/#5773537320632320 GSoC 2020 program].
-[[Category:Files]]
+== The previous implementation ==
-== Problem ==
+== The current implementation ==
-We make sure that the files we restore on are "the same" (whatever it means) as they were during dump. Especially this is critical for ELF-s (execulable and libs) Checksum is the way to go, but calculation of checksum over all files is very time and resource consuming, we shouldn't do it. Instead there should be some partial checksum for files that's both -- fast enough not to kill the system but at the same time string enough to bear common cases of files change. The primary goal is to prevent restoring on executables and libraries of wrong version.
+The file size method should be a preliminary method, if it fails there’s no need to do any of the more intensive checks and immediately give out an error and stop restoring. Stronger checks are used only if it passes.
-== Possible solutions ==
+The simplest and strongest check is to calculate the checksum for the entire file but this would be very intensive for large files and therefore not always feasible. A reasonable compromise would be to calculate the checksum only for certain parts of the file. This is the checksum method and it is one of the two methods that have been implemented in CRIU.
-=== Size checks ====
+The other method is the build-ID method. The build-ID is a "strongly unique embedded identifier" that (If present) is stored in a particular note section of ELF files.
-First and obvious -- check the file size is the same. This is what currently criu does
+== Build-ID ==
-=== Build-id ===
+The position of the program headers is stored as an offset in <code>phoff</code>. Since all the program headers are stored in an arbitrary order, each program header needs to be checked. If a program header of type <code>PT_NOTE</code> is found, the position of this note section is stored as an offset in <code>p_offset</code>. The notes are stored in an arbitrary order as well, so each note needs to be checked. If a note of type <code>NT_GNU_BUILD_ID</code> is found, the build-ID is present in its description.
-The ld linker supports the --build-id option that requests creation of ".note.gnu.build-id" ELF note section. If we find one in .so file, we can (probably) not count the checksub, but just read one and save in images and require it to match on restore. Need to check if standard (libc, libssl, and alike) libraries of modern distros use it.
+== Checksum ==
-=== Check-summing ===
+The file is mapped 10 MB at a time and the checksum is calculated on the required bytes (Depending on the configuration set - Entire file, First N bytes of the file, or Every Nth byte of the file). N is the checksum parameter.
-Other than this we may try to checksum parts of the file.
+Adding a new configuration is quite simple and only requires the iterator to be moved to the necessary bytes (0 refers to the first byte of the file and 1 refers to the second byte and so on):

← Older revision		Revision as of 15:56, 10 March 2020
Line 2:		Line 2:

	[[Category:Files]]		[[Category:Files]]
		+
		+	== Problem ==
		+	We make sure that the files we restore on are "the same" (whatever it means) as they were during dump. Especially this is critical for ELF-s (execulable and libs) Checksum is the way to go, but calculation of checksum over all files is very time and resource consuming, we shouldn't do it. Instead there should be some partial checksum for files that's both -- fast enough not to kill the system but at the same time string enough to bear common cases of files change. The primary goal is to prevent restoring on executables and libraries of wrong version.
		+
		+	== Possible solutions ==
		+
		+	=== Size checks ====
		+
		+	First and obvious -- check the file size is the same. This is what currently criu does
		+
		+	=== Build-id ===
		+
		+	The ld linker supports the --build-id option that requests creation of ".note.gnu.build-id" ELF note section. If we find one in .so file, we can (probably) not count the checksub, but just read one and save in images and require it to match on restore. Need to check if standard (libc, libssl, and alike) libraries of modern distros use it.
		+
		+	=== Check-summing ===
		+
		+	Other than this we may try to checksum parts of the file.