Kubernetes
Container checkpointing was introduced as an alpha feature in Kubernetes v1.25 and graduated to beta in Kubernetes v1.30. This functionality allows running containers to be transparently checkpointed to persistent storage and later restored to resume execution, or migrated across nodes and clusters. The content of container checkpoints can be further analyzed with the checkpointctl tool. This allows to perform forensic analysis in case of security incidents (e.g., suspected compromise, data exfiltration) or application failures by inspecting the saved process memory, open files, sockets, and execution context captured in the checkpoint.
Kubelet Checkpoint API
This functionality is exposed through a node-local kubelet checkpoint API (enabled by default in Kubernetes v1.30). A checkpoint can be triggered by sending an HTTP POST Request to the kubelet as follows:
curl -X POST "https://localhost:10250/checkpoint/<Namespace>/<Pod>/<Container>"
Triggering this kubelet API will request the creation of a checkpoint from the container runtime (e.g., containerd or CRI-O). In tern, the container runtime requests a checkpoint from the low-level runtime (e.g., runc) that invokes CRIU.
Once the checkpointing has been created, it will be saved as a tar archive with the following name checkpoint-<pod>_<namespace>-<container>-<timestamp>.tar in /var/lib/kubelet/checkpoints.
Usage Example
1. Creating a Pod with a single container
cat > pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: counters
spec:
containers:
- name: counter
image: busybox:latest
command: ['sh', '-c', 'i=0; while true; do echo $i; i=$((i+1)); sleep 1; done']
EOF
kubectl apply -f pod.yaml
The following command can be used to verify that the container is running:
kubectl logs -f -c counter counters
2. Create client-admin.crt and client-admin.key files
These certificate and key files will be used to authorize the use of the checkpoint API:
kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-certificate-data}' \
| base64 -d > client-admin.crt
kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-key-data}' \
| base64 -d > client-admin.key
chmod 600 client-admin.key
3. Creating a checkpoint of the running container
Note that the --insecure option is necessary for curl to accept the kubelet's self-signed certificate.
curl --insecure \ --cert client-admin.crt \ --key client-admin.key \ -X POST "https://localhost:10250/checkpoint/default/counters/counter"
Forensic Analysis
Once a container checkpoint has been created, it's content can be analysed with the help of the checkpointctl tool.
Overview of Checkpoints
checkpointctl provides list and show commands that display an overview of checkpoints stored in /var/lib/kubelet/checkpoints.
$ sudo checkpointctl list Listing checkpoints in path: /var/lib/kubelet/checkpoints/ NAMESPACE POD CONTAINER ENGINE TIME CHECKPOINTED CHECKPOINT NAME --------- --- --------- ------ ----------------- --------------- default counters counter containerd 07 Nov 25 11:58 UTC checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar default counters counter containerd 07 Nov 25 12:09 UTC checkpoint-counters_default-counter-2025-11-07T12:09:07Z.tar default counters counter containerd 07 Nov 25 12:30 UTC checkpoint-counters_default-counter-2025-11-07T12:30:00Z.tar $ sudo checkpointctl show /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar Displaying container checkpoint data from /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar CONTAINER IMAGE ID RUNTIME CREATED ENGINE CHKPT SIZE ROOT FS DIFF SIZE --------- ----- -- ------- ------- ------ ---------- ----------------- counter docker.io/library/busybox:latest 52d907dc8f75 io.containerd.runc.v2 2025-11-07T11:48:41Z containerd 306.8 KiB 270 B
Low-level Analysis
The checkpointctl inspect command can be used to perform low-level analysis of the checkpoint data.
$ sudo checkpointctl inspect --files --ps-tree --metadata /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar
Displaying container checkpoint tree view from /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar
counter
├── Image: docker.io/library/busybox:latest
├── ID: 52d907dc8f75c8a60b366c2fca70839b9505c9da909ef4ae4f90a1c59ccd69ba
├── Runtime: io.containerd.runc.v2
├── Created: 2025-11-07T11:48:41Z
├── Checkpointed: 2025-11-07T11:58:58Z
├── Engine: containerd
├── Checkpoint size: 306.8 KiB
│ └── Memory pages size: 292.0 KiB
├── Root FS diff size: 270 B
├── Metadata
│ ├── Pod name: counters
│ ├── Kubernetes namespace: default
│ └── Annotations
│ ├── io.kubernetes.cri.sandbox-name: counters
│ ├── io.kubernetes.cri.sandbox-namespace: default
│ ├── io.kubernetes.cri.sandbox-uid: 430de1f2-cb7b-4c96-8ea7-ba51d335845f
│ ├── io.kubernetes.cri.container-name: counter
│ ├── io.kubernetes.cri.container-type: container
│ ├── io.kubernetes.cri.image-name: busybox:latest
│ └── io.kubernetes.cri.sandbox-id: ee53903d4146165817d0a95e3cfd95340cb9f3bc1852ff28031e43d97e765d88
└── Process tree
└── [1] sh
├── Open files
│ ├── [REG 0] /dev/null
│ ├── [PIPE 1] pipe[4398338]
│ ├── [PIPE 2] pipe[4398339]
│ ├── [cwd] /
│ └── [root] /
└── [623] sleep
└── Open files
├── [REG 0] /dev/null
├── [PIPE 1] pipe[4398338]
├── [PIPE 2] pipe[4398339]
├── [cwd] /
└── [root] /
Memory Forensics
The checkpointctl memparse command can be used to analyze the memory pages of individual processes in the container checkpoint. When used without any options, this command will display a table with an overview of the processes: their names, IDs, and memory sizes. The --pid option can be used to specify a process to analyze. The --search and --search-regex options can be used to search for a string or regex pattern in the memory pages.