Kubernetes

Container checkpointing was introduced as an alpha feature in Kubernetes v1.25 and graduated to beta in Kubernetes v1.30. This functionality allows running containers to be transparently checkpointed to persistent storage and later restored to resume execution, or migrated across nodes and clusters. The content of container checkpoints can be further analyzed with the checkpointctl tool. This allows to perform forensic analysis in case of security incidents (e.g., suspected compromise, data exfiltration) or application failures by inspecting the saved process memory, open files, sockets, and execution context captured in the checkpoint.

Kubelet Checkpoint API

This functionality is exposed through a node-local kubelet checkpoint API (enabled by default in Kubernetes v1.30). A checkpoint can be triggered by sending an HTTP POST Request to the kubelet as follows:

curl -X POST "https://localhost:10250/checkpoint/<Namespace>/<Pod>/<Container>"

Triggering this kubelet API will request the creation of a checkpoint from the container runtime (e.g., containerd or CRI-O). In tern, the container runtime requests a checkpoint from the low-level runtime (e.g., runc) that invokes CRIU.

Once the checkpointing has been created, it will be saved as a tar archive with the following name checkpoint-<pod>_<namespace>-<container>-<timestamp>.tar in /var/lib/kubelet/checkpoints.

Usage Example

1. Creating a Pod with a single container

cat > pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: counters
spec:
  containers:
    - name: counter
      image: busybox:latest
      command: ['sh', '-c', 'i=0; while true; do echo $i; i=$((i+1)); sleep 1; done']
EOF

kubectl apply -f pod.yaml

The following command can be used to verify that the container is running:

kubectl logs -f -c counter counters

2. Create client-admin.crt and client-admin.key files

These certificate and key files will be used to authorize the use of the checkpoint API:

kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-certificate-data}' \
| base64 -d > client-admin.crt

kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-key-data}' \
| base64 -d > client-admin.key
chmod 600 client-admin.key

3. Creating a checkpoint of the running container

Note that the --insecure option is necessary for curl to accept the kubelet's self-signed certificate.

curl --insecure \
  --cert client-admin.crt \
  --key client-admin.key \
  -X POST "https://localhost:10250/checkpoint/default/counters/counter"

Forensic Analysis

Once a container checkpoint has been created, it's content can be analysed with the help of the checkpointctl tool.

Overview of Checkpoints

checkpointctl provides list and show commands that display an overview of checkpoints stored in /var/lib/kubelet/checkpoints.

$ sudo checkpointctl list
Listing checkpoints in path: /var/lib/kubelet/checkpoints/
NAMESPACE   POD        CONTAINER   ENGINE       TIME CHECKPOINTED     CHECKPOINT NAME
---------   ---        ---------   ------       -----------------     ---------------
default     counters   counter     containerd   07 Nov 25 11:58 UTC   checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar
default     counters   counter     containerd   07 Nov 25 12:09 UTC   checkpoint-counters_default-counter-2025-11-07T12:09:07Z.tar
default     counters   counter     containerd   07 Nov 25 12:30 UTC   checkpoint-counters_default-counter-2025-11-07T12:30:00Z.tar

$ sudo checkpointctl show /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar
Displaying container checkpoint data from /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar

CONTAINER   IMAGE                              ID             RUNTIME                 CREATED                ENGINE       CHKPT SIZE   ROOT FS DIFF SIZE
---------   -----                              --             -------                 -------                ------       ----------   -----------------
counter     docker.io/library/busybox:latest   52d907dc8f75   io.containerd.runc.v2   2025-11-07T11:48:41Z   containerd   306.8 KiB    270 B

Low-level Analysis

The inspect command can be used to perform low-level analysis of the checkpoint data.

$ sudo checkpointctl inspect --files --ps-tree --metadata /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar

Displaying container checkpoint tree view from /var/lib/kubelet/checkpoints/checkpoint-counters_default-counter-2025-11-07T11:58:58Z.tar

counter
├── Image: docker.io/library/busybox:latest
├── ID: 52d907dc8f75c8a60b366c2fca70839b9505c9da909ef4ae4f90a1c59ccd69ba
├── Runtime: io.containerd.runc.v2
├── Created: 2025-11-07T11:48:41Z
├── Checkpointed: 2025-11-07T11:58:58Z
├── Engine: containerd
├── Checkpoint size: 306.8 KiB
│   └── Memory pages size: 292.0 KiB
├── Root FS diff size: 270 B
├── Metadata
│   ├── Pod name: counters
│   ├── Kubernetes namespace: default
│   └── Annotations
│       ├── io.kubernetes.cri.sandbox-name: counters
│       ├── io.kubernetes.cri.sandbox-namespace: default
│       ├── io.kubernetes.cri.sandbox-uid: 430de1f2-cb7b-4c96-8ea7-ba51d335845f
│       ├── io.kubernetes.cri.container-name: counter
│       ├── io.kubernetes.cri.container-type: container
│       ├── io.kubernetes.cri.image-name: busybox:latest
│       └── io.kubernetes.cri.sandbox-id: ee53903d4146165817d0a95e3cfd95340cb9f3bc1852ff28031e43d97e765d88
└── Process tree
    └── [1]  sh
        ├── Open files
        │   ├── [REG 0]  /dev/null
        │   ├── [PIPE 1]  pipe[4398338]
        │   ├── [PIPE 2]  pipe[4398339]
        │   ├── [cwd]  /
        │   └── [root]  /
        └── [623]  sleep
            └── Open files
                ├── [REG 0]  /dev/null
                ├── [PIPE 1]  pipe[4398338]
                ├── [PIPE 2]  pipe[4398339]
                ├── [cwd]  /
                └── [root]  /