Kubernetes

Container checkpointing was introduced as an alpha feature in Kubernetes v1.25 and graduated to beta in Kubernetes v1.30. This functionality allows running containers to be transparently checkpointed to persistent storage and later restored to resume execution, or migrated across nodes and clusters. The content of container checkpoints can be further analyzed with the checkpointctl tool. This allows to perform forensic analysis in case of security incidents (e.g., suspected compromise, data exfiltration) or application failures by inspecting the saved process memory, open files, sockets, and execution context captured in the checkpoint.

Kubelet Checkpoint API

This functionality is exposed through a node-local kubelet checkpoint API (enabled by default in Kubernetes v1.30). A checkpoint can be triggered by sending an HTTP POST Request to the kubelet as follows:

curl -X POST "https://localhost:10250/checkpoint/<Namespace>/<Pod>/<Container>"

Triggering this kubelet API will request the creation of a checkpoint from the container runtime (e.g., containerd or CRI-O). In tern, the container runtime requests a checkpoint from the low-level runtime (e.g., runc) that invokes CRIU.

Once the checkpointing has been created, it will be saved as a tar archive with the following name checkpoint-<pod>_<namespace>-<container>-<timestamp>.tar in /var/lib/kubelet/checkpoints.

Example

1. Creating a Pod with a single container

cat > pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: counters
spec:
  containers:
    - name: counter
      image: busybox:latest
      command: ['sh', '-c', 'i=0; while true; do echo $i; i=$((i+1)); sleep 1; done']
EOF

kubectl apply -f pod.yaml

The following command can be used to verify that the container is running:

kubectl logs -f -c counter counters

2. Create client-admin.crt and client-admin.key files

These certificate and key files will be used to authorize the use of the checkpoint API:

kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-certificate-data}' \
| base64 -d > client-admin.crt

kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-key-data}' \
| base64 -d > client-admin.key
chmod 600 client-admin.key

3. Creating a checkpoint of the running container

curl --insecure \
  --cert client-admin.crt \
  --key client-admin.key \
  -X POST "https://localhost:10250/checkpoint/default/counters/counter"

The --insecure command-line option is necessary for curl to accept the kubelet's self-signed certificate.