Kubernetes

Container checkpointing was introduced as an alpha feature in Kubernetes v1.25 and graduated to beta in Kubernetes v1.30. This functionality allows running containers to be transparently checkpointed to persistent storage and later restored to resume execution, or migrated across nodes and clusters. The content of container checkpoints can be further analyzed with the checkpointctl tool. This allows to perform forensic analysis in case of security incidents (e.g., suspected compromise, data exfiltration) or application failures by inspecting the saved process memory, open files, sockets, and execution context captured in the checkpoint.

Kubelet Checkpoint API

This functionality is exposed through a node-local kubelet checkpoint API (enabled by default in Kubernetes v1.30). A checkpoint can be triggered by sending an HTTP POST Request to the kubelet as follows:

curl -X POST "https://localhost:10250/checkpoint/<Namespace>/<Pod>/<Container>"

Triggering this kubelet API will request the creation of a checkpoint from the container runtime (e.g., containerd or CRI-O). In tern, the container runtime requests a checkpoint from the low-level runtime (e.g., runc) that invokes CRIU.

Once the checkpointing has been created, it will be saved as a tar archive with the following name checkpoint-<pod>_<namespace>-<container>-<timestamp>.tar in /var/lib/kubelet/checkpoints.

Example

1. Creating a Pod called counters with a single container called counter

cat > pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: counters
spec:
  containers:
    - name: counter
      image: busybox:latest
      command: ['sh', '-c', 'i=0; while true; do echo $i; i=$((i+1)); sleep 1; done']
EOF

kubectl apply -f pod.yaml

The following command can be used to verify that the container is running:

kubectl logs -f -c counter counters

2. Creating a checkpoint of the running container

curl -X POST "https://localhost:10250/checkpoint/default/counters/counter"

The following command-line options are necessary for curl to accept the kubelet's self-signed certificate (--insecure) and authorize the use of the checkpoint API (--cert, --key).

--insecure --cert client-admin.crt --key client-admin.key

The client-admin.crt and client-admin.key files can be created as follows:

kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-certificate-data}'
| base64 -d > client-admin.crt

kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-key-data}' \
| base64 -d > client-admin.key
chmod 600 client-admin.key