CKSKubernetes 1.35

Lessons

Container immutability at runtime

updated 2026-10-22 by seed

What "immutable" means here

An attacker who lands in a container should not be able to install a tool, drop a
binary, or edit a config on disk. Two levers:

  1. The image — distroless/scratch: no shell, no package manager, nothing to write to for most of the filesystem.
  2. The Pod spec — readOnlyRootFilesystem: true, which makes the container's root mount read-only at the runtime level.

The manifest, done properly

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 2000
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: nginx
    image: nginxinc/nginx-unprivileged:1.27
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities: {drop: ["ALL"]}
    volumeMounts:
    - {name: run, mountPath: /var/run}
    - {name: cache, mountPath: /var/cache/nginx}
    - {name: tmp, mountPath: /tmp}
  volumes:
  - {name: run, emptyDir: {}}
  - {name: cache, emptyDir: {}}
  - {name: tmp, emptyDir: {}}

The emptyDir mounts are not optional decorating: an app that writes anywhere at
startup (pid file, cache, temp) crash-loops without them, and a crash-looping Pod
scores zero.

Prove it

kubectl exec -n prod nginx -- touch /file         # Read-only file system
kubectl exec -n prod nginx -- touch /tmp/file     # works (emptyDir)
kubectl get pod nginx -n prod -o jsonpath='{.spec.containers[0].securityContext.readOnlyRootFilesystem}'

The triage task

"Find every Pod that is not immutable and fix or delete it." The markers look at:

kubectl get pods -A -o json | jq -r '
 .items[]
 | select(.spec.hostNetwork==true or .spec.hostPID==true or .spec.hostIPC==true
          or any(.spec.containers[]; .securityContext.privileged==true)
          or any(.spec.containers[]; (.securityContext.readOnlyRootFilesystem//false)==false))
 | "\(.metadata.namespace)/\(.metadata.name)"'

Then, for each one, decide: patch it (Deployment) or delete it (bare Pod), per the
task wording.

Config changes without a rebuild

The other half of immutability is "do not bake config into the image":

    volumeMounts:
    - {name: conf, mountPath: /etc/app, readOnly: true}
  volumes:
  - name: conf
    configMap:
      name: app-conf

Editing the ConfigMap does not change the running container until restart — which
is the property you want when the task says "the configuration must not be part of
the image".

Traps

  • readOnlyRootFilesystem is container-level only — putting it in the Pod securityContext is silently ignored.
  • Setting it without the emptyDir for a cache: CrashLoopBackOff.
  • runAsNonRoot: true plus an image with USER root → container refuses to start.
  • Distroless images have no shell, so kubectl exec -- sh fails: that is success, not a bug — but do not try to "verify" with a shell you removed.