Audit logging: policy, static pod, proof

What an audit record is

One JSON line per API request, written after the request is processed, with
who did what to which object and what the authoriser decided.

The policy: four levels

Rules are evaluated top-down, first match wins, and the top-level
omitStages: ["RequestReceived"] halves the volume.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
# 1. never log the noisy stuff
- level: None
  users: ["system:kube-proxy"]
  verbs: ["watch"]
  resources:
  - {group: "", resources: ["endpoints", "services", "services/status"]}
# 2. no bodies for authentication decisions
- level: Metadata
  resources:
  - {group: "authentication.k8s.io", resources: ["tokenreviews"]}
# 3. secrets and configmaps: log the body, in the namespace that matters
- level: RequestResponse
  namespaces: ["config-data"]
  verbs: ["create", "update", "patch", "delete"]
  resources:
  - {group: "", resources: ["configmaps", "secrets"]}
# 4. everything in prod at metadata level
- level: Metadata
  namespaces: ["prod"]
# 5. catch-all
- level: Metadata
  omitStages: ["RequestReceived"]

Scoping knobs: namespaces, verbs, users, userGroups, resources,
resourceNames, nonResourceURLs, omitStages.

Wiring the log backend

# /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    volumeMounts:
    - {name: audit-policy, mountPath: /etc/kubernetes/audit/policy.yaml, readOnly: true}
    - {name: audit-log, mountPath: /var/log/kubernetes, readOnly: false}
  volumes:
  - name: audit-policy
    hostPath: {path: /etc/kubernetes/audit/policy.yaml, type: File}
  - name: audit-log
    hostPath: {path: /var/log/kubernetes, type: DirectoryOrCreate}

Details that decide the marks:

• type: File for the policy — if the file is missing the API server will not start.
• type: DirectoryOrCreate for the log directory.
• The log mount must be rw.
• No systemctl restart — the kubelet restarts the static pod when the manifest changes. sleep 20 && kubectl get --raw /healthz to confirm.

Prove it

kubectl run audit-test --image=nginx:1.27 -n prod
sudo tail -1 /var/log/kubernetes/audit.log | jq .

You want: "level":"Metadata", "stage":"ResponseComplete",
"verb":"create", "user":{"username":"kubernetes-admin"},
"objectRef":{"resource":"pods","namespace":"prod","name":"audit-test"} and
"annotations":{"authorization.k8s.io/decision":"allow"}.

# did anybody get denied?
sudo grep '"decision":"forbid"' /var/log/kubernetes/audit.log | jq -r '.user.username' | sort | uniq -c

Webhook backend

Same policy, different sink:

    - --audit-webhook-config-file=/etc/kubernetes/audit/webhook.yaml
    - --audit-webhook-batch-max-wait=5s

The kubeconfig points at a collector. If a task mentions a webhook, it wants the
same policy file plus this flag and config — not a new policy.

Traps

• Putting a broad Metadata rule before a narrow RequestResponse one: the narrow rule never matches. First match wins, so specific rules go first.
• Forgetting the volume mounts: the flags then point at files that are not inside the container, and the API server crash-loops.
• --audit-log-path on a directory that does not exist → no log, no error you will notice.
• Expecting kubectl get events to show audit records. It never will; the log is a file on the node.
• Logging level RequestResponse for all secrets cluster-wide fills the disk and writes secrets in clear. Scope it.