Secrets: types, consumption, and encryption at rest

What a Secret is (and is not)

A Secret is a base64-encoded value in etcd — not encrypted by default, and
readable by anyone who can read the object or the etcd data directory. base64 is
encoding, not protection.

kubectl create secret generic db --from-literal=password='S3cr3t!' -n prod
kubectl get secret db -n prod -o jsonpath='{.data.password}' | base64 -d

Typed secrets, and why the type matters

A task that says "create a TLS secret" and finds an Opaque one has failed, even
with the right key names.

Consumption: env vs volume

# environment variable — ends up in the process environment, visible in /proc
    env:
    - name: DB_PASS
      valueFrom:
        secretKeyRef: {name: db, key: password}

# volume — a file, mode 0400, never in the environment
    volumeMounts:
    - {name: db, mountPath: /etc/db, readOnly: true}
  volumes:
  - name: db
    secret:
      secretName: db
      defaultMode: 0400
      items:
      - {key: password, path: password}

Prefer volumes: environment variables leak into crash dumps, kubectl describe
of a badly written app, and child processes.

Encryption at rest — the full task

1. Generate a key.

head -c 32 /dev/urandom | base64

2. Write the EncryptionConfiguration.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets"]
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <the base64 key>
  - identity: {}       # fallback: read what is not yet encrypted

Order matters: the first provider encrypts new writes; the last is what reads
legacy/unencrypted data. identity last means "read plaintext too", which is what
you want during migration — and never as the first entry, or nothing gets
encrypted.

3. Give the API server the file and a volume.

# /etc/kubernetes/manifests/kube-apiserver.yaml
    - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
    volumeMounts:
    - {name: enc, mountPath: /etc/kubernetes/enc, readOnly: true}
  volumes:
  - {name: enc, hostPath: {path: /etc/kubernetes/enc, type: DirectoryOrCreate}}

The file is a hostPath because it must survive the container restart. Wait for
the API server to come back, then:

4. Rewrite the existing secrets — they are still plaintext until re-saved.

kubectl get secrets -A -o json | kubectl replace -f -

5. Prove it, from the node.

sudo ETCDCTL_API=3 etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/prod/db | strings | head

Encrypted output starts with k8s:enc:aescbc:v1:key1. Plaintext shows the value
in clear. That prefix is the proof; kubectl get secret looks identical
either way, which is why the task asks for etcdctl.

Traps

• Forgetting step 4 — the config is enabled, no secret is encrypted, the marker reads plaintext in etcd.
• Putting identity first: writes stay plaintext.
• The key must be base64 of exactly 32 bytes for aescbc (16/24/32 for aesgcm).
• A wrong hostPath type and the API server will not start.
• Deleting the key later makes existing secrets unreadable. In the exam, don't.