CKSKubernetes 1.35

Lessons

Falco: rules, overrides, and output channels

updated 2026-10-22 by seed

What Falco does

It reads kernel events (eBPF or a kernel module), evaluates rules written against
those events, and emits alerts. It is the "detect the attacker who is already
inside" tool: shell in a container, write to /etc, unexpected outbound
connection, privileged container started, sensitive file opened.

Install and look at what it is doing

# the exam usually has it installed
systemctl status falco --no-pager
journalctl -u falco -n 20 --no-pager
cat /etc/falco/falco.yaml | head -40
ls /etc/falco/
# falco.yaml  falco_rules.yaml  falco_rules.local.yaml  k8s_audit_rules.yaml

Rule anatomy

- rule: Terminal shell in container
  desc: A shell was used as the entrypoint or spawned inside a container
  condition: >
    spawned_process and container and shell_procs
    and proc.tty != 0 and container_entrypoint
  output: >
    A shell was spawned in a container with an attached terminal
    (user=%user.name container_id=%container.id image=%container.image.repository
     shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: NOTICE
  tags: [container, shell, mitre_execution]
  • condition is a filter over events, built from macros.
  • output is a format string — %field placeholders.
  • priority — EMERGENCY … DEBUG.

The override the exam asks for

Never edit /etc/falco/falco_rules.yaml — it is replaced on upgrade. Copy the
whole rule into /etc/falco/falco_rules.local.yaml, keep the same rule: name,
change what the task wants, restart Falco. Last rule with that name wins.

# /etc/falco/falco_rules.local.yaml
- rule: Write below etc
  desc: an attempt to write to any file below /etc
  condition: write_etc_common
  output: "File below /etc opened for writing (time=%evt.time.s user=%user.name container=%container.id file=%fd.name)"
  priority: ERROR
  tags: [filesystem, mitre_persistence]
sudo systemctl restart falco
sudo falco --validate /etc/falco/falco_rules.local.yaml   # if the version supports it

Partial overrides do not work: the rule is matched by name, the newer
definition replaces the old one entirely, and a missing condition makes it
invalid.

Output channels

# /etc/falco/falco.yaml
stdout_output:
  enabled: false
file_output:
  enabled: true
  keep_alive: false
  filename: /var/log/falco/events.log

Then sudo tail -f /var/log/falco/events.log.

Other channels: syslog_output, program_output (a shell command),
grpc_output, http_output.

Trigger an alert and prove it

kubectl exec -n prod web -- sh -c 'echo intrusion >> /etc/threat'
sudo tail -5 /var/log/falco/events.log
# 12:41:03: Warning File below /etc opened for writing (user=root container=... file=/etc/threat)

A shell in a container is the other reliable trigger:

kubectl exec -it -n prod web -- bash

Traps

  • Editing falco_rules.yaml directly: works until upgrade, and the marker may compare the file's mtime or the local file.
  • Forgetting systemctl restart falco — the running process has the old rules in memory.
  • A rule that never matches because the condition references a macro that was renamed between Falco versions. Copy the condition from the existing rule and change only what the task names.
  • Writing output to a file whose directory does not exist: Falco fails at start, and the service shows failed rather than an alert.