CKSKubernetes 1.35

Lessons

Upgrading Kubernetes without breaking it

updated 2026-10-22 by seed

The rules that make the order non-negotiable

  • Versioning is vMAJOR.MINOR.PATCH; support is the three most recent minors.
  • Skew: the API server may be at most one minor ahead of kubelets, and a kubelet may never be newer than the API server. Control plane first, always.
  • Three releases a year, ~14 months of support per minor.
  • Never skip a minor: 1.33 → 1.35 must pass through 1.34.

The runbook

# --- control plane ---
sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.35.0-1.1
sudo apt-mark hold kubeadm
kubeadm version
sudo kubeadm upgrade plan                      # shows the target and node order
sudo kubeadm upgrade apply v1.35.0             # answer "yes"

# --- every node, one at a time ---
kubectl drain node1 --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.35.0-1.1 kubectl=1.35.0-1.1
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
sudo kubeadm upgrade node
kubectl uncordon node1

kubeadm upgrade apply on the first control-plane node, kubeadm upgrade node on
every other control-plane node and on every worker.

What the exam actually asks

  • "Upgrade the cluster to v1.35.0" — the runbook above, control plane only.
  • "Cordon and drain a node before upgrading the kubelet" — the two commands, and --ignore-daemonsets.
  • "Renew the certificates" — kubeadm certs renew all then restart the control plane components; kubeadm certs check-expiration proves it.
  • "The kubelet is at the wrong version" — pin the version, daemon-reload, restart, kubectl get nodes -o wide to show the version column.

Certificates, because an upgrade task often hides one

sudo kubeadm certs check-expiration
sudo kubeadm certs renew all
sudo systemctl restart kubelet
# static pods pick up the new certs when their containers restart:
sudo crictl ps | grep -E 'apiserver|controller|scheduler|etcd'

If kubectl starts failing with x509: certificate has expired mid-task, this is
the fix, and it is why you keep /etc/kubernetes/pki out of your cleanup.

Draining without losing marks

kubectl drain node2 --ignore-daemonsets --delete-emptydir-data --force
# ... upgrade ...
kubectl uncordon node2
kubectl get nodes            # Ready, and the right version

Leaving a node cordoned is a classic silent failure: everything you do next still
works, and the marker finds the node unschedulable.