1. Upgrade Strategy Overview
Kubernetes only supports upgrading 1 minor version at a time (v1.28 → v1.29, not v1.28 → v1.30). Always upgrade the control plane before worker nodes.
Upgrade sequence:
1. Upgrade kubeadm (on control plane node)
2. kubeadm upgrade apply v1.29.0 (control plane components)
3. Upgrade kubelet + kubectl (on control plane)
4. For each worker node:
a. kubectl drain NODE --ignore-daemonsets
b. Upgrade kubeadm + kubelet + kubectl on node
c. kubeadm upgrade node
d. kubectl uncordon NODE2. Drain, Cordon & Uncordon
| Command | Effect | When to use |
|---|---|---|
kubectl cordon NODE | Mark node Unschedulable (no new pods) | Preparing for maintenance |
kubectl drain NODE | Cordon + evict all non-DaemonSet pods | Upgrade / replace node |
kubectl uncordon NODE | Mark node Schedulable again | After maintenance is done |
Common kubectl drain flags:
--ignore-daemonsets # DaemonSet pods cannot be evicted, ignore them
--delete-emptydir-data # Evict pods using emptyDir volume
--force # Evict pods not managed by any controllerExam tip:
kubectl drainwithout--ignore-daemonsetswill fail if the node has DaemonSet pods. Always add this flag. If pods useemptyDir, also add--delete-emptydir-data.
3. Detailed Upgrade Steps
# ====== CONTROL PLANE NODE ======
# Step 1: Upgrade kubeadm
apt-mark unhold kubeadm
apt-get install -y kubeadm=1.29.0-00
apt-mark hold kubeadm
# Step 2: Verify upgrade plan
kubeadm upgrade plan
# Step 3: Apply upgrade
kubeadm upgrade apply v1.29.0
# Step 4: Upgrade kubelet + kubectl
apt-mark unhold kubelet kubectl
apt-get install -y kubelet=1.29.0-00 kubectl=1.29.0-00
apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet
# ====== WORKER NODE ======
(SSH into the worker node)
# Step 1: Drain node from control plane
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
# Step 2: Upgrade packages on worker
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=1.29.0-00 kubelet=1.29.0-00 kubectl=1.29.0-00
# Step 3: Upgrade node config
kubeadm upgrade node
# Step 4: Restart kubelet
systemctl daemon-reload && systemctl restart kubelet
# Step 5: Uncordon from control plane
kubectl uncordon worker-14. Version Skew Policy
| Component | Allowed skew vs kube-apiserver |
|---|---|
| kube-apiserver | Must be same version as other control plane |
| kubelet | Can be 2 minor versions older |
| kubectl | ±1 minor version from apiserver |
| kube-scheduler | Must match apiserver version |
Exam tip: Worker node kubelet can run an older version during upgrade. This is why nodes can be upgraded one at a time while the cluster remains operational.
5. Cheat Sheet
| Task | Command |
|---|---|
| Check upgrade plan | kubeadm upgrade plan |
| Apply control plane upgrade | kubeadm upgrade apply v1.XX.0 |
| Drain node (safe) | kubectl drain NODE --ignore-daemonsets --delete-emptydir-data |
| Mark node schedulable | kubectl uncordon NODE |
| Check node versions | kubectl get nodes -o wide |
6. Practice Questions
Q1: You need to upgrade a worker node. Before running the upgrade commands on the node, what step must you perform from the control plane?
- A) kubectl cordon the node to prevent new Pod scheduling
- B) kubectl drain the node to evict running Pods ✓
- C) kubectl delete the node and re-add it after upgrade
- D) Run kubeadm upgrade plan to verify compatibility
Explanation: kubectl drain both cordons (marks unschedulable) AND evicts all Pods gracefully. This ensures the node has no running workloads before maintenance begins. kubeadm upgrade plan is run on control plane, not required per-worker-node.
Q2: The kubectl drain command fails with "cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet". What flag resolves this?
- A) --ignore-daemonsets
- B) --delete-emptydir-data
- C) --force ✓
- D) --grace-period=0
Explanation: --force is required to evict Pods that are not managed by any controller. Without a controller, the Pod won't be rescheduled elsewhere, so kubectl warns you and requires --force to confirm you accept potential data loss.
Q3: What is the maximum kubelet version skew allowed compared to the kube-apiserver?
- A) ±1 minor version
- B) 2 minor versions older ✓
- C) Any version
- D) Must be identical version
Explanation: Per Kubernetes version skew policy, kubelet can be at most 2 minor versions older than kube-apiserver. This allows rolling upgrades where nodes are upgraded one at a time while the control plane is already on the new version.