Bài 2: Cluster Upgrade với kubeadm

1. Upgrade Strategy Overview

Kubernetes chỉ hỗ trợ upgrade 1 minor version một lần (v1.28 → v1.29, không nhảy v1.28 → v1.30). Luôn upgrade control plane trước worker nodes.

Upgrade sequence:
  1. Upgrade kubeadm (on control plane node)
  2. kubeadm upgrade apply v1.29.0 (control plane components)
  3. Upgrade kubelet + kubectl (on control plane)
  4. For each worker node:
     a. kubectl drain NODE --ignore-daemonsets
     b. Upgrade kubeadm + kubelet + kubectl on node
     c. kubeadm upgrade node
     d. kubectl uncordon NODE

2. Drain, Cordon & Uncordon

Command	Effect	Dùng khi
kubectl cordon NODE	Mark node Unschedulable (no new pods)	Chuẩn bị maintenance
kubectl drain NODE	Cordon + evict tất cả non-DaemonSet pods	Upgrade / replace node
kubectl uncordon NODE	Mark node Schedulable lại	Sau khi maintenance xong

kubectl drain flags thường dùng:
  --ignore-daemonsets     # DaemonSet pods không thể evict, ignore them
  --delete-emptydir-data  # Evict pods dùng emptyDir volume
  --force                 # Evict pods không managed by controller

Exam tip: kubectl drain mà không có --ignore-daemonsets sẽ fail nếu node có DaemonSet pods. Luôn thêm flag này. Nếu có pods dùng emptyDir, cũng cần --delete-emptydir-data.

3. Chi tiết Upgrade Steps

# ====== CONTROL PLANE NODE ======

# Step 1: Upgrade kubeadm
apt-mark unhold kubeadm
apt-get install -y kubeadm=1.29.0-00
apt-mark hold kubeadm

# Step 2: Verify upgrade plan
kubeadm upgrade plan

# Step 3: Apply upgrade
kubeadm upgrade apply v1.29.0

# Step 4: Upgrade kubelet + kubectl  
apt-mark unhold kubelet kubectl
apt-get install -y kubelet=1.29.0-00 kubectl=1.29.0-00
apt-mark hold kubelet kubectl
systemctl daemon-reload && systemctl restart kubelet

# ====== WORKER NODE ======
(SSH vào worker node)

# Step 1: Drain node from control plane
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# Step 2: Upgrade packages on worker
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y kubeadm=1.29.0-00 kubelet=1.29.0-00 kubectl=1.29.0-00

# Step 3: Upgrade node config
kubeadm upgrade node

# Step 4: Restart kubelet
systemctl daemon-reload && systemctl restart kubelet

# Step 5: Uncordon from control plane
kubectl uncordon worker-1

4. Version Skew Policy

Component	Allowed skew vs kube-apiserver
kube-apiserver	Must be same version as other control plane
kubelet	Can be 2 minor versions older
kubectl	±1 minor version from apiserver
kube-scheduler	Must match apiserver version

Exam tip: Worker node kubelet có thể chạy phiên bản cũ hơn trong quá trình upgrade. Đây là lý do upgrade từng node một mà cluster vẫn hoạt động.

5. Cheat Sheet

Task	Command
Check upgrade plan	kubeadm upgrade plan
Apply control plane upgrade	kubeadm upgrade apply v1.XX.0
Drain node (safe)	kubectl drain NODE --ignore-daemonsets --delete-emptydir-data
Mark node schedulable	kubectl uncordon NODE
Check node versions	kubectl get nodes -o wide

6. Practice Questions

Q1: You need to upgrade a worker node. Before running the upgrade commands on the node, what step must you perform from the control plane?

• A) kubectl cordon the node to prevent new Pod scheduling
• B) kubectl drain the node to evict running Pods ✓
• C) kubectl delete the node and re-add it after upgrade
• D) Run kubeadm upgrade plan to verify compatibility

Explanation: kubectl drain both cordons (marks unschedulable) AND evicts all Pods gracefully. This ensures the node has no running workloads before maintenance begins. kubeadm upgrade plan is run on control plane, not required per-worker-node.

Q2: The kubectl drain command fails with "cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet". What flag resolves this?

• A) --ignore-daemonsets
• B) --delete-emptydir-data
• C) --force ✓
• D) --grace-period=0

Explanation: --force is required to evict Pods that are not managed by any controller. Without a controller, the Pod won't be rescheduled elsewhere, so kubectl warns you and requires --force to confirm you accept potential data loss.

Q3: What is the maximum kubelet version skew allowed compared to the kube-apiserver?

• A) ±1 minor version
• B) 2 minor versions older ✓
• C) Any version
• D) Must be identical version

Explanation: Per Kubernetes version skew policy, kubelet can be at most 2 minor versions older than kube-apiserver. This allows rolling upgrades where nodes are upgraded one at a time while the control plane is already on the new version.