xDev Asia

etcd backup với etcdctl snapshot. Restore cluster từ backup. TLS certificates cho etcd. Critical CKA exam task — cần thành thạo hoàn toàn.

1. etcd — Tổng quan

etcd là distributed key-value store lưu trữ toàn bộ cluster state: Pods, Services, Secrets, ConfigMaps, Nodes. Mất etcd = mất toàn bộ cluster.

etcd info từ kube-apiserver manifest:
  cat /etc/kubernetes/manifests/etcd.yaml

Key paths:
  --data-dir=/var/lib/etcd          # Data directory
  --cert-file=/etc/kubernetes/pki/etcd/server.crt
  --key-file=/etc/kubernetes/pki/etcd/server.key
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
  --listen-client-urls=https://127.0.0.1:2379

2. etcdctl Setup

# Set API version (always use v3)
export ETCDCTL_API=3

# Find etcd certs
ls /etc/kubernetes/pki/etcd/
# ca.crt, server.crt, server.key, healthcheck-client.*

# Test connection
etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Exam tip: Phải set ETCDCTL_API=3 trước khi dùng etcdctl. API v2 dùng lệnh khác và không tương thích. Trên exam, nếu quên certs path: cat /etc/kubernetes/manifests/etcd.yaml | grep cert hoặc kubectl describe pod etcd -n kube-system.

3. Backup etcd

ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify backup
ETCDCTL_API=3 etcdctl snapshot status /opt/etcd-backup.db \
  --write-out=table

# Output:
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| abcdef12 |    12345 |       1234 |     4.5 MB |
+----------+----------+------------+------------+

4. Restore etcd

# Step 1: Restore to new data directory
ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore

# Step 2: Update etcd manifest to use new data dir
vi /etc/kubernetes/manifests/etcd.yaml

# Change --data-dir và hostPath volume:
spec:
  containers:
  - command:
    - --data-dir=/var/lib/etcd-restore  # Changed
  volumes:
  - hostPath:
      path: /var/lib/etcd-restore       # Changed
      type: DirectoryOrCreate
    name: etcd-data

# Step 3: kubelet detects manifest change → restarts etcd
# Wait for etcd to restart (may take 2-3 min)
kubectl get pods -n kube-system | grep etcd

Exam tip: Sau khi restore, cần đợi toàn bộ control plane restart và sync. Có thể cần restart: systemctl restart kubelet. Nếu API server không lên, xem logs: crictl logs $(crictl ps -a --name kube-apiserver -q).

5. Cheat Sheet — etcd Backup/Restore

# BACKUP (4 required flags):
ETCDCTL_API=3 etcdctl snapshot save BACKUP_PATH \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=CA_CERT \
  --cert=SERVER_CERT \
  --key=SERVER_KEY

# RESTORE (minimal):
ETCDCTL_API=3 etcdctl snapshot restore BACKUP_PATH \
  --data-dir=NEW_DATA_DIR

# Then update /etc/kubernetes/manifests/etcd.yaml → data-dir + volume path

Cert File	Path	Flag
CA cert	/etc/kubernetes/pki/etcd/ca.crt	--cacert
Server cert	/etc/kubernetes/pki/etcd/server.crt	--cert
Server key	/etc/kubernetes/pki/etcd/server.key	--key

6. Practice Questions

Q1: You perform an etcd snapshot restore to /var/lib/etcd-new. The cluster does not recover. What step is most likely missing?

A) You need to re-run kubeadm init
B) The etcd static Pod manifest data-dir and volume path must be updated to point to the new directory ✓
C) etctdl restore must be run with --force flag
D) The kube-apiserver certificate must be rotated

Explanation: After restoring to a new directory, etcd's static Pod manifest (/etc/kubernetes/manifests/etcd.yaml) must be updated: change --data-dir flag AND the hostPath volume path to the new directory. Otherwise, etcd still reads the old (broken) data directory.

Q2: What environment variable must be set to use etcdctl v3 API commands?

A) ETCD_VERSION=3
B) ETCDCTL_API=3 ✓
C) KUBECONFIG=/etc/kubernetes/etcd.conf
D) ETCD_ENDPOINT=localhost:2379

Explanation: ETCDCTL_API=3 enables v3 API commands (snapshot save, snapshot restore). Without it, etcdctl defaults to v2, which uses different command syntax and is incompatible with etcd v3 clusters (which all Kubernetes clusters use).

Q3: Which of the following contains the TLS certificates required for etcdctl to communicate with the etcd server?

A) /etc/kubernetes/pki/apiserver*.crt
B) /etc/kubernetes/pki/etcd/ directory ✓
C) ~/.kube/config
D) /var/lib/etcd/certs/