Lesson 9: etcd Backup & Restore

1. etcd — Overview

etcd is a distributed key-value store that holds the entire cluster state: Pods, Services, Secrets, ConfigMaps, Nodes. Losing etcd = losing the entire cluster.

etcd info from kube-apiserver manifest:
  cat /etc/kubernetes/manifests/etcd.yaml

Key paths:
  --data-dir=/var/lib/etcd          # Data directory
  --cert-file=/etc/kubernetes/pki/etcd/server.crt
  --key-file=/etc/kubernetes/pki/etcd/server.key
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
  --listen-client-urls=https://127.0.0.1:2379

2. etcdctl Setup

# Set API version (always use v3)
export ETCDCTL_API=3

# Find etcd certs
ls /etc/kubernetes/pki/etcd/
# ca.crt, server.crt, server.key, healthcheck-client.*

# Test connection
etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Exam tip: You must set ETCDCTL_API=3 before using etcdctl. API v2 uses different commands and is incompatible. On the exam, if you forget the cert paths: cat /etc/kubernetes/manifests/etcd.yaml | grep cert or kubectl describe pod etcd -n kube-system.

3. Backup etcd

ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify backup
ETCDCTL_API=3 etcdctl snapshot status /opt/etcd-backup.db \
  --write-out=table

# Output:
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| abcdef12 |    12345 |       1234 |     4.5 MB |
+----------+----------+------------+------------+

4. Restore etcd

# Step 1: Restore to new data directory
ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup.db \
  --data-dir=/var/lib/etcd-restore

# Step 2: Update etcd manifest to use new data dir
vi /etc/kubernetes/manifests/etcd.yaml

# Change --data-dir and hostPath volume:
spec:
  containers:
  - command:
    - --data-dir=/var/lib/etcd-restore  # Changed
  volumes:
  - hostPath:
      path: /var/lib/etcd-restore       # Changed
      type: DirectoryOrCreate
    name: etcd-data

# Step 3: kubelet detects manifest change → restarts etcd
# Wait for etcd to restart (may take 2-3 min)
kubectl get pods -n kube-system | grep etcd

Exam tip: After restoring, you need to wait for the entire control plane to restart and sync. You may need to restart: systemctl restart kubelet. If the API server doesn't come up, check logs: crictl logs $(crictl ps -a --name kube-apiserver -q).

5. Cheat Sheet — etcd Backup/Restore

# BACKUP (4 required flags):
ETCDCTL_API=3 etcdctl snapshot save BACKUP_PATH \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=CA_CERT \
  --cert=SERVER_CERT \
  --key=SERVER_KEY

# RESTORE (minimal):
ETCDCTL_API=3 etcdctl snapshot restore BACKUP_PATH \
  --data-dir=NEW_DATA_DIR

# Then update /etc/kubernetes/manifests/etcd.yaml → data-dir + volume path

Cert File	Path	Flag
CA cert	/etc/kubernetes/pki/etcd/ca.crt	--cacert
Server cert	/etc/kubernetes/pki/etcd/server.crt	--cert
Server key	/etc/kubernetes/pki/etcd/server.key	--key

6. Practice Questions

Q1: You perform an etcd snapshot restore to /var/lib/etcd-new. The cluster does not recover. What step is most likely missing?

• A) You need to re-run kubeadm init
• B) The etcd static Pod manifest data-dir and volume path must be updated to point to the new directory ✓
• C) etcdctl restore must be run with --force flag
• D) The kube-apiserver certificate must be rotated

Explanation: After restoring to a new directory, etcd's static Pod manifest (/etc/kubernetes/manifests/etcd.yaml) must be updated: change --data-dir flag AND the hostPath volume path to the new directory. Otherwise, etcd still reads the old (broken) data directory.

Q2: What environment variable must be set to use etcdctl v3 API commands?

• A) ETCD_VERSION=3
• B) ETCDCTL_API=3 ✓
• C) KUBECONFIG=/etc/kubernetes/etcd.conf
• D) ETCD_ENDPOINT=localhost:2379

Explanation: ETCDCTL_API=3 enables v3 API commands (snapshot save, snapshot restore). Without it, etcdctl defaults to v2, which uses different command syntax and is incompatible with etcd v3 clusters (which all Kubernetes clusters use).

Q3: Which of the following contains the TLS certificates required for etcdctl to communicate with the etcd server?

• A) /etc/kubernetes/pki/apiserver*.crt
• B) /etc/kubernetes/pki/etcd/ directory ✓
• C) ~/.kube/config
• D) /var/lib/etcd/certs/

Explanation: etcd certificates are stored in /etc/kubernetes/pki/etcd/. Important files: ca.crt (CA), server.crt and server.key (for etcdctl). These paths are also defined in the etcd static Pod manifest.