Deploy Microservices On-Premises với Kubernetes HA

Phần 2: Kubernetes HA Cluster với kubeadm

xdev.asia

🎯 MỤC TIÊU BÀI HỌC

Sau khi hoàn thành bài học này, bạn sẽ:

• ✅ Hiểu Raft consensus algorithm và etcd internals
• ✅ Thành thạo etcdctl cho operations hàng ngày
• ✅ Setup automated backup với CronJob
• ✅ Thực hành restore từ snapshot (disaster recovery)
• ✅ Defragmentation, compaction, và performance tuning

---

PHẦN 1: ETCD INTERNALS

1.1. Raft Consensus Algorithm

graph TB
    subgraph CLUSTER["etcd 3-Node Cluster"]
        subgraph N1["etcd-1 — LEADER"]
            WAL1["WAL
Write-Ahead Log"]
            SNAP1["Snapshot"]
        end
        subgraph N2["etcd-2 — FOLLOWER"]
            WAL2["WAL"]
            SNAP2["Snapshot"]
        end
        subgraph N3["etcd-3 — FOLLOWER"]
            WAL3["WAL"]
            SNAP3["Snapshot"]
        end
    end

    N1 -->|"Replicate"| N2
    N1 -->|"Replicate"| N3

    style N1 fill:#15803d,stroke:#22c55e,color:#fff
    style N2 fill:#1e3a5f,stroke:#3b82f6,color:#e2e8f0
    style N3 fill:#1e3a5f,stroke:#3b82f6,color:#e2e8f0

sequenceDiagram
    participant C as Client
    participant L as Leader (etcd-1)
    participant F1 as Follower (etcd-2)
    participant F2 as Follower (etcd-3)

    Note over C,F2: Write Flow — Raft Consensus

    C->>L: 1. Write request
    L->>L: 2. Ghi vào WAL
    par Replicate
        L->>F1: 3. Replicate entry
        L->>F2: 3. Replicate entry
    end
    F1-->>L: ACK ✅
    F2-->>L: ACK ✅
    Note over L: 4. MAJORITY (2/3) confirm → COMMIT
    L->>L: 5. Apply vào state machine
    L-->>C: 6. Response: success ✅

⚠️ etcd tolerates (N-1)/2 failures:

• 3 nodes → tolerate 1 failure (quorum = 2)
• 5 nodes → tolerate 2 failures (quorum = 3)

1.2. Data Model

# etcd lưu trữ key-value pairs trong B+ Tree
# Kubernetes objects → etcd keys:
# /registry/deployments/default/nginx
# /registry/pods/kube-system/coredns-xxx
# /registry/secrets/default/my-secret
# /registry/configmaps/kube-system/kubeadm-config
# /registry/services/default/kubernetes
etcdctl alias (thêm vào ~/.bashrc):
alias etcdctl='kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key'

---

PHẦN 2: ETCDCTL OPERATIONS

2.1. Cluster Health Monitoring

# Tạo script helper (chạy trên master1):
cat > /usr/local/bin/etcd-check.sh << 'EOF'
#!/bin/bash
ETCDCTL_CMD="etcdctl \
  --endpoints=https://10.10.20.11:2379,https://10.10.20.12:2379,https://10.10.20.13:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key"
echo "=== MEMBER LIST ==="
$ETCDCTL_CMD member list -w table
echo ""
echo "=== ENDPOINT HEALTH ==="
$ETCDCTL_CMD endpoint health -w table
echo ""
echo "=== ENDPOINT STATUS ==="
$ETCDCTL_CMD endpoint status -w table
echo ""
echo "=== ALARM LIST ==="
$ETCDCTL_CMD alarm list
EOF
chmod +x /usr/local/bin/etcd-check.sh

2.2. Performance Check

# etcd performance benchmark:
kubectl -n kube-system exec etcd-master1 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  check perf --load="s"
Output:
60/60: 99% PASS  (write: 2000/2000, read: 8000/8000)
Passed ✅
Disk performance (CRITICAL cho etcd):
etcd yêu cầu p99 write latency < 10ms
Nếu > 10ms → cần NVMe SSD
fio --rw=write --ioengine=sync 

--fdatasync=1 --directory=/var/lib/etcd 

--size=22m --bs=2300 --name=etcd-test
Kiểm tra: fsync/fdatasync p99 latency

2.3. Xem Kubernetes data trong etcd

# List tất cả keys (registry prefix):
kubectl -n kube-system exec etcd-master1 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry --prefix --keys-only | head -20
Output:
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
/registry/clusterrolebindings/cluster-admin
/registry/clusterroles/admin
/registry/configmaps/kube-system/kubeadm-config
...
Đếm số keys:
kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

get /registry --prefix --keys-only | wc -l
~500-1000 keys cho cluster mới

---

PHẦN 3: ETCD BACKUP

3.1. Manual Snapshot

# Tạo snapshot trên master1:
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd/snapshot-$(date +%Y%m%d-%H%M%S).db
Verify snapshot:
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd/snapshot-*.db -w table
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| abc12345 |   14523  |        832 |     3.3 MB |
+----------+----------+------------+------------+

3.2. Automated Backup Script

# /usr/local/bin/etcd-backup.sh:
cat > /usr/local/bin/etcd-backup.sh << 'SCRIPT'
#!/bin/bash
set -euo pipefail
BACKUP_DIR="/backup/etcd"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d-%H%M%S)
SNAPSHOT="${BACKUP_DIR}/snapshot-${DATE}.db"
Tạo backup directory:
mkdir -p "${BACKUP_DIR}"
Snapshot:
ETCDCTL_API=3 etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

snapshot save "${SNAPSHOT}"
Verify snapshot:
ETCDCTL_API=3 etcdctl snapshot status "${SNAPSHOT}" -w json | jq .
gzip compress:
gzip "${SNAPSHOT}"
Backup certificates (cần cho restore):
tar czf "${BACKUP_DIR}/pki-${DATE}.tar.gz" /etc/kubernetes/pki/
Copy to remote storage (NFS, S3, etc.):
rsync -avz "${BACKUP_DIR}/" backup-server:/etcd-backups/
Cleanup old backups:
find "${BACKUP_DIR}" -name "snapshot-.db.gz" -mtime +${RETENTION_DAYS} -delete
find "${BACKUP_DIR}" -name "pki-.tar.gz" -mtime +${RETENTION_DAYS} -delete
echo "[$(date)] Backup completed: ${SNAPSHOT}.gz"
SCRIPT
chmod +x /usr/local/bin/etcd-backup.sh

3.3. Cron Backup (mỗi 6 giờ)

# Crontab trên master1:
crontab -e
# Add:
0 */6 * * * /usr/local/bin/etcd-backup.sh >> /var/log/etcd-backup.log 2>&1
Hoặc dùng Kubernetes CronJob:

# etcd-backup-cronjob.yaml:
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 */6 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
            - key: "node-role.kubernetes.io/control-plane"
              effect: "NoSchedule"
          containers:
            - name: etcd-backup
              image: registry.k8s.io/etcd:3.5.15-0
              command:
                - /bin/sh
                - -c
                - |
                  DATE=$(date +%Y%m%d-%H%M%S)
                  etcdctl snapshot save /backup/snapshot-${DATE}.db \
                    --endpoints=https://127.0.0.1:2379 \
                    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                    --cert=/etc/kubernetes/pki/etcd/server.crt \
                    --key=/etc/kubernetes/pki/etcd/server.key
                  etcdctl snapshot status /backup/snapshot-${DATE}.db -w table
                  # Cleanup old backups
                  find /backup -name "snapshot-*.db" -mtime +30 -delete
              volumeMounts:
                - name: etcd-certs
                  mountPath: /etc/kubernetes/pki/etcd
                  readOnly: true
                - name: backup
                  mountPath: /backup
          restartPolicy: OnFailure
          volumes:
            - name: etcd-certs
              hostPath:
                path: /etc/kubernetes/pki/etcd
            - name: backup
              hostPath:
                path: /backup/etcd
                type: DirectoryOrCreate

---

PHẦN 4: DISASTER RECOVERY — RESTORE

4.1. Restore Scenario

graph TD
    DISASTER["🔴 DISASTER
etcd data bị corrupt
trên tất cả 3 nodes"]
    S1["1️⃣ Stop kube-apiserver + etcd
trên TẤT CẢ masters"]
    S2["2️⃣ Restore snapshot trên MỖI master
different initial-cluster settings"]
    S3["3️⃣ Restart etcd
→ form new cluster"]
    S4["4️⃣ Restart kube-apiserver"]
    S5["5️⃣ Verify cluster"]
    OK["✅ Cluster restored"]
    WARN["⚠️ Tất cả resources sau
thời điểm snapshot sẽ BỊ MẤT"]

    DISASTER --> S1 --> S2 --> S3 --> S4 --> S5 --> OK
    S2 --> WARN

    style DISASTER fill:#dc2626,stroke:#fca5a5,color:#fff
    style OK fill:#15803d,stroke:#22c55e,color:#fff
    style WARN fill:#92400e,stroke:#f59e0b,color:#fef3c7

4.2. Restore Step-by-Step

# === BƯỚC 1: Stop tất cả trên MỖI master ===
# Trên master1, master2, master3:
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
Đợi pods terminate:
crictl ps | grep -E "etcd|apiserver"
Phải trống (không có container nào)
Backup data cũ:
mv /var/lib/etcd /var/lib/etcd.bak
=== BƯỚC 2: Restore snapshot trên MỖI master ===
Trên master1:
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/snapshot-20250402-060000.db 

--name=master1 

--initial-cluster="master1=https://10.10.20.11:2380,master2=https://10.10.20.12:2380,master3=https://10.10.20.13:2380" 

--initial-advertise-peer-urls=https://10.10.20.11:2380 

--data-dir=/var/lib/etcd
Trên master2:
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/snapshot-20250402-060000.db 

--name=master2 

--initial-cluster="master1=https://10.10.20.11:2380,master2=https://10.10.20.12:2380,master3=https://10.10.20.13:2380" 

--initial-advertise-peer-urls=https://10.10.20.12:2380 

--data-dir=/var/lib/etcd
Trên master3:
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd/snapshot-20250402-060000.db 

--name=master3 

--initial-cluster="master1=https://10.10.20.11:2380,master2=https://10.10.20.12:2380,master3=https://10.10.20.13:2380" 

--initial-advertise-peer-urls=https://10.10.20.13:2380 

--data-dir=/var/lib/etcd
=== BƯỚC 3: Restart etcd và API server trên MỖI master ===
mv /tmp/etcd.yaml /etc/kubernetes/manifests/
Đợi etcd start:
crictl ps | grep etcd
Khi etcd running → start API server:
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
=== BƯỚC 4: Verify ===
kubectl get nodes
kubectl get pods -A
etcd-check.sh

---

PHẦN 5: COMPACTION VÀ DEFRAGMENTATION

5.1. Compaction

# etcd giữ tất cả revisions (history).
# Compaction xóa history cũ, giảm DB size.
Lấy current revision:
kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

endpoint status -w json | jq '.[0].Status.header.revision'
Output: 14523
Compact tới revision cũ (giữ lại last 1000 revisions):
kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

compact 13523
compacted revision 13523
⚠️ Kubernetes kube-apiserver tự động compact etcd
Thường không cần manual compact

5.2. Defragmentation

# Sau compaction, disk space không được giải phóng ngay
# Defrag giải phóng disk space:
⚠️ Defrag BLOCK etcd trong vài giây
Chạy trên TỪNG node, KHÔNG đồng thời:
Defrag master1:
kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://10.10.20.11:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

defrag
Finished defragmenting etcd member[https://10.10.20.11:2379]
Repeat cho master2, master3 (lần lượt, không đồng thời)

5.3. Alarm Management

# etcd tự động raise alarm khi:
# - NOSPACE: DB size > quota (default 2GB)
# - CORRUPT: Data corruption detected
List alarms:
kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

alarm list
(empty = no alarms ✅)
Nếu NOSPACE alarm:
1. Compact old revisions
2. Defrag
3. Disarm alarm:
etcdctl alarm disarm

---

PHẦN 6: ETCD MONITORING

6.1. Prometheus Metrics

# etcd expose metrics tại port 2381 (đã cấu hình ở Bài 6):
curl -sk https://10.10.20.11:2381/metrics | head -20
Key metrics to monitor:
etcd_server_has_leader                    = 1 (phải = 1)
etcd_server_leader_changes_seen_total     = number of leader changes
etcd_disk_wal_fsync_duration_seconds      = WAL fsync latency
etcd_disk_backend_commit_duration_seconds = backend commit latency
etcd_mvcc_db_total_size_in_bytes         = DB size
etcd_network_peer_round_trip_time_seconds = peer RTT

6.2. PrometheusRule (sẽ dùng ở Bài 32)

# etcd-alerts.yaml (tham khảo):
groups:
  - name: etcd
    rules:
      - alert: EtcdNoLeader
        expr: etcd_server_has_leader == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "etcd member has no leader"
  - alert: EtcdHighDiskLatency
    expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 0.01
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "etcd WAL fsync p99 > 10ms"

  - alert: EtcdDatabaseSizeHigh
    expr: etcd_mvcc_db_total_size_in_bytes > 6*1024*1024*1024
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "etcd DB size > 6GB (quota 8GB)"

---

💡 KEY TAKEAWAYS

1. Raft consensus: majority quorum (2/3) required cho writes
2. Backup mỗi 6h là minimum cho production — lưu cả snapshot + certificates
3. Restore = Tạo cluster mới từ snapshot, tất cả data sau snapshot bị mất
4. Defrag lần lượt từng member, KHÔNG đồng thời (blocks I/O)
5. NOSPACE alarm: compact → defrag → disarm
6. NVMe SSD là must-have cho etcd (p99 write latency < 10ms)

---

🎯 BÀI TẬP

Bài tập 1: Backup & Restore Lab

• Tạo Deployment nginx (3 replicas)
• Backup etcd snapshot
• Xóa Deployment nginx
• Restore từ snapshot, verify Deployment nginx trở lại

Bài tập 2: Monitoring

• Chạy etcd-check.sh, ghi nhận DB size, leader, health
• Compact và defrag, so sánh DB size trước/sau

Bài tập 3: Setup CronJob backup

• Deploy etcd-backup CronJob
• Verify Job chạy thành công
• Kiểm tra snapshot file trong /backup/etcd/

---

📚 BÀI TIẾP THEO

Trong Bài 11: Kiến trúc Distributed Storage với Rook-Ceph, chúng ta sẽ bắt đầu Phần 3 — cài đặt Rook-Ceph cho persistent storage.