Deploy Microservices On-Premises với Kubernetes HA

Phần 1: Nền tảng & Thiết kế Hạ tầng On-Premises

xdev.asia

🎯 MỤC TIÊU BÀI HỌC

Sau khi hoàn thành bài học này, bạn sẽ:

• ✅ Hiểu tại sao cần Load Balancer cho K8s API Server trong HA setup
• ✅ Cài đặt và cấu hình keepalived cho Virtual IP (VIP) failover
• ✅ Cài đặt và cấu hình HAProxy cho API server load balancing
• ✅ Cấu hình health checks và failover tự động
• ✅ Test HA bằng cách simulate failures
• ✅ So sánh keepalived+HAProxy vs kube-vip

---

PHẦN 1: TẠI SAO CẦN LOAD BALANCER CHO API SERVER?

1.1. Vấn đề với single API endpoint

graph LR
    subgraph PROBLEM["❌ Hardcode master1"]
        kubectl1["kubectl"] -->|":6443"| master1a["master1
kube-apiserver"]
        master1a -.->|"DOWN!"| X["❌ Toàn bộ cluster
mất quản lý"]
    end

    subgraph SOLUTION["✅ Virtual IP"]
        kubectl2["kubectl"] -->|":6443"| VIP["🔷 VIP
10.10.20.100"]
        VIP --> m1["master1"] & m2["master2"] & m3["master3"]
        m1 -.->|"DOWN"| VIP
    end

    style PROBLEM fill:#450a0a,stroke:#dc2626,color:#fca5a5
    style SOLUTION fill:#052e16,stroke:#22c55e,color:#bbf7d0
    style X fill:#dc2626,stroke:#fca5a5,color:#fff
    style VIP fill:#1d4ed8,stroke:#60a5fa,color:#fff

1.2. Kiến trúc Load Balancer

graph TD
    VIP["🔷 VIP: 10.10.20.100:6443
keepalived floating IP"]
    HAProxy["⚡ HAProxy
L4 TCP proxy · Listen :6443
Health check: /healthz"]

    VIP --> HAProxy

    HAProxy --> m1["master1:6443
kube-apiserver"] & m2["master2:6443
kube-apiserver"] & m3["master3:6443
kube-apiserver"]

    subgraph FAILOVER["🔄 Failover"]
        lb1["lb1
keepalived MASTER
holds VIP"]
        lb2["lb2
keepalived BACKUP"]
        lb1 -.->|"lb1 down → lb2 takes VIP
trong < 3 giây"| lb2
    end

    style VIP fill:#1d4ed8,stroke:#60a5fa,color:#fff
    style HAProxy fill:#7c3aed,stroke:#a78bfa,color:#fff
    style lb1 fill:#15803d,stroke:#22c55e,color:#fff
    style lb2 fill:#78716c,stroke:#a8a29e,color:#fff

1.3. Lựa chọn Architecture

Option	Ưu điểm	Nhược điểm	Recommendation
Dedicated LB nodes	Isolation, simple, clear separation	Cần thêm 2 servers	✅ Production
HAProxy trên masters	Không cần thêm servers	Resource contention, complexity	Lab/Small
kube-vip (DaemonSet)	Không cần external LB	Runs inside K8s → chicken-egg	Simple setups

Chúng ta sẽ dùng dedicated LB nodes (lb1 + lb2) cho production-grade setup.

---

PHẦN 2: CÀI ĐẶT VÀ CẤU HÌNH HAPROXY

2.1. Cài đặt HAProxy

# Trên CẢ HAI lb1 và lb2:
Cài đặt HAProxy (version 2.8+ LTS)
sudo apt install -y haproxy
Verify version
haproxy -v
Output: HAProxy version 2.8.x ...

2.2. Cấu hình HAProxy

# Backup config gốc
sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
Tạo cấu hình mới
cat > /etc/haproxy/haproxy.cfg << 'EOF'
#---------------------------------------------------------------------
Global settings
#---------------------------------------------------------------------
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# SSL/TLS settings
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
ssl-default-bind-options ssl-min-ver TLSv1.2

# Tuning
maxconn 50000
tune.ssl.default-dh-param 2048

#---------------------------------------------------------------------
Defaults
#---------------------------------------------------------------------
defaults
log     global
mode    tcp                    # Layer 4 (TCP) mode cho K8s API
option  tcplog
option  dontlognull
timeout connect 5000ms
timeout client  50000ms
timeout server  50000ms
retries 3
default-server inter 10s downinter 5s rise 2 fall 3 slowstart 60s
#---------------------------------------------------------------------
Stats page (cho monitoring)
#---------------------------------------------------------------------
listen stats
bind *:9000
mode http
stats enable
stats uri /stats
stats realm HAProxy\ Statistics
stats auth admin:SecureP@ssw0rd    # Đổi password trong production!
stats refresh 10s
#---------------------------------------------------------------------
Kubernetes API Server Frontend
#---------------------------------------------------------------------
frontend k8s-api
bind *:6443
mode tcp
option tcplog
default_backend k8s-api-backend
#---------------------------------------------------------------------
Kubernetes API Server Backend
#---------------------------------------------------------------------
backend k8s-api-backend
mode tcp
option tcp-check
balance roundrobin
# Health check: TCP connection check to port 6443
# (K8s API server responds to TCP SYN on this port)
server master1 10.10.20.11:6443 check
server master2 10.10.20.12:6443 check
server master3 10.10.20.13:6443 check

#---------------------------------------------------------------------
etcd (optional - cho external etcd client access)
#---------------------------------------------------------------------
frontend etcd
bind *:2379
mode tcp
default_backend etcd-backend

backend etcd-backend
mode tcp
balance roundrobin
server etcd1 10.10.20.11:2379 check
server etcd2 10.10.20.12:2379 check
server etcd3 10.10.20.13:2379 check
EOF

2.3. Advanced Health Check (HTTP)

TCP check chỉ kiểm tra port open. Để check API server thực sự healthy, dùng HTTP health check:

# Thay thế backend section bằng advanced version:
backend k8s-api-backend
    mode tcp
    option tcp-check
    balance roundrobin

    # HTTP health check qua /healthz endpoint
    option httpchk GET /healthz
    http-check expect status 200

    # default-server: check interval 3s, fall after 3 failures, rise after 2 success
    default-server inter 3s fall 3 rise 2

    server master1 10.10.20.11:6443 check check-ssl verify none
    server master2 10.10.20.12:6443 check check-ssl verify none
    server master3 10.10.20.13:6443 check check-ssl verify none

⚠️ Lưu ý: HTTP health check yêu cầu check-ssl verify none vì K8s API dùng self-signed certificates. Trong production có thể cấu hình CA cert cho verification.

2.4. Start HAProxy

# Validate config
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
# Output: Configuration file is valid
Enable và start
sudo systemctl enable haproxy
sudo systemctl start haproxy
Verify
sudo systemctl status haproxy
ss -tlnp | grep 6443
Output:
LISTEN  0  50000  *:6443  users:(("haproxy",pid=1234,...))
Kiểm tra stats page
curl http://localhost:9000/stats
Mở browser: http://lb1:9000/stats → login admin/SecureP@ssw0rd

---

PHẦN 3: CÀI ĐẶT VÀ CẤU HÌNH KEEPALIVED

3.1. keepalived là gì?

keepalived sử dụng VRRP (Virtual Router Redundancy Protocol) để quản lý một Virtual IP giữa 2+ servers:

sequenceDiagram
    participant lb1 as lb1 (MASTER, priority 101)
    participant VIP as VIP 10.10.20.100
    participant lb2 as lb2 (BACKUP, priority 100)

    Note over lb1,lb2: 1. Ban đầu — lb1 giữ VIP

    lb1->>VIP: Giữ VIP
    loop Mỗi 1 giây
        lb1->>lb2: VRRP Advertisement
        lb2-->>lb1: Xác nhận alive
    end

    Note over lb1: 2. lb1 crash!
    lb1--xlb2: ❌ Không gửi VRRP

    Note over lb2: 3+ giây không nhận VRRP
    lb2->>lb2: Promote → MASTER
    lb2->>VIP: Gửi Gratuitous ARP
    lb2->>VIP: Nhận VIP

    Note over lb1,lb2: Failover hoàn tất ~3 giây

    Note over lb1: 4. lb1 recover
    lb1->>lb2: Lên lại, thấy lb2 MASTER
    lb1->>lb1: Trở thành BACKUP (nopreempt)

3.2. Cài đặt keepalived

# Trên CẢ HAI lb1 và lb2:
sudo apt install -y keepalived
Enable non-local IP binding (cho VIP)
echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.d/99-keepalived.conf
sudo sysctl -p /etc/sysctl.d/99-keepalived.conf

3.3. Cấu hình keepalived trên lb1 (MASTER)

cat > /etc/keepalived/keepalived.conf << 'EOF'
! Configuration for keepalived
global_defs {
router_id LB1              # Unique identifier for this node
enable_script_security      # Required cho script execution
script_user root
}
Health check script cho HAProxy
vrrp_script check_haproxy {
script "/usr/bin/killall -0 haproxy"   # Check if haproxy process exists
interval 2                              # Check every 2 seconds
weight -30                              # Reduce priority by 30 if check fails
fall 3                                  # Mark DOWN after 3 consecutive failures
rise 2                                  # Mark UP after 2 consecutive successes
}
vrrp_instance K8S_API {
state MASTER                # Initial state: MASTER on lb1
interface eth1              # Network interface for VRRP (cluster network)
virtual_router_id 51        # Same ID on both LBs (0-255)
priority 101                # Higher priority = preferred MASTER
advert_int 1                # VRRP advertisement interval (seconds)
nopreempt                   # Không tự preempt khi recovered (recommended)
authentication {
    auth_type PASS
    auth_pass K8sHA2026     # Shared password (max 8 chars)
}

virtual_ipaddress {
    10.10.20.100/24 dev eth1 label eth1:vip
}

track_script {
    check_haproxy           # Track HAProxy health
}

# Notification scripts (optional)
notify_master "/etc/keepalived/notify.sh MASTER"
notify_backup "/etc/keepalived/notify.sh BACKUP"
notify_fault  "/etc/keepalived/notify.sh FAULT"

}
EOF

3.4. Cấu hình keepalived trên lb2 (BACKUP)

cat > /etc/keepalived/keepalived.conf << 'EOF'
global_defs {
    router_id LB2
    enable_script_security
    script_user root
}
vrrp_script check_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight -30
fall 3
rise 2
}
vrrp_instance K8S_API {
state BACKUP                # Initial state: BACKUP on lb2
interface eth1
virtual_router_id 51        # PHẢI GIỐNG lb1
priority 100                # Thấp hơn lb1 (101)
advert_int 1
nopreempt
authentication {
    auth_type PASS
    auth_pass K8sHA2026     # PHẢI GIỐNG lb1
}

virtual_ipaddress {
    10.10.20.100/24 dev eth1 label eth1:vip
}

track_script {
    check_haproxy
}

notify_master "/etc/keepalived/notify.sh MASTER"
notify_backup "/etc/keepalived/notify.sh BACKUP"
notify_fault  "/etc/keepalived/notify.sh FAULT"

}
EOF

3.5. Notification Script (Optional)

# Trên cả lb1 và lb2:
cat > /etc/keepalived/notify.sh << 'SCRIPT'
#!/bin/bash
STATE=$1
DATETIME=$(date '+%Y-%m-%d %H:%M:%S')
HOSTNAME=$(hostname)
echo "${DATETIME} - ${HOSTNAME} transitioned to ${STATE}" >> /var/log/keepalived-state.log
Optional: Send notification (Slack, email, etc.)
curl -X POST -H 'Content-type: application/json' \
--data "{"text":"keepalived: ${HOSTNAME} → ${STATE}"}" \
https://hooks.slack.com/services/YOUR/WEBHOOK/URL
SCRIPT
chmod +x /etc/keepalived/notify.sh

3.6. Start keepalived

# Trên lb1 (start MASTER trước):
sudo systemctl enable keepalived
sudo systemctl start keepalived
Trên lb2 (start BACKUP sau):
sudo systemctl enable keepalived
sudo systemctl start keepalived
Verify trên lb1 (MASTER):
ip addr show eth1
Output:
inet 10.10.20.9/24 brd 10.10.20.255 scope global eth1
inet 10.10.20.100/24 scope global secondary eth1:vip  ← VIP!
Verify trên lb2 (BACKUP):
ip addr show eth1
Output:
inet 10.10.20.10/24 brd 10.10.20.255 scope global eth1
(Không có VIP)
Check keepalived state
sudo journalctl -u keepalived -f

---

PHẦN 4: TESTING HA FAILOVER

4.1. Test 1: VIP Failover khi lb1 down

# Terminal 1: Continuous ping tới VIP từ workstation
ping 10.10.20.100
Terminal 2: Tắt keepalived trên lb1 (MASTER)
ssh lb1 "sudo systemctl stop keepalived"
Kết quả expected trên Terminal 1:
64 bytes from 10.10.20.100: icmp_seq=45 ttl=64 time=0.4ms
64 bytes from 10.10.20.100: icmp_seq=46 ttl=64 time=0.4ms
Request timeout for icmp_seq 47      ← 1-3 packets lost
Request timeout for icmp_seq 48
64 bytes from 10.10.20.100: icmp_seq=49 ttl=64 time=0.5ms  ← VIP on lb2 now
64 bytes from 10.10.20.100: icmp_seq=50 ttl=64 time=0.5ms
Verify VIP chuyển sang lb2:
ssh lb2 "ip addr show eth1 | grep 'inet '"
Output: inet 10.10.20.100/24 scope global secondary eth1:vip  ✅
Restore lb1:
ssh lb1 "sudo systemctl start keepalived"
VIP stays on lb2 (nopreempt mode)

4.2. Test 2: HAProxy failure → keepalived demotion

# Stop HAProxy trên lb1 (hiện đang MASTER):
ssh lb1 "sudo systemctl stop haproxy"
keepalived health check detects HAProxy down
→ Priority giảm từ 101 → 71 (101 - 30)
→ lb2 priority 100 > 71 → lb2 becomes MASTER
Check log:
ssh lb1 "sudo journalctl -u keepalived --since '1 min ago'"
Output:
VRRP_Script(check_haproxy) failed (exited with status 1)
VRRP_Instance(K8S_API) Changing effective priority from 101 to 71
VRRP_Instance(K8S_API) Received advert with higher priority 100
VRRP_Instance(K8S_API) Entering BACKUP STATE
Restore HAProxy:
ssh lb1 "sudo systemctl start haproxy"

4.3. Test 3: API Server Backend Failover

# Sau khi K8s cluster đã chạy (Bài 6-7):
Continuous API call qua VIP
while true; do
curl -sk https://10.10.20.100:6443/healthz && echo " OK $(date)"
sleep 1
done
Tắt kube-apiserver trên master1:
ssh master1 "sudo crictl stop $(sudo crictl ps --name kube-apiserver -q)"
Output expected:
ok OK Wed Apr 02 10:00:01 2026
ok OK Wed Apr 02 10:00:02 2026   ← HAProxy routes to master2/3
ok OK Wed Apr 02 10:00:03 2026   ← No interruption!
Kiểm tra HAProxy stats:
curl -s "http://lb1:9000/stats;csv" | grep k8s-api-backend
master1 → DOWN, master2/3 → UP

---

PHẦN 5: ALTERNATIVE — kube-vip

5.1. kube-vip là gì?

kube-vip chạy như static pod trên control plane nodes, kết hợp VIP + load balancing trong 1 component:

keepalived + HAProxy (external):
  ├── 2 dedicated LB servers
  ├── keepalived manages VIP
  └── HAProxy load balances → API servers

kube-vip (internal):
  ├── Chạy như DaemonSet/static pod trên masters
  ├── Leader election qua Raft hoặc ARP
  ├── Leader giữ VIP + local LB
  └── Không cần external servers

5.2. So sánh chi tiết

Tiêu chí	keepalived + HAProxy	kube-vip
Extra servers	Cần 2 LB servers	Không cần
Complexity	2 components cần quản lý	1 component
Independence	✅ Độc lập với K8s cluster	❌ Chạy trong K8s (chicken-egg)
Chicken-egg problem	✅ Không có	⚠️ cần init trước K8s
Health checks	✅ Advanced (HTTP, TCP, script)	Basic
Monitoring	✅ HAProxy stats, Prometheus	Limited
Production proven	✅ 20+ years	Newer, less battle-tested
Lab/Development	Overkill	✅ Perfect

💡 Recommendation:

• Production: keepalived + HAProxy (mature, independent, observable)
• Lab/Small: kube-vip (simpler, no extra servers)

5.3. kube-vip Quick Setup (Tham khảo)

# Tạo kube-vip manifest trước khi kubeadm init
export VIP=10.10.20.100
export INTERFACE=eth1
export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
Generate static pod manifest
ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION
ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip 

manifest pod 

--interface $INTERFACE 

--address $VIP 

--controlplane 

--arp 

--leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml

---

PHẦN 6: PRODUCTION CONSIDERATIONS

6.1. HAProxy Production Tuning

# Thêm vào haproxy.cfg global section:
global
    # Tăng max connections
    maxconn 100000
# Sử dụng nhiều CPU cores
nbthread 4                    # Số threads = số CPU cores

# Enable stats socket cho runtime API
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

# Logging chi tiết
log-send-hostname

6.2. Monitoring HAProxy với Prometheus

# HAProxy built-in Prometheus exporter (HAProxy 2.4+):
# Thêm vào haproxy.cfg:
frontend prometheus
    bind *:8405
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
    no log
Verify:
curl http://lb1:8405/metrics | head -20
Output:
# HELP haproxy_backend_status Current status of the service
haproxy_backend_status{backend="k8s-api-backend",server="master1"} 1

6.3. Monitoring keepalived

# keepalived exports SNMP metrics
# Hoặc check via logs:
sudo journalctl -u keepalived -f
Custom health check cho keepalived process:
systemctl is-active keepalived && echo "RUNNING" || echo "DOWN"
Check VRRP state:
cat /var/log/keepalived-state.log

---

💡 KEY TAKEAWAYS

1. VIP là critical cho K8s HA — tất cả components kết nối qua VIP, không hardcode master IP
2. keepalived quản lý VIP failover qua VRRP protocol, failover < 3 giây
3. HAProxy load balance TCP traffic tới healthy API servers với health checks
4. nopreempt mode tránh unnecessary VIP flapping khi MASTER recovered
5. Health check script trong keepalived đảm bảo VIP chỉ ở node có HAProxy healthy
6. Test failover TRƯỚC khi deploy K8s: tắt LB, tắt HAProxy, verify VIP migration

---

🎯 BÀI TẬP

Bài tập 1: Deploy HAProxy + keepalived

• Cài đặt HAProxy + keepalived trên lb1 và lb2 theo hướng dẫn
• Verify VIP active trên lb1
• Truy cập HAProxy stats page

Bài tập 2: Failover Testing

• Test 1: Stop keepalived trên lb1, verify VIP chuyển sang lb2
• Test 2: Stop HAProxy trên lb1, verify tự động demotion
• Test 3: Start lại cả hai, verify correct state
• Đo thời gian failover bằng continuous ping

Bài tập 3: Nâng cao

• Thêm Prometheus exporter cho HAProxy
• Viết notification script gửi alert tới Slack khi failover xảy ra
• Cấu hình HAProxy logging chi tiết vào file riêng

---

📚 BÀI TIẾP THEO

Trong Bài 5: Cài đặt containerd và kubeadm trên tất cả nodes, chúng ta sẽ cài đặt container runtime (containerd) và kubeadm tools, chuẩn bị sẵn sàng cho việc khởi tạo K8s HA cluster.