Deploy Microservices On-Premises với Kubernetes HA

Phần 2: Kubernetes HA Cluster với kubeadm

xdev.asia

🎯 MỤC TIÊU BÀI HỌC

Sau khi hoàn thành bài học này, bạn sẽ:

• ✅ Hiểu eBPF và tại sao Cilium vượt trội so với CNI truyền thống
• ✅ Cài đặt Cilium với Helm và verify connectivity
• ✅ Thay thế kube-proxy bằng Cilium eBPF
• ✅ Enable Hubble cho network observability
• ✅ Viết NetworkPolicy L3/L4/L7

---

PHẦN 1: TẠI SAO CHỌN CILIUM?

1.1. So sánh các CNI phổ biến

Tiêu chí	Calico	Flannel	Cilium
Dataplane	iptables/eBPF	VXLAN	eBPF native
NetworkPolicy	L3/L4	❌ Không hỗ trợ	L3/L4/L7
Performance	Tốt	Trung bình	Xuất sắc
Observability	Basic	❌	Hubble (deep)
kube-proxy replacement	❌	❌	✅ Full
Encryption (WireGuard)	✅	❌	✅ Native
Service Mesh integration	❌	❌	✅ Istio/Envoy

1.2. eBPF là gì?

Mô hình truyền thống (iptables):
┌──────────┐    ┌──────────────────────────┐    ┌──────────┐
│  Pod A   │───►│  iptables (userspace)     │───►│  Pod B   │
│          │    │  Chain rules: 100+ rules  │    │          │
│          │    │  Linear scan O(n)         │    │          │
└──────────┘    └──────────────────────────┘    └──────────┘
Mô hình eBPF (Cilium):
┌──────────┐    ┌──────────────────────────┐    ┌──────────┐
│  Pod A   │───►│  eBPF program (kernel)    │───►│  Pod B   │
│          │    │  Hash-based lookup O(1)   │    │          │
│          │    │  XDP fast path            │    │          │
└──────────┘    └──────────────────────────┘    └──────────┘
✅ eBPF chạy trực tiếp trong kernel, không qua iptables
✅ Latency thấp hơn ~20-30%
✅ Scale tốt hơn khi có nhiều services (10,000+)

---

PHẦN 2: CÀI ĐẶT CILIUM

2.1. Cài Helm

# Trên master1 (hoặc bastion host):
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Verify:
helm version
version.BuildInfo{Version:"v3.16.x", ...}

2.2. Xóa kube-proxy (optional — thay thế bằng Cilium)

# Option 1: Xóa kube-proxy hoàn toàn (recommended):
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy
Xóa iptables rules của kube-proxy trên MỖI node:
(SSH vào từng node hoặc dùng script)
iptables-save | grep -v KUBE | iptables-restore
Option 2: Giữ kube-proxy (Cilium sẽ chạy alongside)
→ Không cần xóa, nhưng sẽ có overhead

2.3. Cài Cilium bằng Helm

# Add Cilium Helm repo:
helm repo add cilium https://helm.cilium.io/
helm repo update
Install Cilium:
helm install cilium cilium/cilium --version 1.16.5 

--namespace kube-system 

--set kubeProxyReplacement=true 

--set k8sServiceHost=10.10.20.100 

--set k8sServicePort=6443 

--set ipam.mode=kubernetes 

--set hubble.relay.enabled=true 

--set hubble.ui.enabled=true 

--set hubble.metrics.enableOpenMetrics=true 

--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction}" 

--set operator.replicas=2 

--set bpf.masquerade=true 

--set bgpControlPlane.enabled=false 

--set routingMode=tunnel 

--set tunnelProtocol=vxlan 

--set loadBalancer.algorithm=maglev 

--set bandwidthManager.enabled=true 

--set bandwidthManager.bbr=true

2.4. Giải thích tham số quan trọng

Tham số	Giá trị	Ý nghĩa
kubeProxyReplacement	true	Cilium thay thế hoàn toàn kube-proxy
k8sServiceHost	10.10.20.100	VIP (HAProxy) để Cilium kết nối API server
ipam.mode	kubernetes	Dùng K8s IPAM (phù hợp cho on-prem)
hubble.relay.enabled	true	Enable Hubble Relay cho flow aggregation
hubble.ui.enabled	true	Enable Hubble UI (web dashboard)
operator.replicas	2	HA cho Cilium Operator
bpf.masquerade	true	eBPF masquerading thay thế iptables SNAT
loadBalancer.algorithm	maglev	Consistent hashing cho ít disruption khi scale
bandwidthManager	enabled + bbr	EDT-based rate limiting + BBR congestion control

2.5. Verify Cilium Installation

# Đợi tất cả pods ready:
kubectl -n kube-system get pods -l app.kubernetes.io/part-of=cilium -w
# NAME                               READY   STATUS    RESTARTS   AGE
# cilium-xxxxx                       1/1     Running   0          2m   (DaemonSet, 1 per node)
# cilium-xxxxx                       1/1     Running   0          2m
# cilium-xxxxx                       1/1     Running   0          2m
# cilium-xxxxx                       1/1     Running   0          2m
# cilium-xxxxx                       1/1     Running   0          2m
# cilium-xxxxx                       1/1     Running   0          2m
# cilium-operator-xxxxx-xxxxx        1/1     Running   0          2m
# cilium-operator-xxxxx-xxxxx        1/1     Running   0          2m
# hubble-relay-xxxxx-xxxxx           1/1     Running   0          2m
# hubble-ui-xxxxx-xxxxx              2/2     Running   0          2m
Verify nodes chuyển Ready:
kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
master1   Ready    control-plane   1h    v1.31.0
master2   Ready    control-plane   50m   v1.31.0
master3   Ready    control-plane   48m   v1.31.0
worker1   Ready    worker          30m   v1.31.0
worker2   Ready    worker          29m   v1.31.0
worker3   Ready    worker          28m   v1.31.0
🎉 Tất cả nodes Ready!
Verify CoreDNS running:
kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-xxx-xxx            1/1     Running   0          1h
coredns-xxx-xxx            1/1     Running   0          1h

---

PHẦN 3: CILIUM CLI — CONNECTIVITY TEST

3.1. Cài Cilium CLI

# Download Cilium CLI:
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz
tar xzvf cilium-linux-${CLI_ARCH}.tar.gz -C /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz
Verify:
cilium version

3.2. Cilium Status

cilium status
#     /¯¯\
#  /¯¯\__/¯¯\    Cilium:             OK
#  \__/¯¯\__/    Operator:           OK
#  /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
#  \__/¯¯\__/    Hubble Relay:       OK
#     \__/       ClusterMesh:        disabled
#
# Deployment             cilium-operator    Desired: 2, Ready: 2/2
# DaemonSet              cilium             Desired: 6, Ready: 6/6
# Deployment             hubble-relay       Desired: 1, Ready: 1/1
# Deployment             hubble-ui          Desired: 1, Ready: 1/1
# Containers:            cilium             Running: 6
#                        cilium-operator    Running: 2
#                        hubble-relay       Running: 1
#                        hubble-ui          Running: 1

3.3. Connectivity Test

# Chạy full connectivity test (mất ~5 phút):
cilium connectivity test
Output:
✅ All 46 tests (306 actions) successful, 0 tests skipped, 0 scenarios skipped.

---

PHẦN 4: HUBBLE OBSERVABILITY

4.1. Cài Hubble CLI

# Download Hubble CLI:
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz
tar xzvf hubble-linux-amd64.tar.gz -C /usr/local/bin
rm hubble-linux-amd64.tar.gz

4.2. Xem Network Flows

# Port-forward Hubble Relay:
kubectl -n kube-system port-forward svc/hubble-relay 4245:80 
Observe flows real-time:
hubble observe --follow
Apr  2 07:00:30.123: 10.244.1.5:34567 (ID:12345) -> 10.96.0.1:443 (kube-system/kube-apiserver)
to-stack FORWARDED (TCP Flags: SYN)
Apr  2 07:00:30.124: 10.96.0.1:443 -> 10.244.1.5:34567
to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Filter by namespace:
hubble observe --namespace default --follow
Filter drops:
hubble observe --verdict DROPPED --follow
Xem DNS queries:
hubble observe --protocol dns --follow

4.3. Hubble UI

# Port-forward Hubble UI:
kubectl -n kube-system port-forward svc/hubble-ui 12000:80
# Mở browser: http://localhost:12000
# → Service Map hiển thị tất cả connections giữa pods

---

PHẦN 5: NETWORK POLICY VỚI CILIUM

5.1. Deploy test applications

# Tạo namespace test:
kubectl create namespace policy-demo
Deploy frontend:
kubectl -n policy-demo run frontend --image=nginx:alpine --labels="app=frontend"
kubectl -n policy-demo expose pod frontend --port=80
Deploy backend:
kubectl -n policy-demo run backend --image=nginx:alpine --labels="app=backend"
kubectl -n policy-demo expose pod backend --port=80
Test: frontend → backend (nên hoạt động):
kubectl -n policy-demo exec frontend -- curl -s backend
...  ← OK
Test: backend → frontend (nên hoạt động):
kubectl -n policy-demo exec backend -- curl -s frontend
...  ← OK

5.2. Default Deny All Policy

# deny-all.yaml — khóa toàn bộ traffic trong namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: policy-demo
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

kubectl apply -f deny-all.yaml

# Test lại: frontend → backend (BLOCKED):
kubectl -n policy-demo exec frontend -- curl -s --max-time 3 backend
# curl: (28) Connection timed out  ← BLOCKED! ✅

5.3. Allow frontend → backend

# allow-frontend-to-backend.yaml:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: policy-demo
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 80

kubectl apply -f allow-frontend-to-backend.yaml

# Cần allow DNS egress cho frontend:
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: policy-demo
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - podSelector: {}
      ports:
        - protocol: TCP
          port: 80
EOF

# Test: frontend → backend (ALLOWED):
kubectl -n policy-demo exec frontend -- curl -s --max-time 3 backend
# ...  ← OK! ✅

5.4. Cilium L7 Policy (HTTP-aware)

# cilium-l7-policy.yaml — chỉ cho phép GET /api/*:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: l7-rule-backend
  namespace: policy-demo
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "80"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/.*"

kubectl apply -f cilium-l7-policy.yaml

# GET /api/health → ALLOWED:
kubectl -n policy-demo exec frontend -- curl -s backend/api/health
# (response hoặc 404 nhưng connection OK)

# POST /api/data → BLOCKED:
kubectl -n policy-demo exec frontend -- curl -s -X POST backend/api/data
# Access denied  ← BLOCKED bởi L7 policy! ✅

---

PHẦN 6: TROUBLESHOOTING

6.1. Common Issues

# Issue 1: Cilium pod CrashLoopBackOff
kubectl -n kube-system logs -l k8s-app=cilium --tail=50
# Kiểm tra: kernel modules (bpf), containerd socket, kube-proxy conflict
Issue 2: Nodes vẫn NotReady
cilium status
kubectl -n kube-system describe pod cilium-xxxxx
Kiểm tra: cilium-agent có connect được API server qua VIP
Issue 3: Pod-to-pod connectivity failed
cilium connectivity test --test pod-to-pod
Debug eBPF:
cilium bpf policy get --all
Issue 4: DNS resolution failed
kubectl -n kube-system rollout restart deployment coredns
hubble observe --protocol dns

6.2. Cleanup test resources

kubectl delete namespace policy-demo

---

💡 KEY TAKEAWAYS

1. Cilium eBPF vượt trội: O(1) lookup, thay thế kube-proxy, L7 policy
2. kubeProxyReplacement=true: Cilium handle hết services, không cần kube-proxy
3. Hubble cho deep observability — xem flow, DNS queries, HTTP requests
4. NetworkPolicy: Default deny + whitelist là best practice
5. CiliumNetworkPolicy extend K8s NetworkPolicy thêm L7 rules (HTTP, gRPC, Kafka)
6. cilium connectivity test: Chạy 46 tests tự động verify networking

---

🎯 BÀI TẬP

Bài tập 1: Cài đặt Cilium

• Cài Helm, install Cilium với kubeProxyReplacement
• Verify tất cả nodes Ready
• Chạy cilium status và cilium connectivity test

Bài tập 2: NetworkPolicy Lab

• Tạo namespace và deploy 3 pods (frontend, backend, database)
• Apply default-deny-all
• Tạo policies: frontend→backend (HTTP GET), backend→database (TCP 5432)
• Verify chỉ traffic được cho phép mới hoạt động

---

📚 BÀI TIẾP THEO

Trong Bài 9: MetalLB — LoadBalancer cho On-Premises, chúng ta sẽ cài MetalLB để expose services ra ngoài cluster với type LoadBalancer.