Chuyển đến nội dung chính
BlogCoursesHL7Cert PrepRoadmapGameDomainAboutMCPNEW
Topics
DatabaseDevOpsKiến trúc hệ thốngLinuxProgrammingSecurity

LESSON 6: INITIALIZING THE FIRST KUBERNETES HA CONTROL PLANE

Create kubeadm-config.yaml for HA topology, run kubeadm init on master1 with control-plane-endpoint as VIP, process certificates, copy kubeconfig and check first cluster status.

🔒 DevSecOps — Lesson 6 LESSON 6: INITIALIZING KUBERNETES HA CONTROL FIRST PLANE

Deploy Microservices On-Premises with Kubernetes HA

Part 2: Kubernetes HA Cluster with kubeadm__HTMLTAG_62___

xdev.asia

🎯 LESSON OBJECTIVE__HTMLTAG_68___

After completing this lesson, you will:

  • ✅ Create kubeadm configuration file for HA topology__HTMLTAG_73___
  • ✅ Successfully initialized the first control plane
  • ✅ Understand certificate structure and certificate rotation__HTMLTAG_77___
  • ✅ Copy kubeconfig and access cluster using kubectl
  • ✅ Understanding stacked etcd topology vs external etcd

PART 1: KUBERNETES HA TOPOLOGIES

1.1. Stacked etcd vs External etcd

Option A: Stacked etcd (Recommended for most cases)

graph LR
    subgraph M1["🖥️ master1"]
        A1["API Server<br/>Scheduler<br/>Controller"]
        E1["etcd"]
        A1 --- E1
    end

    subgraph M2["🖥️ master2"]
        A2["API Server<br/>Scheduler<br/>Controller"]
        E2["etcd"]
        A2 --- E2
    end

    subgraph M3["🖥️ master3"]
        A3["API Server<br/>Scheduler<br/>Controller"]
        E3["etcd"]
        A3 --- E3
    end

    E1 <-->|"Raft"| E2
    E2 <-->|"Raft"| E3

    style M1 fill:#0f172a,stroke:#3b82f6,color:#e2e8f0
    style M2 fill:#0f172a,stroke:#3b82f6,color:#e2e8f0
    style M3 fill:#0f172a,stroke:#3b82f6,color:#e2e8f0
    style E1 fill:#15803d,stroke:#22c55e,color:#e2e8f0
    style E2 fill:#15803d,stroke:#22c55e,color:#e2e8f0
    style E3 fill:#15803d,stroke:#22c55e,color:#e2e8f0

✅ Few servers · ✅ Simple · ❌ etcd + API coupled

Option B: External etcd

graph TD
    subgraph MASTERS["Control Plane — 3 masters"]
        MA["API Server<br/>Scheduler<br/>Controller"]
        MB["API Server<br/>Scheduler<br/>Controller"]
        MC["API Server<br/>Scheduler<br/>Controller"]
    end

    subgraph ETCDS["etcd Cluster — 3 dedicated nodes"]
        EA["etcd"]
        EB["etcd"]
        EC["etcd"]
        EA <-->|"Raft"| EB
        EB <-->|"Raft"| EC
    end

    MA --> EA
    MB --> EB
    MC --> EC

    style MASTERS fill:#0f172a,stroke:#3b82f6,color:#e2e8f0
    style ETCDS fill:#1e293b,stroke:#15803d,color:#e2e8f0
    style EA fill:#15803d,stroke:#22c55e,color:#e2e8f0
    style EB fill:#15803d,stroke:#22c55e,color:#e2e8f0
    style EC fill:#15803d,stroke:#22c55e,color:#e2e8f0

✅ Isolation · ✅ Independent scaling · ❌ Needs 6 servers

👉 Choose Stacked etcd for this course — simple, good enough for most production workloads. External etcd is only needed for very large clusters (100+ nodes).

PART 2: CREATE KUBEADM CONFIGURATION

2.1. kubeadm-config.yaml details

# Trên master1, tạo config file:
cat > /root/kubeadm-config.yaml << 'EOF'
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.10.20.11"     # IP của master1 trên cluster network
  bindPort: 6443
nodeRegistration:
  name: master1
  criSocket: unix:///run/containerd/containerd.sock
  taints:
    - key: "node-role.kubernetes.io/control-plane"
      effect: "NoSchedule"



apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: "v1.31.0"          # Exact version
clusterName: "production"
controlPlaneEndpoint: "10.10.20.100:6443"   # ← VIP (HAProxy)!
certificatesDir: /etc/kubernetes/pki


networking:
podSubnet: "10.244.0.0/16"         # Pod CIDR (cho Cilium)
serviceSubnet: "10.96.0.0/12"      # Service CIDR
dnsDomain: "cluster.local"


etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
listen-metrics-urls: "http://0.0.0.0:2381"    # Prometheus metrics


apiServer:
extraArgs:
authorization-mode: "Node,RBAC"
enable-admission-plugins: "NodeRestriction,PodSecurity"
audit-log-path: "/var/log/kubernetes/audit.log"
audit-log-maxage: "30"
audit-log-maxbackup: "10"
audit-log-maxsize: "100"
event-ttl: "4h"
# Encryption at rest (thêm sau khi tạo encryption config)
# encryption-provider-config: "/etc/kubernetes/encryption-config.yaml"
extraVolumes:
- name: audit-log
hostPath: /var/log/kubernetes
mountPath: /var/log/kubernetes
pathType: DirectoryOrCreate


controllerManager:
extraArgs:
bind-address: "0.0.0.0"          # Cho Prometheus scrape
terminated-pod-gc-threshold: "100"


scheduler:
extraArgs:
bind-address: "0.0.0.0"          # Cho Prometheus scrape




apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd                 # Match containerd SystemdCgroup
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
evictionHard:
memory.available: "500Mi"
nodefs.available: "10%"
imagefs.available: "15%"
systemReserved:
cpu: "500m"
memory: "1Gi"
kubeReserved:
cpu: "500m"
memory: "1Gi"
maxPods: 110                          # Default 110, tăng nếu cần
serializeImagePulls: false            # Parallel image pulls




apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"                          # IPVS mode (better than iptables)
ipvs:
strictARP: true                     # Required cho MetalLB
scheduler: "rr"                     # Round-robin
EOF

2.2. Important parameters explained

Parameters Value Meaning
controlPlaneEndpoint 10.10.20.100:6443 HAProxy VIP — ALL components connect through here
podSubnet 10.244.0.0/16 CIDR for pod IPs (65,534 pods max)
serviceSubnet 10.96.0.0/12 CIDR for ClusterIP services (1,048,574 IPs)
mode: ipvs kube-proxy IPVS load balancing (better than iptables when many services)
strictARP: true kube-proxy Required for MetalLB L2 mode
cgroupDriver: systemd kubelet Match containerd SystemdCgroup = true

⚠️ CRITICAL: controlPlaneEndpoint MUST point to VIP (HAProxy), NOT master1's IP. This is a key element for HA.

PART 3: CLUSTER INITIALIZATION

3.1. Create audit log directory

# Trên master1:
mkdir -p /var/log/kubernetes

3.2. Run kubeadm init

# Trên master1:
kubeadm init --config /root/kubeadm-config.yaml --upload-certs

Output (giữ lại CẨN THẬN):


────────────────────────────────────────────────────────────


Your Kubernetes control-plane has initialized successfully!




To start using your cluster, you need to run the following as a regular user:


mkdir -p $HOME/.kube


sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config


sudo chown $(id -u):$(id -g) $HOME/.kube/config




You can now join any number of control-plane node by running:


kubeadm join 10.10.20.100:6443 --token abcdef.0123456789abcdef \


--discovery-token-ca-cert-hash sha256:... \


--control-plane --certificate-key <CERTIFICATE_KEY>




Then you can join any number of worker nodes by running:


kubeadm join 10.10.20.100:6443 --token abcdef.0123456789abcdef \


--discovery-token-ca-cert-hash sha256:...


────────────────────────────────────────────────────────────

⚠️ RECALL immediately:

  • --token: Used for join nodes (expires after 24 hours)
  • --discovery-token-ca-cert-hash: SHA256 hash of CA cert
  • --certificate-key: Used to join control-plane nodes (expires after 2 hours)

3.3. Setup kubeconfig

# Trên master1:
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Verify cluster


kubectl get nodes


Output:


NAME      STATUS     ROLES           AGE   VERSION


master1   NotReady   control-plane   30s   v1.31.0


← NotReady vì chưa cài CNI (Cilium sẽ fix ở Bài 8)


Kiểm tra tất cả system pods


kubectl get pods -n kube-system


Output:


NAME                              READY   STATUS    RESTARTS   AGE


coredns-xxx-xxx                   0/1     Pending   0          30s  ← Pending vì chưa có CNI


coredns-xxx-xxx                   0/1     Pending   0          30s


etcd-master1                      1/1     Running   0          35s


kube-apiserver-master1            1/1     Running   0          35s


kube-controller-manager-master1   1/1     Running   0          35s


kube-scheduler-master1            1/1     Running   0          35s


Kiểm tra etcd health


kubectl -n kube-system exec etcd-master1 -- etcdctl 

--endpoints=https://127.0.0.1:2379 

--cacert=/etc/kubernetes/pki/etcd/ca.crt 

--cert=/etc/kubernetes/pki/etcd/server.crt 

--key=/etc/kubernetes/pki/etcd/server.key 

endpoint health


Output: https://127.0.0.1:2379 is healthy: successfully committed proposal

PART 4: CERTIFICATE STRUCTURE

4.1. Architecture Certificates


/etc/kubernetes/pki/
├── ca.crt                  ─── Kubernetes CA (root cert)
├── ca.key                  ─── CA private key (PROTECT!)
├── apiserver.crt           ─── API Server cert
├── apiserver.key
├── apiserver-kubelet-client.crt
├── apiserver-kubelet-client.key
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
├── front-proxy-ca.crt      ─── Front Proxy CA
├── front-proxy-ca.key
├── front-proxy-client.crt
├── front-proxy-client.key
├── sa.pub                  ─── Service Account public key
├── sa.key                  ─── Service Account private key
└── etcd/
    ├── ca.crt              ─── etcd CA
    ├── ca.key
    ├── server.crt          ─── etcd server cert
    ├── server.key
    ├── peer.crt            ─── etcd peer cert
    ├── peer.key
    ├── healthcheck-client.crt
    └── healthcheck-client.key

4.2. Check Certificate Expiry

# Xem tất cả cert expiry:
kubeadm certs check-expiration
# Output:
# CERTIFICATE                EXPIRES                  RESIDUAL TIME
# admin.conf                 Apr 02, 2027 07:00 UTC   364d
# apiserver                  Apr 02, 2027 07:00 UTC   364d
# apiserver-etcd-client      Apr 02, 2027 07:00 UTC   364d
# ...
# ca                         Mar 30, 2036 07:00 UTC   9y   ← CA valid 10 years
# etcd-ca                    Mar 30, 2036 07:00 UTC   9y

⚠️ Certificates mặc định valid 1 NĂM (trừ CA: 10 năm)


Phải renew trước khi expire!


Renew tất cả certs:


kubeadm certs renew all


(Sẽ học chi tiết ở Bài 46: Nâng cấp Cluster)

PART 5: VERIFY HA READINESS

5.1. Check API Server via VIP

# Verify API server accessible qua VIP (HAProxy):
curl -sk https://10.10.20.100:6443/healthz
# Output: ok

Kiểm tra HAProxy backend status:


curl -s http://lb1:9000/stats\;csv | grep k8s-api


master1 → UP, master2 → DOWN, master3 → DOWN (chưa join)


API server accessible trực tiếp:


curl -sk https://10.10.20.11:6443/healthz


Output: ok

5.2. Save join commands (important!)

# Lưu join commands vào file:
cat > /root/join-commands.sh << 'CMDS'
# === Join Control Plane nodes (master2, master3) ===
# Certificate key expires in 2 HOURS!
kubeadm join 10.10.20.100:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane \
  --certificate-key <CERT_KEY>

=== Join Worker nodes ===


kubeadm join 10.10.20.100:6443 

--token <TOKEN> 

--discovery-token-ca-cert-hash sha256:<HASH>
CMDS


Nếu token hết hạn, tạo mới:


kubeadm token create --print-join-command


Output: kubeadm join 10.10.20.100:6443 --token NEW_TOKEN --discovery-token-ca-cert-hash sha256:HASH


Nếu certificate-key hết hạn, upload certs lại:


kubeadm init phase upload-certs --upload-certs


Output: Using certificate key: NEW_CERTIFICATE_KEY

💡 KEY TAKEAWAYS

  1. controlPlaneEndpoint must point to VIP (HAProxy), not specific IP
  2. Stacked etcdgood enough for most deployments, etcd running on the control plane
  3. --upload-certs flag automatically distribute certificates for join control-plane
  4. Token expires in 24 hours, certificate-key expires in 2 hours — must save and join nodes soon
  5. ipvs mode + strictARP for kube-proxy is a requirement for MetalLB
  6. Node status NotReady is normal before installing CNI (Cilium)

🎯 EXERCISE

Exercise 1: Init control plane__HTMLTAG_243___
  • Create kubeadm-config.yaml on master1
  • Run kubeadm init --config --upload-certs
  • Setup kubeconfig, verify kubectl get nodes
  • RECALL join commands

Exercise 2: Verify certificates

  • List all certificates in /etc/kubernetes/pki/
  • Run kubeadm certs check-expiration
  • Verify API server via VIP: curl -sk https://VIP:6443/healthz

📚 NEXT POST

In Lesson 7: Join Control Plane and Worker Nodes, we will join master2, master3 to the HA control plane and join worker nodes to the cluster.