Deploy Microservices On-Premises with Kubernetes HA

Part 2: Kubernetes HA Cluster with kubeadm__HTMLTAG_62___

xdev.asia

🎯 LESSON OBJECTIVE__HTMLTAG_68___

After completing this lesson, you will:

• ✅ Install containerd container runtime with CRI plugin
• ✅ Configure containerd using systemd cgroup driver__HTMLTAG_75___
• ✅ Install kubeadm, kubelet, kubectl from official repository
• ✅ Install crictl for debugging containers__HTMLTAG_79___
• ✅ Pre-pull sandbox images and verify everything is ready

---

PART 1: CONTAINER RUNTIME — WHY CONTAINERD?

1.1. History of Container Runtime in K8s

Timeline:
  2014-2020: Docker → dockershim → containerd (K8s dùng Docker runtime)
  2020:      K8s 1.20 deprecate dockershim
  2022:      K8s 1.24 remove dockershim hoàn toàn
  2024+:     containerd hoặc CRI-O là standard
Container Runtime Interface (CRI):
kubelet ─── CRI ──→ containerd ──→ runc ──→ Container
│
└──→ CRI-O ──→ runc ──→ Container

1.2. containerd vs CRI-O

Criteria	containerd	CRI-O
Maintainer	CNCF (Docker/Moby origin)	CNCF (Red Hat origin)
Adoption	Very wide (default EKS, GKE, AKS)	OpenShift, RHEL focused
Features	Multi-purpose (docker CLI compatible)	K8s-only (lightweight)
Image build	Supported via nerdctl/buildkit	No (needs podman/buildah)
Stability	Very stable	Stable

👉 Select containerd: Most popular, lots of documentation, compatible with all K8s distribution.

---

PART 2: CONTAINERD SETUP

2.1. Install from Docker Official Repository

# ============================================
# Chạy trên TẤT CẢ nodes (masters + workers + storage)
# ============================================
1. Thêm Docker GPG key và repository
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | 

gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo 

"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] 

https://download.docker.com/linux/ubuntu 

$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | 

tee /etc/apt/sources.list.d/docker.list > /dev/null
2. Cài đặt containerd
apt update
apt install -y containerd.io
3. Verify
containerd --version
Output: containerd containerd.io v2.0.x ...

2.2. Configure containerd

# Tạo default config
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
Cấu hình quan trọng:
1. Enable SystemdCgroup (BẮT BUỘC cho K8s)
2. Set sandbox image đúng version
Chỉnh sửa config.toml:
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
Verify sandbox image version (phải khớp với K8s version):
grep sandbox_image /etc/containerd/config.toml
Output: sandbox_image = "registry.k8s.io/pause:3.10"

🔬 Deep Dive — Why is SystemdCgroup = true?

cgroup driver quản lý resource limits cho containers.
Có 2 drivers: cgroupfs và systemd.
kubelet sử dụng systemd cgroup driver (mặc định từ K8s 1.22+).
containerd PHẢI dùng cùng driver → SystemdCgroup = true.
Nếu mismatch:
kubelet (systemd) vs containerd (cgroupfs)
→ Pods bị OOMKilled bất thường
→ Resource accounting sai
→ kubelet restart liên tục

2.3. Detailed containerd configuration (Optional Tuning)

# /etc/containerd/config.toml — Các section quan trọng:
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.10"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true     # ← CRITICAL

[plugins."io.containerd.grpc.v1.cri".registry]
# Private registry mirrors (nếu cần)
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
# Harbor internal registry:
# [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.internal.com"]
#   endpoint = ["https://harbor.internal.com"]

2.4. Restart and Verify containerd

# Restart containerd
systemctl restart containerd
systemctl enable containerd
Verify containerd running
systemctl status containerd
Output: Active: active (running)
Test containerd CRI
crictl --runtime-endpoint unix:///run/containerd/containerd.sock info
Output: JSON with runtime info

---

PART 3: CRICTL SETUP

3.1. What is crictl?

crictl is CLI tool for CRI-compatible container runtimes. Similar to docker CLI but for CRI:

docker ps          → crictl ps
docker images      → crictl images
docker logs        → crictl logs
docker exec        → crictl exec
docker inspect     → crictl inspect
docker pull        → crictl pull

3.2. Crictl configuration

# crictl đã được cài cùng containerd.io package
# Cấu hình endpoint:
cat > /etc/crictl.yaml << 'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
Test
crictl info
crictl images

---

PART 4: INSTALLING KUBEADM, KUBELET, KUBECTL

4.1. Add Kubernetes Repository

# ============================================
# Chạy trên TẤT CẢ nodes
# ============================================
Kubernetes v1.31 (hoặc latest stable)
KUBE_VERSION="v1.31"
Thêm K8s repository key
curl -fsSL "https://pkgs.k8s.io/core:/stable:/${KUBE_VERSION}/deb/Release.key" | 

gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
Thêm K8s repository
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] 

https://pkgs.k8s.io/core:/stable:/${KUBE_VERSION}/deb/ /" | 

tee /etc/apt/sources.list.d/kubernetes.list
Cài đặt
apt update
apt install -y kubelet kubeadm kubectl
QUAN TRỌNG: Hold version để tránh auto-upgrade
apt-mark hold kubelet kubeadm kubectl
Verify versions
kubeadm version
Output: kubeadm version: &version.Info{Major:"1", Minor:"31", ...}
kubelet --version
Output: Kubernetes v1.31.x
kubectl version --client
Output: Client Version: v1.31.x

4.2. Enable kubelet

# Enable kubelet service (chưa start vì chưa init cluster)
systemctl enable kubelet
kubelet sẽ tự start khi kubeadm init/join chạy
Hiện tại nó sẽ crash-loop → bình thường, bỏ qua

4.3. kubectl Bash Completion

# Setup kubectl autocomplete (trên workstation và masters)
kubectl completion bash | tee /etc/bash_completion.d/kubectl > /dev/null
Alias cho tiện
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -o default -F __start_kubectl k' >> ~/.bashrc
source ~/.bashrc
Test
k get no[TAB]  # → k get nodes

---

PART 5: PRE-PULL IMAGES

5.1. Pull K8s Images before init

# Xem danh sách images cần thiết cho K8s:
kubeadm config images list
# Output:
# registry.k8s.io/kube-apiserver:v1.31.x
# registry.k8s.io/kube-controller-manager:v1.31.x
# registry.k8s.io/kube-scheduler:v1.31.x
# registry.k8s.io/kube-proxy:v1.31.x
# registry.k8s.io/coredns/coredns:v1.11.x
# registry.k8s.io/pause:3.10
# registry.k8s.io/etcd:3.5.x
Pre-pull trên TẤT CẢ control plane nodes (master1, master2, master3):
kubeadm config images pull
Output:
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.31.x
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.31.x
...
Verify
crictl images | grep k8s.io

💡 Tip: If the nodes do not have internet access, pull images on a machine with internet, export to tar, then import on nodes:

# Trên máy có internet:
ctr -n k8s.io images export k8s-images.tar $(kubeadm config images list | tr '\n' ' ')
Copy sang nodes:
scp k8s-images.tar root@master1:/tmp/
Import trên nodes:
ctr -n k8s.io images import /tmp/k8s-images.tar

---

PART 6: PRE-FLIGHT CHECK

6.1. kubeadm preflight check

# Chạy preflight checks (dry-run):
kubeadm init --dry-run 2>&1 | head -30
Hoặc dùng script toàn diện:
#!/bin/bash
echo "=== Pre-flight Check ==="
1. containerd
echo -n "containerd: "
systemctl is-active containerd && echo "✅" || echo "❌"
2. kubelet enabled
echo -n "kubelet enabled: "
systemctl is-enabled kubelet && echo "✅" || echo "❌"
3. Swap disabled
echo -n "Swap disabled: "
[[ $(swapon --show | wc -l) -eq 0 ]] && echo "✅" || echo "❌"
4. br_netfilter
echo -n "br_netfilter: "
lsmod | grep -q br_netfilter && echo "✅" || echo "❌"
5. ip_forward
echo -n "ip_forward: "
[[ $(sysctl -n net.ipv4.ip_forward) -eq 1 ]] && echo "✅" || echo "❌"
6. SystemdCgroup
echo -n "SystemdCgroup: "
grep -q "SystemdCgroup = true" /etc/containerd/config.toml && echo "✅" || echo "❌"
7. Images pulled (control plane only)
echo -n "K8s images: "
REQUIRED=$(kubeadm config images list 2>/dev/null | wc -l)
PULLED=$(crictl images 2>/dev/null | grep -c "registry.k8s.io")
echo "${PULLED}/${REQUIRED} pulled"
8. Ports available
for port in 6443 2379 2380 10250 10259 10257; do
echo -n "Port $port: "
ss -tlnp | grep -q ":$port " && echo "❌ IN USE" || echo "✅ Available"
done
echo ""
echo "=== Check HAProxy VIP ==="
echo -n "VIP reachable: "
ping -c 1 -W 2 10.10.20.100 &>/dev/null && echo "✅" || echo "❌"
echo -n "HAProxy port 6443: "
nc -z -w2 10.10.20.100 6443 &>/dev/null && echo "✅ (will route after API server starts)" || echo "⚠️ Not yet"

---

💡 KEY TAKEAWAYS

1. containerd is the standard container runtime, installed from Docker official repository
2. SystemdCgroup = true is REQUIRED — must match kubelet cgroup driver
3. kubeadm, kubelet, kubectl install from pkgs.k8s.io, then apt-mark hold to avoid auto-upgrade
4. Pre-pull images saves cluster init time, especially important for air-gapped environments
5. Pre-flight checks helps detect issues before running kubeadm init

---

🎯 EXERCISE

Exercise 1: Install on all nodes__HTMLTAG_238___

• Install containerd + kubeadm on all 6 nodes (3 masters + 3 workers)
• Verify SystemdCgroup = true per node
• Pre-pull K8s images on 3 master nodes
• Run pre-flight check script, make sure all PASS

Exercise 2: Air-gapped image transfer

• Export K8s images to tar file
• Transfer to a node without internet__HTMLTAG_255___
• Import and verify images available

---

📚 NEXT POST

In Lesson 6: Initializing the first Kubernetes HA Control Plane__HTMLTAG_264___, we will run kubeadm init on master1 with HA configuration, create cluster certificate and kubeconfig.