xDev Asia

Tạo CephBlockPool với replication, StorageClass cho dynamic provisioning, PersistentVolumeClaim, test với StatefulSet, snapshot và clone volumes.

Deploy Microservices On-Premises với Kubernetes HA

Phần 3: Distributed Storage — Rook-Ceph

xdev.asia

🎯 MỤC TIÊU BÀI HỌC

Sau khi hoàn thành bài học này, bạn sẽ:

✅ Tạo CephBlockPool với replication factor 3
✅ Tạo StorageClass cho dynamic PV provisioning
✅ Tạo PVC và mount vào Pod/StatefulSet
✅ Volume snapshot và restore
✅ Volume cloning

PHẦN 1: CEPH BLOCK POOL

1.1. Tạo CephBlockPool

# ceph-block-pool.yaml:
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host              # Replicate across different hosts
  replicated:
    size: 3                         # 3 copies
    requireSafeReplicaSize: true    # Không cho write nếu < 3 replicas
  parameters:
    compression_mode: aggressive    # zstd compression
    target_size_ratio: "0.8"        # Pool chiếm tối đa 80% cluster
  mirroring:
    enabled: false

kubectl apply -f ceph-block-pool.yaml

# Verify pool:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail
# pool 1 'replicapool' replicated size 3 min_size 2 ...

1.2. StorageClass cho RBD

# ceph-block-sc.yaml: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ceph-block annotations: storageclass.kubernetes.io/is-default-class: "true" # Default StorageClass provisioner: rook-ceph.rbd.csi.ceph.com parameters: clusterID: rook-ceph pool: replicapool imageFormat: "2" imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete # PV bị xóa khi PVC bị xóa allowVolumeExpansion: true # Cho phép resize PVC volumeBindingMode: Immediate

kubectl apply -f ceph-block-sc.yaml

# Verify:
kubectl get storageclass
# NAME                   PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE
# ceph-block (default)   rook-ceph.rbd.csi.ceph.com        Delete          Immediate

PHẦN 2: PVC VÀ PODS

2.1. Tạo PVC

# test-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce               # RBD chỉ hỗ trợ RWO
  storageClassName: ceph-block
  resources:
    requests:
      storage: 5Gi

kubectl apply -f test-pvc.yaml

# Verify PVC Bound:
kubectl get pvc test-pvc
# NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
# test-pvc   Bound    pvc-xxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxxxx   5Gi        RWO            ceph-block

# Verify PV tạo tự động:
kubectl get pv
# NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
# pvc-xxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxxxx   5Gi        RWO            Delete           Bound    default/test-pvc

# Verify RBD image trên Ceph:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rbd ls replicapool
# csi-vol-xxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxxxx

2.2. Mount PVC vào Pod

# test-pod.yaml:
apiVersion: v1
kind: Pod
metadata:
  name: test-storage
  namespace: default
spec:
  containers:
    - name: test
      image: busybox
      command: ["sh", "-c", "while true; do date >> /data/log.txt; sleep 5; done"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: test-pvc

kubectl apply -f test-pod.yaml

# Verify data persistence:
kubectl exec test-storage -- cat /data/log.txt
# Mon Apr  2 07:00:05 UTC 2025
# Mon Apr  2 07:00:10 UTC 2025
# ...

# Delete pod:
kubectl delete pod test-storage

# Recreate pod → data vẫn còn:
kubectl apply -f test-pod.yaml
kubectl exec test-storage -- cat /data/log.txt
# Data cũ vẫn có! ✅ Persistent storage works

2.3. StatefulSet với volumeClaimTemplates

# statefulset-test.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: default
spec:
  serviceName: "web"
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: ceph-block
        resources:
          requests:
            storage: 2Gi

kubectl apply -f statefulset-test.yaml

# Mỗi replica có PVC riêng:
kubectl get pvc
# NAME        STATUS   VOLUME    CAPACITY   STORAGECLASS
# data-web-0  Bound    pvc-xxx   2Gi        ceph-block
# data-web-1  Bound    pvc-yyy   2Gi        ceph-block
# data-web-2  Bound    pvc-zzz   2Gi        ceph-block

PHẦN 3: VOLUME EXPANSION

# Resize PVC (allowVolumeExpansion: true):
kubectl patch pvc test-pvc -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

# Verify:
kubectl get pvc test-pvc
# NAME       STATUS   VOLUME    CAPACITY   STORAGECLASS
# test-pvc   Bound    pvc-xxx   10Gi       ceph-block
# ↑ Đã resize từ 5Gi → 10Gi ✅

# ⚠️ Chỉ có thể EXPAND, không thể SHRINK

PHẦN 4: VOLUME SNAPSHOT

4.1. VolumeSnapshotClass

# ceph-snapshot-class.yaml:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ceph-block-snapshot
driver: rook-ceph.rbd.csi.ceph.com
deletionPolicy: Delete
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph

4.2. Tạo Snapshot

# snapshot.yaml:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: test-pvc-snapshot
  namespace: default
spec:
  volumeSnapshotClassName: ceph-block-snapshot
  source:
    persistentVolumeClaimName: test-pvc

kubectl apply -f snapshot.yaml

# Verify:
kubectl get volumesnapshot
# NAME                READYTOUSE   SOURCEPVC   RESTORESIZE   AGE
# test-pvc-snapshot   true         test-pvc    10Gi          30s

4.3. Restore từ Snapshot

# restore-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-block
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: test-pvc-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

kubectl apply -f restore-pvc.yaml

# Verify restored PVC has data:
kubectl run restore-test --image=busybox \
  --overrides='{"spec":{"containers":[{"name":"test","image":"busybox","command":["cat","/data/log.txt"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"restored-pvc"}}]}}' \
  --restart=Never
kubectl logs restore-test
# Data from snapshot! ✅

PHẦN 5: VOLUME CLONING

# clone-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cloned-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-block
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: test-pvc                  # Source PVC
    kind: PersistentVolumeClaim

kubectl apply -f clone-pvc.yaml

# Clone tạo copy-on-write duplicate nhanh chóng
kubectl get pvc cloned-pvc
# STATUS: Bound ✅

PHẦN 6: MONITORING STORAGE

# Ceph cluster capacity:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df
# --- RAW STORAGE ---
# CLASS    SIZE      AVAIL     USED      RAW USED   %RAW USED
# ssd      300 GiB   285 GiB   5.0 GiB   15 GiB     5.0
#
# --- POOLS ---
# POOL           ID   PGS   STORED    OBJECTS   USED      %USED
# replicapool    1    32    1.5 GiB   400       4.5 GiB   1.5

# OSD utilization:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df
# ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META   AVAIL    %USE
# 0   ssd    0.09769  1.00000   100 GiB  5.0 GiB  1.5 GiB  0 B  48 MiB  95 GiB  5.0
# 1   ssd    0.09769  1.00000   100 GiB  5.0 GiB  1.5 GiB  0 B  48 MiB  95 GiB  5.0
# 2   ssd    0.09769  1.00000   100 GiB  5.0 GiB  1.5 GiB  0 B  48 MiB  95 GiB  5.0

# Cleanup test resources:
kubectl delete statefulset web
kubectl delete pod test-storage restore-test
kubectl delete pvc test-pvc restored-pvc cloned-pvc data-web-0 data-web-1 data-web-2
kubectl delete volumesnapshot test-pvc-snapshot

💡 KEY TAKEAWAYS

CephBlockPool với replicated.size=3 đảm bảo data an toàn
StorageClass cho dynamic provisioning — PVC automatically creates PV
allowVolumeExpansion cho phép resize PVC mà không cần recreate
VolumeSnapshot tạo point-in-time backup nhanh (copy-on-write)
Volume cloning hữu ích cho dev/test environments
StatefulSet + volumeClaimTemplates = mỗi replica có PVC riêng

🎯 BÀI TẬP

Bài tập 1: Block Storage Lab

Tạo CephBlockPool, StorageClass
Tạo PVC 5Gi, mount vào pod, ghi data
Delete pod, recreate, verify data persist
Resize PVC lên 10Gi

Bài tập 2: Snapshot & Clone

Tạo VolumeSnapshot từ PVC
Restore PVC từ snapshot, verify data
Clone PVC, verify data identical

📚 BÀI TIẾP THEO

Trong Bài 14: CephFS — Shared Filesystem cho ReadWriteMany, chúng ta sẽ cấu hình CephFS cho workloads cần nhiều pod chia sẻ cùng filesystem.

BÀI 13: CEPHBLOCKPOOL, STORAGECLASS VÀ PVC