Chuyển đến nội dung chính
BlogCoursesHL7Cert PrepRoadmapGameDomainAboutMCPNEW
Topics
DatabaseDevOpsKiến trúc hệ thốngLinuxProgrammingSecurity

LESSON 13: CEPHBLOCKPOOL, STORAGECLASS AND PVC

Create CephBlockPool with replication, StorageClass for dynamic provisioning, PersistentVolumeClaim, test with StatefulSet, snapshot and clone volumes.

🔒 DevSecOps — Lesson 13 LESSON 13: CEPHBLOCKPOOL, STORAGECLASS AND PVC

Deploy Microservices On-Premises with Kubernetes HA

Part 3: Distributed Storage — Rook-Ceph

xdev.asia

🎯 LESSON OBJECTIVE__HTMLTAG_66___

After completing this lesson, you will:

  • ✅ Create CephBlockPool with replication factor 3
  • ✅ Create StorageClass for dynamic PV provisioning
  • ✅ Create PVC and mount it in Pod/StatefulSet
  • ✅ Volume snapshot and restore
  • ✅ Volume cloning

PART 1: CEPH BLOCK POOL

1.1. Create CephBlockPool

# ceph-block-pool.yaml:
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host              # Replicate across different hosts
  replicated:
    size: 3                         # 3 copies
    requireSafeReplicaSize: true    # Không cho write nếu < 3 replicas
  parameters:
    compression_mode: aggressive    # zstd compression
    target_size_ratio: "0.8"        # Pool chiếm tối đa 80% cluster
  mirroring:
    enabled: false

kubectl apply -f ceph-block-pool.yaml

# Verify pool:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail
# pool 1 'replicapool' replicated size 3 min_size 2 ...

1.2. StorageClass for RBD

# ceph-block-sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-block
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"   # Default StorageClass
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering,fast-diff,object-map,deep-flatten,exclusive-lock

csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
csi.storage.k8s.io/fstype: ext4


reclaimPolicy: Delete                   # PV bị xóa khi PVC bị xóa
allowVolumeExpansion: true              # Cho phép resize PVC
volumeBindingMode: Immediate


kubectl apply -f ceph-block-sc.yaml

# Verify:
kubectl get storageclass
# NAME                   PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE
# ceph-block (default)   rook-ceph.rbd.csi.ceph.com        Delete          Immediate

PART 2: PVC AND PODS

2.1. Create PVC

# test-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce               # RBD chỉ hỗ trợ RWO
  storageClassName: ceph-block
  resources:
    requests:
      storage: 5Gi

kubectl apply -f test-pvc.yaml

# Verify PVC Bound:
kubectl get pvc test-pvc
# NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
# test-pvc   Bound    pvc-xxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxxxx   5Gi        RWO            ceph-block

# Verify PV tạo tự động:
kubectl get pv
# NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
# pvc-xxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxxxx   5Gi        RWO            Delete           Bound    default/test-pvc

# Verify RBD image trên Ceph:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rbd ls replicapool
# csi-vol-xxxxx-xxxxx-xxxxx-xxxxx-xxxxxxxxxxxx

2.2. Mount PVC to Pod

# test-pod.yaml:
apiVersion: v1
kind: Pod
metadata:
  name: test-storage
  namespace: default
spec:
  containers:
    - name: test
      image: busybox
      command: ["sh", "-c", "while true; do date >> /data/log.txt; sleep 5; done"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: test-pvc

kubectl apply -f test-pod.yaml

# Verify data persistence:
kubectl exec test-storage -- cat /data/log.txt
# Mon Apr  2 07:00:05 UTC 2025
# Mon Apr  2 07:00:10 UTC 2025
# ...

# Delete pod:
kubectl delete pod test-storage

# Recreate pod → data vẫn còn:
kubectl apply -f test-pod.yaml
kubectl exec test-storage -- cat /data/log.txt
# Data cũ vẫn có! ✅ Persistent storage works

2.3. StatefulSet with volumeClaimTemplates

# statefulset-test.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: default
spec:
  serviceName: "web"
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: ceph-block
        resources:
          requests:
            storage: 2Gi

kubectl apply -f statefulset-test.yaml

# Mỗi replica có PVC riêng:
kubectl get pvc
# NAME        STATUS   VOLUME    CAPACITY   STORAGECLASS
# data-web-0  Bound    pvc-xxx   2Gi        ceph-block
# data-web-1  Bound    pvc-yyy   2Gi        ceph-block
# data-web-2  Bound    pvc-zzz   2Gi        ceph-block

PART 3: VOLUME EXPANSION

# Resize PVC (allowVolumeExpansion: true):
kubectl patch pvc test-pvc -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

# Verify:
kubectl get pvc test-pvc
# NAME       STATUS   VOLUME    CAPACITY   STORAGECLASS
# test-pvc   Bound    pvc-xxx   10Gi       ceph-block
# ↑ Đã resize từ 5Gi → 10Gi ✅

# ⚠️ Chỉ có thể EXPAND, không thể SHRINK

PART 4: VOLUME SNAPSHOT

4.1. VolumeSnapshotClass

# ceph-snapshot-class.yaml:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ceph-block-snapshot
driver: rook-ceph.rbd.csi.ceph.com
deletionPolicy: Delete
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph

4.2. Create Snapshot

# snapshot.yaml:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: test-pvc-snapshot
  namespace: default
spec:
  volumeSnapshotClassName: ceph-block-snapshot
  source:
    persistentVolumeClaimName: test-pvc

kubectl apply -f snapshot.yaml

# Verify:
kubectl get volumesnapshot
# NAME                READYTOUSE   SOURCEPVC   RESTORESIZE   AGE
# test-pvc-snapshot   true         test-pvc    10Gi          30s

4.3. Restore from Snapshot

# restore-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restored-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-block
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: test-pvc-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

kubectl apply -f restore-pvc.yaml

# Verify restored PVC has data:
kubectl run restore-test --image=busybox \
  --overrides='{"spec":{"containers":[{"name":"test","image":"busybox","command":["cat","/data/log.txt"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}],"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"restored-pvc"}}]}}' \
  --restart=Never
kubectl logs restore-test
# Data from snapshot! ✅

PART 5: VOLUME CLONING

# clone-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cloned-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-block
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: test-pvc                  # Source PVC
    kind: PersistentVolumeClaim

kubectl apply -f clone-pvc.yaml

# Clone tạo copy-on-write duplicate nhanh chóng
kubectl get pvc cloned-pvc
# STATUS: Bound ✅

PART 6: MONITORING STORAGE

# Ceph cluster capacity:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df
# --- RAW STORAGE ---
# CLASS    SIZE      AVAIL     USED      RAW USED   %RAW USED
# ssd      300 GiB   285 GiB   5.0 GiB   15 GiB     5.0
#
# --- POOLS ---
# POOL           ID   PGS   STORED    OBJECTS   USED      %USED
# replicapool    1    32    1.5 GiB   400       4.5 GiB   1.5

# OSD utilization:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df
# ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META   AVAIL    %USE
# 0   ssd    0.09769  1.00000   100 GiB  5.0 GiB  1.5 GiB  0 B  48 MiB  95 GiB  5.0
# 1   ssd    0.09769  1.00000   100 GiB  5.0 GiB  1.5 GiB  0 B  48 MiB  95 GiB  5.0
# 2   ssd    0.09769  1.00000   100 GiB  5.0 GiB  1.5 GiB  0 B  48 MiB  95 GiB  5.0

# Cleanup test resources:
kubectl delete statefulset web
kubectl delete pod test-storage restore-test
kubectl delete pvc test-pvc restored-pvc cloned-pvc data-web-0 data-web-1 data-web-2
kubectl delete volumesnapshot test-pvc-snapshot

💡 KEY TAKEAWAYS

  1. CephBlockPool with replicated.size=3 ensures data safety
  2. StorageClass for dynamic provisioning — PVC automatically creates PV
  3. allowVolumeExpansion allows PVC to be resized without recreating
  4. VolumeSnapshot creates quick point-in-time backup (copy-on-write)
  5. Volume cloning useful for dev/test environments
  6. StatefulSet + volumeClaimTemplates = each replica has its own PVC

🎯 EXERCISES__HTMLTAG_146___

Exercise 1: Block Storage Lab

  • Create CephBlockPool, StorageClass
  • Create a 5Gi PVC, mount it in pod, write data
  • Delete pod, recreate, verify data persist
  • Resize PVC to 10Gi

Exercise 2: Snapshot & Clone

  • Create VolumeSnapshot from PVC
  • Restore PVC from snapshot, verify data__HTMLTAG_165___
  • Clone PVC, verify data identical__HTMLTAG_167___

📚 NEXT POST

In Lesson 14: CephFS — Shared Filesystem for ReadWriteMany, we will configure CephFS for workloads that need multiple pods to share the same filesystem.