Deploy Microservices On-Premises with Kubernetes HA

Part 3: Distributed Storage — Rook-Ceph

xdev.asia

🎯 LESSON OBJECTIVE__HTMLTAG_68___

After completing this lesson, you will:

• ✅ Create CephFilesystem and MetadataServer (MDS)
• ✅ Create StorageClass for CephFS with ReadWriteMany
• ✅ Mount CephFS from multiple pods at the same time__HTMLTAG_77___
• ✅ Configure quotas and subvolumeGroups
• ✅ Compare RBD vs CephFS use cases

---

PART 1: CEPHFS vs RBD — WHEN TO USE WHAT?

Criteria	RBD (Block)	CephFS (Filesystem)
Access Mode	ReadWriteOnce (RWO)	ReadWriteMany (RWX)
Mount style	1 single pod	Multiple pods at the same time
Use case	Database, single app	Shared content, logs, media
Performance	High (block-level)	Good (POSIX overhead)
Need MDS?	No	Yes

---

PART 2: CREATE CEPHFILESYSTEM

2.1. CephFilesystem CRD

# ceph-filesystem.yaml:
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: cephfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: data0
      failureDomain: host
      replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1                     # 1 active MDS
    activeStandby: true                # 1 standby MDS
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        memory: "4Gi"

kubectl apply -f ceph-filesystem.yaml
Verify MDS pods:
kubectl -n rook-ceph get pods -l app=rook-ceph-mds
NAME                                      READY   STATUS    RESTARTS   AGE
rook-ceph-mds-cephfs-a-xxxxx-xxxxx        2/2     Running   0          60s
rook-ceph-mds-cephfs-b-xxxxx-xxxxx        2/2     Running   0          60s
Verify CephFS:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph fs ls
name: cephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data0]
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph fs status
cephfs - 0 clients
======
RANK   STATE            MDS              ACTIVITY     DNS    INOS   DIRS   CAPS
0     active           cephfs-a         Reqs:    0     10     13     12      0
STANDBY          cephfs-b

2.2. StorageClass for CephFS

# ceph-fs-sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-filesystem
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: cephfs
  pool: cephfs-data0
csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

kubectl apply -f ceph-fs-sc.yaml

kubectl get storageclass
# NAME                   PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE
# ceph-block (default)   rook-ceph.rbd.csi.ceph.com            Delete          Immediate
# ceph-filesystem        rook-ceph.cephfs.csi.ceph.com          Delete          Immediate

---

PART 3: TEST READWRITEMANY

3.1. Create RWX PVC

# shared-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany                  # ← RWX mode!
  storageClassName: ceph-filesystem
  resources:
    requests:
      storage: 5Gi

3.2. Multiple pods with same mount

# shared-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: writer
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: writer
  template:
    metadata:
      labels:
        app: writer
    spec:
      containers:
        - name: writer
          image: busybox
          command:
            - sh
            - -c
            - |
              while true; do
                echo "$(hostname) - $(date)" >> /shared/log.txt
                sleep 5
              done
          volumeMounts:
            - name: shared
              mountPath: /shared
      volumes:
        - name: shared
          persistentVolumeClaim:
            claimName: shared-data

apiVersion: apps/v1
kind: Deployment
metadata:
name: reader
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: reader
template:
metadata:
labels:
app: reader
spec:
containers:
- name: reader
image: busybox
command:
- sh
- -c
- "while true; do tail -5 /shared/log.txt; sleep 10; done"
volumeMounts:
- name: shared
mountPath: /shared
volumes:
- name: shared
persistentVolumeClaim:
claimName: shared-data

kubectl apply -f shared-pvc.yaml -f shared-deployment.yaml

# Verify 4 pods đều mount cùng volume:
kubectl get pods -l 'app in (writer,reader)'
# NAME                      READY   STATUS    RESTARTS   AGE
# writer-xxxxx-xxxxx        1/1     Running   0          30s
# writer-xxxxx-xxxxx        1/1     Running   0          30s
# writer-xxxxx-xxxxx        1/1     Running   0          30s
# reader-xxxxx-xxxxx        1/1     Running   0          30s

# Check reader sees all writers:
kubectl logs -l app=reader --tail=10
# writer-xxxxx-xxxxx - Mon Apr  2 07:00:05 UTC 2025
# writer-xxxxx-xxxxx - Mon Apr  2 07:00:05 UTC 2025
# writer-xxxxx-xxxxx - Mon Apr  2 07:00:05 UTC 2025
# ← 3 writer pods đều ghi vào cùng file! ✅

---

PART 4: QUOTAS AND SUBVOLUMES__HTMLTAG_158___

# CephFS quotas (set via Ceph toolbox):
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- bash

# Set quota trên directory:
# setfattr -n ceph.quota.max_bytes -v $((10*1024*1024*1024)) /mnt/cephfs/subdir
# setfattr -n ceph.quota.max_files -v 100000 /mnt/cephfs/subdir

# CSI driver tự set quota dựa trên PVC size
# → PVC 5Gi sẽ tự giới hạn 5GiB trên CephFS subvolume

4.1. Cleanup

kubectl delete deployment writer reader
kubectl delete pvc shared-data

---

💡 KEY TAKEAWAYS

1. CephFS for ReadWriteMany — multiple pods read/write the same volume
2. MDS (Metadata Server) manage filesystem metadata
3. activeStandby: true ensure MDS HA
4. Use case RWX: shared content, logs, media, AI training data
5. RBD for databases, CephFS for shared files

---

🎯 EXERCISES__HTMLTAG_188___

Exercise 1: RWX Lab

• Create CephFilesystem, StorageClass, PVC (RWX)
• Deploy 3 writer pods + 1 reader pod__HTMLTAG_195___
• Verify all pods read/write same volume__HTMLTAG_197___

---

📚 NEXT POST

In Lesson 15: Ceph Monitoring, Tuning and Troubleshooting, we will monitor Ceph performance, tuning parameters, and handle common issues.