Chuyển đến nội dung chính
Bài viếtKhoá họcHL7Luyện thiRoadmapGameLĩnh vựcVề tôiMCPNEW
Chủ đề
Bảo mậtCơ sở dữ liệuDevOpsKiến trúc hệ thốngLập trìnhLinux

BÀI 14: CEPHFS — SHARED FILESYSTEM CHO READWRITEMANY

Tạo CephFilesystem, StorageClass cho CephFS (ReadWriteMany), deploy MDS, test shared storage giữa nhiều pods, quotas và subvolumes.

🔒 DevSecOps — Bài 14 BÀI 14: CEPHFS — SHARED FILESYSTEM CHO READWRITEMANY

Deploy Microservices On-Premises với Kubernetes HA

Phần 3: Distributed Storage — Rook-Ceph

xdev.asia

🎯 MỤC TIÊU BÀI HỌC

Sau khi hoàn thành bài học này, bạn sẽ:

  • ✅ Tạo CephFilesystem và MetadataServer (MDS)
  • ✅ Tạo StorageClass cho CephFS với ReadWriteMany
  • ✅ Mount CephFS từ nhiều pods đồng thời
  • ✅ Cấu hình quotas và subvolumeGroups
  • ✅ So sánh RBD vs CephFS use cases

PHẦN 1: CEPHFS vs RBD — KHI NÀO DÙNG GÌ?

Tiêu chí RBD (Block) CephFS (Filesystem)
Access Mode ReadWriteOnce (RWO) ReadWriteMany (RWX)
Mount style 1 pod duy nhất Nhiều pods đồng thời
Use case Database, single app Shared content, logs, media
Performance Cao (block-level) Tốt (POSIX overhead)
Cần MDS? Không Có

PHẦN 2: TẠO CEPHFILESYSTEM

2.1. CephFilesystem CRD

# ceph-filesystem.yaml:
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: cephfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: data0
      failureDomain: host
      replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1                     # 1 active MDS
    activeStandby: true                # 1 standby MDS
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        memory: "4Gi"

kubectl apply -f ceph-filesystem.yaml

# Verify MDS pods:
kubectl -n rook-ceph get pods -l app=rook-ceph-mds
# NAME                                      READY   STATUS    RESTARTS   AGE
# rook-ceph-mds-cephfs-a-xxxxx-xxxxx        2/2     Running   0          60s
# rook-ceph-mds-cephfs-b-xxxxx-xxxxx        2/2     Running   0          60s

# Verify CephFS:
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph fs ls
# name: cephfs, metadata pool: cephfs-metadata, data pools: [cephfs-data0]

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph fs status
# cephfs - 0 clients
# ======
# RANK   STATE            MDS              ACTIVITY     DNS    INOS   DIRS   CAPS
#  0     active           cephfs-a         Reqs:    0     10     13     12      0
#        STANDBY          cephfs-b

2.2. StorageClass cho CephFS

# ceph-fs-sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-filesystem
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: cephfs
  pool: cephfs-data0

csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph


reclaimPolicy: Delete
allowVolumeExpansion: true


kubectl apply -f ceph-fs-sc.yaml

kubectl get storageclass
# NAME                   PROVISIONER                           RECLAIMPOLICY   VOLUMEBINDINGMODE
# ceph-block (default)   rook-ceph.rbd.csi.ceph.com            Delete          Immediate
# ceph-filesystem        rook-ceph.cephfs.csi.ceph.com          Delete          Immediate

PHẦN 3: TEST READWRITEMANY

3.1. Tạo RWX PVC

# shared-pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany                  # ← RWX mode!
  storageClassName: ceph-filesystem
  resources:
    requests:
      storage: 5Gi

3.2. Nhiều pods cùng mount

# shared-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: writer
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: writer
  template:
    metadata:
      labels:
        app: writer
    spec:
      containers:
        - name: writer
          image: busybox
          command:
            - sh
            - -c
            - |
              while true; do
                echo "$(hostname) - $(date)" >> /shared/log.txt
                sleep 5
              done
          volumeMounts:
            - name: shared
              mountPath: /shared
      volumes:
        - name: shared
          persistentVolumeClaim:
            claimName: shared-data



apiVersion: apps/v1
kind: Deployment
metadata:
name: reader
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: reader
template:
metadata:
labels:
app: reader
spec:
containers:
- name: reader
image: busybox
command:
- sh
- -c
- "while true; do tail -5 /shared/log.txt; sleep 10; done"
volumeMounts:
- name: shared
mountPath: /shared
volumes:
- name: shared
persistentVolumeClaim:
claimName: shared-data


kubectl apply -f shared-pvc.yaml -f shared-deployment.yaml

# Verify 4 pods đều mount cùng volume:
kubectl get pods -l 'app in (writer,reader)'
# NAME                      READY   STATUS    RESTARTS   AGE
# writer-xxxxx-xxxxx        1/1     Running   0          30s
# writer-xxxxx-xxxxx        1/1     Running   0          30s
# writer-xxxxx-xxxxx        1/1     Running   0          30s
# reader-xxxxx-xxxxx        1/1     Running   0          30s

# Check reader sees all writers:
kubectl logs -l app=reader --tail=10
# writer-xxxxx-xxxxx - Mon Apr  2 07:00:05 UTC 2025
# writer-xxxxx-xxxxx - Mon Apr  2 07:00:05 UTC 2025
# writer-xxxxx-xxxxx - Mon Apr  2 07:00:05 UTC 2025
# ← 3 writer pods đều ghi vào cùng file! ✅

PHẦN 4: QUOTAS VÀ SUBVOLUMES

# CephFS quotas (set via Ceph toolbox):
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- bash

# Set quota trên directory:
# setfattr -n ceph.quota.max_bytes -v $((10*1024*1024*1024)) /mnt/cephfs/subdir
# setfattr -n ceph.quota.max_files -v 100000 /mnt/cephfs/subdir

# CSI driver tự set quota dựa trên PVC size
# → PVC 5Gi sẽ tự giới hạn 5GiB trên CephFS subvolume

4.1. Cleanup

kubectl delete deployment writer reader
kubectl delete pvc shared-data

💡 KEY TAKEAWAYS

  1. CephFS cho ReadWriteMany — nhiều pods cùng đọc/ghi 1 volume
  2. MDS (Metadata Server) quản lý filesystem metadata
  3. activeStandby: true đảm bảo MDS HA
  4. Use case RWX: shared content, logs, media, AI training data
  5. RBD cho databases, CephFS cho shared files

🎯 BÀI TẬP

Bài tập 1: RWX Lab

  • Tạo CephFilesystem, StorageClass, PVC (RWX)
  • Deploy 3 writer pods + 1 reader pod
  • Verify tất cả pods đọc/ghi cùng volume

📚 BÀI TIẾP THEO

Trong Bài 15: Ceph Monitoring, Tuning và Troubleshooting, chúng ta sẽ monitor Ceph performance, tuning parameters, và xử lý common issues.