Rook-Ceph 1.17 分布式存储：块存储 RBD + CephFS + OSD/MON 实战

Thu, 15 Aug 2024 00:00:00 +0800

为什么用 Rook 部署 Ceph

Ceph 是开源分布式存储"老大哥"，提供 3 种存储接口：

RBD（块存储）：单 Pod 独占，ReadWriteOnce，性能最高
CephFS（共享文件系统）：多 Pod 共享，ReadWriteMany，适合数据共享
RGW（对象存储）：S3 兼容，适合大文件备份

Rook 是 CNCF 毕业项目，把 Ceph 复杂的手工部署变成 K8s Operator 模式：CRD 定义存储集群，Operator 维护所有守护进程。

适用版本：Rook v1.17.4 / Ceph v19.2.2（Reef）/ K8s 1.28.5 内核要求：≥ 4.17（CephFS 配额强制）

1. 架构

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


 ┌─────────────────────────────┐
 │ K8s Control Plane │
 └──────────────┬───────────────┘
 │ Operator 管理
 ▼
 ┌─────────────────────────────┐
 │ Rook Ceph Operator (CRD) │
 └──────┬──────┬──────┬────────┘
 │ │ │
 ┌───────▼──┐ ┌─▼───┐ ┌─▼───────┐
 │ MON │ │ MGR │ │ OSD │
 │ (3 副本) │ │ 2 │ │ (N 块盘)│
 └──────────┘ └─────┘ └─────────┘
 │ │
 └──────┬──────┘
 ▼
 ┌────────────────────────┐
 │ RBD / CephFS / RGW │
 └────────────────────────┘

Rook 部署后产出的 Operator 守护：

Operator：控制端，监控存储守护进程
Agent：每个存储节点
OSD：每块硬盘一个 OSD，提供存储
MON：监控集群，记录拓扑
MDS：CephFS 元数据服务
RGW：S3 接口（可选）
MGR：统一管理入口

2. 部署 Rook Operator

2.1 克隆代码

1
2


git clone --single-branch --branch v1.17.4 https://github.com/rook/rook.git
cd rook/deploy/examples

2.2 镜像替换

默认镜像从 quay.io / registry.k8s.io 拉，生产环境要先 tag 到私有 harbor：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


docker login -u admin -p {{HARBOR_PASSWORD}} <harbor-ip>:13001

# operator
docker pull quay.io/cephcsi/cephcsi:v3.14.0
docker tag quay.io/cephcsi/cephcsi:v3.14.0 <harbor-ip>:13001/base/cephcsi/cephcsi:v3.14.0
docker push <harbor-ip>:13001/base/cephcsi/cephcsi:v3.14.0

docker pull registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
docker tag registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0 <harbor-ip>:13001/base/sig-storage/csi-node-driver-registrar:v2.13.0
docker push <harbor-ip>:13001/base/sig-storage/csi-node-driver-registrar:v2.13.0

docker pull registry.k8s.io/sig-storage/csi-resizer:v1.13.2
docker tag registry.k8s.io/sig-storage/csi-resizer:v1.13.2 <harbor-ip>:13001/base/sig-storage/csi-resizer:v1.13.2
docker push <harbor-ip>:13001/base/sig-storage/csi-resizer:v1.13.2

docker pull registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
docker tag registry.k8s.io/sig-storage/csi-provisioner:v5.2.0 <harbor-ip>:13001/base/sig-storage/csi-provisioner:v5.2.0
docker push <harbor-ip>:13001/base/sig-storage/csi-provisioner:v5.2.0

docker pull registry.k8s.io/sig-storage/csi-snapshotter:v8.2.1
docker tag registry.k8s.io/sig-storage/csi-snapshotter:v8.2.1 <harbor-ip>:13001/base/sig-storage/csi-snapshotter:v8.2.1
docker push <harbor-ip>:13001/base/sig-storage/csi-snapshotter:v8.2.1

docker pull registry.k8s.io/sig-storage/csi-attacher:v4.8.1
docker tag registry.k8s.io/sig-storage/csi-attacher:v4.8.1 <harbor-ip>:13001/base/sig-storage/csi-attacher:v4.8.1
docker push <harbor-ip>:13001/base/sig-storage/csi-attacher:v4.8.1

docker pull docker.io/rook/ceph:v1.17.4
docker tag docker.io/rook/ceph:v1.17.4 <harbor-ip>:13001/base/rook/ceph:v1.17.4
docker push <harbor-ip>:13001/base/rook/ceph:v1.17.4

docker pull quay.io/ceph/ceph:v19.2.2
docker tag quay.io/ceph/ceph:v19.2.2 <harbor-ip>:13001/base/ceph/ceph:v19.2.2
docker push <harbor-ip>:13001/base/ceph/ceph:v19.2.2

修改 operator.yaml 镜像源：

1
2


sed -i "s#image: rook/ceph:v1.14.8#image: <harbor-ip>:13001/base/rook/ceph:v1.17.4#g" operator.yaml
sed -i "s#image: quay.io/ceph/ceph:v18.2.2#image: <harbor-ip>:13001/base/ceph/ceph:v19.2.2#g" cluster.yaml

2.3 应用

1

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

2.4 创建 CephCluster

cluster.yaml 关键配置：

1
2
3
4
5
6
7


spec:
 cephVersion:
 image: <harbor-ip>:13001/base/ceph/ceph:v19.2.2
 dataDirHostPath: /var/lib/rook
 storage:
 useAllNodes: true # 用所有节点
 useAllDevices: true # 用所有未分区盘

或精确指定：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


spec:
 storage:
 useAllNodes: false
 useAllDevices: false
 config:
 nodes:
 - name: "master1"
 devices:
 - name: "sdb"
 - name: "sdc"
 - name: "master2"
 devices:
 - name: "sdb"
 - name: "sdc"
 - name: "master3"
 devices:
 - name: "sdb"
 - name: "sdc"

1

kubectl create -f cluster.yaml

2.5 验证

1
2


kubectl -n rook-ceph get pods
# 应该有 3 个 mon、2 个 mgr、多个 osd、csi-rbdplugin、csi-cephfsplugin 等

3. 块存储（RBD）StorageClass

3.1 创建 CephBlockPool

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
 name: replicapool
 namespace: rook-ceph
spec:
 failureDomain: host  # host 级容灾
 replicated:
 size: 3 # 3 副本
 requireSafeReplicaSize: true

3.2 创建 StorageClass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
 clusterID: rook-ceph  # namespace:cluster
 pool: replicapool
 imageFormat: "2"
 imageFeatures: layering
 csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
 csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
 csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
 csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
 csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
 csi.storage.k8s.io/fstype: ext4
allowVolumeExpansion: true
reclaimPolicy: Delete

3.3 测试 PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: rbd-pvc
 namespace: rook-ceph
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi
 storageClassName: rook-ceph-block

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


apiVersion: v1
kind: Pod
metadata:
 name: rbd-demo-pod
 namespace: rook-ceph
spec:
 containers:
 - name: rbd-demo-pod
 image: busybox:stable-glibc
 command: ["sleep", "3600"]
 volumeMounts:
 - mountPath: /var/lib/www/html
 name: data
 volumes:
 - name: data
 persistentVolumeClaim:
 claimName: rbd-pvc

验证：

1
2
3
4
5


kubectl exec rbd-demo-pod -n rook-ceph -- df -ah
# 看到 /dev/rbd0 1G

kubectl delete -f rbd-pod.yaml
kubectl delete -f rbd-pvc.yaml

4. CephFS 共享文件系统

4.1 创建 CephFilesystem

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
 name: myfs
 namespace: rook-ceph
spec:
 metadataPool:
 replicated:
 size: 3
 requireSafeReplicaSize: true
 dataPools:
 - failureDomain: host
 replicated:
 size: 3
 requireSafeReplicaSize: true
 metadataServer:
 activeCount: 1 # 生产建议 3
 preserveFilesystemOnDelete: true

4.2 StorageClass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
 clusterID: rook-ceph
 fsName: myfs
 pool: myfs-replicated
 csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
 csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
 csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
 csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
 csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
 csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete

4.3 测试 RWX

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: busybox-data-pvc
spec:
 accessModes:
 - ReadWriteMany
 resources:
 requests:
 storage: 2Gi
 storageClassName: rook-cephfs
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: busybox
spec:
 replicas: 2
 selector:
 matchLabels:
 app: busybox
 template:
 metadata:
 labels:
 app: busybox
 spec:
 containers:
 - name: busybox-container
 image: busybox:stable-glibc
 command: ["sleep", "3600"]
 volumeMounts:
 - name: data-volume
 mountPath: /data
 volumes:
 - name: data-volume
 persistentVolumeClaim:
 claimName: busybox-data-pvc

两个 Pod 都能读写 /data，就是 CephFS 的 RWX 能力。

5. RBD 块存储的局限

RBD 是块设备，只能 ReadWriteOnce 挂载到单个 Pod。Deployment 滚动更新时：

旧 Pod 还挂载着 RBD
新 Pod 调度到不同节点，想再挂载同一个 PV → 失败
旧 Pod 没删除前，RBD 不让别的 Pod 挂

解决方案：

用 CephFS / GlusterFS 这种支持 RWX 的存储
给 Node 加 label，Pod 强制调度到固定 Node（不推荐，单点）

6. Dashboard

1

kubectl apply -f dashboard-external-https.yaml

获取密码：

1

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode

默认用户 admin，可以登录后改密。

7. Toolbox（排障工具）

1
2


kubectl create -f toolbox.yaml
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

常用命令：

1
2
3
4
5
6
7


ceph status # 集群状态
ceph osd status # OSD 状态
ceph osd tree # OSD 拓扑
ceph df # 容量使用
ceph health detail # 健康详情
ceph mgr module enable rook
ceph orch set backend rook

8. 常见问题

8.1 MON_DISK_LOW: mon is low on available space

/var/lib/rook 或 mon 容器所在节点的 /var 满了：

1
2
3
4
5
6


docker system prune -af
journalctl --vacuum-time=1w
journalctl --vacuum-size=500M

# 大头往往是 /var/lib/docker 和 /var/lib/kubelet
du -sh /var/lib/* | sort -hr | head

8.2 MON_CLOCK_SKEW detected on mon.b, mon.c

时钟漂移。检查所有节点 ntp/chrony：

1
2
3
4
5
6
7
8


kubectl -n rook-ceph get cm rook-config-override -o yaml

# 放宽容忍
apiVersion: v1
data:
 config: |
 [global]
 mon clock drift allowed = 1

重启 mon：

1

kubectl -n rook-ceph delete pod $(kubectl -n rook-ceph get pods -o custom-columns=NAME:.metadata.name --no-headers | grep mon)

8.3 TOO_FEW_OSDS

少于 3 块 OSD。Ceph 需要至少 3 副本。检查每台节点上是否真的发现了裸盘：

1
2


lsblk
# 应该是裸盘，不能有分区或文件系统

8.4 卸载卡 Terminating

1
2
3
4
5


NAMESPACE=rook-ceph
kubectl proxy &

kubectl get namespace $NAMESPACE -o json | jq '.spec = {"finalizers":[]}' > temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize

8.5 cephcluster 无法删除

1
2


kubectl edit cephcluster.ceph.rook.io -n rook-ceph
# 删掉 finalizers 字段

8.6 卸载后硬盘数据未清

1
2
3


sgdisk --zap-all /dev/sdb
dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync
blkdiscard /dev/sdb

9. 小结

Rook-Ceph 是 K8s 上跑 Ceph 的事实标准：

RBD 适合 RWO 高性能场景（数据库）
CephFS 适合 RWX 共享场景（多 Pod 数据共享）
MON/OSD 至少 3 副本，生产至少 3 节点 9 块盘
磁盘满、时钟漂移 是最常见的 2 类故障

下一步：GlusterFS 10 分布式文件系统：七种卷类型 + Kadalu CSI 接入。

块存储 on Liangweidong's blog