简介

Ceph是一种高度可扩展的分布式存储解决方案，适用于块存储、对象存储和共享文件系统。

主要特性：

1. 采用 CRUSH 算法，数据分布均衡
2. 能够支持上千个存储节点的规模，支持 TB 到 PB 级的数据
3. 支持故障域分离，数据强一致性
4. 支持三种存储接口：块存储、文件存储、对象存储

部署前

测试集群为1.22版本单机kubernetes， 确保有一块未被使用的硬盘挂载在节点上，方便后续创建ceph集群时检测到该设备

lsblk

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  100G  0 disk 
├─sda1   8:1    0    1G  0 part /boot
└─sda2   8:2    0   99G  0 part 
  ├─centos-root
       253:0    0   50G  0 lvm  /
  ├─centos-swap
       253:1    0  7.9G  0 lvm  
  └─centos-home
       253:2    0 41.1G  0 lvm  /home
sdb      8:16   0   32G  0 disk #未被使用的硬盘

部署Rook Operator

git clone --single-branch --branch v1.11.5 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

检查确认pod 处于running状态

kubectl -n rook-ceph get pod

创建Ceph Cluster

#路径 rook/deploy/examples 
kubectl apply -f cluster-test.yaml

Toolbox

#路径 rook/deploy/examples
kubectl create -f toolbox.yaml

在pod内部可执行命令检查集群状态

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash

ceph status

  cluster:
    id:     974026d9-dd40-4a06-a6bd-36e04ecd7b05
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum a (age 9h)
    mgr: a(active, since 9h)
    osd: 1 osds: 1 up (since 9h), 1 in (since 10h)

  data:
    pools:   2 pools, 64 pgs
    objects: 297 objects, 899 MiB
    usage:   923 MiB used, 31 GiB / 32 GiB avail
    pgs:     64 active+clean

Dashboard

#路径 rook/deploy/examples
kubectl create -f dashboard-external-http.yaml

查看dashboard svc NodePort

kubectl get svc -n rook-ceph |grep external
rook-ceph-mgr-dashboard-external-http   NodePort    10.233.57.168           7000:31705/TCP      10h

访问节点ip:31705 

获取登录密码

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

创建CephBlockPool和StorageClass

#路径 deploy/examples/csi/rbd
kubectl create -f storageclass.yaml

因为单机部署修改其中的size 3为1

spec:
  failureDomain: host
  replicated:
    size: 3

kubectl apply -f storageclass.yaml

使用

部署WordPress进行测试

#路径 deploy/examples
kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml

部署后获取svc 的NodePort 访问节点ip:32669 进入WordPress安装引导

kubectl get svc wordpress

NAME        TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
wordpress   LoadBalancer   10.233.5.198        80:32669/TCP   15h

参考

1. https://www.qikqiak.com/k8strain2/storage/ceph/
2. https://rook.github.io/docs/rook/v1.11/Getting-Started/quickstart/
3. https://rook.github.io/docs/rook/v1.11/Troubleshooting/ceph-toolbox/#interactive-toolbox
4. https://rook.github.io/docs/rook/v1.11/Storage-Configuration/Monitoring/ceph-dashboard/#node-port
5. https://rook.github.io/docs/rook/v1.11/Storage-Configuration/Block-Storage-RBD/block-storage/#advanced-example-erasure-coded-block-storage