kubernetes 部署 ceph-rbd-csi

原创已于 2023-02-17 16:36:26 修改 · 825 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#kubernetes #ceph

于 2023-02-08 18:31:46 首次发布

kubernetes 同时被 2 个专栏收录

21 篇文章

订阅专栏

kubevirt

7 篇文章

订阅专栏

这篇博客介绍了如何在 Kubernetes 1.24 及以上版本上部署 ceph-rbd-csi，以支持 CDI 的 datavolume 功能和 kubevirt 的 images 存储。内容包括：部署准备（如创建池、初始化、用户创建），部署步骤（CSIDriver、RBAC、ConfigMap、csi-encryption-kms、sidecar 和 rbdplugin），测试（secret、storageclass、pvc 的创建和验证），以及数据验证过程。

目的

kubernetes 1.24 上部署 ceph-rbd-csi
用于支持 CDI 中 datavolume 功能
把 kubevirt 的 images 存放至 ceph 中
您可以通过 Ceph csi 在 Kubernetes v1.13 和更高版本中使用 Ceph Block Device images
该映像动态地提供 RBD images 映射为 Kubernete 卷
将这些 RBD images 映射为运行引用 RBD 支持卷的 pod 节点上的块设备
Ceph 将块设备映像作为集群中的对象进行条带化

要求

kubernetes 1.14+
kube api 及 kubelet 服务需要支持 --allow-privileged = true

说明

要使用 ceph block device，需要在 kubenetes 上安装配置 ceph-csi

参考图

在这里插入图片描述

kubenetes 通过 csi 可以通过两种方法访问转换为 rados 协议
内核模块，不建议，因为代码更新比较慢
rbd-nbd 命令工具，通过 librbd 转换为 rados （建议，因为代码更新比较快）
转换为 ceph 协议后就可以与 ceph mon, osd 进行直接沟通

ceph 准备

创建池

 ceph osd pool create kubernetes

初始化池

rbd pool init kubernetes

创建用户

ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
        key = AQAjFtVjmlXdNhAAqM1DHhf2gZEUO2eo05Ka+w==

参考

官方 ceph rbd csi 部署文档
 yaml template

镜像需求

名称	版本
k8s.gcr.io/sig-storage/csi-provisioner	v3.2.1
registry.k8s.io/sig-storage/csi-snapshotter	v6.0.1
registry.k8s.io/sig-storage/csi-attacher	v3.5.0
registry.k8s.io/sig-storage/csi-resizer	v1.5.0
registry.k8s.io/sig-storage/csi-node-driver-registrar	v2.5.1
quay.io/cephcsi/cephcsi	v3.7.2

部署

CSIDriver 对象

# kubectl create -f csidriver.yaml
csidriver.storage.k8s.io/rbd.csi.ceph.com created

sidecar 容器与节点 plugin 部署 RBAC

# kubectl create -f csi-provisioner-rbac.yaml
serviceaccount/rbd-csi-provisioner created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created

# kubectl  apply -f csi-nodeplugin-rbac.yaml
serviceaccount/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created

配置 ConfigMap

需要修改一下，配置 ceph mon 信息

# cat csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "aa0e0954-4abd-41d3-9fcf-b38a8159167a",
        "monitors": [
          "10.189.20.100:6789",
          "10.189.20.101:6789",
          "10.189.20.102:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config

部署 configmap

# kubectl apply -f csi-config-map.yaml
configmap/ceph-csi-config created

配置 ceph global 配置

cat ceph-conf.yaml
---
# This is a sample configmap that helps define a Ceph configuration as required
# by the CSI plugins.

# Sample ceph.conf available at
# https://github.com/ceph/ceph/blob/master/src/sample.ceph.conf Detailed
# documentation is available at
# https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/
apiVersion: v1
kind: ConfigMap
data:
  ceph.conf: |
    [global]
    auth_cluster_required = cephx
    auth_service_required = cephx
    auth_client_required = cephx

  # keyring is a required key and its value should be empty
  keyring: |
metadata:
  name: ceph-config


# kubectl apply -f ceph-conf.yaml
configmap/ceph-config created

部署 csi-encryption-kms 配置

# cat csi-kms-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {}
metadata:
  name: ceph-csi-encryption-kms-config

# kubectl apply -f  csi-kms-config-map.yaml
configmap/ceph-csi-encryption-kms-config created

部署 csi sidecar 容器

# kubectl apply -f  csi-rbdplugin-provisioner.yaml
service/csi-rbdplugin-provisioner created
deployment.apps/csi-rbdplugin-provisioner created

查询 pods

# kubectl get pods
NAME                                         READY   STATUS    RESTARTS   AGE
csi-rbdplugin-provisioner-86b684cdf7-6tq6d   7/7     Running   0          65m
csi-rbdplugin-provisioner-86b684cdf7-hgrrf   7/7     Running   0          65m
csi-rbdplugin-provisioner-86b684cdf7-s7lj8   7/7     Running   0          65m

部署 rbdplugin

# kubectl apply -f csi-rbdplugin.yaml
daemonset.apps/csi-rbdplugin created
service/csi-metrics-rbdplugin created

检查 pods

# kubectl  get pods
NAME                                         READY   STATUS    RESTARTS   AGE
csi-rbdplugin-5l7ds                          3/3     Running   0          24s
csi-rbdplugin-c5ghg                          3/3     Running   0          24s
csi-rbdplugin-d4xpm                          3/3     Running   0          24s
csi-rbdplugin-pgqzg                          3/3     Running   0          24s
csi-rbdplugin-provisioner-86b684cdf7-6tq6d   7/7     Running   0          84m
csi-rbdplugin-provisioner-86b684cdf7-hgrrf   7/7     Running   0          84m
csi-rbdplugin-provisioner-86b684cdf7-s7lj8   7/7     Running   0          84m

检查 service

# kubectl get service
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
csi-metrics-rbdplugin       ClusterIP   10.98.91.245    <none>        8080/TCP   71s
csi-rbdplugin-provisioner   ClusterIP   10.96.131.138   <none>        8080/TCP   85m
kubernetes                  ClusterIP   10.96.0.1       <none>        443/TCP    2d2h=

测试

secret 部署

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: AQAjFtVjmlXdNhAAqM1DHhf2gZEUO2eo05Ka+w==
  encryptionPassphrase: rbd_k8s_passphrase


# kubectl apply -f secret.yaml
secret/csi-rbd-secret created

部署 storageclass

# cat csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: aa0e0954-4abd-41d3-9fcf-b38a8159167a
   pool: kubernetes
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: default
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard



# kubectl apply -f csi-rbd-sc.yaml
storageclass.storage.k8s.io/csi-rbd-sc created


# kubectl get sc
NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   77s

创建 pvc

# cat  pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 50Gi
  storageClassName: csi-rbd-sc

# kubectl apply -f pvc.yaml
persistentvolumeclaim/raw-block-pvc created

验证

# kubectl  get sc
NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   3h14m

# kubectl get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
raw-block-pvc   Bound    pvc-8d6892fe-e730-46d3-9776-7e82d90841a6   50Gi       RWO            csi-rbd-sc     2s

验证导入数据至 datavolume

cat   import-kubevirt-centos7-datavolume.yaml

# This example assumes you are using a default storage class
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: centos7-volume
spec:
  source:
      http:
         url: "http://mirrors.vclound.com/vclound/tmp/openstack/centos7.qcow2"
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: "40Gi"
    storageClassName: "csi-rbd-sc"         <- 必须要定义默认的  storageClassName

导入前验证

导入前 pv 信息

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
pvc-b55fbcf7-5c2c-439e-bf96-1231a885d7b0   40Gi       RWO            Delete           Bound    default/centos7-image1   csi-rbd-sc              2d23h

导入前 pvc 信息

# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
centos7-image1   Bound    pvc-b55fbcf7-5c2c-439e-bf96-1231a885d7b0   40Gi       RWO            csi-rbd-sc     2d23h

导入数据

# kubectl apply -f import-kubevirt-centos7-datavolume.yaml
datavolume.cdi.kubevirt.io/centos7-volume created

会有个 pendding 状态 Pod 用于数据导入， 数据导入完成后会被自动删除
# kubectl get pod
NAME                                         READY   STATUS    RESTARTS   AGE
importer-centos7-volume                      0/1     Pending   0          3s 

验证 pod 状态
Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        19s (x3 over 23s)  default-scheduler        0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Normal   Scheduled               14s                default-scheduler        Successfully assigned default/importer-centos7-volume to ns-yun-020066.vclound.com
  Normal   SuccessfulAttachVolume  13s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-a0466a0b-7541-429a-82d5-b4ec55b60335"
  Normal   Pulled                  4s                 kubelet                  Container image "quay.io/kubevirt/cdi-importer:v1.55.2" already present on machine
  Normal   Created                 4s                 kubelet                  Created container importer
  Normal   Started                 4s                 kubelet                  Started container importer

验证

验证 pv

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
pvc-a0466a0b-7541-429a-82d5-b4ec55b60335   40Gi       RWO            Delete           Bound    default/centos7-volume   csi-rbd-sc              3m4s
pvc-b55fbcf7-5c2c-439e-bf96-1231a885d7b0   40Gi       RWO            Delete           Bound    default/centos7-image1   csi-rbd-sc              2d23h

验证 pvc

# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
centos7-image1   Bound    pvc-b55fbcf7-5c2c-439e-bf96-1231a885d7b0   40Gi       RWO            csi-rbd-sc     2d23h
centos7-volume   Bound    pvc-a0466a0b-7541-429a-82d5-b4ec55b60335   40Gi       RWO            csi-rbd-sc     3m5s