openstack 与 ceph (monitor初始化)

最新推荐文章于 2026-05-08 09:34:26 发布

原创最新推荐文章于 2026-05-08 09:34:26 发布 · 2.7k 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#openstack #ceph #mon #mongodb

ceph 同时被 2 个专栏收录

64 篇文章

订阅专栏

openstack

53 篇文章

订阅专栏

该博客详细介绍了如何在CentOS 7.1环境下初始化并部署ceph监控节点的步骤，包括操作系统初始化、软件安装、配置文件创建、密钥对生成、监控节点添加、数据目录创建、监控进程定义以及启动服务等关键操作。

Monitor 节点初始化

目的

利用三个节点部署 ceph mon 节点

步骤

1. 操作系统初始化

当前环境中, ceph 存储集群统一部署 centos 7.1 操作系统, 并在 /etc/hosts 中登记集群中的服务器主机名, 并完成时间同步

主机名验证

[root@xx-xxx-xxxx-xxxxxxx15-128055 ~]# hostname
xx-xxx-xxxx-xxxxxxx15-128055.vclound.com
[root@xx-xxx-xxxx-xxxxxxx17-128057 ~]# hostname
xx-xxx-xxxx-xxxxxxx17-128057.vclound.com
[root@xx-xxx-xxxx-xxxxxxx24-128074 ~]# hostname
xx-xxx-xxxx-xxxxxxx24-128074.vclound.com

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
XXX.XX.128.55 xx-xxx-xxxx-xxxxxxx15-128055.vclound.com xx-xxx-xxxx-xxxxxxx15-128055
XXX.XX.128.56 xx-xxx-xxxx-xxxxxxx16-128056.vclound.com xx-xxx-xxxx-xxxxxxx16-128056
XXX.XX.128.57 xx-xxx-xxxx-xxxxxxx17-128057.vclound.com xx-xxx-xxxx-xxxxxxx17-128057
XXX.XX.128.73 xx-xxx-xxxx-xxxxxxx23-128073.vclound.com xx-xxx-xxxx-xxxxxxx23-128073
XXX.XX.128.74 xx-xxx-xxxx-xxxxxxx24-128074.vclound.com xx-xxx-xxxx-xxxxxxx24-128074
XXX.XX.128.75 xx-xxx-xxxx-xxxxxxx25-128075.vclound.com xx-xxx-xxxx-xxxxxxx25-128075
XXX.XX.128.76 xx-xxx-xxxx-xxxxxxx26-128076.vclound.com xx-xxx-xxxx-xxxxxxx26-128076

时间同步验证

[root@xx-xxx-xxxx-xxxxxxx24-128074 ~]# chronyc tracking
Reference ID    : 10.199.129.21 (hh-yun-129021.vclound.com)
Stratum         : 12
Ref time (UTC)  : Tue Jul 21 07:45:32 2015
System time     : 0.000002526 seconds fast of NTP time
Last offset     : 0.000005405 seconds
RMS offset      : 0.007192081 seconds
Frequency       : 18.428 ppm slow
Residual freq   : 0.001 ppm
Skew            : 0.060 ppm
Root delay      : 0.000141 seconds
Root dispersion : 0.011696 seconds
Update interval : 386.4 seconds
Leap status     : Normal

2. ceph 软件安装

**注: **

假如之前安装过 ceph, 并计划清除之前的集群配置, 那么执行下面命令可以把 ceph 配置进行安装
清除方法: ceph-deploy purgedata {node-name}, ceph-deploy purge {node-name}

yum 源

yum install -y epel-release.noarch
echo "sslverify=0" >> /etc/yum.conf

ceph 软件安装

yum install -y ceph* bash-completion

3. 创建配置文件, 默认是 /etc/ceph/ceph.conf

vim /etc/ceph/ceph.conf

[global]
fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9
mon initial members = xx-xxx-xxxx-xxxxxxx15-128055, xx-xxx-xxxx-xxxxxxx17-128057, xx-xxx-xxxx-xxxxxxx24-128074
mon host = XXX.XX.128.55, XXX.XX.128.57, XXX.XX.128.74
public network = XXX.XX.128.0/21
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
filestore xattr use omap = true
osd pool default size = 3
osd pool default min size = 1
osd pool default pg num = 1024    <- pg num 值大小参考其他文章
osd pool default pgp num = 1024
osd crush chooseleaf type = 1

4. 创建集群密钥对创建监控密钥

[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
creating /etc/ceph/ceph.mon.keyring
 
[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# cat /etc/ceph/ceph.mon.keyring
[mon.]
				key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==
				caps mon = "allow *"

注意:

ceph-0.87 版本需要把
文件 /etc/ceph/ceph.mon.keyring 重命名为
/etc/ceph/ceph.keyring 或
/etc/ceph/keyring 或 /etc/ceph/ceph.mon.(hostname).keyring
否则无法执行注册

5. 生成管理员密钥, 生成 client.admin 用户并增加用户密钥对

[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
creating /etc/ceph/ceph.client.admin.keyring
 
[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 137 7月  23 16:20 /etc/ceph/ceph.client.admin.keyring
 
[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
				key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==
				auid = 0
				caps mds = "allow"
				caps mon = "allow *"
				caps osd = "allow *"

6. 增加 client.admin 密钥对到 ceph.mon.keyring 中

[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring
 
[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.mon.keyring
-rw------- 1 root root 137 7月  23 16:20 /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 214 7月  23 16:20 /etc/ceph/ceph.mon.keyring

注意

这里 /etc/ceph/ceph.mon.keyring 从之前的 77 字节变成 214 字节, 并自动添加如下的内容

[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]# cat /etc/ceph/ceph.mon.keyring
[mon.]
				key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==
				caps mon = "allow *"
[client.admin]
				key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==
				auid = 0
				caps mds = "allow"
				caps mon = "allow *"
				caps osd = "allow *"

7. 添加 mon 节点

使用主机名, IP 地址及 FSID 生成 monitor map 保存到 /etc/ceph/monmap 注意, 命令创建时候需要一次过把所有的 mon 节点都直接进行定义

[root@xx-xxx-xxxx-xxxxxxx15-128055 tmp]#  monmaptool --create --add  xx-xxx-xxxx-xxxxxxx15-128055 XXX.XX.128.55 --add xx-xxx-xxxx-xxxxxxx17-128057 XXX.XX.128.57 --add xx-xxx-xxxx-xxxxxxx24-128074 XXX.XX.128.74 --fsid dc4f91c1-8792-4948-b68f-2fcea75f53b9 /etc/ceph/monmap
monmaptool: monmap file /etc/ceph/monmap
monmaptool: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
monmaptool: writing epoch 0 to /etc/ceph/monmap (3 monitors)

8. 在对应的监控主机上创建默认数据目录

[root@xx-xxx-xxxx-xxxxxxx15-128055 /]# mkdir -p /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx15-128055
[root@xx-xxx-xxxx-xxxxxxx17-128057 ~]# mkdir -p /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx17-128057
[root@xx-xxx-xxxx-xxxxxxx24-128074 ~]# mkdir -p /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx24-128074

9. 定义监控进程与监控 MAP 及密钥对

[root@xx-xxx-xxxx-xxxxxxx15-128055 ~]# ceph-mon --mkfs -i xx-xxx-xxxx-xxxxxxx15-128055 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx15-128055 for mon.xx-xxx-xxxx-xxxxxxx15-128055

同步 mon 密钥与配置
把 /etc/ceph 目录下的文件: ceph.conf ceph.client.admin.keyring ceph.mon.keyring monmap 同步到另外两台 mon 节点中

[root@xx-xxx-xxxx-xxxxxxx15-128055 ceph]# scp ceph.* monmap xx-xxx-xxxx-xxxxxxx17-128057:/etc/ceph/.
ceph.client.admin.keyring                                                                   100%  137     0.1KB/s   00:00
ceph.conf                                                                                   100%  544     0.5KB/s   00:00
ceph.mon.keyring                                                                            100%  214     0.2KB/s   00:00
monmap                                                                                      100%  550     0.5KB/s   00:00
 
[root@xx-xxx-xxxx-xxxxxxx15-128055 ceph]# scp ceph.* monmap xx-xxx-xxxx-xxxxxxx24-128074:/etc/ceph/.
ceph.client.admin.keyring                                                                   100%  137     0.1KB/s   00:00
ceph.conf                                                                                   100%  544     0.5KB/s   00:00
ceph.mon.keyring                                                                            100%  214     0.2KB/s   00:00
monmap                                                                                      100%  550     0.5KB/s   00:00

初始化 MON 节点数据
为另外两台 mon 节点进行监控数据初始化

[root@xx-xxx-xxxx-xxxxxxx17-128057 ~]# ceph-mon --mkfs -i xx-xxx-xxxx-xxxxxxx17-128057 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx17-128057 for mon.xx-xxx-xxxx-xxxxxxx17-128057
[root@xx-xxx-xxxx-xxxxxxx24-128074 ~]# ceph-mon --mkfs -i xx-xxx-xxxx-xxxxxxx24-128074 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx24-128074 for mon.xx-xxx-xxxx-xxxxxxx24-128074

10. 启动三台 mon 节点

[root@xx-xxx-xxxx-xxxxxxx15-128055 ~]# touch /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx15-128055/done /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx15-128055/sysvinit
[root@xx-xxx-xxxx-xxxxxxx17-128057 ~]# touch /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx17-128057/done /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx17-128057/sysvinit
[root@xx-xxx-xxxx-xxxxxxx24-128074 ~]# touch /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx24-128074/done /var/lib/ceph/mon/ceph-xx-xxx-xxxx-xxxxxxx24-128074/sysvinit

启动 mon 服务

[root@xx-xxx-xxxx-xxxxxxx15-128055 ~]# /etc/init.d/ceph start mon.xx-xxx-xxxx-xxxxxxx15-128055
=== mon.xx-xxx-xxxx-xxxxxxx15-128055 ===
Starting Ceph mon.xx-xxx-xxxx-xxxxxxx15-128055 on xx-xxx-xxxx-xxxxxxx15-128055...
Running as unit run-53061.service.
Starting ceph-create-keys on xx-xxx-xxxx-xxxxxxx15-128055...
 
[root@xx-xxx-xxxx-xxxxxxx17-128057 ~]# /etc/init.d/ceph start mon.xx-xxx-xxxx-xxxxxxx17-128057
=== mon.xx-xxx-xxxx-xxxxxxx17-128057 ===
Starting Ceph mon.xx-xxx-xxxx-xxxxxxx17-128057 on xx-xxx-xxxx-xxxxxxx17-128057...
Running as unit run-52736.service.
Starting ceph-create-keys on xx-xxx-xxxx-xxxxxxx17-128057...
 
[root@xx-xxx-xxxx-xxxxxxx24-128074 ~]# /etc/init.d/ceph start mon.xx-xxx-xxxx-xxxxxxx24-128074
=== mon.xx-xxx-xxxx-xxxxxxx24-128074 ===
Starting Ceph mon.xx-xxx-xxxx-xxxxxxx24-128074 on xx-xxx-xxxx-xxxxxxx24-128074...
Running as unit run-51440.service.
Starting ceph-create-keys on xx-xxx-xxxx-xxxxxxx24-128074...