创建cephfs文件系统时遇到1 pool(s) have non-power-of-two pg_num
在创建前期一切正常集群状态也是健康状态
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# su ceph-admin
[ceph-admin@ceph-node1 root]$ ls
ls: cannot open directory .: Permission denied
[ceph-admin@ceph-node1 root]$ cd
[ceph-admin@ceph-node1 ~]$ cd cluster/
[ceph-admin@ceph-node1 cluster]$ ls
ceph.bootstrap-mds.keyring ceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph-deploy-ceph.log
ceph.bootstrap-mgr.keyring ceph.bootstrap-rgw.keyring ceph.conf ceph.mon.keyring
[ceph-admin@ceph-node1 cluster]$ ll
total 208
-rw-------. 1 ceph-admin ceph-admin 113 Mar 28 00:15 ceph.bootstrap-mds.keyring
-rw-------. 1 ceph-admin ceph-admin 113 Mar 28 00:15 ceph.bootstrap-mgr.keyring
-rw-------. 1 ceph-admin ceph-admin 113 Mar 28 00:15 ceph.bootstrap-osd.keyring
-rw-------. 1 ceph-admin ceph-admin 113 Mar 28 00:15 ceph.bootstrap-rgw.keyring
-rw-------. 1 ceph-admin ceph-admin 151 Mar 28 00:15 ceph.client.admin.keyring
-rw-rw-r--. 1 ceph-admin ceph-admin 268 Mar 28 00:10 ceph.conf
-rw-rw-r--. 1 ceph-admin ceph-admin 181063 Mar 28 00:17 ceph-deploy-ceph.log
-rw-------. 1 ceph-admin ceph-admin 73 Mar 28 00:10 ceph.mon.keyring
[ceph-admin@ceph-node1 cluster]$ ceph-deploy mds create ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph-admin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create ceph-node1 ceph-node2 ceph-node3
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7ff0e6531fc8>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mds at 0x7ff0e6993398>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] mds : [('ceph-node1', 'ceph-node1'), ('ceph-node2', 'ceph-node2'), ('ceph-node3', 'ceph-node3')]
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-node1:ceph-node1 ceph-node2:ceph-node2 ceph-node3:ceph-node3
[ceph-node1][DEBUG ] connection detected need for sudo
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][WARNIN] mds keyring does not exist yet, creating one
[ceph-node1][DEBUG ] create a keyring file
[ceph-node1][DEBUG ] create path if it doesn't exist
[ceph-node1][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-node1 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-node1/keyring
[ceph-node1][INFO ] Running command: sudo systemctl enable ceph-mds@ceph-node1
[ceph-node1][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-node1.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-node1][INFO ] Running command: sudo systemctl start ceph-mds@ceph-node1
[ceph-node1][INFO ] Running command: sudo systemctl enable ceph.target
[ceph-node2][DEBUG ] connection detected need for sudo
[ceph-node2][DEBUG ] connected to host: ceph-node2
[ceph-node2][DEBUG ] detect platform information from remote host
[ceph-node2][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-node2
[ceph-node2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node2][WARNIN] mds keyring does not exist yet, creating one
[ceph-node2][DEBUG ] create a keyring file
[ceph-node2][DEBUG ] create path if it doesn't exist
[ceph-node2][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-node2 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-node2/keyring
[ceph-node2][INFO ] Running command: sudo systemctl enable ceph-mds@ceph-node2
[ceph-node2][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-node2.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-node2][INFO ] Running command: sudo systemctl start ceph-mds@ceph-node2
[ceph-node2][INFO ] Running command: sudo systemctl enable ceph.target
[ceph-node3][DEBUG ] connection detected need for sudo
[ceph-node3][DEBUG ] connected to host: ceph-node3
[ceph-node3][DEBUG ] detect platform information from remote host
[ceph-node3][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-node3
[ceph-node3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node3][WARNIN] mds keyring does not exist yet, creating one
[ceph-node3][DEBUG ] create a keyring file
[ceph-node3][DEBUG ] create path if it doesn't exist
[ceph-node3][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-node3 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-node3/keyring
[ceph-node3][INFO ] Running command: sudo systemctl enable ceph-mds@ceph-node3
[ceph-node3][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@ceph-node3.service to /usr/lib/systemd/system/ceph-mds@.service.
[ceph-node3][INFO ] Running command: sudo systemctl start ceph-mds@ceph-node3
[ceph-node3][INFO ] Running command: sudo systemctl enable ceph.target
[ceph-admin@ceph-node1 cluster]$ ceph -s
cluster:
id: de9f4b5e-7b68-448d-b1d7-db1f233a1249
health: HEALTH_OK
services:
mon: 1 daemons, quorum ceph-node1 (age 6h)
mgr: ceph-node1(active, since 6h)
mds: 2 up:standby
osd: 3 osds: 3 up (since 6h), 3 in (since 6h)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 27 GiB / 30 GiB avail
pgs:
[ceph-admin@ceph-node1 cluster]$ ceph mds stat
3 up:standby
[ceph-admin@ceph-node1 cluster]$ ceph -s
cluster:
id: de9f4b5e-7b68-448d-b1d7-db1f233a1249
health: HEALTH_OK
services:
mon: 1 daemons, quorum ceph-node1 (age 6h)
mgr: ceph-node1(active, since 6h)
mds: 3 up:standby
osd: 3 osds: 3 up (since 6h), 3 in (since 6h)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 27 GiB / 30 GiB avail
pgs: 创建存储池时出现告警
命令为
ceph osd pool create cephfs_data 128
ceph osd pool create cephfs_metadata 60注释
这两条命令是用来在Ceph存储集群中创建存储池的，具体解释如下：
ceph osd pool create cephfs_data 128这条命令用于创建一个名为
cephfs_data的存储池，存储池中的pg_num（Placement Group数量）被设置为128。在Ceph中，每个Placement Group代表一组存储对象，并且负责管理这些对象的复制和分布策略。选择128作为pg_num通常是因为它是2的幂次方，这样有利于Ceph内部的数据分布和负载均衡，提高集群性能和效率。ceph osd pool create cephfs_metadata 60这条命令用于创建一个名为
cephfs_metadata的存储池，其pg_num被设置为60。在这个例子中，60不是一个2的幂次方，这可能导致Ceph集群出现警告，提示存储池的pg_num不是最优配置。在实际应用中，尤其是针对CephFS这样的文件系统场景，元数据存储池的pg_num也应该设置成2的幂次方，以保持最佳实践。在CephFS中，
cephfs_data和cephfs_metadata分别用于存储文件的实际数据内容和关于文件、目录的元数据信息。当准备部署CephFS文件系统时，通常会同时创建这两个池，并在后续步骤中将它们关联起来以构成完整的CephFS。
第二条命令开始告警
解决办法
在您的Ceph集群中，您已经成功创建了两个存储池：
cephfs_data和cephfs_metadata，这两个池分别用于存放CephFS文件系统的数据和元数据。不过，在创建cephfs_metadata池时，集群健康状态变成了HEALTH_WARN，原因是其中一个警告信息指出有一个池的pg_num不是2的幂次方。
在Ceph中，为了优化性能和数据分布，推荐每个存储池的pg_num（Placement Group数量）设置为2的幂次方，例如128、256、512等。非2的幂次方可能会导致数据分布不均匀，影响性能和效率。
要解决这个警告，您可以按照建议重新创建cephfs_metadata存储池，将其pg_num设置为合适的2的幂次方数值，比如64（因为60不是2的幂次方）：ceph osd pool create cephfs_metadata 64然后再次检查集群状态以确认警告是否已消除：
ceph -s接下来，如果您计划创建并使用CephFS文件系统，还需要继续执行创建CephFS的相关步骤，例如：
ceph fs new cephfs cephfs_metadata cephfs_data这将会创建一个新的CephFS文件系统，名为
cephfs，并关联刚刚创建的cephfs_metadata和cephfs_data存储池。
最后，根据需求在客户端节点上挂载CephFS，以便使用它作为文件系统。
执行解决方案第一条命令ceph osd pool create cephfs_metadata 64发现新的问题创建的cephfs_metadata存储池已经存在，并且之前由于pg_num值不是2的幂次方而引发了健康警告
[ceph-admin@ceph-node1 cluster]$ ceph osd pool create cephfs_metadata 64
pool 'cephfs_metadata' already exists解决办法
选择修改现有存储池的pg_num值使其成为2的幂次方。可以通过以下命令完成：ceph osd pool set cephfs_metadata pg_num 64执行完这个命令后，集群应该会自动调整该存储池的pg_num值，并在一段时间后（视集群大小和数据迁移情况而定）恢复到HEALTH_OK状态。您可以稍候执行ceph -s来检查集群健康状况是否恢复正常。
另外，请注意，若要在CephFS中使用这两个存储池，仍需按照之前所述步骤创建CephFS文件系统：ceph fs new cephfs cephfs_metadata cephfs_data
问题解决
[ceph-admin@ceph-node1 cluster]$ ceph osd pool set cephfs_metadata pg_num 64
set pool 2 pg_num to 64
[ceph-admin@ceph-node1 cluster]$ ceph -s
cluster:
id: de9f4b5e-7b68-448d-b1d7-db1f233a1249
health: HEALTH_OK
services:
mon: 1 daemons, quorum ceph-node1 (age 6h)
mgr: ceph-node1(active, since 6h)
mds: 3 up:standby
osd: 3 osds: 3 up (since 6h), 3 in (since 6h)
data:
pools: 2 pools, 192 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 27 GiB / 30 GiB avail
pgs: 1.042% pgs unknown
190 active+clean
2 unknown