ubuntu 20.04 部署 Ceph 15 (Octopus) 存儲

基礎環境

2.環境配置

2.1 OS安裝配置（所有節點）

2.1.1 主機名和IP規劃

步驟1 規劃表

步驟2 配置主機名

hostnamectl --static set-hostname k8s0{1..5}

步驟3 配置文件/etc/hosts

172.27.9.211 k8s01
172.27.9.212 k8s02
172.27.9.213 k8s03
172.27.9.214 k8s04
172.27.9.215 k8s05

2.1.2 時間同步服務

備註：這裏將 k8s01 作爲時間服務器

步驟1 安裝chrony

步驟2 配置chtony

步驟3 所有節點時間同步

2.1.3 配置ssh免密

步驟1 所有節點配置ssh容許root登陸

// 編輯/etc/ssh/sshd_config，修改PermitRootLogin爲yes，修改完後，執行systemctl restart sshd.service或者/etc/init.d/ssh restart重啓ssh服務。

PermitRootLogin yes

步驟2 生成祕鑰

ssh-keygen -t rsa -b 4096 -N '' -f ~/.ssh/id_rsa

步驟3 ssh配置

tee -a ~/.ssh/config<<EOF
Host *
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
    IdentitiesOnly yes
    ConnectTimeout 0
    ServerAliveInterval 300
EOF

步驟4 公鑰發佈

for i in {1..5}; do ssh-copy-id k8s0$i; done;

步驟 4 在任意節點執行：ssh $HOSTNAME（osd1爲具體主機名），遍歷所有主機，均能夠免密碼ssh登陸到對應機器，說明配置成功。

2.1.4 系統參數調整

2.2 依賴軟件安裝

2.2.1 安裝Python3(所有節點)

2.2.2 安裝Cephadm(所有節點)

apt-get install cephadm

3.在Ubuntu 20.04上部署Ceph 15（Octopus）存儲集羣

3.1 引導新的集羣

步驟1 創建配置文件目錄（重建時需要刪除改目錄內的文件）

mkdir -p /etc/ceph

步驟2 引導集羣（創建第一個 mon 節點）

cephadm bootstrap \
  --mon-ip 172.27.9.211 \
  --initial-dashboard-user admin \
  --initial-dashboard-password Str0ngAdminP@ssw0rd

INFO:cephadm:Verifying podman|docker is present...
INFO:cephadm:Verifying lvm2 is present...
INFO:cephadm:Verifying time synchronization is in place...
INFO:cephadm:Unit chrony.service is enabled and running
INFO:cephadm:Repeating the final host check...
INFO:cephadm:podman|docker (/usr/bin/docker) is present
INFO:cephadm:systemctl is present
INFO:cephadm:lvcreate is present
INFO:cephadm:Unit chrony.service is enabled and running
INFO:cephadm:Host looks OK
INFO:root:Cluster fsid: 65dfd2dc-e27d-11ea-bcc8-41f97ad09a47
INFO:cephadm:Verifying IP 172.27.9.211 port 3300 ...
INFO:cephadm:Verifying IP 172.27.9.211 port 6789 ...
INFO:cephadm:Mon IP 172.27.9.211 is in CIDR network 172.27.8.0/22
INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container...
INFO:cephadm:Extracting ceph user uid/gid from container image...
INFO:cephadm:Creating initial keys...
INFO:cephadm:Creating initial monmap...
INFO:cephadm:Creating mon...
INFO:cephadm:Waiting for mon to start...
INFO:cephadm:Waiting for mon...
INFO:cephadm:Assimilating anything we can from ceph.conf...
INFO:cephadm:Generating new minimal ceph.conf...
INFO:cephadm:Restarting the monitor...
INFO:cephadm:Setting mon public_network...
INFO:cephadm:Creating mgr...
INFO:cephadm:Wrote keyring to /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Wrote config to /etc/ceph/ceph.conf
INFO:cephadm:Waiting for mgr to start...
INFO:cephadm:Waiting for mgr...
INFO:cephadm:mgr not available, waiting (1/10)...
INFO:cephadm:mgr not available, waiting (2/10)...
INFO:cephadm:mgr not available, waiting (3/10)...
INFO:cephadm:mgr not available, waiting (4/10)...
INFO:cephadm:mgr not available, waiting (5/10)...
INFO:cephadm:Enabling cephadm module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 5...
INFO:cephadm:Setting orchestrator backend to cephadm...
INFO:cephadm:Generating ssh key...
INFO:cephadm:Wrote public SSH key to to /etc/ceph/ceph.pub
INFO:cephadm:Adding key to root@localhost's authorized_keys...
INFO:cephadm:Adding host k8s01...
INFO:cephadm:Deploying mon service with default placement...
INFO:cephadm:Deploying mgr service with default placement...
INFO:cephadm:Deploying crash service with default placement...
INFO:cephadm:Enabling mgr prometheus module...
INFO:cephadm:Deploying prometheus service with default placement...
INFO:cephadm:Deploying grafana service with default placement...
INFO:cephadm:Deploying node-exporter service with default placement...
INFO:cephadm:Deploying alertmanager service with default placement...
INFO:cephadm:Enabling the dashboard module...
INFO:cephadm:Waiting for the mgr to restart...
INFO:cephadm:Waiting for Mgr epoch 13...
INFO:cephadm:Generating a dashboard self-signed certificate...
INFO:cephadm:Creating initial admin user...
INFO:cephadm:Fetching dashboard port number...
INFO:cephadm:Ceph Dashboard is now available at:

             URL: https://k8s01:8443/
            User: admin
        Password: Str0ngAdminP@ssw0rd

INFO:cephadm:You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid 65dfd2dc-e27d-11ea-bcc8-41f97ad09a47 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

INFO:cephadm:Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/master/mgr/telemetry/

INFO:cephadm:Bootstrap complete.

3.2 安裝ceph工具（所有節點）

cephadm add-repo --release octopus
cephadm install ceph-common

備註：裏面包含了所有的ceph命令，其中包括ceph，rbd，mount.ceph（用於安裝CephFS文件系統）等。

3.3 部署 mon 節點

步驟1 複製Ceph SSH密鑰

ssh-copy-id -f -i /etc/ceph/ceph.pub root@k8s02
ssh-copy-id -f -i /etc/ceph/ceph.pub root@k8s05

步驟2 禁用自動監視器部署：

ceph orch apply mon --unmanaged

步驟3 增加節點到集羣

ceph orch host add k8s02
ceph orch host add k8s05

步驟4 標記mon節點

ceph orch host label add k8s01 mon
ceph orch host label add k8s02 mon
ceph orch host label add k8s05 mon

//查看標記結果
ceph orch host ls

HOST   ADDR   LABELS  STATUS
k8s01  k8s01  mon
k8s02  k8s02  mon
k8s05  k8s05  mon

步驟5 配置cephadm根據標籤部署監視器

ceph orch apply mon label:mon

步驟6 確保將監視器應用於這三個主機

ceph orch apply mon "k8s01,k8s02,k8s05"

3.4 部署Ceph OSD節點

步驟1 複製Ceph SSH密鑰

ssh-copy-id -f -i /etc/ceph/ceph.pub root@k8s03
ssh-copy-id -f -i /etc/ceph/ceph.pub root@k8s04

步驟2 增加節點到集羣

ceph orch host add k8s03
ceph orch host add k8s04

步驟3 標記osd節點

ceph orch host label add k8s03 osd
ceph orch host label add k8s04 osd

步驟4 查看標記結果

ceph orch host ls

HOST   ADDR   LABELS  STATUS
k8s01  k8s01  mon
k8s02  k8s02  mon
k8s03  k8s03  osd
k8s04  k8s04  osd
k8s05  k8s05  mon

查看存儲節點上的所有設備(稍等片刻)

節點：k8s01

ceph orch device ls

HOST   PATH      TYPE   SIZE  DEVICE                               AVAIL  REJECT REASONS
k8s01  /dev/sda  hdd   2794G  ST3000DM001-1ER1_Z500SQMS            True
k8s01  /dev/sdc  hdd   2794G  HGST_HUS724030AL_PN2231P8HU4BHR      True
k8s01  /dev/sdd  hdd   2794G  ST3000NM0033-9ZM_S1Z2EZJ3            True
k8s01  /dev/sdb  ssd    111G  INTEL_SSDSC2BB12_PHWL516200VK120LGN  False  locked
k8s02  /dev/sda  hdd   2794G  ST3000DM001-1ER1_Z500ST11            True
k8s02  /dev/sdb  hdd   2794G  HGST_HUS724030AL_PN2231P8J3GHAR      True
k8s02  /dev/sdc  ssd    111G  INTEL_SSDSC2BB12_PHWL516302PB120LGN  False  locked
k8s05  /dev/sdb  hdd    931G  ST1000NM0033-9ZM_Z1W4JJCN            True
k8s05  /dev/sdc  hdd    931G  ST1000NM0033-9ZM_Z1W33ABA            True
k8s05  /dev/sda  hdd    931G  ST1000NM0033-9ZM_Z1W333J0            False  locked
k8s03  /dev/sda  hdd   2794G  ST3000DM001-1ER1_Z500H0EK            True
k8s03  /dev/sdb  hdd   2794G  ST3000DM001-1ER1_Z500FM04            True
k8s03  /dev/sdc  hdd   2794G  ST3000DM001-1ER1_Z500GZWR            True
k8s03  /dev/sde  hdd   2794G  ST3000DM001-1ER1_Z500GARV            True
k8s03  /dev/sdf  hdd   2794G  ST3000DM001-1ER1_Z500FLGM            True
k8s03  /dev/sdg  hdd   2794G  ST3000DM001-1ER1_Z500GXTB            True
k8s03  /dev/sdd  ssd    111G  INTEL_SSDSC2BW12_PHDA4395000G1207GN  False  locked
k8s03  /dev/sdh  hdd   2794G  ST3000DM001-1ER1_Z500GARJ            False  locked
k8s04  /dev/sdb  hdd   2794G  ST3000NM0033-9ZM_S1Z2EGG4            True
k8s04  /dev/sdc  hdd   2794G  ST3000NM0033-9ZM_Z1Z8DSDW            True
k8s04  /dev/sdd  hdd   2794G  ST3000NM0033-9ZM_S1Z2EYLQ            True
k8s04  /dev/sde  ssd    447G  INTEL_SSDSC2KB48_PHYF9184057M480BGN  True
k8s04  /dev/sdf  ssd    447G  INTEL_SSDSC2KB48_PHYF9086025G480BGN  True
k8s04  /dev/sdg  hdd   2794G  ST3000NM0033-9ZM_S1Z2EGFC            True
k8s04  /dev/sdh  hdd   2794G  ST3000NM0033-9ZM_S1Z2EH3P            True
k8s04  /dev/sdi  hdd   2794G  ST3000NM0033-9ZM_S1Z2EGMS            True
k8s04  /dev/sda  ssd    223G  INTEL_SSDSC2KB24_PHYF906604FP240AGN  False  locked
k8s04  /dev/sdj  hdd   2794G  ST3000NM0033-9ZM_S1Z2EGZN            False  locked

Ceph拒絕在不可用的設備上配置OSD,滿足以下所有條件，則認爲存儲設備可用：
設備必須沒有分區。
設備不得具有任何LVM狀態。
不得安裝設備。
該設備不得包含文件系統。
該設備不得包含Ceph BlueStore OSD。
設備必須大於5 GB。

部署OSD

節點：k8s01

從特定主機上的特定設備創建OSD(本文方式如下)

ceph orch daemon add osd k8s01:/dev/sda
ceph orch daemon add osd k8s02:/dev/sda
ceph orch daemon add osd k8s03:/dev/sda
ceph orch daemon add osd k8s04:/dev/sdb
ceph orch daemon add osd k8s04:/dev/sdc
ceph orch daemon add osd k8s04:/dev/sdd

補充:

Ceph使用任何可用和未使用的存儲設備(不推薦)

ceph orch apply osd --all-available-devices

使用 OSD服務規範基於設備的屬性來描述要使用的設備，例如設備類型（SSD或HDD），設備型號名稱，大小或設備所在的主機

ceph orch apply osd -i spec.yml

檢查ceph狀態

ceph -s

  cluster:
    id:     d750d2c6-e288-11ea-a4d6-114e8a813018
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum k8s01,k8s03,k8s04 (age 11m)
    mgr: k8s01.bcvydy(active, since 34m), standbys: k8s02.zbzfaf
    osd: 6 osds: 6 up (since 76s), 6 in (since 76s)

  data:
    pools:   1 pools, 1 pgs
    objects: 1 objects, 0 B
    usage:   6.0 GiB used, 16 TiB / 16 TiB avail
    pgs:     1 active+clean

訪問Ceph Dashboard

查看mgr節點

ceph -s

  cluster:
    id:     d750d2c6-e288-11ea-a4d6-114e8a813018
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum k8s01,k8s03,k8s04 (age 11m)
    mgr: k8s01.bcvydy(active, since 34m), standbys: k8s02.zbzfaf
    osd: 6 osds: 6 up (since 76s), 6 in (since 76s)

  data:
    pools:   1 pools, 1 pgs
    objects: 1 objects, 0 B
    usage:   6.0 GiB used, 16 TiB / 16 TiB avail
    pgs:     1 active+clean

備註：Ceph儀表板現在在活動MGR服務器的地址處可用，如 mgr: k8s01.bcvydy(active, since 35m), standbys: k8s02.zbzfaf

瀏覽器訪問(初次需要修改密碼)

URL: https://k8s01:8443/
User: admin
Password: Str0ngAdminP@ssw0rd

刪除pool

查看pool

ceph osd lspools

1 device_health_metrics
2 kube

刪除pool（刪除 kube 爲例）

ceph osd pool delete kube kube --yes-i-really-really-mean-it

Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool

參數調整

ceph tell mon.* injectargs --mon-allow-pool-delete=true

mon_allow_pool_delete = 'true'
{}

再次執行刪除操作

ceph osd pool delete kube kube --yes-i-really-really-mean-it

pool 'kube' removed

刪除OSD

查看刪除前的osd進程狀態

ceph osd tree

ID   CLASS  WEIGHT    TYPE NAME       STATUS  REWEIGHT  PRI-AFF
 -1         11.35255  root default
 -3          2.72899      host k8s01
  0    hdd   2.72899          osd.0       up   1.00000  1.00000
 -5          2.72899      host k8s02
  1    hdd   2.72899          osd.1       up   1.00000  1.00000
 -7          2.72899      host k8s03
  2    hdd   2.72899          osd.2       up   1.00000  1.00000
-13          3.16559      host k8s04
  4    hdd   2.72899          osd.4       up   1.00000  1.00000
  3    ssd   0.43660          osd.3       up   1.00000  1.00000

手動移除osd守護程序

ceph orch daemon rm osd.{1..4} --force

查看守護進程狀態

ceph osd tree
ID   CLASS  WEIGHT    TYPE NAME       STATUS  REWEIGHT  PRI-AFF
 -1         11.35255  root default
 -3          2.72899      host k8s01
  0    hdd   2.72899          osd.0       up   1.00000  1.00000
 -5          2.72899      host k8s02
  1    hdd   2.72899          osd.1     down   1.00000  1.00000
 -7          2.72899      host k8s03
  2    hdd   2.72899          osd.2     down   1.00000  1.00000
-13          3.16559      host k8s04
  4    hdd   2.72899          osd.4     down   1.00000  1.00000
  3    ssd   0.43660          osd.3     down   1.00000  1.00000

登錄到 osd.ID 相應主機中手動的將他們的systemd守護進程disable。

//k8s04
systemctl disable --now  ceph-osd@4
//k8s03
systemctl disable --now  ceph-osd@2

刪除CRUSH圖中所對應的OSD節點

ceph osd crush remove osd.1

查看CRUSH中的刪除結果

ceph osd tree

ID   CLASS  WEIGHT   TYPE NAME       STATUS  REWEIGHT  PRI-AFF
 -1         2.72899  root default
 -3         2.72899      host k8s01
  0    hdd  2.72899          osd.0       up   1.00000  1.00000
 -5               0      host k8s02
 -7               0      host k8s03
-13               0      host k8s04
  1               0  osd.1             down   1.00000  1.00000
  2               0  osd.2             down   1.00000  1.00000
  3               0  osd.3             down   1.00000  1.00000
  4               0  osd.4             down   1.00000  1.00000

刪除osd節點

ceph osd rm osd.4 osd.5

刪除osd用戶

//查看用戶
ceph auth ls | grep ^osd -A4

installed auth entries:

osd.0
        key: AQCTPT5fxIcYMxAAGtyz5nHCca8SlBQ2JX/eHQ==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.1
        key: AQChPT5fkM9oDhAAmxojG1xFcYxTdiICg62Biw==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.2
        key: AQCvPT5fb2oAFxAAg7UX3LUfulC+7mnI4lUsEw==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.3
        key: AQC+PT5foF5eJRAAbgv1S1oWeLsALzTTHEBo3A==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.4
        key: AQDLPT5fo/SlBhAAoS/nZoLHnk8bZXo0O0qeuw==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *

//刪除用戶
ceph auth rm osd.4

解除Ceph對磁盤的佔用

//查詢磁盤DM狀態並移除編碼(登陸到不同節點)
dmsetup status

ceph--fda003aa--4bd1--4cdb--85ad--e69c6440605a-osd--block--926d2232--28d0--4029--8130--721d45e04bdc: 0 5860524032 linear

//刪除磁盤DM標記
dmsetup remove ceph--fda003aa--4bd1--4cdb--85ad--e69c6440605a-osd--block--926d2232--28d0--4029--8130--721d45e04bdc

//清理磁盤文件系統
wipefs -a /dev/sda

/dev/sda: 8 bytes were erased at offset 0x00000218 (LVM2_member): 4c 56 4d 32 20 30 30 31

//查看結果
lsblk -f

備註：完成的將 OSD 節點從集羣中移除。可以使用 lsblk 工具或稍後一段時間後在擁有集羣 admin 權限的主機上執行 ceph orch device ls 命令查看這兩個 OSD 節點所使用的磁盤是否是可用狀態。現在可以將這兩個磁盤取出或者在這兩塊磁盤上重新部署 OSD 節點。

本文參考地址： https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-servers/ https://ceph.readthedocs.io/en/latest/cephadm/install/ https://blog.yeefire.com/2020_07/ceph_osd_rm.html