ceph-iscsi原理及部署

作者：吳業亮

博客：wuyeliang.blog.csdn.net

一．架構及方案

Ceph Block框架

2. Ceph iSCSI基本框架

iSCSI gateway的實現主要有TGT && LIO兩種方式。

TGT
TGT：Linux target framework，爲創建、維護SCSI target 驅動（包括iSCSI、FC、SRP等）提供支持。

工作在用戶空間；
在Linux 2.6.38 版本後（含），內核集成了 Linux-IO Target。

在ceph集成原生iscsi之前，通常使用基於用戶空間的"scsi-target-utils"套件實現tgt。

基於centos7.x系列，安裝"scsi-target-utils"套件後，iscsi並不支持ceph rbd後端存儲（通過"tgtadm --lld iscsi --mode system --op show"查看），主要原因是redhat針對套件屏蔽了支持ceph rbd後端存儲的代碼。

解決方案：
通過"rbd map xxx"將ceph rbd掛載到本地後，再通過iscsi tgt的"direct-store"模式發佈ceph塊存儲。

方案缺點：
"rbd map xxx"掛載ceph rbd是通過"ceph rbd kernel module"的形式，tgt在用戶空間實現，導致發佈的ceph rbd在內核態與用戶態之間頻繁切換，影響性能。

LIO
LIO：Linux-IO Target，用軟件實現各種SCIS Target。

工作在內核空間；
支持較多傳輸協議，如Fibre Channel（Qlogic，linux3.5）、FCoE（linux3.0）、iSCSI（linux 3.1）、iSER (Mellanox InfiniBand，linux3.10), SRP (Mellanox InfiniBand，linux3.3), USB等；
對 iSCSI RFC 規範的支持非常好，包括完整的錯誤恢復都有支持；
從內核 3.17 開始引入用戶態後端支持，即 TCMU(Target Core Module in Userspace)
在Linux 2.6.38 版本後（含），內核集成了 Linux-IO Target。

本文主要介紹基於LIO的ceph原生iscsi 實現方式，LIO利用用戶空間直通（即TCMU）與ceph的librbd庫進行交互（tcmu-runner處理LIO TCM後端存儲的用戶空間端的守護進程，在內核之上多了一個用戶態的驅動層，這樣只需要根據tcmu的標準來對接接口即可，而不用去直接與內核進行交互），並將rbd image暴露給iSCSI客戶端。

二、前提條件
1、版本

Ceph Luminous 版本的集羣或者更新的版本
RHEL/CentOS 7.5或者Linux kernel v4.16或者更新版本的內核
其他控制軟件
targetcli-2.1.fb47 or newer package
python-rtslib-2.1.fb68 or newer package
tcmu-runner-1.4.0 or newer package
ceph-iscsi-3.2 or newer package

2、針對ceph-mon或osd節點，並沒有特殊的iscsi-gateway參數選項，但降低若干默認的檢測osd宕機時間，可以有效降低initiator的連接超時。可在ceph-mon節點修改ceph.conf文件後分發到所有節點，如：
方法一：

# 新增參數
[osd]
osd client watch timeout = 15
osd heartbeat grace = 20
osd heartbeat interval = 5

重啓服務

systemctl restart ceph.target

方法二：
命令修改方法

# ceph tell osd.* config set osd_client_watch_timeout 15
# ceph tell osd.* config set osd_heartbeat_grace 20
# ceph tell osd.* config set osd_heartbeat_interval 5

驗證方法：

[root@node2 ~]# ceph daemon osd.0 config get osd_client_watch_timeout
{
    "osd_client_watch_timeout": "15"
}
[root@node2 ~]#  ceph daemon osd.0 config get osd_heartbeat_grace
{
    "osd_heartbeat_grace": "20"
}
[root@node2 ~]# ceph daemon osd.0 config get osd_heartbeat_interval
{
    "osd_heartbeat_interval": "5"
}

三、自動化部署
環境
1、通過ceph-ansible安裝

# yum install ceph-ansible

其他ceph-ansible具體安裝參見

https://wuyeliang.blog.csdn.net/article/details/105344968

2、修改/etc/ansible/hosts，新增需要安裝iscsi-gateway的節點主機名

[iscsigws]
node1
node2
node3

3、安裝

# cd /usr/share/ceph-ansible
# ansible-playbook site.yml --limit iscsigws

4、查看狀態

# ceph -s
  cluster:
    id:     8789218f-5fca-4727-9c74-aff3c2cebeab
    health: HEALTH_OK
 
  services:
    mon:         3 daemons, quorum node1,node2,node3 (age 14m)
    mgr:         node2(active, since 14m), standbys: node1, node3
    mds:         cephfs:1 {0=node2=up:active} 2 up:standby
    osd:         48 osds: 48 up (since 14m), 48 in (since 65m)
    rgw:         3 daemons active (node1.rgw0, node2.rgw0, node3.rgw0)
    tcmu-runner: 2 daemons active (node1:wyl/disk, node2:wyl/disk)
 
  data:
    pools:   8 pools, 2200 pgs
    objects: 43.01k objects, 107 GiB
    usage:   65 GiB used, 9.3 TiB / 9.4 TiB avail
    pgs:     2200 active+clean
 
  io:
    client:   3.0 KiB/s rd, 2 op/s rd, 0 op/s wr

# gwcli ls
Warning: Could not load preferences file /root/.gwcli/prefs.bin.
o- / .................................................................................................... [...]
  o- cluster .................................................................................... [Clusters: 1]
  | o- ceph ....................................................................................... [HEALTH_OK]
  |   o- pools ..................................................................................... [Pools: 8]
  |   | o- .rgw.root ......................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 1536K]
  |   | o- cephfs_data ....................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 0.00Y]
  |   | o- cephfs_metadata ................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 1536K]
  |   | o- default.rgw.control ............................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 0.00Y]
  |   | o- default.rgw.log ................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 0.00Y]
  |   | o- default.rgw.meta ................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 384K]
  |   | o- rbd ................................................ [(x3), Commit: 0.00Y/3089638M (0%), Used: 192K]
  |   | o- wyl ........................................... [(x3), Commit: 0.00Y/3089638M (0%), Used: 17388864K]
  |   o- topology .......................................................................... [OSDs: 48,MONs: 3]
  o- disks .................................................................................. [0.00Y, Disks: 0]
  o- iscsi-targets .......................................................... [DiscoveryAuth: None, Targets: 0

5、後面具體的服務爲rbd-target-api，可以通過systemctl來管理

# systemctl <start|stop|restart|reload> rbd-target-api

6、登錄web創建iscsi設備

7、服務卸載

cd /usr/share/ceph-ansible/
ansible-playbook purge_gateways.yml

四、手動部署環境

1、安裝軟件
配置源

[root@node1 yum.repos.d]# cat ceph-iscsi.repo 
[ceph-iscsi]
name=ceph-iscsi noarch packages
baseurl=http://download.ceph.com/ceph-iscsi/3/rpm/el7/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
type=rpm-md

[ceph-iscsi-source]
name=ceph-iscsi source packages
baseurl=http://download.ceph.com/ceph-iscsi/3/rpm/el7/SRPMS
enabled=0
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
type=rpm-md
[root@node1 yum.repos.d]# cat python-rtslib-dev.repo
[python-rtslib]
name=python-rtslib packages for $basearch
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/$basearch
enabled=1
gpgcheck=0
type=rpm-md

[python-rtslib-noarch]
name=python-rtslib noarch packages
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/noarch
enabled=1
gpgcheck=0
type=rpm-md

[python-rtslib-source]
name=python-rtslib source packages
baseurl=https://2.chacra.ceph.com/r/python-rtslib/master/67eb1605c697b6307d8083b2962f5170db13d306/centos/7/flavors/default/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
[root@node1 yum.repos.d]# cat tcmu-runner-dev.repo
[tcmu-runner]
name=tcmu-runner packages for $basearch
baseurl=https://1.chacra.ceph.com/r/tcmu-runner/master/9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/$basearch
enabled=1
gpgcheck=0
type=rpm-md

[tcmu-runner-noarch]
name=tcmu-runner noarch packages
baseurl=https://1.chacra.ceph.com/r/tcmu-runner/master/9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/noarch
enabled=1
gpgcheck=0
type=rpm-md

[tcmu-runner-source]
name=tcmu-runner source packages
baseurl=https://1.chacra.ceph.com/r/tcmu-runner/master/9c84f7a4348ac326ac269fbdda507953dba6ec2c/centos/7/flavors/default/SRPMS
enabled=1
gpgcheck=0
type=rpm-md

ceph相關的ISCSI軟件包可以從redhat通過的源代碼進行編譯或者下載centos已經編譯好的。

# yum install ceph-iscsi tcmu-runner targetcli python-rtslib -y
# ceph osd pool create rbd 150 150
# ceph osd pool application enable rbd rbd --yes-i-really-mean-it

2 創建配置文件
創建/etc/ceph/iscsi-gateway.cfg ，此文件主要設置iscsi服務的網關。

[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph

# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring

# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.

# To support the API, the bear minimum settings are:
api_secure = false

# Additional API configuration options are as follows, defaults shown.
# api_user = admin
# api_password = admin
# api_port = 5001
trusted_ip_list = 192.168.70.81,192.168.70.82,192.168.70.83

3、同步文件到其他節點

# scp /etc/ceph/iscsi-gateway.cfg node2:/etc/ceph
# scp /etc/ceph/iscsi-gateway.cfg node3:/etc/ceph

4、啓動API服務

# systemctl daemon-reload
# systemctl enable rbd-target-api
# systemctl start rbd-target-api
# systemctl status rbd-target-api
● rbd-target-api.service - Ceph iscsi target configuration API
   Loaded: loaded (/usr/lib/systemd/system/rbd-target-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-05-31 11:35:04 CST; 4s ago
 Main PID: 25372 (rbd-target-api)
   CGroup: /system.slice/rbd-target-api.service
           └─25372 /usr/bin/python /usr/bin/rbd-target-api

May 31 11:35:04 node1 systemd[1]: Started Ceph iscsi target configuration API.
May 31 11:35:04 node1 systemd[1]: Starting Ceph iscsi target configuration API...
May 31 11:35:05 node1 rbd-target-api[25372]: Started the configuration object watcher
May 31 11:35:05 node1 rbd-target-api[25372]: Checking for config object changes every 1s
May 31 11:35:05 node1 rbd-target-api[25372]:  * Running on http://0.0.0.0:5000/

5、配置ISCSI服務

5.1 創建target

# gwcli
/> cd iscsi-targets
/iscsi-target> create iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
/iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/
/iscsi-target...-gw:iscsi-igw> cd gateways

5.2 創建ISCSI網關

/iscsi-target...-igw/gateways> create node1 192.168.70.81 
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...-igw/gateways> create node2 192.168.70.82
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...-igw/gateways> create node3 192.168.70.83 
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...-igw/gateways> ls
o- gateways .................................................................................................. [Up: 3/3, Portals: 3]
  o- node1 .................................................................................................. [192.168.70.81 (UP)]
  o- node2 .................................................................................................. [192.168.70.82 (UP)]
  o- node3 .................................................................................................. [192.168.70.83 (UP)]

如果操作系統非Centos或redhat，則需要加skipchecks=true參數。

/iscsi-target...-igw/gateways> create node1 192.168.70.81 skipchecks=true

遇到問題“The first gateway defined must be the local machine”，必須先創建本地節點

/iscsi-target...-igw/gateways>  create node1 10.130.70.91
The first gateway defined must be the local machine
/iscsi-target...-igw/gateways>  create node2 10.130.70.91
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok

遇到問題Failed : Gateway creation failed, gateway(s) unavailable:node2(UNKNOWN state)，需要三個節點都升級內核，升級內核參見：https://wuyeliang.blog.csdn.net/article/details/67644077

/iscsi-target...-igw/gateways> create node1 10.130.70.90  skipchecks=true
OS version/package checks have been bypassed
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
Failed : Gateway creation failed, gateway(s) unavailable:node2(UNKNOWN state)

遇到問題Is the API server running and in the right mode (http/https)?

注意關閉防火牆

systemctl stop firewalld

5.3 創建RBD image

> /iscsi-target...-igw/gateways> cd /disks
> /disks> create pool=rbd image=disk_1 size=90G

5.4 創建客戶端名稱
Linux平臺可以查看/etc/iscsi/initiatorname.iscsi文件獲取InitiatorName。如果修改了默認的名稱，必須重啓iscsid服務，否則在登錄iscsi服務端的時候會報錯。

> /disks> cd /iscsi-target/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts
> /iscsi-target...eph-igw/hosts>  create iqn.1994-05.com.redhat:rh7-client

5.5 設置客戶端認證

> /iscsi-target...at:rh7-client>  auth username=myiscsiusername password=myiscsipassword

5.6 客戶端映射磁盤

/iscsi-target...at:rh7-client> disk add rbd/disk_1

五、windows的多路徑連接

下面分享下Windows下多路徑功能配置
如果沒有添加MultipathI/O功能，在iSCSI發起程序屬性——》設備——》中看到“MPIO”顯示會是灰色。

安裝MPIO

配置MPIO支持ISCSI

連接ISCSI配置MPIO支持

然後選擇連接的存儲-選擇屬性配置另一條路徑

此時再查看屬性

六．iSCSI initiator

安裝initiator與multipath工具

iscsi-initiator-utils是通用initiator套件；
device-mapper-multipath是多路徑工具

# yum install iscsi-initiator-utils device-mapper-multipath -y

設置multipath服務，啓用multipath服務，生成”/etc/multipath.conf”文件

# mpathconf --enable --with_multipathd  y

在”/etc/multipath.conf”文件新增配置，針對LIO後端存儲設置多路徑ha

devices {
        device {
                vendor                "LIO-ORG"
                hardware_handler       "1 alua"
                path_grouping_policy    "failover"
                path_selector           "queue-length 0"
                failback                60
                path_checker           tur
                prio                   alua
                prio_args               exclusive_pref_bit
                fast_io_fail_tmo         25
                no_path_retry           queue
        }
}

重新加載multinpath服務

# systemctl reload multipathd

iscsi discovery
1）設置chap認證

# 開啓initiator的chap認證，並設置username/password，與iscsi-target設置保持一致；
# CHAP Settings部分，涉及57/61/62行
# vim /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = iscsiname
node.session.auth.password = iscsipassword

2）設置initiatoe-name

# 設置initiator-name，保持與iscsi-target設置的initiator-name一致
# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-04.192.168.70.50:iscsi-initiator

3）發現iscsi-target

# 發現iscsi存儲：iscsiadm -m discovery -t st -p ISCSI_IP，ISCSI_IP默認採用3260端口；
# 查看iscsi發現記錄：iscsiadm -m node
# 刪除iscsi發現記錄：iscsiadm -m node -o delete -T LUN_NAME -p ISCSI_IP
# iscsiadm -m discovery -t st -p 192.168.70.57

4）登陸iscsi-target

# 登錄iscsi存儲：iscsiadm -m node -T LUN_NAME -p ISCSI_IP -l
# 登出iscsi存儲：iscsiadm -m node -T LUN_NAME -p ISCSI_IP -u
# 顯示會話情況：iscsiadm -m session
# iscsiadm -m node -T iqn.2018-09.192.168.70.5x:iscsi-gw -l

# 1個後端存儲，通過3條路徑連接
# multipath -ll


# 通過多路徑連接後端存儲，生成多個盤符；
# 通過multipath服務匯聚，生成盤符/dev/mapper/mpathx，mount時間直接使用；
# 或：lsscsi
# fdisk -l

mount驗證

# mkfs.xfs /dev/mapper/mpatha

# 掛載分區
# mount /dev/mapper/mpatha /mnt

# 查看掛載情況
# df -Th

修改/etc/fstab設置開機啓動掛載

/dev/mapper/mpatha1 /mnt                    xfs    noatime,_netdev        0 0
參數說明：
# filesystem parameters列設置掛載時間；
# noatime：禁止更新文件與目錄的inode訪問時間，以獲得更快的訪問速度；
# _netdev：標識文件系統位於網絡上，防止網絡啓動前掛載
# rbd

參考：

https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/
https://www.lagou.com/lgeduarticle/43742.html
https://blog.51cto.com/candon123/2125049
https://blog.51cto.com/ityunwei2017/1916244
https://www.cnblogs.com/netonline/p/10432653.html

ceph-iscsi原理及部署

號稱能打敗MLP的KAN到底行不行？數學核心原理全面解析

同事使用 insert into select 遷移數據，開開心心上線，上線後被公司開除！

DeepFilterNet復現

ceph-iscsi原理及部署

rabbitmq刪除隊列

docker構建jre鏡像

openstack批量創建虛擬機

linux下格式磁盤爲ntfs格式

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結