Cinder塊節點重啓後,在Dashboard上不能啓動附帶了雲硬盤(券)的實例,報錯:
Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf
iscsiadm -m node -T iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78 -p block_node_IP:3260 --rescan
Exit code: 21 Stdout: u'' Stderr: u'iscsia].
錯誤代碼21表示:ISCSI_ERR_NO_OBJS_FOUND - no records/targets/sessions/portals found to execute operation on,找不到目標對象。而在實例關機狀態下,不能卸載掛載的卷。
一、計算節點
# tail /var/log/nova/nova-compute.log -n 500 | grep iscsi /*沒有發現會話*/
Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf
iscsiadm -m node -T iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78 -p block_node_IP:3260 --rescan
Exit code: 21
Stdout: u''
Stderr: u'iscsiadm: No session found.
<pre name="code" class="html"><pre name="code" class="html">......
# tail -n 50 /var/log/messages | grep auth /*登陸拒絕*/
Jan 25 14:53:43 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
Jan 25 14:57:29 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
Jan 25 16:06:40 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
Jan 25 17:10:51 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
Jan 25 17:17:37 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
Jan 25 17:41:02 compute3 iscsiadm: iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
Jan 25 17:41:02 compute3 iscsiadm: iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
Jan 25 17:41:03 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
Jan 25 17:41:03 compute3 iscsid: conn 0 login rejected: initiator failed authorization with target
# systemctl status iscsid -l /*認證失敗*/1月 25 17:17:37 compute3 iscsid[1730]: conn 0 login rejected: initiator failed authorization with target
1月 25 17:17:37 compute3 iscsid[1730]: Connection63:0 to
[target: iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78, portal: block_node_IP,3260]
through [iface: default] is shutdown.
......
其中:
1. iqn.2010-10.org.openstack是塊節點的iscsi的target標示
2. volume-369865bb-0714-4ab2-a96c-7a91b7483e78是塊節點上爲實例創建的卷名(lvdisplay可以看到)
# iscsiadm -m node -T iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78 -p block_node_IP:3260 --login /*手工登陸驗證*/
Logging in to [iface: default, target: iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78,
portal: block_node_IP,3260] (multiple)
iscsiadm: Could not login to [iface: default,
target: iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78, portal: block_node_IP,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals
二、塊節點
# tail -n 1000 /var/log/messages | grep auth /*系統日誌*/
Jan 25 17:10:50 block1 kernel: iSCSI Initiator Node: iqn.1994-05.com.redhat:a5fd80c5a912 is not authorized to access iSCSI target portal group: 1.
Jan 25 17:17:37 block1 kernel: iSCSI Initiator Node: iqn.1994-05.com.redhat:a5fd80c5a912 is not authorized to access iSCSI target portal group: 1.
Jan 25 17:41:02 block1 kernel: iSCSI Initiator Node: iqn.1994-05.com.redhat:a5fd80c5a912 is not authorized to access iSCSI target portal group: 1.
Jan 25 17:41:02 block1 kernel: iSCSI Initiator Node: iqn.1994-05.com.redhat:a5fd80c5a912 is not authorized to access iSCSI target portal group: 1.
其中:
1. iqn.1994-05.com.redhat:a5fd80c5a912是實例所在<strong>計算節點</strong>的iscsi名字,可以在計算節點上查看
$ cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:a5fd80c5a912
看到這裏就是表明:實例所在的計算節點不能成功認證去連接塊節點的卷。
$ targetcli ls /*塊節點上查看target,這裏我有兩個卷,分別500G*/
爲了驗證這些卷最初建立和掛載到實例上時的狀態,這裏新建一個卷並掛載到另一個位於不同計算節點(iqn.1994-05.com.redhat:11f29647866a)的實例上,截圖
/> cd iscsi/
/iscsi> ls /*只看iscsi列表,方便看*/
發現新卷有兩個地方不同:
1. 有acls(訪控列表),指定誰能訪問這個卷,圖中看指明瞭是上面測試用的另一個實例所在的計算節點
2. 下面還有個mapped_lun0,mapping在於指定能訪問誰,也就是上面acls中的主機能夠訪問誰(對應的還有個masking指定不能訪問誰)
類似地,現在對之前的兩個卷,作同樣的更改
# targetcli
/> cd iscsi/iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78/tpg1/acls/ /*轉到對應的卷的acls中*/
/iscsi/iqn.20...e78/tpg1/acls> create iqn.1994-05.com.redhat:a5fd80c5a912 /*指定實例所在計算節點可以訪問*/
Created Node ACL for iqn.1994-05.com.redhat:a5fd80c5a912
Created mapped LUN 0.
/iscsi/iqn.20...e78/tpg1/acls> lso- acls .................................................................................................................. [ACLs: 1]
o- iqn.1994-05.com.redhat:a5fd80c5a912 .............................................................. [1-way auth, Mapped LUNs: 1]
o- mapped_lun0 ......................... [lun0 block/iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78 (rw)]
OK,上面說的兩個東西都有了。對另一個卷作同樣處理。
o- acls .................................................................................................................. [ACLs: 1]
o- iqn.1994-05.com.redhat:a5fd80c5a912 .............................................................. [1-way auth, Mapped LUNs: 1]
o- mapped_lun0 ......................... [lun0 block/iqn.2010-10.org.openstack:volume-446d70fc-c3f8-43cd-a0b9-dfd5eee934b9 (rw)]
/iscsi/iqn.20...4b9/tpg1/acls> exitGlobal pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
# targetcli saveconfig /*再保存下*/Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
# systemctl restart target /*重啓target服務*/
硬重啓實例(步驟看文章底部的控制節點),還是同樣報錯,並且在塊節點上發現:
# tail /var/log/messages
Jan 26 11:06:00 block1 cinder-volume: 2016-01-26 11:06:00.356 2785 INFO cinder.volume.manager [-] Updating volume replication status.
Jan 26 11:06:08 block1 kernel: CHAP user or password not set for Initiator ACL
Jan 26 11:06:08 block1 kernel: Security negotiation failed.
Jan 26 11:06:08 block1 kernel: iSCSI Login negotiation failed.
# vim /etc/iscsi/iscsid.conf /*查看計算節點上的配置,默認就沒有設置用戶名和密碼*/
# *************
# CHAP Settings
# *************
# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
#node.session.auth.authmethod = CHAP
# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
#node.session.auth.username = username
#node.session.auth.password = password
vim /etc/target/saveconfig.json /*回到塊節點,發現新建測試卷有用戶名和密碼,之前的卷在塊節點主機重啓後卻沒有*/
"dev": "/dev/cinder-volumes/volume-01519b87-3036-4b6e-8174-c4a86030b370",
"name": "iqn.2010-10.org.openstack:volume-01519b87-3036-4b6e-8174-c4a86030b370",
"plugin": "block",
"readonly": false,
/password
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0,
"t10_pi": 0
},
"enable": true,
"luns": [
{
"index": 0,
"storage_object": "/backstores/block/iqn.2010-10.org.openstack:volume-01519b87-3036-4b6e-8174-c4a86030b370"
}
],
"node_acls": [
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_password": "5k4DnHHcJd3SyvaF",
"chap_userid": "xZrcAF8GH5P6smJmYceN",
"mapped_luns": [
{
"index": 0,
"tpg_lun": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:11f29647866a"
}
],
# reboot /*驗證塊節點重啓後,新建卷沒有變化,還是能正常識別使用:acls有;默認的用戶名和密碼也都在配置文件中,那麼這個問題沒有重現*/
最後,既然是認證失敗,那麼我就在塊節點上設置這兩個卷的用戶名和密碼,然後在對應計算節點上去配置對應的帳號
/> cd iscsi/iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78/tpg1/acls/iqn.1994-05.com.redhat:a5fd80c5a912/
/iscsi/iqn.20...:a5fd80c5a912> set auth userid=username
Parameter userid is now 'username'.
/iscsi/iqn.20...:a5fd80c5a912> set auth password=password
Parameter password is now 'password'.
/iscsi/iqn.20...:a5fd80c5a912> exit /*另一個卷同樣設置*/Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
# systemctl restart target# ss -napt | grep 3260
LISTEN 0 256 *:3260 *:*
# vim /etc/iscsi/iscsid.conf /*然後在計算節點去配置iscsid.conf,去掉註釋開啓CHAP*/
# *************
# CHAP Settings
# *************
# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP
# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = username
node.session.auth.password = password
# iscsiadm -m discovery -t sendtargets -p block_node_IP /*計算節點上再手工連接驗證*/
# iscsiadm -m node -l
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-446d70fc-c3f8-43cd-a0b9-dfd5eee934b9, portal: block_node_IP,3260] successful.
Login to [iface: default, target: iqn.2010-10.org.openstack:volume-369865bb-0714-4ab2-a96c-7a91b7483e78, portal: block_node_IP,3260] successful.
最後再硬重啓實例,起來了。但是呢這裏會有個潛在問題,以後這個計算節點上的其他實例需要掛載卷的時候,可能就需要對捲進行同樣的用戶名和密碼設置。而如果匿名不需要認證,那就不太安全。
三、控制節點
在Dashboard上硬啓動啓動失敗的實例,直接進入了error狀態,需要reset-state重置狀態再hard reboot
# nova list /*獲取實例ID*/
# nova reset-state 2d6fc5be-a95e-4959-a16a-45f126b0217a --active /*重置爲active活動狀態*/
現在到Dashboard上硬重啓實例。