環境：
兩臺華爲服務器，一臺主存華爲18500，一臺備存5300，多路徑軟件爲華爲的power path
操作系統是RHEL 6.5，做了集羣，集羣軟件是RHCS，應用爲ORACLE，主備模式
已經做完了mirror卷，對應兩臺設備上的/dev/sdb和/dev/sdc lun==接上一篇《lvm2線性卷轉成鏡像卷》
現象：
重起兩個node之後，發現兩個node上/dev/sdb和/dev/sdc互換名字了
node1上
Disk /dev/sdb: 644.2 GB, 644245094400 bytes
255 heads, 63 sectors/track, 78325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdc: 536.9 GB, 536870912000 bytes
255 heads, 63 sectors/track, 65270 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
node2上
Disk /dev/sdb: 536.9 GB, 536870912000 bytes
255 heads, 63 sectors/track, 65270 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdc: 644.2 GB, 644245094400 bytes
255 heads, 63 sectors/track, 78325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
在node2上名字是正確的順序，所以服務運行在node2上
Cluster Status for db @ Wed Apr 18 17:00:02 2018
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1 1 Online, rgmanager
node2 2 Online, Local, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:db-oracle node2 started
現在需要在兩個node上將多路徑聚合出來的名字設置正確，使用udev來解決，解決過程如下：
step1：將rhcs freeze
[root@node1 ~]# clustat
Cluster Status for db @ Thu Apr 19 14:49:00 2018
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1 1 Online, Local, rgmanager
node2 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:db-oracle node2 started
[root@node1 ~]# clusvcadm -Z db-oracle
Local machine freezing service:db-oracle...Success
[root@node1 ~]# clustat
Cluster Status for db @ Thu Apr 19 14:49:19 2018
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1 1 Online, Local, rgmanager
node2 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:db-oracle node2 started [Z]
[root@node1 ~]#
step2：在兩個node上使用udev綁定lun的名字
注意：
多路徑軟件必須實現的功能有2個：1 多路徑聚合，並在兩個node上生成唯一的名字；2 這個名字必須在兩個節點上一致
兩個node的/dev/sdb和/dev/sdc都是經過多路徑聚合後生成的名字。但華爲的iscsi多路徑軟件有問題，只生成了唯一的名字，但在各個node上的名字順序是不統一的，這說明華爲的多路徑軟件很不好，沒有實現多路徑軟件應有的功能。我們必須手動解決這個問題
在兩個node上查看兩個lun的屬性
[root@node1 ~]# udevadm info --query=env --path=/block/sdb
UDEV_LOG=3
DEVPATH=/devices/up_primary/up_adapter/host15/target15:0:0/15:0:0:1/block/sdb
MAJOR=8
MINOR=16
DEVNAME=/dev/sdb
......
ID_MODEL=XSG1
......
ID_REVISION=4303
......
ID_SERIAL=36a8ca7b100808f4e5f3ad82e00000068
......
[root@node1 ~]# udevadm info --query=env --path=/block/sdc
UDEV_LOG=3
DEVPATH=/devices/up_primary/up_adapter/host15/target15:0:1/15:0:1:1/block/sdc
MAJOR=8
MINOR=32
DEVNAME=/dev/sdc
......
ID_MODEL=HVS85T
......
ID_REVISION=4101
......
ID_SERIAL=364482e5100a340a72a7803c900000079
......
[root@node2 ~]# udevadm info --query=env --path=/block/sdb
UDEV_LOG=3
DEVPATH=/devices/up_primary/up_adapter/host15/target15:0:0/15:0:0:1/block/sdb
MAJOR=8
MINOR=16
DEVNAME=/dev/sdb
......
ID_MODEL=HVS85T
......
ID_REVISION=4101
......
ID_SERIAL=364482e5100a340a72a7803c900000079
......
[root@node2 ~]# udevadm info --query=env --path=/block/sdc
UDEV_LOG=3
DEVPATH=/devices/up_primary/up_adapter/host15/target15:0:1/15:0:1:1/block/sdc
MAJOR=8
MINOR=32
DEVNAME=/dev/sdc
......
ID_MODEL=XSG1
......
ID_REVISION=4303
......
ID_SERIAL=36a8ca7b100808f4e5f3ad82e00000068
......
經過對比，可以發現
node1 /dev/sdb的ID_SERIAL等於node2 /dev/sdc的ID_SERIAL
node1 /dev/sdc的ID_SERIAL等於node2 /dev/sdb的ID_SERIAL
node1
/dev/sdb
ID_SERIAL=36a8ca7b100808f4e5f3ad82e00000068
/dev/sdc
ID_SERIAL=364482e5100a340a72a7803c900000079
node2
/dev/sdb
ID_SERIAL=364482e5100a340a72a7803c900000079
/dev/sdc
ID_SERIAL=36a8ca7b100808f4e5f3ad82e00000068
step3：將數據庫關掉
SYS@nbora2> shutdown immediate;
[oracle@node2 ~]$ lsnrctl stop
step4：在step2中找到唯一標識符後，在step3中關庫、關監聽，之後就可以配置udev了
node1上
[root@node1 ~]# more /etc/udev/rules.d/99-z-multipath.rules
KERNEL=="sd*",SUBSYSTEM=="block",ENV{ID_MODEL}=="HVS85T",ENV{ID_REVISION}=="4101",ENV{ID_SERIAL}=="364482e5100a340a72a7803c900000079",NAME="sdb"
KERNEL=="sd*",SUBSYSTEM=="block",ENV{ID_MODEL}=="XSG1",ENV{ID_REVISION}=="4303",ENV{ID_SERIAL}=="36a8ca7b100808f4e5f3ad82e00000068",NAME="sdc"
node2上相同，把/etc/udev/rules.d/99-z-multipath.rules拷貝到node2的/etc/udev/rules.d中
之後在兩個node上分別執行如下命令即可==重新識別所有udev規則
start_udev
注意：
我之前在兩個node的udev中只寫了ENV{ID_SERIAL}，重啓後發現還是亂的，於是就又在兩個node添加了兩個屬性ENV{ID_MODEL}和ENV{ID_REVISION}，這樣，兩個lun在兩個node上就是唯一的了
[root@node1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 536.9 GB, 536870912000 bytes
255 heads, 63 sectors/track, 65270 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@node1 ~]# fdisk -l /dev/sdc
Disk /dev/sdc: 644.2 GB, 644245094400 bytes
255 heads, 63 sectors/track, 78325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@node2 rules.d]# fdisk -l /dev/sdb
Disk /dev/sdb: 536.9 GB, 536870912000 bytes
255 heads, 63 sectors/track, 65270 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@node2 rules.d]# fdisk -l /dev/sdc
Disk /dev/sdc: 644.2 GB, 644245094400 bytes
255 heads, 63 sectors/track, 78325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
可以看到，兩個node上的/dev/sdb和/dev/sdc就完全一致了
step5：現在將cluster unfreeze
[root@node2 ~]# clusvcadm -U db-oracle
之後切換測試一下
clusvcadm -r db-oracle -m node1
[root@node1 ~]# clustat
Cluster Status for db @ Thu Apr 19 15:17:53 2018
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
node1 1 Online, Local, rgmanager
node2 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:db-oracle node1 started
集羣切換正常，可以正常切換到node1上
查看兩個node上lvm信息
[root@node1 ~]# lvs -a -o name,copy_percent,devices vg_oracle
LV Cpy%Sync Devices
lv_oracle 100.00 lv_oracle_mimage_0(0),lv_oracle_mimage_1(0)
[lv_oracle_mimage_0] /dev/sdb(0)
[lv_oracle_mimage_1] /dev/sdc(0)
[lv_oracle_mlog] /dev/sdc(127999)
[root@node2 ~]# lvs -a -o name,copy_percent,devices vg_oracle
LV Cpy%Sync Devices
lv_oracle 100.00 lv_oracle_mimage_0(0),lv_oracle_mimage_1(0)
[lv_oracle_mimage_0] /dev/sdb(0)
[lv_oracle_mimage_1] /dev/sdc(0)
[lv_oracle_mlog] /dev/sdc(127999)
至此，線性卷修改爲mirror卷OK，華爲多路徑導致lun名字混亂問題解決，cluster切換測試完畢