前段時間學習了關於Redhat Linux上的多路徑Multipath，以及爲了方便學習在本機電腦上安裝了Openfiler模擬存儲，配置多塊網卡實現多路徑冗餘識別存儲。在這裏以我個人的理解簡單的描述我所認知的理論知識。
Multipath是Linux系統上開源免費的多路徑軟件，實現服務器可以通過多條物理路徑連接到塊存儲。
Openfiler是底層以Linux系統模擬的存儲架構，可以配置多塊網卡，同時可以配置軟件Raid。
多路徑是避免設備之間單條鏈路的單點故障隱患，同時可以實現流量分攤，實現性能提升。
當因爲主機HBA卡、線纜、交換機或者存儲設備的RAID控制器故障等原因造成一條物理路徑失效時,服務器可以將通過此物理路徑的I/O轉移到其他正常的物理路徑上面,而應用程序不會覺察到底層的改變，從而提高系統的可用性。
在這裏通過模擬現網環境配置Multipath實現多路徑冗餘；
實驗環境：
Redhat Enterprise Linux 7.2 //安裝配置Multipath多路徑識別ISCSI存儲設備; Openfileresa-2.99.1-x86_64 //配置多網卡，模擬配置ISCSI存儲;
在VMworkstation上Openfiler系統已經安裝完成並且配置了四塊物理網卡用於實現存儲多路徑，相應的IP地址已經配置規劃；
首先配置存儲設備的磁盤陣列Raid，由於在這裏用的是模擬存儲設備沒有相應的RAID控制器，所以在這裏配置軟件Raid，模擬現網中存儲設備的配置；
在Openfiler中配置添加邏輯卷，以及分配相應的空間大小給卷組。在這裏配置了四個lv磁盤組，爲了便於後續Oracle ASM使用；
創建相應的ISCSI並關聯LUN，並且配置允許192.168.60.0/24網段訪問，在此Openfiler存儲相應的配置就完成了；
在Redhat Linux上首先掛載並配置YUM源，安裝iscsi initiator軟件；
[root@bogon ~]# mount /dev/sr0 /mnt mount: /dev/sr0 is write-protected, mounting read-only [root@bogon ~]# vi /etc/yum.conf [local] baseurl=file:///mnt gpgcheck=0 enabled=1 [root@bogon ~]# yum repolist Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Repository 'local' is missing name in configuration, using id local | 4.1 kB 00:00:00 (1/2): local/group_gz | 136 kB 00:00:00 (2/2): local/primary_db | 3.6 MB 00:00:00 repo id repo name status local local 4,620 repolist: 4,620 [root@bogon ~]# cd /mnt/Packages/ [root@bogon Packages]# rpm -ivh iscsi-initiator-utils-6.2.0.873-32.el7.x86_64.rpm warning: iscsi-initiator-utils-6.2.0.873-32.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY Preparing... ################################# [100%] package iscsi-initiator-utils-6.2.0.873-32.el7.x86_64 is already installed
啓動iSCSI相關服務，並且配置開機啓動項；
[root@bogon Packages]# systemctl list-unit-files | grep iscsi iscsi-shutdown.service static iscsi.service enabled iscsid.service disabled iscsiuio.service disabled iscsid.socket enabled iscsiuio.socket enabled [root@bogon Packages]# systemctl enable iscsi.service
掃描並識別ISCSI存儲信息，由於Openfiler所用端口號爲3260，所以在此掃描添加端口號；
[root@bogon Packages]# iscsiadm -m discovery -t st -p 192.168.60.11:3260 192.168.60.11:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 192.168.60.12:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 192.168.60.13:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 192.168.60.14:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06
修改配置ISCSI軟件開機掃描信息；
vi /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2006-01.com.openfiler:tsn.6524fccc7d06
對掃描到的ISCSI識別存儲進行登陸；
[root@bogon Packages]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6524fccc7d06 -p 192.168.60.11:3260 -l Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.11,3260] (multiple) Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.11,3260] successful. [root@bogon Packages]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6524fccc7d06 -p 192.168.60.12:3260 -l Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.12,3260] (multiple) Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.12,3260] successful. [root@bogon Packages]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6524fccc7d06 -p 192.168.60.13:3260 -l Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.13,3260] (multiple) Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.13,3260] successful. [root@bogon Packages]# iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6524fccc7d06 -p 192.168.60.14:3260 -l Logging in to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.14,3260] (multiple) Login to [iface: default, target: iqn.2006-01.com.openfiler:tsn.6524fccc7d06, portal: 192.168.60.14,3260] successful.
查看ISCSI當前登陸會話是否正常；
[root@bogon Packages]# iscsiadm -m session tcp: [1] 192.168.60.11:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 (non-flash) tcp: [2] 192.168.60.12:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 (non-flash) tcp: [3] 192.168.60.13:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 (non-flash) tcp: [4] 192.168.60.14:3260,1 iqn.2006-01.com.openfiler:tsn.6524fccc7d06 (non-flash)
安裝Multipath多路徑軟件，並配置開機自動啓動；
[root@bogon Packages]# rpm -ivh device-mapper-libs-1.02.107-5.el7.x86_64.rpm warning: device-mapper-libs-1.02.107-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY Preparing... ################################# [100%] package device-mapper-libs-7:1.02.107-5.el7.x86_64 is already installed [root@bogon Packages]# rpm -ivh device-mapper-multipath-0.4.9-85.el7.x86_64.rpm warning: device-mapper-multipath-0.4.9-85.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY Preparing... ################################# [100%] package device-mapper-multipath-0.4.9-85.el7.x86_64 is already installed [root@bogon Packages]# systemctl enable multipathd.service [root@bogon Packages]# systemctl list-unit-files | grep multipath multipathd.service enabled [root@bogon Packages]# modprobe dm-multipath [root@bogon Packages]# modprobe dm-round-robin [root@bogon Packages]# systemctl start multipathd.service
查看並生成Multipath配置文件；
[root@bogon Packages]# multipath -ll Jun 09 11:28:53 | /etc/multipath.conf does not exist, blacklisting all devices. Jun 09 11:28:53 | A default multipath.conf file is located at Jun 09 11:28:53 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf Jun 09 11:28:53 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf [root@bogon Packages]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf [root@bogon Packages]#multipath start
查看多路徑所產生的磁盤並獲取其WWID，爲配置Multipath.conf做準備；
[root@bogon Packages]# fdisk -l | grep " 10.7 GB" Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdd: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdf: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdg: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdh: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdj: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdk: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdl: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdn: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdo: 10.7 GB, 10737418240 bytes, 20971520 sectors Disk /dev/sdp: 10.7 GB, 10737418240 bytes, 20971520 sectors [root@bogon Packages]# fdisk -l | grep " 21.5 GB" Disk /dev/sde: 21.5 GB, 21474836480 bytes, 41943040 sectors Disk /dev/sdi: 21.5 GB, 21474836480 bytes, 41943040 sectors Disk /dev/sdm: 21.5 GB, 21474836480 bytes, 41943040 sectors Disk /dev/sdq: 21.5 GB, 21474836480 bytes, 41943040 sectors [root@bogon Packages]# ls -l /dev/disk/by-id/ total 0 lrwxrwxrwx. 1 root root 9 Jun 9 11:09 ata-VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 -> ../../sr0 lrwxrwxrwx. 1 root root 10 Jun 9 11:09 dm-name-rhel-root -> ../../dm-0 lrwxrwxrwx. 1 root root 10 Jun 9 11:09 dm-name-rhel-swap -> ../../dm-1 lrwxrwxrwx. 1 root root 10 Jun 9 11:09 dm-uuid-LVM-lqbtrHjC1YDN1WRYptlr8UJDHwBSKqgAgdEPZJZeAyoqfX9sJjEm7gO9aIji71Gb -> ../../dm-0 lrwxrwxrwx. 1 root root 10 Jun 9 11:09 dm-uuid-LVM-lqbtrHjC1YDN1WRYptlr8UJDHwBSKqgAweG1tRVIKfpakJ0csD3TE1coXJbDB8r2 -> ../../dm-1 lrwxrwxrwx. 1 root root 10 Jun 9 11:09 lvm-pv-uuid-JuUZYx-8bPC-zoP1-4riV-qVxw-4qhJ-2ddb6j -> ../../sda2 lrwxrwxrwx. 1 root root 9 Jun 9 11:23 scsi-14f504e46494c45524e77537957762d34737a312d4d716339 -> ../../sdn lrwxrwxrwx. 1 root root 9 Jun 9 11:23 scsi-14f504e46494c4552554c536b71612d474539552d4942357a -> ../../sdo lrwxrwxrwx. 1 root root 9 Jun 9 11:23 scsi-14f504e46494c45526a33566a58622d6b6d4b562d73776f64 -> ../../sdq lrwxrwxrwx. 1 root root 9 Jun 9 11:23 scsi-14f504e46494c45527748533276382d707978772d686c7a43 -> ../../sdp [root@bogon Packages]#multipath -v3 .......... 14f504e46494c45524e77537957762d34737a312d4d716339 33:0:0:0 sdb 8:16 1 undef 14f504e46494c4552554c536b71612d474539552d4942357a 33:0:0:1 sdc 8:32 1 undef 14f504e46494c45527748533276382d707978772d686c7a43 33:0:0:2 sdd 8:48 1 undef 14f504e46494c45526a33566a58622d6b6d4b562d73776f64 33:0:0:3 sde 8:64 1 undef 14f504e46494c45524e77537957762d34737a312d4d716339 34:0:0:0 sdf 8:80 1 undef 14f504e46494c4552554c536b71612d474539552d4942357a 34:0:0:1 sdg 8:96 1 undef 14f504e46494c45527748533276382d707978772d686c7a43 34:0:0:2 sdh 8:112 1 undef 14f504e46494c45526a33566a58622d6b6d4b562d73776f64 34:0:0:3 sdi 8:128 1 undef 14f504e46494c45524e77537957762d34737a312d4d716339 35:0:0:0 sdj 8:144 1 undef 14f504e46494c4552554c536b71612d474539552d4942357a 35:0:0:1 sdk 8:160 1 undef 14f504e46494c45527748533276382d707978772d686c7a43 35:0:0:2 sdl 8:176 1 undef 14f504e46494c45526a33566a58622d6b6d4b562d73776f64 35:0:0:3 sdm 8:192 1 undef 14f504e46494c45524e77537957762d34737a312d4d716339 36:0:0:0 sdn 8:208 1 undef 14f504e46494c4552554c536b71612d474539552d4942357a 36:0:0:1 sdo 8:224 1 undef 14f504e46494c45527748533276382d707978772d686c7a43 36:0:0:2 sdp 8:240 1 undef 14f504e46494c45526a33566a58622d6b6d4b562d73776f64 36:0:0:3 sdq 65:0 1 undef
在此上面的查找WWID，可以識別爲：
14f504e46494c45524e77537957762d34737a312d4d716339 sdb sdf sdj sdn 10.7GB 14f504e46494c4552554c536b71612d474539552d4942357a sdc sdg sdk sdo 10.7GB 14f504e46494c45527748533276382d707978772d686c7a43 sdd sdh sdl sdp 10.7GB 14f504e46494c45526a33566a58622d6b6d4b562d73776f64 sde sdi sdm sdq 21.5GB
緊接着就可以配置Multipath.conf配置文件了；
[root@bogon Packages]# vi /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
}
#blacklist {
# wwid 26353900f02796769
# devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# devnode "^hd[a-z]"
#}
multipaths {
multipath {
wwid 14f504e46494c45524e77537957762d34737a312d4d716339
alias oracle-grid
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 14f504e46494c4552554c536b71612d474539552d4942357a
alias oracle-sys
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 14f504e46494c45527748533276382d707978772d686c7a43
alias oracle-arch
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 14f504e46494c45526a33566a58622d6b6d4b562d73776f64
alias oracle-data
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
}
devices {
device {
vendor "openfiler"
product "virtual-disk"
path_grouping_policy multibus
path_checker readsector0
path_selector "round-robin 0"
hardware_handler "0"
failback 15
rr_weight priorities
no_path_retry queue
}
}配置完成後重新啓動Multipath並檢查是否正常；
[root@bogon Packages]# multipath stop [root@bogon Packages]# multipath start [root@bogon Packages]# multipath -F [root@bogon Packages]# multipath -v2 create: oracle-grid (14f504e46494c45524e77537957762d34737a312d4d716339) undef OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=1 status=undef |- 33:0:0:0 sdb 8:16 undef ready running |- 34:0:0:0 sdf 8:80 undef ready running |- 35:0:0:0 sdj 8:144 undef ready running `- 36:0:0:0 sdn 8:208 undef ready running create: oracle-sys (14f504e46494c4552554c536b71612d474539552d4942357a) undef OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=1 status=undef |- 33:0:0:1 sdc 8:32 undef ready running |- 34:0:0:1 sdg 8:96 undef ready running |- 35:0:0:1 sdk 8:160 undef ready running `- 36:0:0:1 sdo 8:224 undef ready running create: oracle-arch (14f504e46494c45527748533276382d707978772d686c7a43) undef OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=1 status=undef |- 33:0:0:2 sdd 8:48 undef ready running |- 34:0:0:2 sdh 8:112 undef ready running |- 35:0:0:2 sdl 8:176 undef ready running `- 36:0:0:2 sdp 8:240 undef ready running create: oracle-data (14f504e46494c45526a33566a58622d6b6d4b562d73776f64) undef OPNFILER,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=undef `-+- policy='round-robin 0' prio=1 status=undef |- 33:0:0:3 sde 8:64 undef ready running |- 34:0:0:3 sdi 8:128 undef ready running |- 35:0:0:3 sdm 8:192 undef ready running `- 36:0:0:3 sdq 65:0 undef ready running [root@bogon Packages]# multipath -ll oracle-sys (14f504e46494c4552554c536b71612d474539552d4942357a) dm-3 OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:1 sdc 8:32 active ready running |- 34:0:0:1 sdg 8:96 active ready running |- 35:0:0:1 sdk 8:160 active ready running `- 36:0:0:1 sdo 8:224 active ready running oracle-data (14f504e46494c45526a33566a58622d6b6d4b562d73776f64) dm-5 OPNFILER,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:3 sde 8:64 active ready running |- 34:0:0:3 sdi 8:128 active ready running |- 35:0:0:3 sdm 8:192 active ready running `- 36:0:0:3 sdq 65:0 active ready running oracle-arch (14f504e46494c45527748533276382d707978772d686c7a43) dm-4 OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:2 sdd 8:48 active ready running |- 34:0:0:2 sdh 8:112 active ready running |- 35:0:0:2 sdl 8:176 active ready running `- 36:0:0:2 sdp 8:240 active ready running oracle-grid (14f504e46494c45524e77537957762d34737a312d4d716339) dm-2 OPNFILER,VIRTUAL-DISK size=10G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 33:0:0:0 sdb 8:16 active ready running |- 34:0:0:0 sdf 8:80 active ready running |- 35:0:0:0 sdj 8:144 active ready running `- 36:0:0:0 sdn 8:208 active ready running
檢查相應的多路徑對應是否正常；
[root@bogon Packages]# cd /dev/mapper/ [root@bogon mapper]# ls -lsa total 0 0 drwxr-xr-x. 2 root root 180 Jun 9 12:05 . 0 drwxr-xr-x. 20 root root 3940 Jun 9 12:05 .. 0 crw-------. 1 root root 10, 236 Jun 9 11:09 control 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:05 oracle-arch -> ../dm-4 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:05 oracle-data -> ../dm-5 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:05 oracle-grid -> ../dm-2 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:05 oracle-sys -> ../dm-3 0 lrwxrwxrwx. 1 root root 7 Jun 9 11:09 rhel-root -> ../dm-0 0 lrwxrwxrwx. 1 root root 7 Jun 9 11:09 rhel-swap -> ../dm-1
格式化相應的磁盤，爲後續ASM磁盤使用；
[root@bogon mapper]# fdisk -c /dev/mapper/oracle-arch [root@bogon mapper]# fdisk -c /dev/mapper/oracle-data [root@bogon mapper]# fdisk -c /dev/mapper/oracle-grid [root@bogon mapper]# fdisk -c /dev/mapper/oracle-sys [root@bogon mapper]# kpartx -a /dev/mapper/oracle-arch [root@bogon mapper]# kpartx -a /dev/mapper/oracle-sys [root@bogon mapper]# kpartx -a /dev/mapper/oracle-data [root@bogon mapper]# kpartx -a /dev/mapper/oracle-grid [root@bogon mapper]# ls -lsa total 0 0 drwxr-xr-x. 2 root root 260 Jun 9 12:22 . 0 drwxr-xr-x. 20 root root 4020 Jun 9 12:22 .. 0 crw-------. 1 root root 10, 236 Jun 9 11:09 control 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:16 oracle-arch -> ../dm-4 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:21 oracle-arch1 -> ../dm-6 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:17 oracle-data -> ../dm-5 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:22 oracle-data1 -> ../dm-8 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:17 oracle-grid -> ../dm-2 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:22 oracle-grid1 -> ../dm-9 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:17 oracle-sys -> ../dm-3 0 lrwxrwxrwx. 1 root root 7 Jun 9 12:22 oracle-sys1 -> ../dm-7 0 lrwxrwxrwx. 1 root root 7 Jun 9 11:09 rhel-root -> ../dm-0 0 lrwxrwxrwx. 1 root root 7 Jun 9 11:09 rhel-swap -> ../dm-1
在此，實驗所用的模擬存儲Openfiler使用四塊物理網卡，配置軟件Raid，邏輯卷配置多塊Lun。Redhat Enterprise Linux配置Multipath多路徑，所有均已實現。