單路徑變更多路徑
某客戶一套生產系統，由於施工期間不規範，數據庫服務器直接直連存儲，
現需要將服務器通過光交與存儲連接，並配置多路徑保證鏈路的冗餘和負載均衡
具體思路如下:
1.收集系統信息
1.1 覈對服務器磁盤歸屬信息(本地磁盤，存儲lun)
1.2 收集所有磁盤設備的WWID信息，fdisk信息
1.3 收集文件系統fstab掛載信息(備份fstab)
1.4 收集磁盤卷、文件系統uuid信息
1.5 收集所有oracle數據文件信息(位置，大小，個數，狀態)
2.配置多路徑、覈對信息
2.1 檢查配置multipath服務
2.2 配置multipath.conf，綁定新設備名
2.3 重啓操作系統
2.4 刷新multipath服務
2.5 檢查校驗多路徑狀態，新設備掛在信息
2.6 檢查卷、文件系統UUID信息
2.7 啓動數據庫
#############
具體實施步驟#
#############
####在進行操作前，收集現有系統配置信息並對其進行備份，以便出現問題時候回退###
1.1 服務器磁盤歸屬信息
經覈查sda爲本地磁盤，其他設備均爲直連存儲的lun(sbd,sdc,sdd,sdg,sde,sdf)
1.2查看現有系統wwid信息
[root@oradb ~]# for i in `cat /proc/partitions | awk {'print $4'} | grep sd`; do echo "### $i: `scsi_id --whitelist /dev/$i`"; done
### sda: 361866da08f0a8c0020d6a2c31e4628f7
### sdb: 36006016093e24100920c1b587959b56c
### sdb2:
### sdc: 36006016093e241007b0c1b583729b956
### sdc1:
### sdd: 36006016093e24100c069285a00444b52
### sdg: 36006016042704500966bc2336892e911
### sdf: 360060160427045006e1c1baad56be711
### sdf1:
### sde: 36006016093e24100a3595b5b716a5de0
### sde1:
##############
fsisk -l信息 #
##############
[root@oradb ~]# fdisk -l
Disk /dev/sda: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x6ad0d56a
Device Boot Start End Blocks Id System
/dev/sda1 1 36405 292421631+ ee GPT
Disk /dev/sdb: 1649.3 GB, 1649267441664 bytes
255 heads, 63 sectors/track, 200512 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb2 1 200512 1610612608+ 83 Linux
Disk /dev/sdc: 1649.3 GB, 1649267441664 bytes
255 heads, 63 sectors/track, 200512 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdc1 1 200512 1610612608+ 83 Linux
Disk /dev/sdd: 1099.5 GB, 1099511627776 bytes
255 heads, 63 sectors/track, 133674 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdg: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdf: 644.2 GB, 644245094400 bytes
255 heads, 63 sectors/track, 78325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdf1 1 78325 629145531 83 Linux
Disk /dev/sde: 1099.5 GB, 1099511627776 bytes
255 heads, 63 sectors/track, 133674 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcd0c0083
Device Boot Start End Blocks Id System
/dev/sde1 1 133674 1073736373+ 83 Linux
Disk /dev/mapper/backupvg-backup_lv: 3247.0 GB, 3246995275776 bytes
255 heads, 63 sectors/track, 394758 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
1.3 收集fstab信息
當前文件系統使用uuid信息掛載，這樣的好處是新設備綁定多路徑後，原文件系統的fstab信息不需要再修改
[root@oradb ~]# more /etc/fstab
# /etc/fstab
# Created by anaconda on Tue Sep 19 21:19:56 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=94882757-4b38-4318-a08b-6f7f712ab516 / ext4 defaults 1 1
UUID=594fa55a-19d0-445f-8e27-4a40b0da0640 /boot ext4 defaults 1 2
UUID=d4a51b71-f7ec-404a-9870-8495c0accf64 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
UUID=7f2f7a67-7b97-4d12-99c2-8fdd1d7bcdf0 /emcbakup ext3 defaults 0 0
UUID=5de2da02-0d90-484d-a4e2-2c68c42485c9 /dbfile ext3 defaults 0 0
UUID=498cfb57-c47c-4411-8ccb-406969e51fe4 /dbtempspace ext3 defaults 0 0
UUID=f818248d-3c9a-4473-9358-04967f422633 /newiscsi ext3 defaults 0 0
/dev/backupvg/backup_lv /backup ext4 defaults 0 0
1.4 收集磁盤、卷、文件系統uuid信息
當前vg是由sdd與sdg創建的物理卷組成(當設備綁定多路徑後，pvname自動變成多路徑設備名，無需對vg進行export與import)
[root@oradb ~]# pvdisplay
--- Physical volume ---
PV Name /dev/sdg
VG Name backupvg
PV Size 2.00 TiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 524287
Free PE 0
Allocated PE 524287
PV UUID shnHNI-jo6U-kqm2-qFgY-wPs7-25p7-e2Xq8l
--- Physical volume ---
PV Name /dev/sdd
VG Name backupvg
PV Size 1.00 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 262143
Free PE 12286
Allocated PE 249857
PV UUID fylY8E-FKuJ-g6yn-EpUS-wYMA-OvuW-seX2oI
[root@oradb ~]# vgdisplay
--- Volume group ---
VG Name backupvg
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 8
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 3.00 TiB
PE Size 4.00 MiB
Total PE 786430
Alloc PE / Size 774144 / 2.95 TiB
Free PE / Size 12286 / 47.99 GiB
VG UUID wCOfCY-fuYD-nwEL-EpAQ-wCji-XCtH-GzOY7j
[root@oradb ~]# lvdisplay
--- Logical volume ---
LV Path /dev/backupvg/backup_lv
LV Name backup_lv
VG Name backupvg
LV UUID f5LqSU-Dd6H-VEAq-MhO2-wov8-nk4G-dD38pw
LV Write Access read/write
LV Creation host, time oradb, 2019-06-30 18:26:38 +0800
LV Status available
# open 1
LV Size 2.95 TiB
Current LE 774144
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
查看當前文件系統UUID信息，以便後續變更後覈對校準
[root@oradb backup]# blkid
/dev/sda3: UUID="94882757-4b38-4318-a08b-6f7f712ab516" TYPE="ext4"
/dev/sda1: UUID="594fa55a-19d0-445f-8e27-4a40b0da0640" TYPE="ext4"
/dev/sda2: UUID="d4a51b71-f7ec-404a-9870-8495c0accf64" TYPE="swap"
/dev/sde1: UUID="f818248d-3c9a-4473-9358-04967f422633" TYPE="ext3"
/dev/sdf1: UUID="498cfb57-c47c-4411-8ccb-406969e51fe4" TYPE="ext3"
/dev/sdb2: UUID="7f2f7a67-7b97-4d12-99c2-8fdd1d7bcdf0" TYPE="ext3"
/dev/sdc1: UUID="5de2da02-0d90-484d-a4e2-2c68c42485c9" TYPE="ext3"
/dev/sdd: UUID="fylY8E-FKuJ-g6yn-EpUS-wYMA-OvuW-seX2oI" TYPE="LVM2_member"
/dev/sdg: UUID="shnHNI-jo6U-kqm2-qFgY-wPs7-25p7-e2Xq8l" TYPE="LVM2_member"
/dev/mapper/backupvg-backup_lv: UUID="9c67c793-9f02-4fd2-834d-bbee0dbc3d57" TYPE="ext4"
當前文件系統掛載信息
[root@oradb ~]# mount
/dev/sda3 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/sdb2 on /emcbakup type ext3 (rw)
/dev/sdc1 on /dbfile type ext3 (rw)
/dev/sdf1 on /dbtempspace type ext3 (rw)
/dev/sde1 on /newiscsi type ext3 (rw)
/dev/mapper/backupvg-backup_lv on /backup type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
1.5 收集所有數據文件信息(位置，大小，個數，狀態)
SQL > select file_name,status from dba_data_files
2.配置多路徑
2.1 檢查multipath服務
[root@oradb ~]# service multipathd status
multipathd (pid 8105) is running...
如果沒安裝multipath，可以使用yum自行安裝
#yum –y install device-mapper* device-mapper-multipath*
2.2 配置multipath.conf，綁定新設備名
[root@oradb ~]# vi /etc/multipath.conf
blacklist {
devnode "^sda"
}
defaults {
user_friendly_names yes
udev_dir /dev
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
multipaths {
multipath {
wwid 36006016093e24100920c1b587959b56c
alias mulsdb2
path_grouping_policy multibus
path_selector "round-robin 0"
}
multipath {
wwid 36006016093e241007b0c1b583729b956
alias mulsdc1
path_grouping_policy multibus
path_selector "round-robin 0"
}
multipath {
wwid 360060160427045006e1c1baad56be711
alias mulsdf1
path_grouping_policy multibus
path_selector "round-robin 0"
}
multipath {
wwid 36006016093e24100a3595b5b716a5de0
alias mulsde1
path_grouping_policy multibus
path_selector "round-robin 0"
}
multipath {
wwid 36006016093e24100c069285a00444b52
alias mulsdd
path_grouping_policy multibus
path_selector "round-robin 0"
}
multipath {
wwid 36006016042704500966bc2336892e911
alias mulsdg
path_grouping_policy multibus
path_selector "round-robin 0"
}
}
devices {
device {
vendor "COMPELNT"
product "Virtual Disk"
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
path_checker readsector0
path_selector "round-robin 0"
}
}
2.3 重啓操作系統
配置好光交與多路徑後，重啓現有系統
2.4 刷新multipath服務，打印當前多路徑信息
系統重啓成功後，查看當前多路徑信息，如下所示所有設備已經成功綁定
[root@oradb ~]# multipath -v2
[root@oradb ~]# multipath -ll
mulsdf1 (360060160427045006e1c1baad56be711) dm-0 DGC,VRAID
size=600G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=70 status=active
|- 9:0:0:0 sdb 8:16 active ready running
`- 10:0:1:0 sdm 8:192 active ready running
mulsdb2 (36006016093e24100920c1b587959b56c) dm-1 DGC,VRAID
size=1.5T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
|- 8:0:0:0 sdc 8:32 active ready running
`- 10:0:0:0 sdg 8:96 active ready running
mulsdg (36006016042704500966bc2336892e911) dm-4 DGC,VRAID
size=2.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
|- 9:0:0:1 sde 8:64 active ready running
`- 10:0:1:1 sdn 8:208 active ready running
mulsde1 (36006016093e24100a3595b5b716a5de0) dm-5 DGC,VRAID
size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=70 status=active
|- 8:0:0:3 sdj 8:144 active ready running
`- 10:0:0:3 sdl 8:176 active ready running
mulsdd (36006016093e24100c069285a00444b52) dm-3 DGC,VRAID
size=1.0T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=70 status=active
|- 8:0:0:2 sdh 8:112 active ready running
`- 10:0:0:2 sdk 8:160 active ready running
mulsdc1 (36006016093e241007b0c1b583729b956) dm-2 DGC,VRAID
size=1.5T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 emc' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
|- 8:0:0:1 sdf 8:80 active ready running
`- 10:0:0:1 sdi 8:128 active ready running
2.5 檢查校驗多路徑狀態，多路徑新設備掛載信息
當前掛載設備已經變更爲多路徑設備名,當前的fstab信息是以文件系統的UUID與邏輯卷設備名掛載，當存儲變更多路徑時，無需對fstab其更改
檢查覈對所有的信息，保證系統存儲使用多路徑設備名掛載後的信息與原來的掛載位置保持一致
[root@oradb ~]# mount
/dev/sda3 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/mulsdb2p2 on /emcbakup type ext3 (rw)
/dev/mapper/mulsdc1p1 on /dbfile type ext3 (rw)
/dev/mapper/mulsdf1p1 on /dbtempspace type ext3 (rw)
/dev/mapper/mulsde1p1 on /newiscsi type ext3 (rw)
/dev/mapper/backupvg-backup_lv on /backup type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
2.6 檢查卷、文件系統UUID信息
所有的卷設備，已經變成多路徑的設備名
[root@oradb ~]# pvdisplay
--- Physical volume ---
PV Name /dev/mapper/mulsdg
VG Name backupvg
PV Size 2.00 TiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 524287
Free PE 0
Allocated PE 524287
PV UUID shnHNI-jo6U-kqm2-qFgY-wPs7-25p7-e2Xq8l
--- Physical volume ---
PV Name /dev/mapper/mulsdd
VG Name backupvg
PV Size 1.00 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 262143
Free PE 12286
Allocated PE 249857
PV UUID fylY8E-FKuJ-g6yn-EpUS-wYMA-OvuW-seX2oI
[root@oradb ~]# vgdisplay
--- Volume group ---
VG Name backupvg
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 8
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 3.00 TiB
PE Size 4.00 MiB
Total PE 786430
Alloc PE / Size 774144 / 2.95 TiB
Free PE / Size 12286 / 47.99 GiB
VG UUID wCOfCY-fuYD-nwEL-EpAQ-wCji-XCtH-GzOY7j
[root@oradb ~]# lvdisplay
--- Logical volume ---
LV Path /dev/backupvg/backup_lv
LV Name backup_lv
VG Name backupvg
LV UUID f5LqSU-Dd6H-VEAq-MhO2-wov8-nk4G-dD38pw
LV Write Access read/write
LV Creation host, time oradb, 2019-06-30 18:26:38 +0800
LV Status available
# open 1
LV Size 2.95 TiB
Current LE 774144
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:10
文件系統UUID信息
[root@oradb ~]# blkid
/dev/sda3: UUID="94882757-4b38-4318-a08b-6f7f712ab516" TYPE="ext4"
/dev/sda1: UUID="594fa55a-19d0-445f-8e27-4a40b0da0640" TYPE="ext4"
/dev/sda2: UUID="d4a51b71-f7ec-404a-9870-8495c0accf64" TYPE="swap"
/dev/sdb1: UUID="498cfb57-c47c-4411-8ccb-406969e51fe4" TYPE="ext3"
/dev/sdc2: UUID="7f2f7a67-7b97-4d12-99c2-8fdd1d7bcdf0" TYPE="ext3"
/dev/sdh: UUID="fylY8E-FKuJ-g6yn-EpUS-wYMA-OvuW-seX2oI" TYPE="LVM2_member"
/dev/sde: UUID="shnHNI-jo6U-kqm2-qFgY-wPs7-25p7-e2Xq8l" TYPE="LVM2_member"
/dev/sdl1: UUID="f818248d-3c9a-4473-9358-04967f422633" TYPE="ext3"
/dev/sdj1: UUID="f818248d-3c9a-4473-9358-04967f422633" TYPE="ext3"
/dev/mapper/mulsdd: UUID="fylY8E-FKuJ-g6yn-EpUS-wYMA-OvuW-seX2oI" TYPE="LVM2_member"
/dev/mapper/mulsdg: UUID="shnHNI-jo6U-kqm2-qFgY-wPs7-25p7-e2Xq8l" TYPE="LVM2_member"
/dev/mapper/mulsdf1p1: UUID="498cfb57-c47c-4411-8ccb-406969e51fe4" TYPE="ext3"
/dev/mapper/mulsdb2p2: UUID="7f2f7a67-7b97-4d12-99c2-8fdd1d7bcdf0" TYPE="ext3"
/dev/mapper/mulsdc1p1: UUID="5de2da02-0d90-484d-a4e2-2c68c42485c9" TYPE="ext3"
/dev/mapper/mulsde1p1: UUID="f818248d-3c9a-4473-9358-04967f422633" TYPE="ext3"
/dev/mapper/backupvg-backup_lv: UUID="9c67c793-9f02-4fd2-834d-bbee0dbc3d57" TYPE="ext4"
/dev/sdg2: UUID="7f2f7a67-7b97-4d12-99c2-8fdd1d7bcdf0" TYPE="ext3"
/dev/sdi1: UUID="5de2da02-0d90-484d-a4e2-2c68c42485c9" TYPE="ext3"
/dev/sdk: UUID="fylY8E-FKuJ-g6yn-EpUS-wYMA-OvuW-seX2oI" TYPE="LVM2_member"
/dev/sdm1: UUID="498cfb57-c47c-4411-8ccb-406969e51fe4" TYPE="ext3"
/dev/sdn: UUID="shnHNI-jo6U-kqm2-qFgY-wPs7-25p7-e2Xq8l" TYPE="LVM2_member"
You have mail in /var/spool/mail/root
2.7 啓動數據庫
所有信息檢查無誤後，啓動數據庫，檢查數據庫數據文件大小，狀態
