有一臺高配的R730xd服務器，帶有滿配大容量硬盤以及SSD和兩張Qlogic的HBA卡。有天朋友討論能不能把這臺服務器配置成一臺FC存儲服務器推給vmware或者其他客戶端使用。這個我之前沒有做過，很感興趣，所以我們花了一個多星期的進行研究最後成功實現，故我們把過程和遇到的問題及解決方法寫出來分享，希望幫到其他有這個需求的運維。
剛開始Google了良久，最後決定思路是：在Linux上部署zfs系統，把SSD定義爲zfs的緩存磁盤，然後把HBA卡設置爲target模式，使用targetcli在zfs下的一個文件塊推送至vmware或者其他客戶端。
尋找良好兼容性的Linux發行版我們搞了很久，發現CentOS6使用的2.6內核不能把HBA卡設置爲target模式；使用CentOS7.1能支持設置爲target模式，但在targetcli下無法看到qla2xxx的路徑；自己試過在CentOS6下編譯3.x或者4.x的內核也是在targetcli下無法看到qla2xxx的路徑，一直無法解決。後面安裝Ubuntu15.10，就沒有這個問題，但是在裝zfs的時候發現有個包沒有兼容內核，無法安裝。最後是使用了Ubuntu14.04 LTS版本才順利完成。詳見本文最後的報錯分享。
最後的環境：DellR730xd+QLE2650(8G)+Intel SSD 750+Ubuntu14.04LTS
先安裝ZFS文件系統支持
sudo su - apt-add-repository --yes ppa:zfs-native/stable apt-get update apt-get install debootstrap spl-dkms zfs-dkms ubuntu-zfs #系統啓動掛載zfs模塊 vi /etc/rc.local /sbin/modprobe zfs #安裝SSD驅動 apt-get install git-core build-essential libncurses5-dev git clone cd nvme-cli make && make install #創建pool zpool create zfspool /dev/sdb #添加緩存盤 zpool add zfspool cache /dev/nvme0n1
安裝targetcli
apt-get install targetcli
設置HBA卡爲target mode
vi /etc/modprobe.d/qla2xxx.conf options qla2xxx qlini_mode="disabled"
重啓，檢查targetcli是否正確加載qla2xxx模塊，下面有qla2xxx輸出即爲正常。
targetcli /> ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- fileio ............................................... [0 Storage Object] | o- iblock ............................................... [0 Storage Object] | o- pscsi ................................................ [0 Storage Object] | o- rd_dr ................................................ [0 Storage Object] | o- rd_mcp ............................................... [0 Storage Object] o- ib_srpt ........................................................ [0 Target] o- iscsi .......................................................... [0 Target] o- loopback ....................................................... [0 Target] o- qla2xxx ........................................................ [0 Target] />
創建backstore
#官方文檔說明設備、文件、閃存盤可以作爲backstore，此處我創建文件型backstore /> cd backstores/ /backstores> fileio/ create name=test file_or_dev=/zfspool/test size=1T Using buffered mode. Created fileio test. /backstores> ls o- backstores ............................................................ [...] o- fileio ................................................. [1 Storage Object] | o- test ............................... [1.0T, /zfspool/test, not in use] o- iblock ................................................. [0 Storage Object] o- pscsi .................................................. [0 Storage Object] o- rd_mcp ................................................. [0 Storage Object]
創建target，插了兩塊HBA，可以看到有兩個WWN
/> qla2xxx/ info Fabric module name: qla2xxx ConfigFS path: /sys/kernel/config/target/qla2xxx Allowed WWNs list (free type): 21:00:00:24:ff:0e:1e:30, 21:00:00:24:ff:0e:7c:f5 Fabric module specfile: /var/target/fabric/qla2xxx.spec Fabric module features: acls Corresponding kernel module: tcm_qla2xxx /> qla2xxx/ create 21:00:00:24:ff:0e:1e:30 Created target 21:00:00:24:ff:0e:1e:30. /> qla2xxx/ create 21:00:00:24:ff:0e:7c:f5 Created target 21:00:00:24:ff:0e:7c:f5.
推LUN
/> cd qla2xxx/21:00:00:24:ff:0e:1e:30/ /qla2xxx/21:0...4:ff:0e:1e:30> luns/ create /backstores/fileio/test Selected LUN 0. Created LUN 0. /qla2xxx/21:0...4:ff:0e:1e:30> cd ../ /qla2xxx> cd 21:00:00:24:ff:0e: 21:00:00:24:ff:0e:1e:30/ 21:00:00:24:ff:0e:7c:f5/ .............path /qla2xxx> cd 21:00:00:24:ff:0e:7c:f5/ /qla2xxx/21:0...4:ff:0e:7c:f5> luns/ create /backstores/fileio/test Selected LUN 0. Created LUN 0.
設置接入權限,接入端也是兩個HBA卡，所以每個target設置兩個ACL
/qla2xxx/21:0...4:ff:0e:7c:f5> acls/ create 20:00:74:e6:e2:65:c8:0b Created Node ACL for 20:00:74:e6:e2:65:c8:0b Created mapped LUN 0. /qla2xxx/21:0...4:ff:0e:7c:f5> acls/ create 20:01:74:e6:e2:65:c8:0b Created Node ACL for 20:01:74:e6:e2:65:c8:0b Created mapped LUN 0. /qla2xxx/21:0...4:ff:0e:7c:f5> cd .. /qla2xxx> cd 21:00:00:24:ff:0e:1e:30/ /qla2xxx/21:0...4:ff:0e:1e:30> acls/ create 20:00:74:e6:e2:65:c8:0b Created Node ACL for 20:00:74:e6:e2:65:c8:0b Created mapped LUN 0. /qla2xxx/21:0...4:ff:0e:1e:30> acls/ create 20:01:74:e6:e2:65:c8:0b Created Node ACL for 20:01:74:e6:e2:65:c8:0b Created mapped LUN 0.
最後整個targetcli的樹目錄是這樣的
/> ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- fileio ............................................... [1 Storage Object] | | o- test .................................... [1.0T, /zfspool/test, in use] | o- iblock ............................................... [0 Storage Object] | o- pscsi ................................................ [0 Storage Object] | o- rd_mcp ............................................... [0 Storage Object] o- ib_srpt ....................................................... [0 Targets] o- iscsi ......................................................... [0 Targets] o- loopback ...................................................... [0 Targets] o- qla2xxx ....................................................... [2 Targets] | o- 21:00:00:24:ff:0e:1e:30 ....................................... [enabled] | | o- acls ......................................................... [2 ACLs] | | | o- 20:00:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN] | | | | o- mapped_lun0 ........................................... [lun0 (rw)] | | | o- 20:01:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN] | | | o- mapped_lun0 ........................................... [lun0 (rw)] | | o- luns .......................................................... [1 LUN] | | o- lun0 .................................. [fileio/test (/zfspool/test)] | o- 21:00:00:24:ff:0e:7c:f5 ....................................... [enabled] | o- acls ......................................................... [2 ACLs] | | o- 20:00:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN] | | | o- mapped_lun0 ........................................... [lun0 (rw)] | | o- 20:01:74:e6:e2:65:c8:0b .............................. [1 Mapped LUN] | | o- mapped_lun0 ........................................... [lun0 (rw)] | o- luns .......................................................... [1 LUN] | o- lun0 .................................. [fileio/test (/zfspool/test)] o- tcm_fc ........................................................ [0 Targets] o- usb_gadget .................................................... [0 Targets] o- vhost ......................................................... [0 Targets]
最後別忘了保存，在保存的過程中遇到以下錯誤，查詢資料發現是targetcli的一個bug，也給出了修復方法。
/> saveconfig Save configuration? [Y/n]: y Saving new startup configuration Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 990, in run_interactive self._cli_loop() File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 820, in _cli_loop self.run_cmdline(cmdline) File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 934, in run_cmdline self._execute_command(path, command, pparams, kparams) File "/usr/lib/python2.7/dist-packages/configshell/shell.py", line 909, in _execute_command result = target.execute_command(command, pparams, kparams) File "/usr/lib/python2.7/dist-packages/targetcli/ui_node.py", line 104, in execute_command pparams, kparams) File "/usr/lib/python2.7/dist-packages/configshell/node.py", line 1416, in execute_command result = method(*pparams, **kparams) File "/usr/lib/python2.7/dist-packages/targetcli/ui_node.py", line 123, in ui_command_saveconfig CliConfig.save_running_config() File "/usr/lib/python2.7/dist-packages/targetcli/cli_config.py", line 65, in save_running_config config.load_live() File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 565, in load_live source=source, allow_new_attrs=True) File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 190, in _load_parse_tree token = self.validate_obj(token, cur) File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 377, in validate_obj valid_value = self.validate_val(valid_token['key'][1], id_type) File "/usr/lib/python2.7/dist-packages/rtslib/config.py", line 355, in validate_val % (val_type, value)) ConfigError: Unknown value type 'qla2xxx_wwn' when validating 21:00:00:24:ff:0e:7c:f5
修復方法，見https://github.com/bootc/rtslib/commit/727c345bd18137c424e4fba62bfab7bcfabfc024
vi /usr/share/pyshared/rtslib/config.py
#第349行增加
elif val_type == 'naa':
if is_valid_wwn('naa', value):
valid_value = value
+ elif val_type == 'qla2xxx_wwn':
+ if is_valid_wwn('qla2xxx_wwn', value):
+ valid_value = value
elif val_type == 'backend':
if is_valid_backend(value, parent):
valid_value = value
vi /usr/share/pyshared/rtslib/utils.py
#第562行增加
and re.match(
"[0-9A-Fa-f]{8}(-[0-9A-Fa-f]{4}){3}-[0-9A-Fa-f]{12}$", wwn):
return True
+ elif wwn_type == 'qla2xxx_wwn' \
+ and re.match(
+ "[0-9A-Fa-f]{2}(:[0-9A-Fa-f]{2}){7}$", wwn):
+ return True
else:
return False再次保存，就不出錯了。
附上這個環境下的虛擬機與運行在存儲上和vmware vsan下的虛擬機性能對比。
R730
NetAPP FC存儲
VMware VSAN
得益於Intel SSD 750的緩存，可以看到性能比FC存儲有了好幾倍的提升。
感謝以下參考資料：
http://linux-iscsi.org/wiki/Fibre_Channel
http://linux-iscsi.org/wiki/Targetcli
http://iori.tw/透過targetcli設定linux-io的fiber-channel-w-qlogic-cards/
https://github.com/bootc/rtslib/commit/727c345bd18137c424e4fba62bfab7bcfabfc024
常見錯誤解決：
qla2xxx: Unknown parameter `qlini_mode' #我在CentOS下遇到這個錯誤，最後發現是內核不支持，後面通過裝CentOS7.1或者編譯內核至3.x以上可以解決這個錯誤，但是在targetcli下看不到qla2xxx，最後換成了Ubuntu解決。
E: Package 'ubuntu-zfs' has no installation candidate #這個錯誤在Ubuntu安裝zfs文件系統的時候出現，原因是我裝的時候這個包還沒有支持ubuntu15.10，需要換舊版的Ubuntu