centos7部署lustre文件系統

1. 概述

本文使用kvm虛擬機node[19-24]在centos7.9操作系統，部署lustre文件系統2.12

集羣規劃如下：

節點名稱	磁盤數目	服務
node19/node20	共享/dev/vbd	mgt/mgs
node21/node22	共享/dev/vd{b,c}	mdt/mds
node23/node24	共享/dev/vd{b,c,d,e,f,g}	oss/ost
node25		client

2. 虛擬機準備

說明：本篇使用kvm的克隆功能，實現快速部署。

也可以創建多個節點以後，使用批處理命令完成操作

2.1 創建一個lustre內核操作系統模板

virsh start node19

啓動node19節點，創建yum源：vim /etc/yum.repo.d/whamcloud.repo

[lustre_latest_el7_client]
name=whamcloud_lustre_client
baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.9.2009/client
enabled=1
priority=5
gpgcheck=0

[lustre_latest_el7_server]
name=whamcloud_lustre_server
baseurl=https://downloads.whamcloud.com/public/lustre/latest-release/el7.9.2009/server
enabled=1
priority=5
gpgcheck=0

[e2fsprogs_latest]
name=whamcloud_e2fsprogs
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/
enabled=1
priority=5
gpgcheck=0

執行安裝yum插件：

yum -y install yum-plugin-priorities

執行安裝lustre相關軟件：

yum -y install e2fsprogs e2fsprogs-libs libss libcom_err kernel-*lustre kmod-lustre kmod-lustre-osd-ldiskfs kmod-lustre-tests lustre-osd-ldiskfs lustre-osd-ldiskfs-mount lustre-modules lustre-ldiskfs-dkms lustre

編輯內核配置文件：vim /etc/modprobe.d/lnet.conf

optinos lnet networks=tcp0(eth0)

編輯啓動腳本vim /etc/rc.local

modprobe lnet
modprobe lustre
modprobe ldiskfs

chmod +x /etc/rc.local

關閉node19節點：virsh destroy node19

2.2 使用模板克隆生成新節點

 for node in {20..24}
 do
 	virt-clone -o node19 -n node${node} -f /var/lib/libvirt/images/node${node}.qcow2 -m 00:00:00:80:00:${node}
 done 

2.3 爲虛擬機添加共享磁盤

本篇添加共享磁盤的方式爲：virsh edit xxx。

也可以通過修改xyz.xml後，使用virsh define /etc/libvirt/qemu/xyz.xml導入修改

先創建虛擬磁盤：

# 1. 創建lustre-mgt-000.img，掛載給node19和node20,盤符/dev/vdb
qemu-img create -f raw lustre-mgt-000.img 10G
# 2. 創建lustre-mdt-000.img和lustre-mdt-001.img，掛載給node21和node22,盤符/dev/vd{b,c}
qemu-img create -f raw lustre-mdt-000.img 10G
qemu-img create -f raw lustre-mdt-001.img 10G
# 3. 創建lustre-ost-000.img - lustre-ost-006.img，掛載給node23,node24，盤符/dev/vd{b,c,d,e,f,g}
for i in {000..006}
do
qemu-img create -f raw lustre-ost-${i}.img 10G
done

爲node19和node20添加共享磁盤

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-mgt-000.img'/>
      <target dev='vdb' bus='virtio'/>
      <shareable/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
    </disk>

爲node21和node22添加共享磁盤

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-mdt-000.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-mdt-001.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>

爲node23和node24添加共享磁盤

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-000.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-001.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-002.img'/>
      <target dev='vdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-003.img'/>
      <target dev='vde' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-004.img'/>
      <target dev='vdf' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-005.img'/>
      <target dev='vdg' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>

3.創建文件系統

3.1 mgs服務創建

在node19上執行：

# 格式化設備
mkfs.lustre --fsname=mylustre --reformat --mgs --servicenode=192.168.80.19@tcp0 --servicenode=192.168.80.20@tcp0 /dev/vdb
# 創建掛載點
mkdir -p /mnt/mgt000
# 掛載設備
mount.lustre /dev/vdb	/mnt/mgt000

node20上面不用創建，直接掛載即可

3.2 mds服務創建

在node21上執行：

# 格式化設備
mkfs.lustre --fsname=mylustre  --reformat --mdt --index=0  --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.21@tcp0 --servicenode=192.168.80.22@tcp0  /dev/vdb
mkfs.lustre --fsname=mylustre  --reformat --mdt --index=1  --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.21@tcp0 --servicenode=192.168.80.22@tcp0  /dev/vdc
# 創建掛載點
mkdir -p /mnt/mdt{000,001}
# 掛載設備
mount.lustre /dev/vdb /mnt/mdt000

在node22上執行：

# 創建掛載點
mkdir -p /mnt/mdt{000,001}
# 掛載設備
mount.lustre /dev/vdc /mnt/mdt001

3.3 oss服務創建

在node23上執行：

#!/bin/bash
#/dev/vdb
mkfs.lustre --fsname=mylustre --reformat --ost --index=0 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdb
#/dev/vdc
mkfs.lustre --fsname=mylustre --reformat --ost --index=1 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdc
#/dev/vdd
mkfs.lustre --fsname=mylustre --reformat --ost --index=2 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdd
# mkdir
mkdir -p /mnt/{ost000,ost001,ost002,ost003,ost004,ost005}
# mount
mount.lustre /dev/vdb /mnt/ost000
mount.lustre /dev/vdc /mnt/ost001
mount.lustre /dev/vdd /mnt/ost002

在node24上執行：

#!/bin/bash
#/dev/vde
mkfs.lustre --fsname=mylustre --reformat --ost --index=3 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vde
#/dev/vdf
mkfs.lustre --fsname=mylustre --reformat --ost --index=4 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdf
#/dev/vdg
mkfs.lustre --fsname=mylustre --reformat --ost --index=5 --mgsnode=192.168.80.19@tcp0:192.168.80.20@tcp0 --servicenode=192.168.80.23@tcp0 --servicenode=192.168.80.24@tcp0 /dev/vdg
# mkdir
mkdir -p /mnt/{ost000,ost001,ost002,ost003,ost004,ost005}
# mount
mount.lustre /dev/vde /mnt/ost003
mount.lustre /dev/vdf /mnt/ost004
mount.lustre /dev/vdg /mnt/ost005

4. 創建客戶端虛擬機並掛載

node25執行yum -y install lustre-client

創建掛載節點：mkdir -p /home/export/online1

掛載文件系統:mount.lustre 192.168.80.19@tcp0:192.168.80.20@tcp0:/mylustre /home/export/online1/

執行命令：lfs df -h

UUID                       bytes        Used   Available Use% Mounted on
mylustre-MDT0000_UUID        5.5G        2.0M        5.0G   1% /home/export/online1[MDT:0]
mylustre-MDT0001_UUID        5.5G        2.0M        5.0G   1% /home/export/online1[MDT:1]
mylustre-OST0000_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:0]
mylustre-OST0001_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:1]
mylustre-OST0002_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:2]
mylustre-OST0003_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:3]
mylustre-OST0004_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:4]
mylustre-OST0005_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:5]

filesystem_summary:        55.1G        8.1M       52.0G   1% /home/export/online1