GPFS centos7部署實戰
準備工作(每個節點)
集羣時間同步
yum install -y ntpdate
ntpdate ntp.sjtu.edu.cn
關閉不必要的服務
systemctl stop firewalld
systemctl disable firewalld
systemctl stop NetworkManager
systemctl disable NetworkManager
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
免密
root 本地和節點之間免密認證,ssh自己也要做免密的
ssh-keygen -t rsa
ssh-copy-id -i .ssh/id_rsa.pub gpfs1
ssh-copy-id -i .ssh/id_rsa.pub gpfs2
安裝(每個節點)
執行自動解壓安裝包
./Spectrum_Scale_Advanced-4.2.3.9-x86_64-Linux-install
進入解壓完的rpm包文件夾
cd /usr/lpp/mmfs/4.2.3.9/gpfs_rpms
yum安裝解決依賴問題
yum install -y gpfs.base-4.2.3-9.x86_64.rpm gpfs.docs-4.2.3-9.noarch.rpm gpfs.gpl-4.2.3-9.noarch.rpm gpfs.msg.en_US-4.2.3-9.noarch.rpm gpfs.gskit-8.0.50-86.x86_64.rpm gpfs.ext-4.2.3-9.x86_64.rpm
編譯前的準備(記得安裝內核頭文件和gcc gcc++等編譯器,建議使用本地光盤yum源)
yum install kernel-headers kernel-devel -y
yum groupinstall "Development Tools" -y
編譯安裝(如果編譯出錯,請檢查編譯器版本和內核版本)
cd /usr/lpp/mmfs/src/
make Autoconfig && make World && make InstallImages
環境變量導入
echo 'export PATH=$PATH:/usr/lpp/mmfs/bin' >> /etc/profile
source /etc/profile
創建集羣(管理節點)
編輯node描述文件 node.desc
目前認爲:manager-quorum屬於管理節點
gpfs_test1:manager-quorum
gpfs_test2:quorum
生成集羣
mmcrcluster -A -N node.desc -p gpfs_test1 -s gpfs_test2 -r /usr/bin/ssh -R /usr/bin/scp
查看集羣
mmlscluster
集羣改名
mmchcluster -C simple_test
配置許可
mmchlicense server --accept -N gpfs_test1,gpfs_test2
啓動集羣
mmstartup -a
查看集羣狀態,如果狀態爲active則爲正常
mmgetstate -a
NSD磁盤配置
編輯nsd描述文件 nsd.desc
/dev/vda2:gpfs_test1::dataAndMetadata::nsd1:system
/dev/vda2:gpfs_test2::dataAndMetadata::nsd2:system
創建nsd
mmcrnsd -F nsd.desc -v no
查看nsd狀態
mmlsnsd
配置仲裁盤(摸索中)
仲裁盤作用:當定義的仲裁盤有一半的磁盤不可用時,該集羣不可用。
仲裁節點和仲裁盤只能配置一種。
mmchconfig tiebreakerDisks="gpfs1nsd;gpfs2nsd;gpfs3nsd"
mmchconfig tiebreakerDisks="nsd1,nsd2,nsd3 "
但是,悲傷的問題就是會報錯,好難過555
絕招,設置個default 就不報錯了
mmchconfig tiebreakerDisks=DEFAULT
創建文件系統
在之前建立nsd後,nsd的描述文件會改變,如下
# /dev/vda2:gpfs_test1::dataAndMetadata::nsd1:system
nsd1:::dataAndMetadata:-1::system
# /dev/vda2:gpfs_test2::dataAndMetadata::nsd2:system
nsd2:::dataAndMetadata:-1::system
創建文件系統
mmcrfs wahaha -F nsd.desc -A yes -T /wahaha
第一個wahaha是gpfs的文件名,第二個/wahaha是文件系統的掛載點
查看文件系統的詳細信息
mmlsfs all
File system attributes for /dev/wahaha:
=======================================
flag value description
------------------- ------------------------ -----------------------------------
-f 8192 Minimum fragment size in bytes
-i 4096 Inode size in bytes
-I 32768 Indirect block size in bytes
-m 1 Default number of metadata replicas
-M 2 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 2 Maximum number of data replicas
-j cluster Block allocation type
-D nfs4 File locking semantics in effect
-k all ACL semantics in effect
-n 32 Estimated number of nodes that will mount file system
-B 262144 Block size
-Q none Quotas accounting enabled
none Quotas enforced
none Default quotas enabled
--perfileset-quota No Per-fileset quota enforcement
--filesetdf No Fileset df enabled?
-V 17.09 (4.2.3.9) File system version
--create-time Tue Dec 15 10:22:59 2020 File system creation time
-z No Is DMAPI enabled?
-L 4194304 Logfile size
-E Yes Exact mtime mount option
-S No Suppress atime mount option
-K whenpossible Strict replica allocation option
--fastea Yes Fast external attributes enabled?
--encryption No Encryption enabled?
--inode-limit 65792 Maximum number of inodes
--log-replicas 0 Number of log replicas
--is4KAligned Yes is4KAligned?
--rapid-repair Yes rapidRepair enabled?
--write-cache-threshold 0 HAWC Threshold (max 65536)
--subblocks-per-full-block 32 Number of subblocks per full block
-P system Disk storage pools in file system
-d nsd1;nsd2 Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /wahaha Default mount point
--mount-priority 0 Mount priority
掛載,掛載成功如下圖
mmmount wahaha -a
維護
開機啓動
echo "/usr/lpp/mmfs/bin/mmstartup -a" >> /etc/rc.local
或者
mmchconfig autoload=yes
常用命令
mmstartup
mmshutdown
mmmount
mmumount
mmlscluster
mmlsnsd
mmlsfs
mmlsconfig
參考:
https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_dnoupsp.htm
https://blog.csdn.net/cici_dh/article/details/46786387
https://www.jianshu.com/p/445753a851a4
https://blog.csdn.net/weixin_33755554/article/details/90252311