1. 環境部署
- 我使用VirtualBox創建了六臺虛擬機，其中監控節點一個，元數據節點兩個，數據節點三個，並使用監控節點作爲客戶端。機器名和ip如下：
|
mon1.ihep.ac.cn |
192.168.56.107 |
|
mds1.ihep.ac.cn |
192.168.56.108 |
|
mds2.ihep.ac.cn |
192.168.56.109 |
|
osd1.ihep.ac.cn |
192.168.56.110 |
|
osd2.ihep.ac.cn |
192.168.56.111 |
|
osd3.ihep.ac.cn |
192.168.56.112 |
-
操作系統使用的的Fedora14，（這個內核版本是2.26.35，小小意見，最好使用2.26.34以上的版本，由於其內核包含Ceph客戶端以及btrfs，低版本折騰起來太費時間，當然這也取決於你追求啥，我就是想搭個環境測一測ceph)
-
所有虛擬機特別要注意的是一定要關掉iptables和selinux，反正這是測試環境，有時候的問題往往是他們引起的。關掉省心！（這只是測試，生產環境悠着點）
-
機器之間不用密碼直接登錄。在任一臺機器運行如下命令：
#ssh-keygen
#cd ~/.ssh; cat id_dsa.pub >>authorized_keys
#複製.ssh 文件夾到其他機器用戶home目錄上
2. 在六個節點都安裝ceph，步驟基本一致(出去數據節點）
-
下載ceph
#wget http://ceph.newdream.net/download/ceph-0.27.1.tar.gz
-
解壓，配置，安裝
#tar xzvf ceph-0.27.1.tar.gz
#cd ceph-0.27.1
# ./autogen.sh
#./configure ##可能有些會提示你有些依賴包沒裝，裝上就ok
# make
# make install
#cp ./src/sample.* /usr/local/etc/ceph/ ##拷貝ceph的配置文件，後面要修改
#mv /usr/local/etc/ceph/sample.ceph.conf /usr/local/etc/ceph/ceph.conf
#mv /usr/local/etc/ceph/sample.fetch_config /usr/local/etc/ceph/fetch_config
#cp ./src/init-ceph /etc/init.d/ceph
#mkdir /var/log/ceph; ##存放log，現在ceph自己還不自動建這個目錄
#mkdir /data ##存儲相應信息，下面配置會用到
-
針對數據節點安裝btrfs文件系統，特別注意，這一步只針對數據節點，其他類型節點沒必要
#yum install btrfs-progs
#fdisk /dev/sda (進入以後做一些選擇，不懂就man吧） ##這主要就是創建一個新的分區，我這裏sda對應的磁盤還有一部分空餘，所以在其上創建了一個/dev/sda4
#mkdf.btrfs /dev/sda4
3.配置。主要要修改的就是/usr/local/etc/ceph/下的兩個文件ceph.conf和fetch_config。
-
我使用的ceph.conf文件的內容;
; global
[global]
; enable secure authentication
;auth supported = cephx ##這個註釋掉吧，用來授權訪問的一個東東，測試沒關係的
; allow ourselves to open a lot of files
max open files = 131072
; set up logging
log file = /var/log/ceph/$name.log
; set up pid files
pid file = /var/run/ceph/$name.pid
; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /data/mon$id
; logging, for debugging monitor crashes, in order of
; their likelihood of being helpful :)
;debug ms = 1
;debug mon = 20
;debug paxos = 20
;debug auth = 20
[mon.0]
host = mon1.ihep.ac.cn
mon addr = 192.168.56.107:6789
;[mon.1]
; host = beta
; mon addr = 192.168.0.11:6789
;[mon.2]
; host = gamma
; mon addr = 192.168.0.12:6789
; mds
; You need at least one. Define two to get a standby.
[mds]
; where the mds keeps it's secret encryption keys
keyring = /data/keyring.$name
; mds logging to debug issues.
;debug ms = 1
;debug mds = 20
[mds.alpha]
host = mds1.ihep.ac.cn
[mds.beta]
host = mds2.ihep.ac.cn
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
; This is where the btrfs volume will be mounted.
osd data = /data/osd$id
; Ideally, make this a separate disk or partition. A few
; hundred MB should be enough; more if you have fast or many
; disks. You can use a file under the osd data dir if need be
; (e.g. /data/osd$id/journal), but it will be slower than a
; separate disk or partition.
; This is an example of a file-based journal.
osd journal = /data/osd$id/journal
osd journal size = 1000 ; journal size, in megabytes
; osd logging to debug osd issues, in order of likelihood of being
; helpful
;debug ms = 1
;debug osd = 20
;debug filestore = 20
;debug journal = 20
[osd.0]
host = osd1.ihep.ac.cn
; if 'btrfs devs' is not specified, you're responsible for
; setting up the 'osd data' dir. if it is not btrfs, things
; will behave up until you try to recover from a crash (which
; usually fine for basic testing).
btrfs devs = /dev/sda4
[osd.1]
host = osd2.ihep.ac.cn
btrfs devs = /dev/sda4
[osd.2]
host = osd3.ihep.ac.cn
btrfs devs = /dev/sda4
;[osd.3]
; host = eta
; btrfs devs = /dev/sdy
-
我使用的ceph.conf文件的內容
#!/bin/sh
conf="$1"
## fetch ceph.conf from some remote location and save it to $conf.
##
## make sure this script is executable (chmod +x fetch_config)
##
## examples:
##
## from a locally accessible file
## from a URL:
# wget -q -O $conf http://somewhere.com/some/ceph.conf
## via scp
# scp -i /path/to/id_dsa user@host:/path/to/ceph.conf $conf
scp [email protected]:/usr/local/etc/ceph/ceph/conf $conf
4. 創建文件系統並啓動。下面都做都在監控節點做。
#mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
#mkdir /etc/ceph; cp /usr/local/etc/ceph/* /etc/ceph ##這個我記不清楚了，反正似乎有用
#/etc/init.d/ceph -a start
5. 掛載
#mkdir /ceph
#mount.ceph 192.168.56.107:/ /ceph
參考文獻
1.http://ceph.newdream.net/wiki/
2. http://blog.sina.com.cn/s/blog_62218fc80100ntj5.html
3.http://blog.csdn.net/polisan/archive/2010/05/26/5624207.aspx