分佈式文件系統FastDFS配置與使用

       FastDFS 是一個用C語言編寫的分佈式文件系統，剛開始接觸FastDFS的時候，是因爲文件同步。後來發現自己完全理解錯誤，FastDFS並不是Rsync那樣文件同步，但是既然開始了就繼續深入瞭解她。FastDFS支持的文件同步是把文件上傳到調度器（tracker），然後按照管理員設定把文件下發到其中一臺文件存儲器（storage），成功後把文件同步到其他各臺機器。FastDFS也可以作爲文件存儲、文件上傳下載等，可以解決文件服務集羣的分佈式部署與負載均衡。一個典型的例子是充當圖片服務器，爲論壇、電商等提供高速可靠的圖片服務。FastDFS 客戶端開發也支持多語言（C、python、java、php等）API，爲項目開發提供便利。

1、系統環境

       三臺Centos_X64_6.5,一個tracker，兩個storage，其中一臺機器也模仿client的角色，FastDFS源碼安裝5.05版本。注意：三臺機器都要安裝FastDFS，而根據對應所在系統的角色配置相關配置文件，然後啓動fdfs_trackerd、fdfs_storaged服務。

2、FastDFS安裝過程

       以源碼方式安裝，需要提前安裝C語言開發環境，還要安裝cmake環境等，使用yum命令安裝以上開發工具也比較方便，如：yum install cmake gcc git。

2.1安裝依賴庫

安裝libevent命令行輸入下命令完成安裝：

[root@localhost Desktop]# mkdir FastNFS
[root@localhost Desktop]# cd FastNFS/
[root@localhost FastNFS]# git clone https://github.com/libevent/libevent.git
[root@localhost libevent]# rpm -qa libevent
[root@localhost libevent]# rpm -qa|grep libevent rpm -e libevent*
[root@localhost FastNFS]# cd libevent
[root@localhost libevent]# cd cmake
[root@localhost cmake]# yum install cmake
[root@localhost cmake]# cmake ..
[root@localhost cmake]# make
[root@localhost cmake]# make install

安裝libfastcommon，命令行輸入下命令完成安裝：

[root@localhost FastNFS]# git clone https://github.com/happyfish100/libfastcommon.git
[root@localhost FastNFS]# cd libfastcommon
[root@localhost libfastcommon]# ./make.sh
[root@localhost libfastcommon]# ./make.sh install

打印如下面內容即安裝成功：

mkdir -p /usr/lib64
mkdir -p /usr/lib
install -m 755 libfastcommon.so /usr/lib64
install -m 755 libfastcommon.so /usr/lib
mkdir -p /usr/include/fastcommon
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h /usr/include/fastcommon

2.2安裝FastDFS

下載源碼包、解壓、進入目錄完成安裝：

[root@localhost FastNFS]# wget https://github.com/happyfish100/fastdfs/archive/V5.05.tar.gz
[root@localhost FastNFS]# tar -zxvf V5.05.tar.gz
[root@localhost FastNFS]# cd fastdfs-5.05
[root@localhost fastdfs-5.05]# ./make.sh
[root@localhost fastdfs-5.05]# ./make.sh install

打印如下面內容即安裝成功：

mkdir -p /usr/bin
mkdir -p /etc/fdfs
cp -f fdfs_trackerd /usr/bin
if [ ! -f /etc/fdfs/tracker.conf.sample ]; then cp -f ../conf/tracker.conf /etc/fdfs/tracker.conf.sample; fi
mkdir -p /usr/bin
mkdir -p /etc/fdfs
cp -f fdfs_storaged  /usr/bin
if [ ! -f /etc/fdfs/storage.conf.sample ]; then cp -f ../conf/storage.conf /etc/fdfs/storage.conf.sample; fi
mkdir -p /usr/bin
mkdir -p /etc/fdfs
mkdir -p /usr/lib64
cp -f fdfs_monitor fdfs_test fdfs_test1 fdfs_crc32 fdfs_upload_file fdfs_download_file fdfs_delete_file fdfs_file_info fdfs_appender_test fdfs_appender_test1 fdfs_append_file fdfs_upload_appender /usr/bin
if [ 0 -eq 1 ]; then cp -f libfdfsclient.a /usr/lib64; fi
if [ 1 -eq 1 ]; then cp -f libfdfsclient.so /usr/lib64; fi
mkdir -p /usr/include/fastdfs
cp -f ../common/fdfs_define.h ../common/fdfs_global.h ../common/mime_file_parser.h ../common/fdfs_http_shared.h ../tracker/tracker_types.h ../tracker/tracker_proto.h ../tracker/fdfs_shared_func.h ../storage/trunk_mgr/trunk_shared.h tracker_client.h storage_client.h storage_client1.h client_func.h client_global.h fdfs_client.h /usr/include/fastdfs
if [ ! -f /etc/fdfs/client.conf.sample ]; then cp -f ../conf/client.conf /etc/fdfs/client.conf.sample; fi

查看FastDFS安裝後的配置文件：

[root@localhost fastdfs-5.05]# ll /etc/fdfs/

打印輸出下面信息：

total 20
-rw-r--r--. 1 root root 1461 Mar 21 03:01 client.conf.sample
-rw-r--r--. 1 root root 7829 Mar 21 03:01 storage.conf.sample
-rw-r--r--. 1 root root 7102 Mar 21 03:01 tracker.conf.sample

3、FastDFS簡單配置

3.1配置tracker

創建配置文件：

[root@localhost fastdfs-5.05]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf

創建base目錄

[root@localhost github.com]# mkdir -p /home/yuqing/fastdfs

打開配置文件，主要bind_addr、port、base_path配置幾個選項

[root@localhost fastdfs-5.05]# vi /etc/fdfs/tracker.conf

根據說明配置成下面內容：

disabled：是否關閉服務，false爲不關閉服務
bind_addr：綁定的IP地址
port：綁定的端口地址

base_path：指定文件存儲目錄

附上配置文件內容，方便查看解析說明：

# is this config file disabled
# false for enabled
# true for disabled
disabled=false


# bind an address of this host
# empty for bind all addresses of this host
bind_addr=


# the tracker server port
port=22122


# connect timeout in seconds
# default value is 30s
connect_timeout=30


# network timeout in seconds
# default value is 30s
network_timeout=60


# the base path to store data and log files
base_path=/home/yuqing/fastdfs


# max concurrent connections this server supported
max_connections=256


# accept thread count
# default value is 1
# since V4.07
accept_threads=1


# work thread count, should <= max_connections
# default value is 4
# since V2.00
work_threads=4


# the method of selecting group to upload files
# 0: round robin
# 1: specify group
# 2: load balance, select the max free space group to upload file
store_lookup=2


# which group to upload file
# when store_lookup set to 1, must set store_group to the group name
store_group=group2


# which storage server to upload file
# 0: round robin (default)
# 1: the first server order by ip address
# 2: the first server order by priority (the minimal)
store_server=0


# which path(means disk or mount point) of the storage server to upload file
# 0: round robin
# 2: load balance, select the max free space path to upload file
store_path=0


# which storage server to download file
# 0: round robin (default)
# 1: the source storage server which the current file uploaded to
download_server=0


# reserved storage space for system or other applications.
# if the free(available) space of any stoarge server in 
# a group <= reserved_storage_space, 
# no file can be uploaded to this group.
# bytes unit can be one of follows:
### G or g for gigabyte(GB)
### M or m for megabyte(MB)
### K or k for kilobyte(KB)
### no unit for byte(B)
### XX.XX% as ratio such as reserved_storage_space = 10%
reserved_storage_space = 10%


#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info


#unix group name to run this program, 
#not set (empty) means run by the group of current user
run_by_group=


#unix username to run this program,
#not set (empty) means run by current user
run_by_user=


# allow_hosts can ocur more than once, host can be hostname or ip address,
# "*" means match all ip addresses, can use range like this: 10.0.1.[1-15,20] or
# host[01-08,20-25].domain.com, for example:
# allow_hosts=10.0.1.[1-15,20]
# allow_hosts=host[01-08,20-25].domain.com
allow_hosts=*


# sync log buff to disk every interval seconds
# default value is 10 seconds
sync_log_buff_interval = 10


# check storage server alive interval seconds
check_active_interval = 120


# thread stack size, should >= 64KB
# default value is 64KB
thread_stack_size = 64KB


# auto adjust when the ip address of the storage server changed
# default value is true
storage_ip_changed_auto_adjust = true


# storage sync file max delay seconds
# default value is 86400 seconds (one day)
# since V2.00
storage_sync_file_max_delay = 86400


# the max time of storage sync a file
# default value is 300 seconds
# since V2.00
storage_sync_file_max_time = 300


# if use a trunk file to store several small files
# default value is false
# since V3.00
use_trunk_file = false 


# the min slot size, should <= 4KB
# default value is 256 bytes
# since V3.00
slot_min_size = 256


# the max slot size, should > slot_min_size
# store the upload file to trunk file when it's size <=  this value
# default value is 16MB
# since V3.00
slot_max_size = 16MB


# the trunk file size, should >= 4MB
# default value is 64MB
# since V3.00
trunk_file_size = 64MB


# if create trunk file advancely
# default value is false
# since V3.06
trunk_create_file_advance = false


# the time base to create trunk file
# the time format: HH:MM
# default value is 02:00
# since V3.06
trunk_create_file_time_base = 02:00


# the interval of create trunk file, unit: second
# default value is 38400 (one day)
# since V3.06
trunk_create_file_interval = 86400


# the threshold to create trunk file
# when the free trunk file size less than the threshold, will create 
# the trunk files
# default value is 0
# since V3.06
trunk_create_file_space_threshold = 20G


# if check trunk space occupying when loading trunk free spaces
# the occupied spaces will be ignored
# default value is false
# since V3.09
# NOTICE: set this parameter to true will slow the loading of trunk spaces 
# when startup. you should set this parameter to true when neccessary.
trunk_init_check_occupying = false


# if ignore storage_trunk.dat, reload from trunk binlog
# default value is false
# since V3.10
# set to true once for version upgrade when your version less than V3.10
trunk_init_reload_from_binlog = false


# the min interval for compressing the trunk binlog file
# unit: second
# default value is 0, 0 means never compress
# FastDFS compress the trunk binlog when trunk init and trunk destroy
# recommand to set this parameter to 86400 (one day)
# since V5.01
trunk_compress_binlog_min_interval = 0


# if use storage ID instead of IP address
# default value is false
# since V4.00
use_storage_id = false


# specify storage ids filename, can use relative or absolute path
# since V4.00
storage_ids_filename = storage_ids.conf


# id type of the storage server in the filename, values are:
## ip: the ip address of the storage server
## id: the server id of the storage server
# this paramter is valid only when use_storage_id set to true
# default value is ip
# since V4.03
id_type_in_filename = ip


# if store slave file use symbol link
# default value is false
# since V4.01
store_slave_file_use_link = false


# if rotate the error log every day
# default value is false
# since V4.02
rotate_error_log = false


# rotate error log time base, time format: Hour:Minute
# Hour from 0 to 23, Minute from 0 to 59
# default value is 00:00
# since V4.02
error_log_rotate_time=00:00


# rotate error log when the log file exceeds this size
# 0 means never rotates log file by log file size
# default value is 0
# since V4.02
rotate_error_log_size = 0


# keep days of the log files
# 0 means do not delete old log files
# default value is 0
log_file_keep_days = 0


# if use connection pool
# default value is false
# since V4.05
use_connection_pool = false


# connections whose the idle time exceeds this time will be closed
# unit: second
# default value is 3600
# since V4.05
connection_pool_max_idle_time = 3600


# HTTP port on this tracker server
http.server_port=8080


# check storage HTTP server alive interval seconds
# <= 0 for never check
# default value is 30
http.check_alive_interval=30


# check storage HTTP server alive type, values are:
#   tcp : connect to the storge server with HTTP port only, 
#        do not request and get response
#   http: storage check alive url must return http status 200
# default value is tcp
http.check_alive_type=tcp


# check storage HTTP server alive uri/url
# NOTE: storage embed HTTP server support uri: /status.html
http.check_alive_uri=/status.html

3.2配置storage

創建配置文件：

[root@localhost fastdfs-5.05]# cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf

創建base目錄：

[root@localhost fastdfs-5.05]# mkdir -p /home/yuqing/fastdfs

主要配置：group_name、bind_addr、base_path、store_path0、tracker_server

[root@localhost fastdfs-5.05]# vi /etc/fdfs/storage.conf

group_name：所在文件組
bind_addr：服務地址
port：服務端口
base_path：文件數據保存與日誌目錄
store_path0：文件數據保存目錄，覆蓋base_path的文件數據保存目錄配置
tracker_server：調度器的地址端口，IP:Port的形式

其他選項與調度器類是，請繼續查看storage的配置文件。

3.3配置client

創建配置文件：

[root@localhost fdfs]# cp /etc/fdfs/client.conf.sample client.conf

主要修改：base_path、tracker_server

[root@localhost fdfs]# vi /etc/fdfs/client.conf

配置文件如下：

# connect timeout in seconds
# default value is 30s
connect_timeout=30


# network timeout in seconds
# default value is 30s
network_timeout=60


# the base path to store log files
base_path=/home/yuqing/fastdfs


# tracker_server can ocur more than once, and tracker_server format is
#  "host:port", host can be hostname or ip address
tracker_server=192.168.213.161:22122


#standard log level as syslog, case insensitive, value list:
### emerg for emergency
### alert
### crit for critical
### error
### warn for warning
### notice
### info
### debug
log_level=info


# if use connection pool
# default value is false
# since V4.05
use_connection_pool = false


# connections whose the idle time exceeds this time will be closed
# unit: second
# default value is 3600
# since V4.05
connection_pool_max_idle_time = 3600


# if load FastDFS parameters from tracker server
# since V4.05
# default value is false
load_fdfs_parameters_from_tracker=false


# if use storage ID instead of IP address
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# default value is false
# since V4.05
use_storage_id = false


# specify storage ids filename, can use relative or absolute path
# same as tracker.conf
# valid only when load_fdfs_parameters_from_tracker is false
# since V4.05
storage_ids_filename = storage_ids.conf




#HTTP settings
http.tracker_server_port=80


#use "#include" directive to include HTTP other settiongs
##include http.conf

4、啓動服務

4.1啓動調度器

命令行輸入：

[root@localhost fastdfs-5.05]# fdfs_trackerd /etc/fdfs/tracker.conf 

查看日誌

[root@localhost fastdfs-5.05]# cat  /home/yuqing/fastdfs/logs/trackerd.log 

打印如下：

[2016-03-21 23:12:16] INFO - FastDFS v5.05, base_path=/home/yuqing/fastdfs, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=192.168.213.161, max_connections=256, accept_threads=1, work_threads=4, store_lookup=2, store_group=, store_server=0, store_path=0, reserved_storage_space=10.00%, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, use_storage_id=0, id_type_in_filename=ip, storage_id_count=0, rotate_error_log=0, error_log_rotate_time=00:00, rotate_error_log_size=0, log_file_keep_days=0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s

查看進程：

[root@localhost fastdfs-5.05]# ps -ef |grep track 

打印如下：

root     12839     1  0 23:12 ?        00:00:00 fdfs_trackerd /etc/fdfs/tracker.conf
root     12884  5571  0 23:15 pts/2    00:00:00 grep track

查看fdfs服務

[root@localhost fastdfs-5.05]# netstat -ntpl |grep fdfs 

打印如下：

tcp        0      0 192.168.213.161:22122       0.0.0.0:*                   LISTEN      12839/fdfs_trackerd 

4.2啓動文件存儲

命令行輸入：

[root@localhost fastdfs-5.05]# fdfs_storaged /etc/fdfs/storage.conf

查看日誌

[root@localhost fastdfs-5.05]# cat  /home/yuqing/fastdfs/logs/storaged.log 

打印如下：

[2016-03-21 23:31:50] INFO - FastDFS v5.05, base_path=/home/yuqing/fastdfs, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=192.168.213.136, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
data path: /home/yuqing/fastdfs/data, mkdir sub dir...
mkdir data path: 00 ...
mkdir data path: 01 ...
mkdir data path: 02 ...
mkdir data path: 03 ...
mkdir data path: 04 ...
mkdir data path: 05 ...
mkdir data path: 06 ...

查看進程：

[root@localhost fastdfs-5.05]# ps -ef |grep fdfs

打印如下：

root     16280     1  0 23:32 ?        00:00:00 fdfs_storaged /etc/fdfs/storage.conf
root     16310  1185  0 23:33 pts/2    00:00:00 grep fdfs

查看fdfs服務：

[root@localhost fastdfs-5.05]# netstat -ntpl |grep fdfs 

打印如下：

tcp        0      0 192.168.213.136:23000       0.0.0.0:*                   LISTEN      16280/fdfs_storaged 

5、測試FastDFS功能

在配置了client配置文件的機器上面進行上傳文件測試，結果應該返回上傳文件的保存路徑以及文件名，查看兩臺文件存儲器上相同目錄下以相同的文件名保存上傳的文件。

使用client上傳文件命令,將返回文件上傳結果，文件的保存目錄、文件名等信息：

上傳文件：

[root@localhost Desktop]# fdfs_upload_file /etc/fdfs/client.conf index.html

打印信息，是文件上傳路徑與文件名：

group1/M00/00/00/wKjVh1bw-1KADEceAAAsSKZJhlk53.html

驗證測試結果，兩臺文件服務器擁有相同的文件。

storage查看上傳文件：

[root@localhost 00]# ll /home/yuqing/fastdfs/data/00/00/

打印信息存在client上傳的文件wKjVh1bw-1KADEceAAAsSKZJhlk53.html：

total 24
-rw-r--r--. 1 root root 11336 Mar 22 00:59 wKjVh1bw-1KADEceAAAsSKZJhlk53.html
-rw-r--r--. 1 root root 11336 Mar 22 01:05 wKjViFbw_LyAIv-oAAAsSKZJhlk82.html

   

6、說明

FastDFS自帶訪問文件服務http功能，在安裝的時候有相關配置選項，如果打開了該http服務功能，可以按照client返回的上傳文件保存路徑以及文件名，在瀏覽器可以直接訪問。