FastDFS數據遷移及(常用命令)

前提：所用鏡像image: luhuiguo/fastdfs
其他鏡像命令自行修改
思維流程導向：old-tracker storage ——>加一個新的storage做同步用——>查看狀態等待數據同步完成——>斷開連接——>修改tracker爲新的地址——>修改storage0/data/.data_init_flag配置文件（sync_src_server=）置爲空——>重新啓動storage——>成功！！！

1、安裝fastdfs（開始）

舊tracker：192.168.180.46
新tracker：192.168.180.48
eg：存儲一個文件返回路徑爲-做爲後續查找使用：http://192.168.180.46/group1/M00/00/00/wKi0Ll3bji6AKxJiAAAABncc3SA697_big.txt

2、修改新的Storage配置文件，部署在新的服務器②192.168.180.48上（只安裝storage，tracker先用舊的tracker）

vim leo_docker-compose.yaml

version: '3.0'

services:
   fastdfs:
     image: luhuiguo/fastdfs
     container_name: tracker
     network_mode: host
     command: tracker
     volumes:
     - ./fdfs/tracker:/var/fdfs

   storage0:
     container_name: storage0
     image: luhuiguo/fastdfs
     command: storage
     network_mode: host
     environment:
       - TRACKER_SERVER=192.168.180.46:22122
       - GROUP_NAME=group1
     volumes:
     - ./fdfs/storage0:/var/fdfs

注：TRACKER_SERVER配置爲舊的tracker

3、啓動storage和新的tracker

docker-compose -f leo_docker-compose.yaml up -d

4、查看監控狀態

新的storage節點已經加進來了
且ip_addr = 192.168.180.48 (test3) ACTIVE
如果ip_addr = 192.168.180.48 (test3) SYNCING表示正在同步數據
ip_addr = 192.168.180.48 (test3) WAIT_SYNC等待同步

docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf

[root@test1 luhuiguo_fastdfs]# docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf
[2019-11-25 08:31:31] DEBUG - base_path=/var/fdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 192.168.180.46:22122

group count: 2

Group 1:
group name = group1
disk total space = 120819 MB
disk free space = 100004 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.180.46
                ip_addr = 192.168.180.46 (test1)  ACTIVE
                ……
        Storage 2:
                id = 192.168.180.48
                ip_addr = 192.168.180.48 (test3)  ACTIVE
               ……

Group 2:
group name = group2
disk total space = 120819 MB
disk free space = 100004 MB
trunk free space = 0 MB
storage server count = 1
active server count = 1
storage server port = 22222
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.180.46
                ip_addr = 192.168.180.46 (test1)  ACTIVE
                ……

5、這時我們在新的服務器上查看之前在舊的fastdfs上傳的文件（根據之前存儲的返回路徑http://192.168.180.46/group1/M00/00/00/wKi0Ll3bji6AKxJiAAAABncc3SA697_big.txt）

[root@test3 data]# cat 00/00/wKi0Ll3bji6AKxJiAAAABncc3SA697_big.txt
hello

可以看到已經同步到這臺新的服務器上了

6、同步完成後,停掉新的Storage服務器進程

docker-compose -f leo_docker-compose.yaml stop storage0

再次查看狀態

docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf

# 可以看到已經下線了ip_addr = 192.168.180.48 (test3) OFFLINE

[2019-11-25 08:31:31] DEBUG - base_path=/var/fdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 192.168.180.46:22122

group count: 2

Group 1:
group name = group1
disk total space = 120819 MB
disk free space = 100004 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.180.46
                ip_addr = 192.168.180.46 (test1)  ACTIVE
                ……
        Storage 2:
                id = 192.168.180.48
                ip_addr = 192.168.180.48 (test3)  OFFLINE
               ……

Group 2:
group name = group2
disk total space = 120819 MB
disk free space = 100004 MB
trunk free space = 0 MB
storage server count = 1
active server count = 1
storage server port = 22222
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.180.46
                ip_addr = 192.168.180.46 (test1)  ACTIVE
                ……

7、（docker-compose啓動不用執行此步驟，二進制需要）修改.data_init_flag文件，將文件中sync_src_server配置項留空，其它配置項不變

sync_src_server=

[root@test3 data]# pwd
/home/fastdfs/fdfs/storage0/data
[root@test3 data]# ll -a
[root@test3 data]# vim .data_init_flag
storage_join_time=1574668867
sync_old_done=1
sync_src_server=
sync_until_timestamp=1574668870
last_ip_addr=192.168.180.48
last_server_port=23000
last_http_port=8888
current_trunk_file_id=0
trunk_last_compress_time=0

8、重新啓動storage

docker-compose -f leo_docker-compose.yaml up -d storage0

9、查看狀態已經切換到了新的tracker（結束）

docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf

[root@test3 fastdfs]# docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf
[2019-11-25 09:46:45] DEBUG - base_path=/var/fdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 192.168.180.48:22122

group count: 1

Group 1:
group name = group1
disk total space = 120819 MB
disk free space = 107597 MB
trunk free space = 0 MB
storage server count = 1
active server count = 1
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.180.48
                ip_addr = 192.168.180.48 (test3)  ACTIVE
                http domain =

驗證1、如果想要再次驗證，就再找一臺服務器創建storage連接新的tracker。
查看連接狀態（ip_addr = 192.168.180.47 (test2) ACTIVE ）爲驗證的服務器

[root@test3 fastdfs]# docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf
[2019-11-25 09:53:30] DEBUG - base_path=/var/fdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 192.168.180.48:22122

group count: 1

Group 1:
group name = group1
disk total space = 120819 MB
disk free space = 101093 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

        Storage 1:
                id = 192.168.180.47
                ip_addr = 192.168.180.47 (test2)  ACTIVE
                ……
        Storage 2:
                id = 192.168.180.48
                ip_addr = 192.168.180.48 (test3)  ACTIVE
                http domain =
                version = 5.11
                ……

1.1、在新加的storage上查看我們遷移過來的數據——因爲是同一個group所以是同步的，根據之前返回的數據路徑，在新加的storage服務器上查看。

[root@test2 cluster]# cat fdfs/storage0/data/00/00/wKi0Ll3bji6AKxJiAAAABncc3SA697_big.txt
hello

可以看到數據已經同步過來了！
使用備註：

docker-compose修改配置文件需要重啓其中一個service需要
docker-compose -f leo_docker-compose.yaml stop
docker-compose -f leo_docker-compose.yaml up -d storage0 #不能用start啓動

驗證2、

①、上傳
eg：
/root@localhost:/# usr/bin/fdfs_test /etc/fdfs/client.conf upload a.txt

②、下載
eg：[root@test1 fastdfs]# /usr/bin/fdfs_test /etc/fdfs/client.conf download group1 M00/00/00/wKi0Ll3bji6AKxJiAAAABncc3SA697_big.txt

Usage: /usr/bin/fdfs_test <config_file> download <group_name> <remote_filename>

③、刪除（所有storage中的文件都會刪除）
eg：[root@test1 fastdfs]# /usr/bin/fdfs_test /etc/fdfs/client.conf delete group1 M00/00/00/wKi0Ll3cwVCAHq1oAAAADj9UaMQ484.txt

Usage: /usr/bin/fdfs_test <config_file> delete <group_name> <remote_filename>

驗證3、不同情況下同步合併驗證

===========46===========
①上傳example file url: http://192.168.180.46/group1/M00/00/00/wKi0Ll3c2tGACRe1AAAACfdYteI663_big.txt
leojiang

②上傳example file url: http://192.168.180.46/group1/M00/00/00/wKi0Ll3c4vGAeuJnAAAACBgItVc245_big.txt
maosiyu

③同步lihongbao

④上傳example file url: http://192.168.180.46/group1/M00/00/00/wKi0Ll3c6C2AUFlvAAAAA7UVEzU990_big.txt
46
④同步48

===========48===========
①同步leojiang

②同步maosiyu

③上傳example file url: http://192.168.180.48/group1/M00/00/00/wKi0MF3c5R2AA5oKAAAACit7gaU604_big.txt
lihongbao

④上傳example file url: http://192.168.180.48/group1/M00/00/00/wKi0MF3c52SAVkcXAAAAA9SWPrs177_big.txt
48
④同步46

命令參考：

查看tracker的配置
docker exec -it tracker fdfs_monitor /etc/fdfs/client.conf
參數說明：
tracker_server_count：2 --表示2個Tracker Server
tracker server is 198.168.1.121:22122 --表示Leader Tracker
group count: 1	--表示有1個group
group name = group1	--組名稱是group1
storage server count = 2	--組內有2個storage
active server count = 2	--活動的storage有2個
storage server port = 23002	--storage的端口
storage HTTP port = 9101	--storage的文件訪問端口
store path count = 1	--storage只掛了一個存儲目錄
total_upload_count = 11	--總共上傳了多少個文件
total_upload_bytes = 691405	--總共上傳了多少字節
success_upload_bytes = 691405 --成功上傳了多少字節
total_download_count = 2	--總共下載了多少文件（使用java客戶端）

查看狀態：
docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf
# 從集羣中刪除storage
docker exec -it storage0 fdfs_monitor /etc/fdfs/client.conf delete group1 10.1.8.101
創建文件：
root@localhost:/# echo hello>a.txt
上傳文件：
/usr/bin/fdfs_test /etc/fdfs/client.conf upload a.txt
/usr/bin/fdfs_test /etc/fdfs/client.conf download group1 M00/02/32/CgAUD1s6716ATKtBAAMd3jd5Lxc105.png

1）wKjyglxkBuSAbT8iAAAAAc3XBpM982.txt 是主文件 。
2）wKjyglxkBuSAbT8iAAAAAc3XBpM982_big.txt 是從文件。
3)wKjyglxkBuSAbT8iAAAAAc3XBpM982.txt-m 文件 是 wKjyglxkBuSAbT8iAAAAAc3XBpM982.txt文件 屬性文件，意思就是：wKjyglxkBuSAbT8iAAAAAc3XBpM982.txt-m中存放的是wKjyglxkBuSAbT8iAAAAAc3XBpM982.txt文件的各項屬性的信息，比如文件的類型，高度和寬度等等屬性的信息。


upload：上傳普通文件，包括主文件
upload_appender：上傳appender文件，後續可以對其進行append、modify和truncate操作
upload_slave：上傳從文件
download：下載文件
delete：刪除文件
append：在appender文件後追加內容
modify：appender文件修改
set_metadata：設置文件附加屬性
get_metadata：獲取文件附加屬性

重啓跟蹤器：
/usr/bin/restart.sh /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf

重啓存儲器
/usr/bin/restart.sh /usr/bin/fdfs_storaged /etc/fdfs/storage.conf

參數形式如下：
volume0/M00/00/02/Cs8b8lFJIIyAH841AAAbpQt7xVI4715674

volume0：組名
M00：磁盤名
00/02：目錄
Cs8b8lFJIIyAH841AAAbpQt7xVI4715674：文件名，採用base64編碼，信息包含源storage server Ip、文件創建時間、文件大小、文件CRC32效驗碼和隨機數

（以上是個人驗證的結果，僅供參考）