實戰nginx前端反代mogfilefs及負載均衡
===============================================================================
實驗描述：
使用Nginx代理請求至tackers，實現通過鍵就可以訪問到文件；
在配置nginx做反代將用戶的請求調度至後端的MogileFS，此實驗依賴於nginx-mogilefs-module模塊，需在編譯時加上此模塊。
實驗環境：
再準備一臺CentOS 7的虛擬主機，來作爲前端nginx反代服務器，因爲要想支持反代MogileFS需要
nginx-mogilefs-module模塊，所以，要編譯安裝nginx；
node1節點仍作爲tracker，mogstored和MySQL服務器；node2作爲tracker和mogstored節點；
IP地址規劃：
node1：192.168.1.112
node2：192.168.1.113
nginx：192.168.1.114
實驗拓撲圖：
實驗步驟如下：
要實現Nginx代理tackers，需要在編譯Nginx的時候加入第三方模塊nginx_mogilefs_module.
下載地址：http://www.grid.net.ru/nginx/mogilefs.en.html
1.準備開發環境，並解決依賴關係
yum groupinstall "Development Tools" "Server Platfrom Development" -y yum install pcre-devel openssl-devel zlib-devel -y
2.下載編譯安裝的nginx和第三方模塊nginx_mogilefs_module.的程序包，如下：
[root@centos7 nginx-mogilefs]# ls nginx-1.10.0.tar.gz nginx_mogilefs_module-1.0.4.tar.gz
3.編譯安裝Nginx，並且加入第三方模塊nginx_mogilefs_module.
# 解壓縮程序包 [root@centos7 nginx-mogilefs]# tar xf nginx-1.10.0.tar.gz [root@centos7 nginx-mogilefs]# tar xf nginx_mogilefs_module-1.0.4.tar.gz [root@centos7 nginx-mogilefs]# ls nginx-1.10.0 nginx-1.10.0.tar.gz nginx_mogilefs_module-1.0.4 nginx_mogilefs_module-1.0.4.tar.gz
創建nginx用戶，進入解壓之後的nginx-1.10.0目錄，執行編譯，如下：
# 首先添加nginx用戶 [root@centos7 nginx-mogilefs]# useradd -r nginx [root@centos7 nginx-mogilefs]# cd nginx-1.10.0/ [root@centos7 nginx-1.10.0]# ls auto CHANGES CHANGES.ru conf configure contrib html LICENSE man README src [root@centos7 nginx-1.10.0]# ./configure \ --prefix=/usr/local/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=nginx \ --group=nginx \ --with-http_ssl_module \ --with-http_flv_module \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-client-body-temp-path=/var/tmp/nginx/client/ \ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \ --http-scgi-temp-path=/var/tmp/nginx/scgi \ --with-pcre \ --with-debug \ --add-module=/root/nginx-mogilefs/nginx_mogilefs_module-1.0.4 #指明第三方模塊的位置即可 # [root@centos7 nginx-1.10.0]# make && make install
注意：
在執行make時提示了一個錯誤，如下：
通過在網上查詢得知，找到nginx目錄中/objs/Makefile文件，找到 -Werror 字段，去掉-Werror，重新編譯，則問題解決!
編譯完成後爲nginx提供PATH環境變量
[root@centos7 nginx-1.10.0]# cd /usr/local/nginx/ [root@centos7 nginx]# ls html sbin [root@centos7 nginx]# cd sbin/ [root@centos7 sbin]# ./nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: [emerg] mkdir() "/var/tmp/nginx/client/" failed (2: No such file or directory) nginx: configuration file /etc/nginx/nginx.conf test failed [root@centos7 sbin]# mkdir -p /var/tmp/nginx/client/ # 創建文件 [root@centos7 sbin]# ./nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful #======================================================================================= # 提供環境變量 [root@centos7 ~]# vim /etc/profile.d/nginx.sh export PATH=/usr/local/nginx/sbin:$PATH [root@centos7 ~]# . /etc/profile.d/nginx.sh [root@centos7 ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
4.配置nginx可以支持mogilefs模塊，在server塊中添加location僅使p_w_picpaths向後端代理，編輯nginx的配置文件/etc/nginx/nginx.conf 如下：
5.在node1或node2tracker節點上傳文件
[root@centos7 ~]# mogupload --trackers=192.168.1.112 --domain=p_w_picpaths --key='man.jpg' --file='/root/man.jpg' [root@centos7 ~]# mogupload --trackers=192.168.1.113 --domain=p_w_picpaths --key='new.jpg' --file='/root/new.jpg' [root@centos7 ~]# mogfileinfo --trackers=192.168.1.112 --domain=p_w_picpaths --key='man.jpg' - file: man.jpg class: default devcount: 2 domain: p_w_picpaths fid: 14 key: man.jpg length: 3401017 - http://192.168.1.112:7500/dev1/0/000/000/0000000014.fid - http://192.168.1.113:7500/dev3/0/000/000/0000000014.fid [root@centos7 ~]# mogfileinfo --trackers=192.168.1.113 --domain=p_w_picpaths --key='new.jpg' - file: new.jpg class: default devcount: 2 domain: p_w_picpaths fid: 15 key: new.jpg length: 1267947 - http://192.168.1.113:7500/dev3/0/000/000/0000000015.fid - http://192.168.1.112:7500/dev1/0/000/000/0000000015.fid
使用瀏覽器訪問nginx服務器，location後面跟key
注意：
上傳文件時， –key不能使用 –key='/man.jpg' ,要使用-key='man.jpg' 因爲nginx配置文件中直接定義了 location 爲 /p_w_picpaths/ ；如果，要使用-key='/man.jpg'，則在配置文件中的location就要定義爲/p_w_picpaths；
由於版本bug使用瀏覽器上傳功能並不能實現，但是刪除功能是可以實現的，這裏就不再演示，實際生產環境中應用程序都是通過調用API直接聯繫mogilefs上傳的，故此處不必過於糾結；
6.配置upstream，開啓負載均衡模式：
# 在http的配置段添加upstream
upstream mogtrackers {
server 192.168.1.112:7001;
server 192.168.1.113:7001;
}
# 修改server配置段中定義的location /p_w_picpaths/
location /p_w_picpaths/ {
mogilefs_tracker mogtrackers;
mogilefs_domain p_w_picpaths;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}模擬node1節點的tracker故障（killall mogilefsd），使用瀏覽器再次訪問nginx服務器，發現也可以正常訪問，如下：
nginx 完整的配置文件如下：
[root@centos7 ~]# cat /etc/nginx/nginx.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
upstream mogtrackers {
server 192.168.1.112:7001;
server 192.168.1.113:7001;
}
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
location /p_w_picpaths/ {
mogilefs_tracker mogtrackers;
mogilefs_domain p_w_picpaths;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_name localhost;
# ssl_certificate cert.pem;
# ssl_certificate_key cert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# location / {
# root html;
# index index.html index.htm;
# }
#}
}
如上，就是整個Nginx反代MogileFS的整個過程。。。
補充一下mogadm命令的額外指令：
mogadm slave //由於文件元數據信息是保存在MySQL中的，每次請求文件需要讀取MySQL一旦請求量太大，MySQL必定會成爲性能瓶頸，這時我們要給MySQL做主從，這個指令就是用來添加從節點的，可以實現tracker與主從架構的MySQL交互時，實現讀寫分離。
mogadm fsck //文件系統檢測，一般不要使用，除非集羣意外斷電，在上線時執行。
mogadm rebalance // 重新平衡，當數據出現熱區時使用，在執行之前需要定義平衡策略。
mogadm rebalance start // 啓動平衡策略。
mogadm rebalance policy //定義平衡策略。
mogadm rebalance test //用於測試是否出現數據不平衡的問題。
mogadm settings //定義mogilefs工作屬性。
mogadm class modify <domain> <class> --mindevcount= //定義文件最小的副本份數
[root@centos7 ~]# mogadm --help
Usage: (enter any command prefix, leaving off options, for further help)
mogadm check Check the state of the MogileFS world.
mogadm stats Show MogileFS system statistics. (DEPRECATED: use mogstats instead)
mogadm host ...
host add ... Add a host to MogileFS.
host delete ... Delete a host.
host list List all hosts.
host mark ... Change the status of a host. (equivalent to 'modify --status')
host modify ... Modify a host's properties.
mogadm device ...
device add ... Add a device to a host.
device list ... List all devices, for each host.
device mark ... Mark a device as {alive,dead,down,drain,readonly}
device modify ... Modify a device's properties.
device next Show the next available devid.
device summary ... List the summary of devices, for each host.
mogadm domain ...
domain add ... Add a domain (namespace)
domain delete ... Delete a domain.
domain list List all hosts.
mogadm class ...
class add ... Add a file class to a domain.
class delete ... Delete a file class from a domain.
class list List all classes, for each domain.
class modify ... Modify properties of a file class.
mogadm slave ...
slave add ... Add a slave node for store usage
slave delete ... Delete a slave node for store usage
slave list List current store slave nodes.
slave modify ... Modify a slave node for store usage
mogadm fsck ...
fsck clearlog Clear the fsck log
fsck printlog Display the fsck log
fsck reset ... Reset fsck position back to the beginning
fsck start Start (or resume) background fsck
fsck status Show fsck status
fsck stop Stop (pause) background fsck
fsck taillog Tail the fsck log
mogadm rebalance ...
rebalance policy ... Add or adjust the current policy
rebalance reset Reset an existing policy
rebalance settings Display rebalance settings
rebalance start Start a rebalance job
rebalance status Show status of current rebalance job
rebalance stop Stop a rebalance job
rebalance test Show what devices the current policy would match
mogadm settings ...
settings list List all server settings
settings set ... Set server setting 'key' to 'value'.另附三篇相關博客，在我配置過程中給了我很大幫助：