前言:
上回講到,《lvs-dr模型負載均衡高可用Discuz》,但是由於lvs過於重量級,小公司使用有點不合時宜,本回咱們使用nginx實現此功能。望各位博友笑納。
此次試驗使用nginx代替lvs作爲前端調度器,使用keepalived對nginx做雙主模型高可用,向後做反向代理。RS服務器使用httpd+php來處理php頁面請求,httpd和php以模塊形式結合。暫時不考慮會話保持和動靜資源分離的問題。我們下一篇博客會做補充。一次寫太多會影響品質!!!
規劃:
前端使用兩臺nginx服務器作爲前端調度器,使用keepalived對nginx做雙主模型高可用,兩臺RS服務器,後端一臺共享存儲提供mariadb和NFS服務,因爲不再使用LVS調度,而是通過nginx反代來實現,所以兩臺RS服務器不需要在本地迴環接口的別名上配置VIP
各個主機的IP地址
Host VS1
eno16777736 10.0.0.201/8 (DIP)
gateway: 10.0.0.254
Host VS2
eno16777736 10.0.0.203/8(DIP)
gateway: 10.0.0.254
Host RS1
eno16777736 10.0.0.101/8 (RIP1)
gateway: 10.0.0.254
Host RS2
eno16777736 10.0.0.102/8 (RIP2)
gateway: 10.0.0.254
Host DB
eno16777736 10.0.0.202/8
gateway: 10.0.0.254
時間同步:
# ntpdate cn.pool.ntp.org # hwclock --sysohc
安裝軟件:
HostDB
安裝二進制mariadb-5.5.46
詳細配置請入傳送門:http://wscto.blog.51cto.com/11249394/1783131
安裝NFS
# yum install -y nfs-utils
HostRS1
安裝nginx,注意nginx屬於epel源
# yum install-y nginx php-fpm php-mbstring php-mysql nfs-utils mariadb
HostRS2
安裝nginx,注意nginx屬於epel源
# yum install-y nginx php-fpm php-mbstring php-mysql nfs-utils mariadb
Host VS1
安裝keepalived是爲了實現高可用,nginx實現反代 nfs-utils
# yum install -y keepalived nfs-utils # wget http://nginx.org/packages/centos/6/x86_64/RPMS/nginx-1.8.1-1.el6.ngx.x86_64.rpm # rpm -ivh nginx-1.8.1-1.el6.ngx.x86_64.rpm
注意:CentOS 6中nginx版本,比較低,所以可以這樣下載,在7中無需考慮
Host VS2
安裝keepalived是爲了實現高可用,nginx實現反代 nfs-utils
# yum install -y keepalived nfs-utils # wget http://nginx.org/packages/centos/6/x86_64/RPMS/nginx-1.8.1-1.el6.ngx.x86_64.rpm # rpm -ivh nginx-1.8.1-1.el6.ngx.x86_64.rpm
HostDB
完成安全初始化後,創建discuz數據庫和discuz用戶,並授權其可以遠程操作數據庫
# mysql_secure_installation # mysql -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 3 Server version: 5.5.46-MariaDB-log MariaDB Server Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database discuz; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> create user 'discuz'@'localhost' identified by 'magedu'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> grant all privileges on discuz.* to 'discuz'@'%' identified by 'magedu'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> flush privileges; Query OK, 0 rows affected (0.00 sec)
配置NFS
# mkdir /nfshare/ # ls -dl /nfshare/ drwxr-xr-x 2 root root 6 May 9 17:01 /nfshare/ # echo "/nfshare/ 10.0.0.101(rw,no_root_squash,async) 10.0.0.102(rw,no_root_squash,async)" > /etc/exports # systemctl start rpcbind <--如果是CentOS-6要使用/etc/init.d/rpcbind start Starting rpcbind: [ OK ] [root@34400575 ~]# /etc/init.d/nfs start Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS mountd: [ OK ] Starting NFS daemon: [ OK ] Starting RPC idmapd: [ OK ] # systemctl enable rpcbind <--如果是6 chkconfig rpcbind on # systemctl enable rpcbind <--如果是6 chkconfig nfs on [root@34400575 ~]# chkconfig rpcbind --list rpcbind 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@34400575 ~]# chkconfig nfs --list nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@34400575 ~]# showmount -e 127.0.0.1 Export list for 127.0.0.1: /nfshare 172.18.71.102,172.18.71.101 此步驟如果出現問題,請查看《lvs-dr負載均衡Discuz》
解壓discuz的程序包至/nfsshare/目錄
# mkdir /nfshare/discuz # unzip /Discuz_X3.2_SC_UTF8.zip -d /nfshare/discuz/ # ls /nfshare/discuz/ readme upload utility
Host RS1
測試鏈接HostDB上的mariadb
# mysql -h 10.0.0.202 -u discuz -p
掛載HostDB上的NFS共享存儲目錄,屬主屬組改爲apache
# showmount -e 10.0.0.202 Export list for 10.0.0.202: /nfshare 10.0.0.102,10.0.0.101 [root@b9cf468b ~]# mkdir /htdocs [root@b9cf468b ~]# ls -ld /htdocs/ drwxr-xr-x 2 root root 6 May 9 17:05 /htdocs/ [root@b9cf468b ~]# mount -t nfs 10.0.0.202:/nfshare /htdocs [root@b9cf468b ~]# ls /htdocs/ discuz [root@b9cf468b ~]# chown -R apache:apache /htdocs/discuz/
配置httpd,測試沒有語法錯誤,啓動httpd
# vim /etc/httpd/conf.d/vhosts.conf ... <VirtualHost *:80> ServerName admin.ws.com DocumentRoot "/htdocs/discuz/upload/" <Directory "/htdocs/discuz/upload/"> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> </VirtualHost> # systemctl start httpd.service # ss -tnl | grep 80 LISTEN 0 128 :::80 :::*
客戶端訪問:http://10.0.0.102/install/
注意一定要在這裏安裝完成,否則搭建完反代再安裝就晚了。可能會不識別靜態內容
Host RS2
與Host RS1的配置相同,此處不再複製
Host VS1
掛在HostDB上的NFS共享存儲到Host VS1上
# showmount -e 10.0.0.202 Export list for 10.0.0.202: /nfshare 10.0.0.0/8 # mkdir /htdocs # mount -t nfs 10.0.0.202:/nfshare /htdocs
使用upstream模塊爲後端RS主機做反代,併爲其配置SorryServer,在後端RS主機宕機的時候,能夠給用戶一些友好的提示
# vim /etc/nginx/conf.d/default.conf upstream backend { server 10.0.0.101 weight=1 max_fails=3 fail_timeout=3; server 10.0.0.102 weight=1 max_fails=3 fail_timeout=3; server 10.0.0.111:8080 backup; <--當所有RS不可用時,backup可以定義一臺備用主機 server 10.0.0.112:8080 backup; } server { listen 80; server_name localhost; # 當nginx將php代碼送至後端RS處理時請求頭中的Host值會是backend. # php代碼在RS上處理時,其內部代碼會去請求圖片/層疊樣式表等靜態資源以填充頁面. # 而php代碼去請求靜態資源時使用的是如http://backend/xxxx.gif這樣的url,自然是取不到的. # 所以我們要在nginx向後代理遇到Host爲backend時,將其轉換爲127.0.0.1. set $my_host $http_host; if ($http_host = "backend") { set $my_host "127.0.0.1"; } location / { proxy_pass http://backend; proxy_set_header Host $my_host; } } server { listen 8080; server_name localhost; charset utf-8; root /usr/local/nginx/html; index index.html index.htm; # sorry_server僅提供主頁面, 訪問其它資源也轉至主頁面. location ~ .* { error_page 404 /; } }
生成sorry_server的頁面,測試語法無誤後啓動nginx
# echo "服務維護中,請稍後訪問." > /usr/share/nginx/html/index.html # nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@8a028eb8 ~]# nginx [root@8a028eb8 nginx]# ss -tnl | grep 80 LISTEN 0 128 *:8080 *:* LISTEN 0 128 *:80 *:*
修改keepalived配置文件,此處無需像lvs一樣定義virtual_server和real_server
調度算法這些東西了,這些工作全由nginx完成
# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost # 郵件給本機root用戶 } notification_email_from [email protected] smtp_server 127.0.0.1 # 使用本機作爲smtp服務器 smtp_connect_timeout 30 router_id 8a028eb8 # 標識主機,可以使用主機名. vrrp_mcast_group4 224.0.71.18 # 多播地址,用於發送心跳信號.儘量讓集羣內的主機處於同一個獨立的多播地址. } # nginx進程監控腳本.如果進程不在,降低自身權重,使從節點主機優先級高於自身,將VRRP漂移至從節點主機. vrrp_script chk_nginx { script "killall -0 nginx" interval 2 weight -8 } vrrp_instance VI_1 { state MASTER # vrrp實例VI_1中HostA作爲主 interface eth0 virtual_router_id 71 # 0-255範圍內的數字,用於區分vrrp實例,所以兩個實例不能一致. priority 100 # MASTER的優先級要高一些 advert_int 1 authentication { auth_type PASS auth_pass uWjblY61 # 簡單字符認證, 8位任意字符串. } virtual_ipaddress { 10.0.0.0.111/8 brd 10.0.0.111 dev eno16777736 label eno16777736:0 # VIP1 } # 在此處調用nginx進程監控腳本 track_script { chk_nginx } # 關閉爭用. 爭用是指當高優先級節點上線會立即爭奪成爲MASTER # 而不管其它節點是否正在給用戶提供服務. #nopreempt # 開啓爭用時,會延遲一段時間纔開始. #preempt_delay 300 } vrrp_instance VI_2 { state BACKUP # vrrp實例VI_2中HostA作爲備 interface eth0 virtual_router_id 171 priority 95 # BACKUP的優先級要低一些 advert_int 1 authentication { auth_type PASS auth_pass uWjblY62 } virtual_ipaddress { 10.0.0.112/8 brd 10.0.0.112 dev eno16777736 label eno16777736:1 # VIP2 } # 在此處調用nginx進程監控腳本 track_script { chk_nginx } }
啓動keepalived服務,並檢查ip地址和日誌,可以看到HostVS1 VI_2實例的BACKUP,優先級較低,但是由於作爲VI_2實例MASTER的HostVS2主機還沒有啓動keepalived服務,所以HostVS1獲取到了VI_2實例的VIP 10.0.0.112
Host VS2
掛載HostDB上的NFS共享存儲目錄
# showmount -e 10.0.0.202 Export list for 10.0.0.202: /nfshare 10.0.0.0/8 # mkdir /htdocs # mount -t nfs 10.0.0.202:/nfshare /htdocs
使用upstream模塊爲後端RS主機做反代,併爲其配置SorryServer,在後端RS主機宕機的時候,能夠給用戶一些友好的提示
# vim /etc/nginx/conf.d/default.conf upstream backend { server 10.0.0.101 weight=1 max_fails=3 fail_timeout=3; server 10.0.0.102 weight=1 max_fails=3 fail_timeout=3; server 10.0.0.111:8080 backup; server 10.0.0.112:8080 backup; } server { listen 80; server_name localhost; # 當nginx將php代碼送至後端RS處理時請求頭中的Host值會是backend. # php代碼在RS上處理時,其內部代碼會去請求圖片/層疊樣式表等靜態資源以填充頁面. # 而php代碼去請求靜態資源時使用的是如http://backend/xxxx.gif這樣的url,自然是取不到的. # 所以我們要在nginx向後代理遇到Host爲backend時,將其轉換爲127.0.0.1. set $my_host $http_host; if ($http_host = "backend") { set $my_host "127.0.0.1"; } location / { proxy_pass http://backend; proxy_set_header Host $my_host; } } server { listen 8080; server_name localhost; charset utf-8; root /usr/local/nginx/html; index index.html index.htm; # sorry_server僅提供主頁面, 訪問其它資源也轉至主頁面. location ~ .* { error_page 404 /; } }
生成sorry_server的頁面,測試語法無誤後啓動nginx
# echo "服務維護中,請稍後訪問." > /usr/share/nginx/html/index.html # nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@8a028eb8 ~]# nginx [root@8a028eb8 nginx]# ss -tnl | grep 80 LISTEN 0 128 *:8080 *:* LISTEN 0 128 *:80
修改Host VS2的keepalived配置文件,配置文件無需大改,只需要修改二者的主從關係和優先級即可
# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost # 郵件給本機root用戶 } notification_email_from [email protected] smtp_server 127.0.0.1 # 使用本機作爲smtp服務器 smtp_connect_timeout 30 router_id 8a028eb8 # 標識主機,可以使用主機名. vrrp_mcast_group4 224.0.71.18 # 多播地址,用於發送心跳信號.儘量讓集羣內的主機處於同一個獨立的多播地址. } # nginx進程監控腳本.如果進程不在,降低自身權重,使從節點主機優先級高於自身,將VRRP漂移至從節點主機. vrrp_script chk_nginx { script "killall -0 nginx" interval 2 weight -8 } vrrp_instance VI_1 { state BACKUP # vrrp實例VI_1中HostA作爲主 interface eno16777736 virtual_router_id 71 # 0-255範圍內的數字,用於區分vrrp實例,所以兩個實例不能一致. priority 98 # MASTER的優先級要低一些 advert_int 1 authentication { auth_type PASS auth_pass uWjblY61 # 簡單字符認證, 8位任意字符串. } virtual_ipaddress { 10.0.0.0.111/8 brd 10.0.0.111 dev eno16777736 label eno16777736:0 # VIP1 } # 在此處調用nginx進程監控腳本 track_script { chk_nginx } # 關閉爭用. 爭用是指當高優先級節點上線會立即爭奪成爲MASTER # 而不管其它節點是否正在給用戶提供服務. #nopreempt # 開啓爭用時,會延遲一段時間纔開始. #preempt_delay 300 } vrrp_instance VI_2 { state BACKUP # vrrp實例VI_2中HostA作爲備 interface eno16777736 virtual_router_id 171 priority 100 # BACKUP的優先級要高一些 advert_int 1 authentication { auth_type PASS auth_pass uWjblY62 } virtual_ipaddress { 10.0.0.112/8 brd 10.0.0.112 dev eno16777736 label eno16777736:1 # VIP2 } # 在此處調用nginx進程監控腳本 track_script { chk_nginx } }
啓動keepalived服務,並檢查IP地址和日誌,可以看到HostVS2作爲VI_2實例的MASTER,優先級較高,在爭用模式下,HostVS2立刻獲取到了VI_2的VIP 10.0.0.112
測試:分別訪問http://10.0.0.111/和10.0.0.112/
輪流關閉Host VS1和Host VS2的keepalived服務和nginx服務,都不會影響用戶訪問
由於nginx對後端RS主機的健康狀態檢測模塊health_check已經被商業化,我們只能考慮使用由淘寶技術團隊開發的tengine,所以我們要想實現,某一臺RS宕機之後,仍然能繼續工作,以及RS全部宕機還能夠提供Sorry-server頁面就需要使用tengine,我們下一篇博客細細講解