Nginx+Keepalived實現Nginx負載均衡及高可用WEB服務器集羣

原創 fslc2004 2018-09-12 02:31


環境：
操作系統Centos 6.5 X86_64(final)
Nginx-Master:192.168.2.32
Nginx-Backup:192.168.3.31
VIP:192.168.2.33
Web3:192.168.2.29
Web4:192.168.2.30
1、分別在Nginx-Master和Nginx-Backup上安裝nginx
[root@Nginx-Master ~]# rpm -i http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm
[root@Nginx-Backup ~]# rpm -i http://nginx.org/packages/centos/6/noarch/RPMS/nginx-release-centos-6-0.el6.ngx.noarch.rpm

[root@Nginx-Master ~]# yum install nginx

[root@Nginx-Backup ~]# yum install nginx

2、先配置Nginx-Master上的nginx，建立配置一個proxy.conf文件。並複製一份到Nginx-Backup上，刪除兩臺機器上默認的default.conf,啓動nginx服務；並將服務添加到開機啓動。
[root@Nginx-Master ~]# vi /etc/nginx/conf.d/proxy.conf
    upstream web {
    ＃ip_hash;
        server 192.168.2.29:80;   #默認爲rr輪詢，如需解決session的問題採有哈希（ip_hash)模塊。
        server 192.168.2.30:80;
}
        server {
        listen 80;
        index index.php index.html index.htm;
        location / {
        proxy_pass http://web;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

[root@Nginx-Master ~]# scp /etc/nginx/conf.d/proxy.conf [email protected]:/etc/nginx/conf.d/

[root@Nginx-Master ~]# service nginx start
Starting nginx:                                    [  OK  ]

[root@Nginx-Backup ~]# service nginx start
Starting nginx:                                    [  OK  ]

[root@Nginx-Master ~]# chkconfig nginx on

[root@Nginx-Backup ~]# chkconfig nginx on

3、分別在Nginx-Master和Nginx-Backup上安裝keepalived並配置。

[root@Nginx-Master ~]# yum install keepalived
[root@Nginx-Backup ~]# yum install keepalived

[root@Nginx-Master ~]# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_nginx {                         #監控nginx服務進程腳本
    script "/root/nginx.sh"          
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    mcast_src_ip 192.168.2.32
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {                                    #調用監控腳本
        chk_nginx
}
    virtual_ipaddress {
        192.168.2.33
    }
}


[root@Nginx-Backup ~]# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_nginx {                        #監控nginx服務進程腳本
    script "/root/nginx.sh"       
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90
    mcast_src_ip 192.168.2.31
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {                                  #調用監控腳本
        chk_nginx

    virtual_ipaddress {
        192.168.2.33
    }
}

[root@Nginx-Master ~]# service keepalived start
Starting keepalived:                                   [  OK  ]

[root@Nginx-Backup ~]# service keepalived start
Starting keepalived:                                   [  OK  ]

3.1.對keepalived的不足寫的一個腳本，用來檢測本機的nginx是否正常的運行，如果nginx掛掉試着重新啓動，如果啓動後又掛掉，那麼就直接停止keepalived進程，keepalived將轉移到另一臺備用上，實現故障轉移
[root@Nginx-Master ~]# vi /root/nginx.sh
#!/bin/bash
A=`ps -C nginx --no-header |wc -l`                      # 查看是否有 nginx進程 把值賦給變量A
if [ $A -eq 0 ];then                                                #如果沒有進程值爲零
                /usr/sbin/nginx
                sleep 1
                if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
                       /etc/init.d/keepalived stop             # 則停止keepalived 進程
                fi
fi


[root@Nginx-Master ~]# chmod 755 /root/nginx.sh

[root@Nginx-Master ~]# scp /root/nginx.sh [email protected]:/root/nginx.sh

3.2.重啓keepalived服務，查看日誌腳本是否正常
Nginx-Master日誌
[root@Nginx-Master ~]# tail -f /var/log/messages
Sep  5 15:20:48 Nginx-Master Keepalived_vrrp[5136]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Sep  5 15:20:48 Nginx-Master Keepalived_vrrp[5136]: VRRP_Script(chk_nginx) succeeded                                                                     ＃檢查腳本成功，表示正常
Sep  5 15:20:48 Nginx-Master Keepalived_vrrp[5136]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep  5 15:20:49 Nginx-Master Keepalived_vrrp[5136]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep  5 15:20:49 Nginx-Master Keepalived_vrrp[5136]: VRRP_Instance(VI_1) setting protocol VIPs.

Nginx-Backup日誌
[root@Nginx-Backup ~]# tail -f /var/log/messages
Sep  5 15:20:41 Nginx-Backup Keepalived_vrrp[7670]: Using LinkWatch kernel netlink reflector...
Sep  5 15:20:41 Nginx-Backup Keepalived_vrrp[7670]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep  5 15:20:41 Nginx-Backup Keepalived_vrrp[7670]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Sep  5 15:20:41 Nginx-Backup Keepalived_vrrp[7670]: VRRP_Script(chk_nginx) succeeded                                                                     #檢查腳本成功，表示正常

[root@Nginx-Backup ~]# tail -f /var/log/messages
Sep  5 15:10:39 Nginx-Backup Keepalived_vrrp[5265]: Process [5913] didn't respond to SIGTERM
Sep  5 15:10:41 Nginx-Backup Keepalived_vrrp[5265]: Process [5923] didn't respond to SIGTERM                                                                        #如出現這種，看下腳本是否正確。
Sep  5 15:10:43 Nginx-Backup Keepalived_vrrp[5265]: Process [5933] didn't respond to SIGTERM

3.3.另外一種查看腳本是否生效方法,停止nginx服務，馬上會自動啓動   

[root@Nginx-Master ~]# service nginx status
nginx (pid  1114) is running...
[root@Nginx-Master ~]# service nginx stop
Stopping nginx:                                            [  OK  ]
[root@Nginx-Master ~]# service nginx status
nginx (pid  28751) is running...

4、測試VIP，檢查是否能主備切換

4.1.當兩臺主機同時啓動時，只有Nginx-Master服務器擁有VIP地址，備服務器沒有。

[root@Nginx-Master ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:a6:00:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.32/24 brd 192.168.2.255 scope global eth0
    inet 192.168.2.33/32 scope global eth0                        #可以看到Nginx-Master服務器上擁有192.168.2.33這個VIP地址
    inet6 fe80::250:56ff:fea6:13/64 scope link
       valid_lft forever preferred_lft forever

[root@Nginx-Backup ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 08:00:27:77:d3:82 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.31/24 brd 192.168.2.255 scope global eth0
    inet6 fe80::a00:27ff:fe77:d382/64 scope link                #Nginx-Backup服務器上沒有  
       valid_lft forever preferred_lft forever

4.2.當停止Nginx-Master服務器的keepalived服務，再查看下兩臺主機的VIP地址，發現VIP地址已從主服務器轉移到了備服務器

[root@Nginx-Master ~]# service keepalived stop
Stopping keepalived:                  [  OK  ]

[root@Nginx-Master ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:a6:00:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.32/24 brd 192.168.2.255 scope global eth0
    inet6 fe80::250:56ff:fea6:13/64 scope link                #Nginx-Master服務器VIP地址已移除      
       valid_lft forever preferred_lft forever

[root@Nginx-Backup ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 08:00:27:77:d3:82 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.31/24 brd 192.168.2.255 scope global eth0
    inet 192.168.2.33/32 scope global eth0         #VIP地地已自動轉移到Nginx-Backup服務器上            
    inet6 fe80::a00:27ff:fe77:d382/64 scope link
       valid_lft forever preferred_lft forever

5、日誌查看主備切換過程

5.1.當停止Nginx-Master上的keepalived服務時
root@Nginx-Master ~]# service keepalived stop
Nginx-Master日誌
[root@Nginx-Master ~]# tail -f /var/log/messages
Sep  4 18:04:06 Nginx-Master Keepalived[3278]: Stopping Keepalived v1.2.7 (02/21,2013)                                                                                  ＃主服務器已停掉
Sep  4 18:04:06 Nginx-Master Keepalived_vrrp[3281]: VRRP_Instance(VI_1) sending 0 priority    
Sep  4 18:04:06 Nginx-Master Keepalived_vrrp[3281]: VRRP_Instance(VI_1) removing protocol VIPs.

Nginx-Backup日誌
[root@Nginx-Backup ~]# tail -f /var/log/messages
Sep  4 18:04:07 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep  4 18:04:08 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) Entering MASTER STATE                                                                  #Nginx-Backup轉爲MASTER STATE
Sep  4 18:04:08 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep  4 18:04:08 Nginx-Backup Keepalived_healthcheckers[1427]: Netlink reflector reports IP 192.168.2.33 added
Sep  4 18:04:08 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33
Sep  4 18:04:13 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33

5.2.當Nginx-Master的keepalived服務再次啓動時
[root@Nginx-Master ~]# service keepalived restart
Stopping keepalived:                     [  OK  ]
Starting keepalived:                     [  OK  ]
Nginx-Master日誌
[root@Nginx-Master ~]# tail -f /var/log/messages
Sep  4 18:06:47 Nginx-Master Keepalived_vrrp[3316]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep  4 18:06:48 Nginx-Master Keepalived_vrrp[3316]: VRRP_Instance(VI_1) Entering MASTER STATE                                                   #Nginx-Master轉回MASTER STATE
Sep  4 18:06:48 Nginx-Master Keepalived_vrrp[3316]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep  4 18:06:48 Nginx-Master Keepalived_healthcheckers[3315]: Netlink reflector reports IP 192.168.2.33 added
Sep  4 18:06:48 Nginx-Master Keepalived_vrrp[3316]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33
Sep  4 18:06:53 Nginx-Master Keepalived_vrrp[3316]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.33

Nginx-Backup日誌
[root@Nginx-Backup ~]# tail -f /var/log/messages
Sep  4 18:06:47 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) Received higher prio advert
Sep  4 18:06:47 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) Entering BACKUP STATE                                                   #Nginx-Backup轉回BACKUP STATE
Sep  4 18:06:47 Nginx-Backup Keepalived_vrrp[1428]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep  4 18:06:47 Nginx-Backup Keepalived_healthcheckers[1427]: Netlink reflector reports IP 192.168.2.33 removed

6、安裝後端兩臺httpd服務器，並添加內容測試
[root@web3 ~]# yum install httpd

[root@web3 ~]# vi /var/www/html/index.html
<h1>Welcome to web3(192.168.2.29)</h1>

[root@web3 ~]# service httpd start

[root@web4 ~]# yum install httpd
[root@web4 ~]# vi /var/www/html/index.html
<h1>Welcome to web4(192.168.2.30)</h1>

[root@web4 ~]# service httpd start

7.查看後端web服務器日誌

7.1.後端web服務器日誌，訪問IP還是代理服務器的，不是真實客戶端IP
[root@web3 ~]# tail -f /var/log/httpd/access_log
192.168.2.31 - - [05/Sep/2014:15:47:46 +0800] "GET / HTTP/1.0" 200 39 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.3.1.2000 Chrome/30.0.1599.101 Safari/537.36"
192.168.2.32 - - [05/Sep/2014:15:58:02 +0800] "GET / HTTP/1.0" 200 39 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.3.1.2000 Chrome/30.0.1599.101 Safari/537.36"

[root@web4 ~]# tail -f /var/log/httpd/access_log
192.168.2.31 - - [05/Sep/2014:15:48:47 +0800] "GET / HTTP/1.0" 200 39 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.3.1.2000 Chrome/30.0.1599.101 Safari/537.36"
192.168.2.32 - - [05/Sep/2014:15:59:55 +0800] "GET / HTTP/1.0" 200 39 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.3.1.2000 Chrome/30.0.1599.101 Safari/537.36"

7.2.雖然在配置proxy.conf時已配置了proxy_set_header X-Real-IP $remote_addr；但還需要在web端修改配圍起文件顯示
[root@web3 ~]# vi /etc/httpd/conf/httpd.conf
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
改爲:
LogFormat "%{X-Real-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
web4也同樣修改以上數據後重啓

[root@web3 ~]# service httpd restart
[root@web4 ~]# service httpd restart

7.3.最後再查看後端web服務器日誌爲真實客戶端IP

[root@web3 ~]# tail -f /var/log/httpd/access_log
192.168.2.200 - - [05/Sep/2014:16:58:36 +0800] "GET / HTTP/1.0" 200 39 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.3.1.2000 Chrome/30.0.1599.101 Safari/537.36"

[root@web4 ~]# tail -f /var/log/httpd/access_log
192.168.2.200 - - [05/Sep/2014:16:58:42 +0800] "GET / HTTP/1.0" 200 39 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Maxthon/4.3.1.2000 Chrome/30.0.1599.101 Safari/537.36"

最後測試主keepalived掛掉，會轉移到備用keepalived上，nginx負載均衡服務，keepalived會一直調用腳本檢查機制，如果nginx服務掛掉或不能從新啓動，都會停止掉keepalived並立即轉移到備用上繼續工作，後端web服務器出現問題，nginx負載均衡能自動切換後端有故障的web服務器。



參考：http://network.51cto.com/art/201007/209823.htm

          http://friendlinux.blog.51cto.com/6249249/1433295

發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

bat記錄遠程桌面連接登錄信息

jason1982 2019-02-24 12:57:09

Exchange Server 2010 POP3&IMAP設置詳解

lingping 2019-02-23 14:05:54

虛擬化技術--服務器虛擬化

dawei818 2019-02-23 14:05:39

windows 2008 全新仲裁模式

qyh282110204 2019-02-23 14:05:36

軟路由建立PPPOE服務器

ct19871125 2019-02-23 14:04:24

MySQL性能優化的21個最佳實踐

fdb2b 2019-02-23 14:01:03

centos下簡單實現日誌切割，並上傳至日誌服務器。

隨風上升 2019-02-23 13:59:02

遠程訪問及控制

曉晶 2019-02-23 13:58:40

web網站服務1

曉晶 2019-02-23 13:58:27

郵件服務器

曉晶 2019-02-23 13:58:27

web網站服務2

曉晶 2019-02-23 13:58:27

iscsi存儲

samplelife 2019-02-23 13:57:35

Error response from daemon: OCI runtime create failed

uiz 2019-02-24 13:44:10

Vmware支持的操作系統

dawei818 2019-02-23 14:05:37

恢復IOS操作系統

wpgs 2019-02-23 14:05:20