Nginx+Keepalived簡單構建高可用集羣

以前一直用heartbeat或corosync+pacemaker構建高可用集羣，現在發現keepalived實現起來更簡單。
keepalived的master向backup發送廣播，當backup一段時間收不到對方傳來的VRRP廣播時，backup會通過競選一個master,master就會重新持有資源。具體的理論知識參見http://bbs.ywlm.net/thread-790-1-1.html

實驗目標:2臺Nginx+Keepalived 2臺Lamp構建高可用Web集羣

規劃：

ng1.laoguang.me 192.168.1.22 ng1
ng2.laoguang.me 192.168.1.23 ng2
lamp1.laoguang.me   192.168.1.24 lamp1
lamp2.laoguang.me   192.168.1.25 lamp2

拓撲：

一.基本環境準備
ng1,ng2上安裝nginx
lamp1,lamp2上構建LAMP或只安裝httpd，我只安裝了Httpd，這裏不給大家演示了，有需要請看我的其它博文，更改lamp1,lamp2的index.html的內容分別爲lamp1和lamp2,以容易區分，實際集羣中內容應該是一致的，由共享存儲提供。

二.ng1,ng2上安裝配置keepalived
下載地址:http://www.keepalived.org/download.html
2.1 安裝keepalived

tar xvf keepalived-1.2.7.tar.gz
cd keepalived-1.2.7
./configure --prefix=/usr/local/keepalived
##可能會提示安裝popt-devel包，yum即可
make && make install

2.2 整理配置文件與腳本

mkdir /etc/keepalived
##keepalived默認配置文件從/etc/keepalived下讀取
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
##就一個二進制文件，直接拷貝過去即可，多的話就更改PATH吧
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
##腳本的額外配置文件讀取位置
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
##啓動腳本你懂得
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
##我們關鍵的keepalived配置文件

2.3 修改ng1的/etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   notification_email {
     [email protected]         ##出故障發送郵件給誰
   }
   notification_email_from keepalived@localhost ##故障用哪個郵箱發送郵件
   smtp_server 127.0.0.1   ##SMTP_Server IP
   smtp_connect_timeout 30 ##超時時間
   router_id LVS_DEVEL     ##服務器標識
}
vrrp_instance VI_1 {
    state BACKUP
##狀態，都爲BACKUP，它們會推選Master，如果你寫MASTER，它就會是Master，
    ##當Master故障時Backup會成爲Master,當原來的Master恢復後，原來的Master會成爲Master
    interface eth0        ##發送VRRP的接口，仔細看你的是不是eth0
    virtual_router_id 51  ##虛擬路由標識，同一個組應該用一個，即Master與Backup同一個
    priority 100   ##重要的優先級哦
    nopreempt      ##不搶佔，一個故障時，重啓後恢復後不搶佔意資源
    advert_int 1   ##同步間隔時長
    authentication {              ##認證
        auth_type PASS            ##認證方式
        auth_pass www.laoguang.me ##密鑰
    }
    virtual_ipaddress {
        192.168.1.18/24 dev eth0              ##VIP
    }
}
##後面的刪除吧，LVS上纔有用

拷貝到ng2上一份，只修改priority 90 即可

scp /etc/keepalived/keepalived.conf 192.168.1.23:/etc/keepalived/
##Ng2上
vi /etc/keepalived/keepalived.conf  priority 90   ##其它一致

2.4 ng1,ng2上啓動keepalived

service keepalived start

查看日誌

tail /var/log/messages
Nov 27 08:07:54 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 27 08:07:54 localhost Keepalived_vrrp[41871]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Nov 27 08:07:54 localhost Keepalived_healthcheckers[41870]: Using LinkWatch kernel netlink reflector...
Nov 27 08:07:54 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) forcing a new MASTER election
Nov 27 08:07:55 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 27 08:07:56 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 27 08:07:56 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 27 08:07:56 localhost Keepalived_healthcheckers[41870]: Netlink reflector reports IP 192.168.1.18 added
Nov 27 08:07:56 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.18
Nov 27 08:08:01 localhost Keepalived_vrrp[41871]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.18

查看vip綁定到哪臺機器上了

ip addr     ##ng1上
....省略
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e8:90:0b brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.22/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.18/32 scope global eth0
    inet6 fe80::20c:29ff:fee8:900b/64 scope link
       valid_lft forever preferred_lft forever

由此可知vip綁定到ng1上了
三，Keepalived測試

3.1 關閉ng1上的keepalived或者直接關閉ng1 查看vip轉移情況

service keepalived stop
ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e8:90:0b brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.22/24 brd 192.168.1.255 scope global eth0
    inet6 fe80::20c:29ff:fee8:900b/64 scope link
       valid_lft forever preferred_lft forever

3.2 查看ng2上是否綁定了vip

ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:dd:00:77 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.23/24 brd 192.168.1.255 scope global eth0
    inet 192.168.1.18/32 scope global eth0
    inet6 fe80::20c:29ff:fedd:77/64 scope link
       valid_lft forever preferred_lft forever

由此可知ip轉移正常，keepalived設置成功

四.配置Nginx做反向代理

4.1 修改nginx配置文件

vi /etc/nginx/nginx.conf
user  nginx nginx;   ##運行nginx的用戶和組
worker_processes  2; ##啓動進程數
error_log /var/log/nginx/error.log  notice;  ##錯誤日誌記錄
pid        /tmp/nginx.pid;                   ##pid存放位置
worker_rlimit_nofile 65535;                  ##線程最大打開文件數，須配合ulimit -SHn使用
events {
    use epoll;                 ##工作模型
    worker_connections  65536; ##單進程最大連接數
}
http {                         ##http模塊
    include       mime.types;  ##包含進來
    default_type  application/octet-stream; ##默認類型
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
##日誌格式
    access_log  /var/logs/nginx/http.access.log  main; ##訪問日誌
    client_max_body_size 20m;  ##最大請求文件大小
    client_header_buffer_size 16k; ##來自客戶端請求header_buffer大小
    large_client_header_buffers 4 16k; ##較大請求緩衝個數與大小
    sendfile       on;                 ##內核空間直接發送到tcp隊列
    tcp_nopush     on;
    tcp_nodelay    on;
    keepalive_timeout  65;     ##長連接時長
    gzip  on;                  ##啓用壓縮
    gzip_min_length 1k;        ##最小壓縮大小
    gzip_buffers 4 16k;        ##壓縮緩衝
    gzip_http_version 1.1;     ##支持協議
    gzip_comp_level 2;         ##壓縮等級
    gzip_types text/plain application/x-javascript text/css application/xml;      ##壓縮類型
    gzip_vary on;              ##前端緩存服務器可以緩存壓縮過的頁面
    upstream laoguang.me {     ##用upstream模塊定義集羣與RS
        server 192.168.1.24:80 max_fails=3fail_timeout=10s;   ##RS的地址，最大錯誤數與超時時間，超過了自動剔除
        server 192.168.1.25:80 max_fails=3fail_timeout=10s;
}
 server {
        listen       80;           ##監聽端口
        server_name  192.168.1.18; ##servername
        root   html;               ##根目錄
        index  index.html index.htm; ##你懂得
        #charset koi8-r;
        access_log  logs/192.168.1.18.access.log  main;
  ##這個server的訪問日誌
        location / {
                proxy_pass http://laoguang.me;  ##反向代理
                proxy_redirect off;
                proxy_set_header X-Real-IP $remote_addr;
 ##真實客戶ip告訴後端
                proxy_set_header X-Forwarded-For Proxy_add_x_forwarded_for;
        }
        location /nginx {
                access_log off;
                stub_status on;  ##狀態頁面
        }
        error_page   500 502 503 504  /50x.html;
location = /50x.html {
            root   html;
        }
    }
}

4.2 拷貝到ng2上一份

scp /etc/nginx/nginx.conf 192.168.1.23:/etc/nginx/

4.3 測試反向代理能否負載均衡

lamp1,lamp2啓動httpd

service httpd start

ng1重啓nginx

service nginx restart

用RealIp訪問測試能否輪詢
http://192.168.1.22

同樣測試ng2,如果都能實現負載均衡，那麼繼續

五.測試keepalived與nginx配合運行

現在192.168.1.18在 ng2上，        訪問 http://192.168.1.18 測試能否輪詢
ng2上 service keepalived stop     訪問測試 http://192.168.1.18 能否輪詢
關閉lamp1上的service httpd stop   訪問測試http://192.168.1.18 是否會報錯

到此高可用webserver構建完畢，沒有單點故障，任何一點故障不影響業務。