詳細講解:http://freeloda.blog.51cto.com/2033581/1294094
HAProxy提供高可用性、負載均衡以及基於TCP和HTTP應用的代理,支持虛擬主機,它是免費、快速並且可靠的一種解決方案。HAProxy特別適用於那些負載特大的web站點,這些站點通常又需要會話保持或七層處理。HAProxy運行在當前的硬件上,完全可以支持數以萬計的併發連接。並且它的運行模式使得它可以很簡單安全的整合進您當前的架構中,同時可以保護你的web服務器不被暴露到網絡上。
HAProxy實現了一種事件驅動, 單一進程模型,此模型支持非常大的併發連接數。多進程或多線程模型受內存限制、系統調度器限制以及無處不在的鎖限制,很少能處理數千併發連接。事件驅動模型因爲在有更好的資源和時間管理的用戶端(User-Space)實現所有這些任務,所以沒有這些問題。此模型的弊端是,在多核系統上,這些程序通常擴展性較差。這就是爲什麼他們必須進行優化以使每個CPU時間片(Cycle)做更多的工作。
拓撲介紹:前端兩臺服務器提供高可用和haproxy,,後端有動態和靜態服務器組,haproxy將動態和靜態頁面請求分發到不同的服務器上;在測試情況下有異常發生,當跟蹤的腳本失敗後,沒有按照預定的動作執行;但有一點是可以肯定的,當一臺服務器宕機後,服務可以正常提供;由MASTER上的keepalived提供VIP和後端服務器的網關;(在本例中用不到後端服務器網關,因爲後端服務器只需與代理服務器交互,它們是在同一個網段的,不需要網關。本例中給出配置,以備不時之需)
一、keepalived配置:(配置選項含義請參考上篇)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | [root@node1 keepalived] # cat keepalived.conf ! Configuration File for keepalived global_defs {
notification_email {
root@localhost
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id Node1 }
vrrp_script chk_state_down {
script "[ -f /etc/keepalived/down ] && exit 1 || exit 0"
interval 1
weight 30 }
vrrp_script chk_Haproxy {
script "killall -0 haproxy"
interval 1
fall 2
rise 1
weight 30 } vrrp_instance Haproxy {
state MASTER
interface eth0
virtual_router_id 10
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass Haproxy
}
virtual_ipaddress {
172.16.1.100
}
track_script {
chk_Haproxy
chk_state_down
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault" } vrrp_instance Gateway {
state MASTER
interface eth1
virtual_router_id 20
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass Gateway
}
virtual_ipaddress {
192.168.10.250
}
track_script {
chk_Haproxy
chk_state_down
} } [root@node2 keepalived] # cat keepalived.conf ! Configuration File for keepalived global_defs {
notification_email {
root@localhost
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id Node2 }
vrrp_script chk_state_down {
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
interval 1
weight 10 }
vrrp_script chk_Haproxy {
script "killall -0 haproxy"
interval 1
fall 2
rise 1
weight 10 } vrrp_instance Haproxy {
state BACKUP
interface eth0
virtual_router_id 10
priority 95
advert_int 1
authentication {
auth_type PASS
auth_pass Haproxy
}
virtual_ipaddress {
172.16.1.100
}
track_script {
chk_Haproxy
chk_state_down
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault" } vrrp_instance Gateway {
state BACKUP
interface eth1
virtual_router_id 20
priority 95
advert_int 1
authentication {
auth_type PASS
auth_pass Gateway
}
virtual_ipaddress {
192.168.10.250
}
track_script {
chk_Haproxy
chk_state_down
} } [root@node2 keepalived] # cat notify.sh #!/bin/bash # vip=172.16.1.100 contact= 'root@localhost' notify () {
mailsubject= "`hostname` became to $1 , $vip floated."
mailbody= "`date +" %F %T "`: vrrp status changed. `hostname` became $1"
echo $mailbody | mail -s "$mailsubject" $contact } case $1 in
master)
notify master ;;
backup)
notify backup;;
fault)
notify fault ;;
*)
echo "Usage: `basename $0` {master|backup|fault}"
exit 1;; esac |
二、Haproxy配置分析:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | global
chroot /var/haproxy #chroot運行的路徑,增加安全性
uid 99 #程序運行的用戶id
gid 99 #程序運行的用戶組id
daemon #以後臺形式運行haproxy
nbproc 1 #number of process進程數量,不建議修改
pidfile /var/run/haproxy .pid #haproxy的pid存放路徑
maxconn 4000 #默認最大連接數 defaults
mode http ;工作模式
option dontlognull
log global;記錄日誌
option http-server-close ;啓用服務器端關閉
option forwardfor except 127.0.0.0 /8 ;傳遞客戶端ip
option redispatch ;當服務器組中的某臺設備故障後,自動將請求重定向到組內其他主機。
retries 3 ;請求重試的次數
timeout http-request 10s;http請求超時時間
timeout queue 1m;一個請求在隊列裏的超時時間·
timeout connect 10s;連接服務器超時時間
timeout client 1m;設置客戶端側最大非活動時間
timeout server 1m;設置服務器側最大非活動時間
timeout http-keep-alive 10s;設置http-keep-alive的超時時間
timeout check 10s;當一個連接建立之後,
maxconn 3000;同時處理的最大連接數
errorfile 403 /etc/haproxy/errorfiles/403 .http
errorfile 500 /etc/haproxy/errorfiles/500 .http
errorfile 502 /etc/haproxy/errorfiles/502 .http
errorfile 503 /etc/haproxy/errorfiles/503 .http
errorfile 504 /etc/haproxy/errorfiles/504 .http Haproxy 負載均衡實例: frontend webserver
bind 172.16.1.100:80
default_backend webservers backend webservers
balance roundrobin
server node3.magedu.com 192.168.10.3:80 check
server node4.magedu.com 192.168.10.4:80 check 或者 listen webservers
bind 172.16.1.100:80
server node3.magedu.com 192.168.10.3 check
server node4.magedu.com 192.168.10.4 check 啓用stats查看,認證等功能: 默認在 /haproxy ?stats
listen stas
bind 172.16.1.100:1099;監聽的地址和端口
stats enable ;啓用狀態監控
stats hide-version;隱藏軟件版本號
stats auth admin:admin;登陸用戶名和密碼
stats realm HAproxy\ stats;提示信息,空格之前加\
stats admin if TRUE;當通過認證纔可管理
stats uri /stats ;訪問路徑
stats refresh 5;頁面自動刷新間隔 |
效果如下:
登錄成功後:
調度算法:
動態算法:
支持動態調整權重,可以在運行中調整而不用重啓服務;
支持慢速啓動(在剛開機時不起作用);
roundrobin 輪調
leastconn 最少連接
source
hash-type : consistent 一致性哈希算法,
靜態算法:
支持權重,不支持動態調整,調整後需重啓服務;
static-rr
source
hash-type :map-based:將source IP進行hash後,對後端服務器的個數取餘算法;
uri 根據uri分配服務器,適用於後端是緩存服務器;也支持2種hash-type;同source算法;還支持參數,len # 和depth # ,用法如下
balance uri [len <len>][depth <depth>]
URL syntax:
<scheme>://<user>:<password>@<host>:<port>/path;<params>?<query>#<frag>
url_param
常用於跟蹤用戶id,將具有特定的用戶標示的GET請求發往特定的服務器;默認是靜態算法,用hash-type修改;用法如下;
balance url_param <param>[check_post [<max_wait>]]
balance url_paramuserid
balance url_param session_id check_post 64
hdr,根據請求頭部中的字段進行調度;
balancehdr(User-Agent)
balance hdr(host)
balance hdr(Host) use_domain_only
自定義日誌輸出位置:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
frontend web
bind 172.16.1.100:80
default_backend webservers
log global
log 127.0.0.1:514 local2 info
option httplog 定義local2的記錄位置:啓用UDP syslog [root@node1 ~] # vi /etc/rsyslog.conf
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514 # Log anything (except mail) of level info or higher. # Don't log private authentication messages!,添加local2.none *.info;mail.none;authpriv.none; cron .none;local2.none /var/log/messages 添加一行,自定義日誌存放位置: local2.* /var/log/haproxy .log 然後重啓syslog服務; [root@node1 ~] # service rsyslog restart 查看日誌情況 [root@node1 ~] # tail /var/log/haproxy.log Sep 29 19:47:17 localhost haproxy[2847]: 172.16.254.52:53660 [29 /Sep/2013 :19:47:17.861] web webservers /node3 .magedu.com 9 /0/0/1/10 200 267 - - ---- 3 /3/0/1/0 0 /0 "GET /4.html HTTP/1.1" Sep 29 19:47:17 localhost haproxy[2847]: 172.16.254.52:53660 [29 /Sep/2013 :19:47:17.861] web webservers /node3 .magedu.com 9 /0/0/1/10 200 267 - - ---- 3 /3/0/1/0 0 /0 "GET /4.html HTTP/1.1" 在後端real server上記錄真實客戶端IP地址: 修改apache配置文件中日誌格式,修改爲: LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined 效果爲: 172.16.254.52 - - [29 /Sep/2013 :14:30:44 +0800] "GET / HTTP/1.1" 200 50485 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.62 Safari/537.36" |
Haproxy中ACL的使用:
acl <aclname> <criterion> [flags] [operator] <value> ...
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | dst <ip_address> dst_port <port> src <ip_address> src_port <port> e.g. #用法一、允許10.0.0.0/24的用戶訪問,其他用戶將禁止 acl goodguys src 10.0.0.0 /24 tcp-request content accept if goodguys tcp-request content reject tcp-request content accept [{ if | unless} <condition>] Accept a connection if /unless a content inspection condition is matched #用法二、將源IP爲172.16.254.52的用戶禁止、將403的錯誤重定向到其他服務器; acl badguy src 172.16.254.52 block if badguy errorloc 403 http: //www .afwing.com/ #用法三、當用戶訪問172.16.1.100時,重定向到http://www.afwing.com acl dstipaddr hdr(Host) 172.16.1.100 redirect location http: //www .afwing.com if dstipaddr #用法四、讀寫分離: acl read method GET acl read method HEAD acl write method PUT acl write method POST use_backend imgservers if read use_backend uploadservers if write #用法五、限制某個IP訪問的路徑(文件) acl badguy src 172.16.254.52 acl denyfile path /index .html http-request deny if denyfile badguy #用法六、動靜分離 acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js #或者 acl url_static path_end -i .jpg$ .gif$ .png$ .css$ .js$ #或者 acl url_static hdr_beg(host) -i www acl url_static hdr_beg(host) -i news. video. download. ftp . use_backend static if url_static default_backend app backend static
balance roundrobin
server static 192.168.10.1:80 check maxconn 6000
server static 192.168.10.2:80 check maxconn 6000 backend app
balance roundrobin
server app1 192.168.10.3:80 check maxconn 1000
server app2 192.168.10.4:80 check maxconn 1000 #Additional examples acl invalid_src src 0.0.0.0 /7 224.0.0.0 /3 acl invalid_src src_port 0:1023 acl local_dst hdr(host) -i localhost Move the login URL only to HTTPS. acl clear dst_port 80 acl secure dst_port 8080 acl login_page url_beg /login acl logout url_beg /logout acl uid_given url_reg /login ?userid=[^&]+ acl cookie_set hdr_sub(cookie) SEEN=1 redirect prefix https: //mysite .com set -cookie SEEN=1 if !cookie_set redirect prefix https: //mysite .com if login_page !secure redirect prefix http: //mysite .com drop-query if login_page !uid_given redirect location http: //mysite .com/ if !login_page secure redirect location / clear -cookie USERID= if logout |
server和default-server options:
backup :當後端服務器都發生故障時,由backup服務器發送錯誤頁面:
在haproxy服務器上啓動http服務以apache爲例,監聽在127.0.0.1的某個端口:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | [root@node1 haproxy] # vi /etc/httpd/conf/httpd.conf Listen 127.0.0.1:8888 [root@node1 haproxy] # service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [root@node1 haproxy] # vi /var/www/html/index.html <h1> Sorry,our site is not in service. Please try again later. < /h1 > [root@node1 haproxy] # vi /etc/haproxy/haproxy.cfg
frontend web
bind 172.16.1.100:80
default_backend webservers
log 127.0.0.1:514 local2 info
option httplog
option forwardfor except 127.0.0.0 /8
rspadd x-via:\ 172.16.1.1
backend webservers
balance uri
hash - type map-based
server node3.magedu.com 192.168.10.3:80 check weight 3
server node4.magedu.com 192.168.10.4:80 check weight 1
server backup.magedu.com 127.0.0.1:8888 backup |
在stats頁面中將後端服務器禁用
結果就是:
fall:<count> 檢測失效的次數;
inter <delay> 檢測的間隔
disabled 手動禁用服務器
maxconn <count>最大連接數
observe <mode> “layer4”“layer7”觀察模式
redir <prefix> 只能重定向HEAD和GET
# server node4.magedu.com 192.168.10.4:80 check weight 1 redir http://www.afwing.com
rise <count>檢測多少次纔算是成功;
slowstart 支持慢啓動,第一次啓動時不算;
weight 權重
monitor-uri,通過訪問monitor-uri顯示集羣總體狀況,
1 2 3 4 5 6 7 8 | frontend www
mode http
acl site_dead nbsrv(appservers) lt 2
monitor-uri /test .html
monitor fail if site_dead backend appservers
server node4.magedu.com 192.168.10.4:80 check
server node3.magedu.com 192.168.10.3:80 check |
下圖所示,當後端主機少於2臺時的狀況。(可以通過手動disable後端一臺主機查看情況)
當後端主機大於2臺時:
更多信息請參考官方主站:http://cbonte.github.io/haproxy-dconv/configuration-1.4.html
我的配置:
global
#log 127.0.0.1 local0
log 127.0.0.1 local3
#log loghost local0 info
maxconn 4096
chroot /usr/local/haproxy
uid nobody
gid nobody
daemon
#debug
#quiet
nbproc 2
pidfile /usr/local/haproxy/run/haproxy.pid
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen http_80_in 0.0.0.0:80
mode http
option httpchk GET /index.html HTTP/1.0\n\nHost:10.0.2.90
# appli1-rewrite 0.0.0.0:10001
#cookie SERVERID rewrite
balance roundrobin
server app1_1 10.0.2.203:80 check inter 2000 rise 2 fall 5
server app1_2 10.0.2.202:80 check inter 2000 rise 2 fall 5
#server app1_3 10.0.2.202:8080 cookie app1inst3 check inter 2000 rise 2 fall 5
#server app1_4 10.0.2.202:8080 cookie app1inst4 check inter 2000 rise 2 fall 5
listen stats 0.0.0.0:1080
mode http
transparent
stats uri / haproxy-admin
stats realm Haproxy \ statistic
stats auth test:123456