在搭建HAProxy+Keepalived之前先單獨測試HAProxy
一 HAProxy配置測試
（注測試：HAProxy只設內網網卡，Client同樣位於內網）
1.測試web提前配置
yum install httpd -y #安裝httpdecho 192.168.1.188 >/var/www/html/index.html #輸入網站顯示各自IPsystemctl restart httpdfirewall-cmd --permanent --add-port=80/tcp #開啓防火牆80端口firewall-cmd --reload#測試，瀏覽器登錄IP查看，是否正常啓動。
2.HAProxy配置/etc/haproxy/haproxy.cfg（yum install haproxy -y）
global#全區日誌配置 使用rsyslog的local3設備log 127.0.0.1 local3 info#工作目錄（安全）chroot /var/lib/haproxy#pid文件存儲目錄pidfile /var/run/haproxy.pid#後臺進程數量nbproc 1#每個進程最大併發數maxconn 40000user haproxygroup haproxy#後臺程序模式工作daemon- defaults
mode http#後端連接重試次數，超出標識不可用retries 3#連接服務器最長等待時間timeout connect 10s#客戶端發送請求最長等待時間timeout client 30s#服務器會復客戶端最長等待時間timeout server 30s#對後端服務器的檢測超時時間timeout check 10s#定義HAProxy監控頁面listen admin_statsbind 0.0.0.0:9188mode httplog 127.0.0.1 local3 err#HAProxy監控頁面統計自動刷新時間。stats refresh 30s#設置監控頁面URL路徑。 http：//IP:9188/haproxy-status可查看stats uri /haproxy-status#統計頁面密碼框提示信息stats realm welcome login\ Haproxy#登錄統計頁面用戶和密碼stats auth admin:123456#隱藏HAProxy版本信息stats hide-version#設置TURE後可在監控頁面手工啓動關閉後端真實服務器stats admin if TRUE#定義前端虛擬節點frontend www#監聽端口bind *:80mode http#啓用日誌記錄HTTP請求。option httplog#啓用後後端服務器可以獲得客戶端IPoption forwardfor#客戶端和服務器完成一次連接請求後，HAProxy主動關閉TCP鏈接（優化選項）option httpclose#使用全局日誌配置log global#指定後端服務池（backend定義htmpool）default_backend htmpool#定義後端真實服務器backend htmpoolmode http#用於cookie保持環境。（如後端服務器故障，客戶端cookie不會刷新，用此來把用戶請求強制定向到正常服務器）option redispatch#負載均衡很高時，自動結束當前隊列處理時間長的連接option abortonclose#負載均衡算法balance roundrobin#允許向cookie插入SERVERID.下面server可以使用cookie定義cookie SERVERID#啓用HTTP服務狀態檢測功能 （後端服務器一定要存在此文件，不然haproxy認爲其故障）option httpchk GET /index.html#後端服務設置server web1 192.168.1.186:80 cookie server1 weight 6 check inter 2000 rise 2 fall 3server web2 192.168.1.188:80 cookie server2 weight 6 check inter 2000 rise 2 fall 3
開啓日誌（日誌如果在主機性能較弱的情況下，最好關閉）
vim /etc/rsyslog.d/haproxy.conf$ModLoad imudp$UDPServerRun 514local3.* /var/log/haproxysystemctl restart rsyslog
開啓防火牆和安全策略
firewall-cmd --permanent --add-port=80/tcpfirewall-cmd --permanent --add-port=9188/tcpfirewall-cmd --reloadsetsebool -P haproxy_connect_any=on#getsebool -a | grep haproxy 查看
啓動HAProxy服務
- systemctl start haproxy
測試均衡能力
[root@test2 ~]# curl http://192.168.1.166192.168.1.188[root@test2 ~]# curl http://192.168.1.166192.168.1.186[root@test2 ~]# curl http://192.168.1.166192.168.1.188[root@test2 ~]# curl http://192.168.1.166192.168.1.186
測試HAProxy監控頁面是否正常（http：//IP:9188/haproxy-status）
![]()
二 HAProxy+Keepalived配置
![]()
HAProxy配置
(HAProxy兩臺Proxy主機配置相同，如上HAProxy的配置即可)
Keepalived配置（/etc/keepalived/keepalived.conf）(yum install keepalived -y)
! Configuration File for keepalivedglobal_defs {#設置報警通知郵件地址，可以設置多個notification_email {msun1996@163.com}#設置郵件的發送地址notification_email_from keepalived@msun.com#設置smtp server的地址,該地址必須是存在的smtp_server 127.0.0.1#設置連接smtp server的超時時間smtp_connect_timeout 30#運行Keepalived服務器的標識，發郵件時顯示在郵件標題中的信息router_id HAProxy_msun}# 檢測haproxy腳本vrrp_script chk_haproxy {script "/etc/keepalived/check_haproxy.sh"interval 2#下面方法相對更優- #script "killall -0 haproxy" #killall (安裝 yum install psmisc -y)
- #interval 2
- #weghit 2 #權值腳本成功時（0）等於priority+weghit #否則爲priority
}#定義VRRP實例,實例名自定義vrrp_instance haproxy_msun {#指定Keepalived的角色，MASTER爲主服務器，BACKUP爲備用服務器state MASTER #從設置爲BACKUP#指定HA監測的接口interface eno16777736#虛擬路由標識，這個標識是一個數字(1-255)，在一個VRRP實例中主備服務器ID必須一樣virtual_router_id 68#優先級，數字越大優先級越高，在一個實例中主服務>器優先級要高於備服務器priority 100 #從設置爲99#設置主備之間同步檢查的時間間隔單位秒advert_int 1#設置驗證類型和密碼authentication {#驗證類型有兩種{PASS|HA}auth_type PASS#設置驗證密碼，在一個實例中主備密碼保持一樣auth_pass 1689}track_script {chk_haproxy # 執行監控的服務}#定義虛擬IP地址,可以有多個，每行一個virtual_ipaddress {192.168.1.160}}
添加HAProxy檢測腳本到vim /etc/keepalived/check_haproxy.sh(執行權限)（這一步必須關閉SElinux,不然vrrp
script 腳本是不可以執行的。下面會介紹開啓selinux下的解決方法）
#!/bin/bash#判斷haproxy是否已經啓動if [ `ps -C haproxy --no-header |wc -l` -eq 0 ] ; then#如果沒有啓動，則啓動haproxy程序systemctl start haproxy#睡眠3秒鐘以等待haproxy完全啓動sleep 3if [ `ps -C haproxy --no-header |wc -l` -eq 0 ] ; thensystemctl stop keepalived#如果haproxy還是沒有啓動，則將keepalived停掉，這樣VIP會自動漂移到另外一臺haproxyfifi
啓動keepalived
systemctl start keepalived
防火牆允許（開放vrrp組播）（不然會導致腦裂）
firewall-cmd --permanent --add-rich='rule family="ipv4" destination address="224.0.0.18" protocol value="vrrp" accept'firewall-cmd --reload
測試
[root@test5 ~]# curl 192.168.1.160192.168.1.186[root@test5 ~]# curl 192.168.1.160192.168.1.188[root@test5 ~]# curl 192.168.1.160192.168.1.186[root@test5 ~]# curl 192.168.1.160192.168.1.188
通過觀察ip
a輸出VIP來判斷是否存在腦裂，和主備故障時是否能正常切換。
三 部署HAProxy遇到的重點問題與解決
1.部署單獨的Haproxy時遇到問題
配置完配置文件後，使用systemctl start haproxy,haproxy無法正常啓動。（啓動後查看狀態沒有對應進程顯示）
（防火牆80端口和9188端口已正常開啓。所以不可能是防火牆問題，再說防火牆也不可能影響啓動啊）
/var/log/message有以下報錯信息
[root@test5 ~]# tail /var/log/messagesMar 12 14:59:29 test5 systemd: Starting HAProxy Load Balancer...Mar 12 14:59:29 test5 haproxy-systemd-wrapper: [ALERT] 070/145929 (3131) : Starting proxy admin_stats: cannot bind socket [0.0.0.0:9188]Mar 12 14:59:29 test5 haproxy-systemd-wrapper: haproxy-systemd-wrapper: exit, haproxy RC=1Mar 12 14:59:29 test5 systemd: haproxy.service: main process exited, code=exited, status=1/FAILUREMar 12 14:59:29 test5 systemd: Unit haproxy.service entered failed state.Mar 12 14:59:29 test5 systemd: haproxy.service failed.
查看haproxy相關selinux政策，發現haproxy默認不允許連接
[root@test5 ~]# getsebool -a | grep haproxyhaproxy_connect_any --> off
重新設置selinux政策後正常啓動
[root@test5 ~]# setsebool -P haproxy_connect_any=on[root@test5 ~]# systemctl start haproxy[root@test5 ~]# netstat -tlunpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3171/haproxytcp 0 0 0.0.0.0:9188 0.0.0.0:* LISTEN 3171/haproxy
2.搭建HAProxy+Keepalived遇到的問題
1.服務器搭建完成後，出現腦裂現象。主備機多有VIP。
解決：防火牆允許（開放vrrp組播）
firewall-cmd --permanent --add-rich='rule family="ipv4" destination address="224.0.0.18" protocol value="vrrp" accept'
2.關於Keepalived中的vrrp script 設置的腳本不執行。
因爲檢測HAproxy狀態的腳本不能運行，導致就算主機HAProxy掛掉，但因爲Keepalived還正常運行，VIP沒有正常從主機正常漂移到從機，導致整個服務垮掉。
直接快速解決方法：關閉SElinux.
SElinux安全策略運行時解決
1)嘗試SElinux允許解決（解決失敗）
開啓SElinux時的錯誤提示：
- [root@test1 ~]# tail -f /var/log/messages
type=AVC msg=audit(1489338470.513:714): avc: denied { getattr } for pid=5174 comm="check_haproxy.s" path="/usr/bin/systemctl" dev="dm-0" ino=33947874 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=filetype=SYSCALL msg=audit(1489338470.513:714): arch=c000003e syscall=4 success=no exit=-13 a0=f72c00 a1=7ffd88bca560 a2=7ffd88bca560 a3=3 items=0 ppid=5173 pid=5174 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)
安裝selinux管理工具（根據報錯顯示可能解決方案）
yum install setroubleshoot -y
查看settoubleshoot給出的解決方案
[root@test1 ~]# tail -f /var/log/messagesMar 13 02:30:27 test1 setroubleshoot: SELinux is preventing /usr/bin/bash from getattr access on the file /usr/bin/systemctl. For complete SELinux messages. run sealert -l 54416ee0-01c3-40e8-8198-675f6f86a7f7Mar 13 02:30:27 test1 python: SELinux is preventing /usr/bin/bash from getattr access on the file /usr/bin/systemctl.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that bash should be allowed getattr access on the systemctl file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys#012# semodule -i my-checkhaproxys.pp#012
嘗試解決：
[root@test1 ~]# sealert -l 54416ee0-01c3-40e8-8198-675f6f86a7f7SELinux is preventing /usr/bin/bash from getattr access on the file /usr/bin/systemctl.***** Plugin catchall (100. confidence) suggests **************************If 確定應默認允許 bash getattr 訪問 systemctl file。Then 應該將這個情況作爲 bug 報告。可以生成本地策略模塊以允許此訪問。Doallow this access for now by executing:# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys# semodule -i my-checkhaproxys.pp
[root@test1 ~]# sealert -l 54416ee0-01c3-40e8-8198-675f6f86a7f7[root@test1 ~]# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys[root@test1 ~]# semodule -i my-checkhaproxys.pp
嘗試後日志提示：
[root@test1 ~]# tail -f /var/log/messagesMar 13 02:36:08 test1 setroubleshoot: SELinux is preventing /usr/bin/bash from open access on the file /usr/bin/systemctl. For complete SELinux messages. run sealert -l 487ebbb3-fefe-4018-8c4c-5be6a185e64bMar 13 02:36:08 test1 python: SELinux is preventing /usr/bin/bash from open access on the file /usr/bin/systemctl.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that bash should be allowed open access on the systemctl file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys#012# semodule -i my-checkhaproxys.pp#012
[root@test1 ~]# tail -f /var/log/audit/audit.logtype=AVC msg=audit(1489352184.678:549): avc: denied { open } for pid=3990 comm="check_haproxy.s" path="/usr/bin/systemctl" dev="dm-0" ino=33724848 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=filetype=SYSCALL msg=audit(1489352184.678:549): arch=c000003e syscall=2 success=no exit=-13 a0=14bec50 a1=0 a2=43 a3=7ffe13e19190 items=0 ppid=3986 pid=3990 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)
繼續按提示輸入
[root@test1 ~]# sealert -l 487ebbb3-fefe-4018-8c4c-[root@test1 ~]# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys[root@test1 ~]# semodule -i my-checkhaproxys.pp
嘗試後日志提示：
[root@test1 ~]# tail -f /var/log/messagesMar 13 05:02:35 test1 setroubleshoot: SELinux is preventing /usr/bin/bash from execute_no_trans access on the file /usr/bin/systemctl. For complete SELinux messages. run sealert -l a3a942ad-2b0e-4b4b-bf1f-b521256f4405Mar 13 05:02:35 test1 python: SELinux is preventing /usr/bin/bash from execute_no_trans access on the file /usr/bin/systemctl.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that bash should be allowed execute_no_trans access on the systemctl file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys#012# semodule -i my-checkhaproxys.pp#012
[root@test1 ~]# tail -f /var/log/audit/audit.logtype=AVC msg=audit(1489352298.756:764): avc: denied { execute_no_trans } for pid=5507 comm="check_haproxy.s" path="/usr/bin/systemctl" dev="dm-0" ino=33724848 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=filetype=SYSCALL msg=audit(1489352298.756:764): arch=c000003e syscall=59 success=no exit=-13 a0=162cc50 a1=162ccc0 a2=162d360 a3=7ffddf4d6190 items=0 ppid=5503 pid=5507 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)
繼續按提示輸入
[root@test1 ~]# sealert -l a3a942ad-2b0e-4b4b-bf1f-b521256f4405[root@test1 ~]# ausearch -c 'check_haproxy.s' --raw | audit2allow -M my-checkhaproxys[root@test1 ~]# semodule -i my-checkhaproxys.pp
嘗試後日志提示：
[root@test1 ~]# tail -f /var/log/messagesMar 13 05:06:42 test1 setroubleshoot: failed to retrieve rpm info for /run/dbus/system_bus_socketMar 13 05:06:42 test1 setroubleshoot: SELinux is preventing /usr/bin/systemctl from connectto access on the unix_stream_socket /run/dbus/system_bus_socket. For complete SELinux messages. run sealert -l e1afcda9-a674-4d76-8aa0-7787404c515eMar 13 05:06:42 test1 python: SELinux is preventing /usr/bin/systemctl from connectto access on the unix_stream_socket /run/dbus/system_bus_socket.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that systemctl should be allowed connectto access on the system_bus_socket unix_stream_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'systemctl' --raw | audit2allow -M my-systemctl#012# semodule -i my-systemctl.pp#012
[root@test1 ~]# tail -f /var/log/audit/audit.logtype=AVC msg=audit(1489352885.238:1505): avc: denied { connectto } for pid=11711 comm="systemctl" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_sockettype=SYSCALL msg=audit(1489352885.238:1505): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=7f42bfa16918 a2=21 a3=7ffe723af2b0 items=0 ppid=11707 pid=11711 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemctl" exe="/usr/bin/systemctl" subj=system_u:system_r:keepalived_t:s0 key=(null)
繼續按提示輸入
[root@test1 ~]# sealert -l e1afcda9-a674-4d76-8aa0-7787404c515e[root@test1 ~]# ausearch -c 'systemctl' --raw | audit2allow -M my-systemctl[root@test1 ~]# semodule -i my-systemctl.pp
嘗試後日志：（最後沒解決 ~。。~）
[root@test1 ~]# tail -f /var/log/messagesMar 13 05:12:09 test1 Keepalived_vrrp[3090]: Process [14395] didn't respond to SIGTERM
[root@test1 ~]# tail -f /var/log/audit/audit.logtype=USER_AVC msg=audit(1489353197.502:2058): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { start } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/haproxy.service" cmdline="systemctl start haproxy" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:haproxy_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'type=USER_AVC msg=audit(1489353198.557:2059): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { stop } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/keepalived.service" cmdline="systemctl stop keepalived" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:keepalived_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'type=USER_AVC msg=audit(1489353198.557:2060): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { status } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/keepalived.service" cmdline="systemctl stop keepalived" scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:keepalived_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=AVC msg=audit(1489354986.919:1591): avc: denied { getattr } for pid=7566 comm="check_haproxy.s" path="/usr/sbin/haproxy" dev="dm-0" ino=68172487 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:haproxy_exec_t:s0 tclass=filetype=SYSCALL msg=audit(1489354986.919:1591): arch=c000003e syscall=4 success=no exit=-13 a0=19cbd50 a1=7ffde08b8720 a2=7ffde08b8720 a3=11 items=0 ppid=7565 pid=7566 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="check_haproxy.s" exe="/usr/bin/bash" subj=system_u:system_r:keepalived_t:s0 key=(null)
觀察：systemctl 相關命令還是因爲selinux運行不了
2）根據1）決定對vrrp script腳本修改如下
#!/bin/bash#判斷haproxy是否已經啓動，去除haproxy試圖啓動部分if [ `ps -C haproxy --no-header |wc -l` -eq 0 ] ; then#如果haproxy還是沒有啓動，則將keepalived停掉，這樣VIP會自動漂移到另外一臺haproxypkill keepalivedfi
（啓動順序必須是haproxy先啓動，負責keepalived如果先啓動會自殺）
也可設置自啓
systemctl enable haproxysystemctl enable keepalived
3）使用文件中註釋的信息完成 vrrp_script （下面的腳本對主從服務器的priority差值有嚴格要求）(最優方法)
-
# 檢測haproxy腳本 -
vrrp_script chk_haproxy{ -
script "killall -0 haproxy" #killall (安裝 yum install psmisc -y)
-
interval 2
-
weghit 2 #權值腳本成功時（0）等於priority+weghit #否則爲priority
-
}