簡介:
haproxy爲高性能的反向代理服務器,在向後端server調度方面支持很多的調度算法:roundrobin,source,uri,url_param,hdr(),leastconn等,且支持動態權重調整的一致性hash,後端爲cache server時很好的提高cache命中率;相比nginx反向代理,haproxy調度功能更爲強大,另外haproxy提供自帶的gui接口,方便直接在web頁面管理前、後端服務器,只需要在配置文件中開啓即可。
nginx自身在處理web靜態頁面方面非常強大,其採用模塊化設計,有較好的擴展性,但不支持模塊動態裝卸載(Tengine支持模塊動態裝卸載),nginx具有高可靠性,支持熱部署,低內存消耗等優點,其在設計之初就吸納了最新的技術,如支持事件驅動、異步IO等,併發處理能力強大。
apache結合php處理動態頁面功能強大,php可直接以模塊的形式加載到httpd,而nginx需要通過fastcgi協議與後端php通信。所以結合各組件優點,我們這裏採用haproxy(支持https)+nginx(靜態)+lamp(動態)架構,最後通過keepalived實現對haproxy高可用。
下圖爲本次實驗架構圖:
圖1
圖2
架構圖說明
這裏有2幅架構圖,圖1中,後端各主機均在同一網段,這也是我們這次實驗所採用的架構;圖2中後端static server group和dynamic server group不在同一網段,二者之間需要路由器相連,Router的地址:eth0 --> 192.168.100.80eth1 --> 192.168.200.80非但如此,採用圖2架構還需在haproxy和各後端server上添加靜態路由:
haproxy1: route add -net 192.168.200.0/24 gw 192.168.100.80 dev eth1
haproxy2: route add -net 192.168.100.0/24 gw 192.168.200.80 dev eth1
Nginx1: routeadd -net 192.168.200.0/24 gw 192.168.100.80 dev eth0
Nginx2: routeadd -net 192.168.200.0/24 gw 192.168.100.80 dev eth0
LAMP1: routeadd -net 192.168.100.0/24 gw 192.168.200.80 dev eth0
LAMP2: routeadd -net 192.168.100.0/24 gw 192.168.200.80 dev eth0
爲簡便,這裏我們採用架構圖1。這裏我們用virtualbox準備6臺linux主機,各主機IP地址、Gateway及所需安裝軟件等如下表所配置,這裏我們將192.168.0.0/24網絡模擬爲公網,192.168.100.0/24爲服務器內網網段。前端爲2臺haproxy組成的反向代理服務器,並實現雙主高可用:所謂雙主,即haproxy1和haproxy2都正常工作,均同時爲後端server提供反向代理和調度功能,用戶可通過DNS解析後的192.168.0.50和192.168.0.51兩個地址訪問我們的網站。客戶端請求的所有phpBB靜態頁面由前端的haproxy調度至後端的2臺nginx組成的服務器組響應;用戶請求的所有phpBB動態頁面由前端的haproxy調度至後端的2臺lamp組成的服務器組響應。
6臺linux主機上所需資源如下:
IP Address | Hosts Name | OS | Software Required | comments |
eth0 -->192.168.0.16; GW -->192.168.0.1 | HAProxy1(active) | centos6.8 | keepalived、haproxy、ntpd、雙網卡 | HAProxy(active) |
eth0:0 -->192.168.0.50(由keepalived配置文件自動生成) | ||||
eth1 -->192.168.100.16 | ||||
eth0 -->192.168.0.66; GW -->192.168.0.1 | HAProxy2(active) | centos6.8 | keepalived、haproxy、雙網卡 | HAProxy(active) |
eth0:1 -->192.168.0.51(由keepalived配置文件自動生成) | ||||
eth1 -->192.168.100.66 | ||||
eth0 -->192.168.100.64; GW -->192.168.100.16 | Nginx1 | centos6.8 | nginx、phpBB、單網卡 | Static Server Group |
eth0 -->192.168.100.65; GW -->192.168.100.66 | Nginx2 | centos6.8 | nginx、phpBB、單網卡 | |
enp0s3 -->192.168.100.70; GW -->192.168.100.16 | LAMP1 | centos7.3 | httpd、mariadb/mysql、php、單網卡、phpBB | Dynamic Server Group |
enp0s3 -->192.168.100.71; GW -->192.168.100.66 | LAMP2 | centos7.3 | httpd、mariadb/mysql、php、單網卡、phpBB |
軟件安裝
按照上面的清單,分別在各主機上部署所需軟件,對於清單中HAProxy需要安裝ntpd服務器的原因在於keepalived的運行需要後端server與前端服務器時鐘保持同步,否則對後端server的健康狀態檢測會出現異常。
①、haproxy
haproxy在centos6.8上可以直接使用yum方式:
[root@centos68 ~]# yum install -y haproxy
目前系統提供的默認版本是1.5.18,且支持openssl:
[root@centos68 ~]# rpm -q haproxy ; ldd $(which haproxy)| grep ssl
haproxy-1.5.18-1.el6.x86_64
libssl.so.10 => /usr/lib64/libssl.so.10(0x00007f05e085c000)
如果是手動編譯安裝,則需要先在官網下載程序包,並添加對openssl的支持,下面是手動編譯步驟:
[root@centos68 ~]# yum install -y openssl openssl-devel readline-devel pcre-devellibssl-dev libpcre3
[root@centos68 ~]# wget http://www.haproxy.org/download/1.6/src/haproxy-1.6.13.tar.gz
[root@centos68 ~]# tar xf haproxy-1.6.13.tar.gz ; cd haproxy-1.6.13
[root@centos68 haproxy-1.6.13]# make TARGET=linux2628 USE_PCRE=1USE_OPENSSL=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LIBCRYPT=1
[root@centos68 haproxy-1.6.13]# ldd $(which haproxy) | grep ssl
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f05e085c000)
[root@centos68 haproxy-1.6.13]# make install PREFIX=/usr/local/haproxy
[root@centos68 haproxy-1.6.13]# vi /etc/profile.d/haproxy.sh
#!/bin/bash
export PATH=$PATH:/usr/local/haproxy/sbin/
[root@centos68 haproxy-1.6.13]# source /etc/profile.d/haproxy.sh
創建配置文件目錄及拷貝並修改啓動腳本:
[root@centos68 haproxy-1.6.13]# mkdir -p /etc/haproxy
[root@centos68 haproxy-1.6.13]# cp examples/haproxy.init/etc/rc.d/init.d/haproxy
[root@centos68 haproxy-1.6.13]# vi /etc/rc.d/init.d/haproxy
BIN=/usr/local/haproxy/sbin/$BASENAME
CFG=/etc/$BASENAME/$BASENAME.cfg
PIDFILE=/var/run/$BASENAME.pid
LOCKFILE=/var/lock/subsys/$BASENAME
至此haproxy已經安裝完畢,這裏我們根據架構圖提供haproxy的配置文件:
[root@centos68 ~]# vi /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 40000
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
# turn on stats unix socket
stats socket/var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend http_frontend
bind *:80
bind *:443 ssl crt /etc/haproxy/haproxy.pem # 網站證書文件
mode http
log global
option httpclose
option logasap
option dontlognull
option forwardfor except127.0.0.1 header X-Client if-none
capture request header Host len20
capture request header Refererlen 60
reqadd X-Forwarded-Proto:\ https
#rspadd Via:\ haproxy.example.com
acl url_static path_beg -i/static /p_w_picpaths /javascript /stylesheets
acl url_static path_end -i .jpg.jpeg .png .bmp .gif .css .js
redirect scheme https if !{ ssl_fc } # 全站採用https協議
use_backend static_servers ifurl_static
default_backend dynamic_servers
#---------------------------------------------------------------------
# static backend for serving up p_w_picpaths, stylesheets and such
#---------------------------------------------------------------------
backend static_servers
mode http
balance roundrobin
#option httpchk HEAD /healthchk.html
server static1 192.168.100.64:80 check inter 1000rise 2 fall 4 weight 1 maxconn 6000
server static2 192.168.100.65:80 check inter 1000rise 2 fall 4 weight 1 maxconn 6000
http-request set-headerX-Forwarded-Port %[dst_port]
http-request add-headerX-Forwarded-Proto https if { ssl_fc }
backend dynamic_servers
mode http
balance roundrobin
#cookie dynamic_cookie insert nocache indirect
server https_dynamic1 192.168.200.70:80 checkinter 2000 rise 2 fall 4 weight 1 maxconn 1000
server https_dynamic2 192.168.200.71:80 checkinter 2000 rise 2 fall 4 weight 1 maxconn 1000
http-request set-headerX-Forwarded-Port %[dst_port]
http-request add-headerX-Forwarded-Proto https if { ssl_fc }
listen stats
bind *:9090
stats enable
stats uri /haproxy?stats
stats hide-version
stats auth admin:yourpassword
stats admin if TRUE
②、haproxy.cfg配置說明
haproxy的配置文件位於:/etc/haproxy/haproxy.cfg,該配置文件中的一個重點是添加對https協議的支持,爲了實現該目的,需要做如下操作:
我們知道,haproxy代理ssl有三種方式:第一種,haproxy 自身提供ssl 證書,僅是客戶端與haproxy之間傳輸https協議,而haproxy與後端的web server仍然傳輸http;第二種爲SSL透傳,haproxy 本身工作於tcp傳輸層,https由後端server直接處理,但haproxy不對https報文做任何修改,這就意味着後端server失去了對客戶端IP、端口及使用協議的記錄;第三種,SSL連接在負載均衡器處終止,按需求調整,然後作爲新的SSL連接代理到後端服務器。這可能會提供最大的安全性和發送客戶端信息的能力。這樣做的代價是更多的CPU能耗和稍複雜的配置。實驗中我們採用第一種,所以我們需要提供haproxy的ssl證書文件:
因爲是實驗,所以我們需要生成自簽署證書cacert.pem、應用程序haproxy證書haproxy.crt,應用程序haproxy私鑰haproxy.key等文件,具體私鑰及證書生成步驟這裏不再詳述。最後將haproxy.crt和haproxy.key合併爲一個文件:
[root@centos68 ~]# cd /etc/haproxy
[root@centos68 haproxy]# cat haproxy.crt haproxy.key | tee haproxy.pem
而上面生成的haproxy.pem正是haproxy.cfg所需要的:
bind *:443 ssl crt /etc/haproxy/haproxy.pem
③、keepalived安裝與配置
我們需要在前端部署有haproxy的兩臺server上安裝keepalived,其安裝過程較爲簡單,直接使用yum安裝即可:
[root@centos68 ~]# yum install -y keepalived
[root@centos68 ~]# rpm -q keepalived
keepalived-1.2.13-5.el6_6.x86_64
haproxy1上的keepalived配置:
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id www.example.com
}
vrrp_script chk_haproxy {
script "killall -0 haproxy&> /dev/null"
interval 1
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 256f6df6
}
virtual_ipaddress {
192.168.0.50/24 dev eth0label eth0:0
}
track_script {
chk_haproxy
}
notify_master"/etc/keepalived/notify.sh master"
notify_backup"/etc/keepalived/notify.sh backup"
notify_fault"/etc/keepalived/notify.sh fault"
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 52
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 19afa2c5
}
virtual_ipaddress {
192.168.0.51/24 dev eth0label eth0:1
}
track_script {
chk_haproxy
}
notify_master"/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.shbackup"
notify_fault"/etc/keepalived/notify.sh fault"
}
haproxy2上的keepalived配置:
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id centos66.example.com
}
vrrp_script chk_haproxy {
script "killall -0 haproxy&> /dev/null"
interval 1
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 256f6df6
}
virtual_ipaddress {
192.168.0.50/24 dev eth0label eth0:0
}
track_script {
chk_haproxy
}
notify_master"/etc/keepalived/notify.sh master"
notify_backup"/etc/keepalived/notify.sh backup"
notify_fault"/etc/keepalived/notify.sh fault"
}
vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 19afa2c5
}
virtual_ipaddress {
192.168.0.51/24 dev eth0label eth0:1
}
track_script {
chk_haproxy
}
notify_master "/etc/keepalived/notify.shmaster"
notify_backup"/etc/keepalived/notify.sh backup"
notify_fault"/etc/keepalived/notify.sh fault"
}
消息通知腳本notify.sh:
#!/bin/bash
#Date:2017-10-07
vip=192.168.0.50
contact="root@localhost"
notify() {
mailsubject="`hostname` tobe $1: $vip floating"
mailbody="`date +'%F%H:%M:%S'`: vrrp transition, `hostname` changed to be $1"
echo "$mailbody" | mail-s "$mailsubject" $contact
}
case "$1" in
master)
notify master
exit 0
;;
backup)
notify backup
exit 0
;;
fault)
notify fault
exit 0
;;
*)
echo "Usage: `basename $0`{master|backup|fault}"
exit 1
;;
esac
④、ntpd的安裝與配置
由於前端keepalived與後端各server通信需要保證時鐘同步,所以我們需要在前端的其中一臺主機上安裝ntpd服務器,注意本次實驗採用的是架構圖1,如果是架構圖2,那麼前端的2臺haproxy服務器上均要部署ntpd。
[root@centos66 ~]# yum install -y ntpd
[root@centos66 ~]# vi /etc/ntpd.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5),ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict ntp1.aliyun.com
restrict ntp2.aliyun.com
restrict ntp3.aliyun.com
restrict ntp4.aliyun.com
restrict ntp5.aliyun.com
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
restrict 192.168.100.0 mask 255.255.255.0 nomodify
server ntp1.aliyun.com prefer
server ntp2.aliyun.com
server ntp3.aliyun.com
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
安裝完成後啓動即可:service ntpd start ;chkconfig ntpd on
後端各server添加crontab任務計劃:
[root@www ~]# crontab -e
MAILTO=""
*/10 * * * * /usr/sbin/ntpdate 192.168.100.16 &> /dev/null
這樣設置完成後,後端的各server每10分鐘就會與前端的ntpd進行一次時鐘同步。
⑤、Static Server Group
靜態服務器組的組成是由各linux主機上部署nginx來完成,在centos6.8上也可以直接使用yum來安裝:
[root@centos66 ~]# yum install -y nginx
然後對後端各主機上的nginx配置文件進行配置:
[root@centos66 ~]# vi /etc/nginx/nginx.conf
# nginx.conf
user nginx;
worker_processes 1;
worker_rlimit_nofile 51200;
pid /var/run/nginx.pid;
events {
worker_connections 51200;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status$body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log/var/log/nginx/error.log notice;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
location / {
root /var/www/html/phpBB;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
⑥、Dynamic Server Group
在後端各server上需要部署httpd+mariadb+php,在centos7上安裝過程比較簡單,直接使用yum安裝即可,這裏不再具體給出各軟件安裝步驟,我們需要在/etc/httpd/conf.d中添加虛擬主機:
[root@www ~]# cd /etc/httpd/conf.d/
[root@www conf.d]# vi vhosts.conf
<VirtualHost *:80>
DocumentRoot"/var/www/html"
ServerName ftp.example.com
</VirtualHost>
<Directory "/var/www/html/phpBB">
Options Includes ExecCGIFollowSymLinks
AllowOverride None
Require all granted
</Directory>
<VirtualHost *:80>
DocumentRoot "/var/www/html/phpBB"
ServerName www.example.com:80
</VirtualHost>
另外需要將配置文件:/etc/httpd/conf/httpd.conf中的"DocumentRoot"這一行註釋掉並修改日誌格式:
#DocumentRoot "/var/www/html"
LogFormat "%{X-Client}i%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b\"%{Referer}i\" \"%{User-Agent}i\"" combined
#SetEnvIf Request_URI "^/test.html(/.*)?$" dontlog
#CustomLog "logs/access_log" combined env=!dontlog
⑦、部署phpBB
從phpBB官網下載安裝包,將安裝包分別拷貝至後端各server:Nginx1,Ngine2,LAMP1,LAMP2上的/var/www/html目錄下,解壓縮並修改所屬用戶和組,這裏以後端的LAMP1來演示,其它各server上的操作步驟與下面類似:
[root@www ~]# cp phpBB-3.2.0.tar.bz2 /var/www/html
[root@www ~]# cd /var/www/html
[root@www html]# tar xf phpBB-3.2.0.tar.bz2
[root@www html]# ls
phpBB3
[root@www html]# chown -R apache:apache phpBB3
[root@www html]# ln -s phpBB3 phpBB
[root@www html]# chown -R apache:apache phpBB
最後我們在LAMP1和LAMP2上創建數據庫
MariaDb[(none)] > create database phpBB
MariaDb[(none)] > grant all privileges on phpBB.* to phpBB@'127.0.0.1'identified by 'yourpassword';
MariaDb[(none)] > flush privileges;
上面創建了phpBB數據庫名爲:phpBB,數據庫用戶名:phpBB,密碼:yourpassword,這在後續的phpBB頁面安裝過程中需要。
至此各server配置已經完成,最後就需要我們在瀏覽器安裝phpBB了,在瀏覽器中輸入www.example.com進入安裝。