tomcat7+apache(mod_jk/mod_proxy)負載均衡

操作目的
前端的web服務器接收請求，若是jsp頁面，就通過mod_jk/mod_proxy連接器轉發至後端的tomcat服務器，並實現tomcat的負載均衡，session會話的保持

簡單拓撲圖
 

準備工作：
兩臺服務器
172.16.220.11  安裝httpd tomcat  把web服務器和tomcat做在一臺服務器上 主機名爲tomcat1
172.16.220.12   安裝tomcat     做爲tomcat服務器   主機名爲tomcat2
同步兩臺服務器的時間
hwclock -s

一、準備兩臺服務器

tomcat1
 編譯安裝httpd
準備的包apr-1.4.6.tar.bz2 apr-util-1.4.1.tar.bz2 httpd-2.4.2.tar.bz2

安裝apr
#tar xf apr-1.4.6.tar.bz2
#cd apr-1.4.6
#./configure --prefix=/usr/local/apr
#make
#make install

#tar xf apr-util-1.4.1.tar.bz2
#cd apr-util-1.4.1
#./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr
#make
#make install

編譯安裝apache
(注意：編譯的時候要啓用--enable-so  指定apr的位置  啓用--enable-proxy --enable-proxy-http --enable-proxy=ajp模塊）
#yum -y install pcre-devel
#tar xf httpd-2.4.2.tar.bz2
#cd httpd-2.4.2
#./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd --enable-so --enable-sll --enable-zlib --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util --enable-proxy --enable-proxy-http --enable-proxy-ajp
#make
#make install

 

添加PidFile文件
 

#vim /etc/httpd/httpd.conf
添加
PidFile "/var/run/httpd.pid"

 

提供服務腳本編輯/etc/rc.d/init.d/httpd

#!/bin/bash
#
# httpd        Startup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: Apache is a World Wide Web server.  It is used to serve \
#        HTML files and CGI.
# processname: httpd
# config: /etc/httpd/conf/httpd.conf
# config: /etc/sysconfig/httpd
# pidfile: /var/run/httpd.pid

# Source function library.
. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/httpd ]; then
        . /etc/sysconfig/httpd
fi

# Start httpd in the C locale by default.
HTTPD_LANG=${HTTPD_LANG-"C"}

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

# Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server
# with the thread-based "worker" MPM; BE WARNED that some modules may not
# work correctly with a thread-based MPM; notably PHP will refuse to start.

# Path to the apachectl script, server binary, and short-form for messages.
apachectl=/usr/local/apache/bin/apachectl
httpd=${HTTPD-/usr/local/apache/bin/httpd}
prog=httpd
pidfile=${PIDFILE-/var/run/httpd.pid}
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}

stop() {
 echo -n $"Stopping $prog: "
 killproc -p ${pidfile} -d 10 $httpd
 RETVAL=$?
 echo
 [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
reload() {
    echo -n $"Reloading $prog: "
    if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
        RETVAL=$?
        echo $"not reloading due to configuration syntax error"
        failure $"not reloading $httpd due to configuration syntax error"
    else
        killproc -p ${pidfile} $httpd -HUP
        RETVAL=$?
    fi
    echo
}

# See how we were called.
case "$1" in
  start)
 start
 ;;
  stop)
 stop
 ;;
  status)
        status -p ${pidfile} $httpd
 RETVAL=$?
 ;;
  restart)
 stop
 start
 ;;
  condrestart)
 if [ -f ${pidfile} ] ; then
  stop
  start
 fi
 ;;
  reload)
        reload
 ;;
  graceful|help|configtest|fullstatus)
 $apachectl $@
 RETVAL=$?
 ;;
  *)
 echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}"
 exit 1
esac

exit $RETVAL

#chmod +x /etc/rc.d/init.d/httpd
#chkconfig --add httpd
#service httpd start

添加apache的路徑
#vim /etc/profile
PATH:$PATH:/usr/local/apache/bin
而後重新載入/etc/profile
#. /etc/profile

# httpd -D DUMP_MODULES  | grep proxy              //查看加載的模塊
 proxy_http_module (shared)
 proxy_fcgi_module (shared)
 proxy_scgi_module (shared)
 proxy_ajp_module (shared)


tomcat1 tomcat2
安裝tomcat
# rpm -ivh jdk-7u5-linux-i586.rpm

爲JVM指定JDK路徑
#vim /etc/profile
JAVA_HOME=/usr/java/jdk1.7.0_05
PATH=$PATH:$JAVA_HOME/bin:$PATH
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC JAVA_HOME
#. /etc/profile
# printenv
有如下內容
JAVA_HOME=/usr/java/jdk1.7.0_05
# java -version              //可以測試jdk是否生效
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) Client VM (build 23.1-b03, mixed mode, sharing)


#tar xf  apache-tomcat-7.0.29.tar.gz -C /usr/local
# cd /usr/local
# ln -sv apache-tomcat-7.0.29 tomcat
# /usr/local/tomcat/bin/catalina.sh start 啓動

至此，兩臺服務器準備完畢

tomcat1和tomcat2上統一做如下配置 

tomcat1
#vim /usr/local/tomcat/conf/server.xml
修改
 <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatA">
演示效果，在TomcatA上某context中提供特定頁面
#cd /usr/local/tomcat/webapps
#mkdir -pv sess/WEB-INF/{lib,classes}
#cd sess
#vim index.jsp
添加如下內容
<%@ page language="java" %>
<html>
  <head><title>TomcatA</title></head>
  <body>
    <h1><font color="red">TomcatA </font></h1>
    <table align="centre" border="1">
      <tr>
        <td>Session ID</td>
        <td><%= session.getId() %></td>
      </tr>
      <tr>
        <td>Created on</td>
        <td><%= session.getCreationTime() %></td>
     </tr>
    </table>
  </body>
</html>
#/usr/local/tomcat/conf/catalina.sh stop
#/usr/local/tomcat/conf/catalina.sh stop


tomcat2
#vim /usr/local/tomcat/conf/server.xml
修改
 <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatB">

演示效果，在TomcatB上某context中，提供特定頁面
#cd /usr/local/tomcat/webapps
#mkdir -pv sess/WEB-INF/{lib,classes}
#cd sess
#vim index.jsp
添加如下
<%@ page language="java" %>
<html>
  <head><title>TomcatB</title></head>
  <body>
    <h1><font color="blue">TomcatB </font></h1>
    <table align="centre" border="1">
      <tr>
        <td>Session ID</td>
        <td><%= session.getId() %></td>
      </tr>
      <tr>
        <td>Created on</td>
        <td><%= session.getCreationTime() %></td>
     </tr>
    </table>
  </body>
</html>

# /usr/local/tomcat/bin/catalina.sh stop
# /usr/local/tomcat/bin/catalina.sh start

 

二、整合apache和tomcat

2.1 基於mod_jk 實現負載均衡

tomcat1:
修改/etc/httpd/extra/httpd-jk.conf爲如下內容：
# Load the mod_jk
LoadModule  jk_module  modules/mod_jk.so
JkWorkersFile  /etc/httpd/extra/workers.properties
JkLogFile  logs/mod_jk.log
JkLogLevel  debug
JkMount  /*  lbcluster1
JkMount  /status  stat1
保存退出

（lbcluster1  這個名字是虛的，專門是爲了實現負載均衡的)


編輯/etc/httpd/extra/workers.properties，添加如下內容：

worker.list=lbcluster1,stat1

worker.TomcatA.port=8009
worker.TomcatA.host=172.16.220.11
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1

worker.TomcatB.port=8009
worker.TomcatB.host=172.16.220.12
worker.TomcatB.type=ajp13
worker.TomcatB.lbfactor=1

worker.lbcluster1.type=lb
worker.lbcluster1.balance_workers=TomcatA,TomcatB
worker.stat1.type=status
worker.lbcluster1.sticky_session = 0

#service httpd restart

 

測試：172.16.220.11/sess 刷新訪問，可以看到兩臺tomcat主機輪詢訪問，實現了負載均衡

 

 

 

問題依然存在，就是會話無法保持。接下來會解決這個問題
（把/etc/httpd/extra/workers.properties中的worker.lbcluster1.sticky_session 的值改爲1，發現請求總是定向到一臺tomcat服務器上）

2.2 基於mod_proxy實現負載均衡

tomcat1:

#vim /etc/httpd/httpd.conf
Include /etc/httpd/extra/httpd-vhosts.conf
#Include /etc/httpd/extra/httpd-jk.conf

#vim /etc/httpd/extra/httpd-vhosts.conf
內容改爲：
ProxyRequests Off
ProxyPass / balancer://lbcluster1/ stickysession=jsessionid
ProxyPa***everse / balancer://lbcluster1/
<proxy balancer://lbcluster1>
        BalancerMember ajp://172.16.220.11:8009 loadfactor=10 route=TomcatA
        BalancerMember ajp://172.16.220.12:8009 loadfactor=10 route=TomcatB
</proxy>

<VirtualHost *:80>
        ServerAdmin [email protected]
        ServerName tomcat.magedu.com
</VirtualHost>

#service httpd restart

測試tomcat.magedu.com/sess 實現了負載均衡

 

 

 

但是會話依然無法保持（若配置worker.lbcluster1.sticky_session = 1 這裏的sticksession=jsessionid 訪問可以實現請求定向到一臺tomcat上）

四、tomcat基於session複製的集羣

（DeltaManager 和BackupManager都可以實現）
在tomcat1主機上：
#vim /usr/local/tomcat/conf/server.xml
<Engine...>行下面添加如下內容：

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                   channelSendOptions="8">
     <Manager className="org.apache.catalina.ha.session.DeltaManager"
                    expireSessionsOnShutdown="false"
                    notifyListenersOnReplication="true"/>
       <Channel className="org.apache.catalina.tribes.group.GroupChannel">
          <Membership className="org.apache.catalina.tribes.membership.McastService"
                   address="228.50.10.1"   bind="172.16.220.11"   port="45564"
                   frequency="500"  dropTime="3000"/>
             <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                                address="172.16.220.11"   port="4000"  autoBind="100"
                                selectorTimeout="5000"   maxThreads="6"/>
             <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
             </Sender>
       <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
       <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
     </Channel>
     <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                        filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"/>
     <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
     <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                     tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/"
                    watchDir="/tmp/war-listen/"  watchEnabled="false"/>
      <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
      <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

#cp /usr/local/tomcat/conf/web.xml /usr/local/tomcat/webapps/sess/WEB-INF
#vim /usr/local/tomcat/webapps/sess/WEB-INF/web.xml
在
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
                      http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
  version="3.0"> 後面添加如下
<distributable/>

#/usr/local/tomcat/bin/catalina.sh stop
#/usr/local/tomcat/bin/catalina.sh start
#service httpd restart

tomcat2上的配置和tocmat1上的配置一樣，只不過是把tomcat1的地址改爲tomcat2的地址爲172.16.220.12，然後複製web.xml，重啓服務

訪問 tomcat.magedu.com/sess 效果是負載均衡，並且會話是保持的

 

此時前端也可以是mod_jk（把apache的配置文件中的虛擬主機取消,mod_jk啓用即可）

---

 

操作過程中遇到了不少問題，這裏分享一下：

1、第五步中啓動httpd時，啓動成功了，但是端口並沒有監聽，什麼原因？
# tail /usr/local/apache/logs/error_log
[Wed Aug 29 12:25:43.918479 2012] [proxy_balancer:emerg] [pid 25008:tid 3086427904] AH01177: Failed to lookup provider 'shm' for 'slotmem': is mod_slotmem_shm loaded?
編譯配置文件啓用這個模塊即可
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

2、 使用mod_jk總是實現不了複製均衡，什麼問題？頁面訪問的一直是TomcatA，鬱悶啊
原因1：worker.lbcluster1.balance_workers = TomcatA,TomcatB  中 “TomcatA和TomcatB之間多了一個空格，導致TomcatA出不來
 2：沒有加worker.lbcluster1.sticky_session = 0 ，導致無法負載均衡，添加上之後就OK了

3、做mod_proxy時遇到了錯誤
重啓httpd 時錯誤
# service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]
# tail /usr/local/apache/logs/error_log
[proxy_balancer:emerg] [pid 4753:tid 3086935808] (22)Invalid argument: AH01186: worker slotmem_grab failed
[Wed Aug 29 20:15:18.609721 2012] [:emerg] [pid 4753:tid 3086935808] AH00020: Configuration Failed, exiting

解決：在虛擬主機的配置文件中，第一次是把
ProxyPass / balancer://lbcluster1/ stickysession=jsessionid
ProxyPa***everse / balancer://lbcluster1/ 這兩項內容定義在了虛擬主機內部，出了錯誤

定義在全局位置就可以了

4、在實現session複製時，定義<distributable/>時，放在其他地方就不行，即便是放在
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
                      http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
  version="3.0"> 的前面就不行，一定要放在後面纔可以，可以是下面的任何一定地方，這是文件讀取時加載的順序