Nginx+Tomcat+memcached負載均衡實現session共享

前言

通過調度器進行調度時，要保證後端不同服務器對客戶的訪問進行狀態保持，也就是說要共享客戶端與後端集羣服務器之間建立的session會話不能丟失，各主機之間在處理用戶請求時，都要能夠對用戶的會話進行處理。
接下來，我們來介紹一種常見的方式，也是實際生產中經常使用到的一種方式，使用memcached配合tomcat搭建Seesion Server。

要用到的jar包
可以到 網絡上去下載以下jar包
https://github.com/magro/memcached-session-manager
javolution-5.4.3.1.jar
memcached-session-manager-1.8.3.jar
memcached-session-manager-tc7-1.8.3.jar
msm-javolution-serializer-1.8.3.jar
spymemcached-2.11.1.jar

三臺主機
A:做調度器，使用nginx
B:安裝tomcat和memcached
C:安裝tomcat和memcached
A：172.18.25.53
B：172.18.25.51
C：172.18.25.52

nginx

vim /etc/nginx/nginx.conf
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
        index index.jsp index.php index.do index.html;#定義優先讀取index.jsp文件
        upstream tcsrvs {
                server 172.18.25.51:8080;   #定義後端B主機
                server 172.18.25.52:8080;   #定義後端A主機
        }
    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        proxy_pass http://tcsrvs;#這裏在定義代理調度
         }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
        include /etc/nginx/default.d/*.conf;

        location / {
        proxy_pass http://tcsrvs;
         }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

這裏的nginx只是起一個簡單的反向代理的負載均衡的調度作用，就沒有在上面做過多的配置。

tomcat

首先還是簡單說一下tomcat的tomcat程序環境

    tomcat的目錄結構
        bin：腳本，及啓動時用到的類；
        conf：配置文件目錄；
        lib：庫文件，Java類庫，jar；
        logs：日誌文件目錄；
        temp：臨時文件目錄；
        webapps：webapp的默認目錄；
        work：工作目錄，存放編譯後的字節碼文件；

    # catalina.sh --help
        debug             Start Catalina in a debugger
        debug -security   Debug Catalina with a security manager
        jpda start        Start Catalina under JPDA debugger
        run               Start Catalina in the current window
        run -security     Start in the current window with security manager
        start             Start Catalina in a separate window
        start  -security   Start in a separate window with security manager
        stop              Stop Catalina, waiting up to 5 seconds for the process to end
        stop n            Stop Catalina, waiting up to n seconds for the process to end
        stop -force       Stop Catalina, wait up to 5 seconds and then use kill -KILL if still running
        stop n -force     Stop Catalina, wait up to n seconds and then use kill -KILL if still running
        configtest        Run a basic syntax check on server.xml - check exit code for result
        version           What version of tomcat are you running?               


    rpm包安裝的程序環境：
        配置文件目錄：/etc/tomcat
            主配置文件：server.xml 
        webapps存放位置：/var/lib/tomcat/webapps/
            examples
            manager
            host-manager
            docs
        Unit File：tomcat.service
        環境配置文件：/etc/sysconfig/tomcat

我這裏爲了方便直接使用的base源裏面的包，對了還有上面的nginx也是直接使用的base源裏面的包。

[ root@B ~ ]# rpm -q tomcat
tomcat-7.0.69-10.el7.noarch
[ root@B ~ ]# rpm -q memcached
memcached-1.4.15-10.el7.x86_64

爲了實驗的隔離，自己創建一個文件夾

[ root@B ~ ]# cd /usr/share/tomcat/webapps/
[ root@B /usr/share/tomcat/webapps ]# mkdir -pv myapp/WEB-INF
[ root@C ~ ]# cd /usr/share/tomcat/webapps/
[ root@C /usr/share/tomcat/webapps ]# mkdir -pv myapp/WEB-INF

在myapp下面創建測試頁面代碼

[ root@B /usr/share/tomcat/webapps/myapp ]# cat index.jsp 
<%@ page language="java" %>
<html>
    <head><title>TomcatA</title></head>
    <body>
        <h1><font color="red">TomcatA</font></h1>
        <table align="centre" border="1">
            <tr>
                <td>Session ID</td>
            <% session.setAttribute("magedu.com","magedu.com"); %>
                <td><%= session.getId() %></td>
            </tr>
            <tr>
                <td>Created on</td>
                <td><%= session.getCreationTime() %></td>
            </tr>
        </table>
    </body>
</html>
[ root@C /usr/share/tomcat/webapps/myapp ]# cat index.jsp 
<%@ page language="java" %>
<html>
    <head><title>TomcatB</title></head>
    <body>
        <h1><font color="blue">TomcatC</font></h1>
        <table align="centre" border="1">
            <tr>
                <td>Session ID</td>
            <% session.setAttribute("magedu.com","magedu.com"); %>
                <td><%= session.getId() %></td>
            </tr>
            <tr>
                <td>Created on</td>
                <td><%= session.getCreationTime() %></td>
            </tr>
        </table>
    </body>
</html>

把/etc/tomcat/下的web.xml文件複製到之前創建WEB-INF下

[ root@B /usr/share/tomcat/webapps/myapp/WEB-INF ]# cp /etc/tomcat/web.xml .
[ root@C /usr/share/tomcat/webapps/myapp/WEB-INF ]# cp /etc/tomcat/web.xml .

接下來修改/etc/tomcat/server.xml文件
分別在兩個tomcat上的某host上定義一個用於測試的context容器，並在其中創建一個會話管理器，如下所示：

[ root@B /etc/tomcat ]# vim server.xml 

<Engine name="Catalina" defaultHost="localhost" jvmRoute="tcA">
     <!--For clustering, please take a look at documentation at:
         /docs/cluster-howto.html  (simple how to)
         /docs/config/cluster.html (reference documentation) -->
     <!--
     <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
     -->
     <!-- Use the LockOutRealm to prevent attempts to guess user passwords
          via a brute-force attack -->
     <Realm className="org.apache.catalina.realm.LockOutRealm">
       <!-- This Realm uses the UserDatabase configured in the global JNDI
            resources under the key "UserDatabase".  Any edits
            that are performed against this UserDatabase are immediately
            available for use by the Realm.  -->
       <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
              resourceName="UserDatabase"/>
     </Realm>
     <Host name="localhost"  appBase="webapps"
           unpackWARs="true" autoDeploy="true">
<Context path="/myapp" docBase="myapp" reloadable="true" >
 <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="m1:172.18.25.51:11211,m2:172.18.25.52:11211"
   failoverNodes="m2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"   transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory" />
</Context>




[ root@C /etc/tomcat ]# vim server.xml 

<Engine name="Catalina" defaultHost="localhost" jvmRoute="tcA">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- Use the LockOutRealm to prevent attempts to guess user passwords
           via a brute-force attack -->
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <!-- This Realm uses the UserDatabase configured in the global JNDI
             resources under the key "UserDatabase".  Any edits
             that are performed against this UserDatabase are immediately
             available for use by the Realm.  -->
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
<Context path="/myapp" docBase="myapp" reloadable="true" >
  <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="m1:172.18.25.51:11211,m2:172.18.25.52:11211"
    failoverNodes="m2"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"    transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory" />
</Context>
      </Host>
    </Engine>
  </Service>
</Server>

到這裏tomcat這邊的配置就完成了。

memcached

memcached的配置文件特別的少，很簡單的一個軟件

[ root@B /etc/tomcat ]# rpm -ql memcached
/etc/sysconfig/memcached
/usr/bin/memcached
/usr/bin/memcached-tool
/usr/lib/systemd/system/memcached.service
/usr/share/doc/memcached-1.4.15
/usr/share/doc/memcached-1.4.15/AUTHORS
/usr/share/doc/memcached-1.4.15/CONTRIBUTORS
/usr/share/doc/memcached-1.4.15/COPYING
/usr/share/doc/memcached-1.4.15/ChangeLog
/usr/share/doc/memcached-1.4.15/NEWS
/usr/share/doc/memcached-1.4.15/README.md
/usr/share/doc/memcached-1.4.15/protocol.txt
/usr/share/doc/memcached-1.4.15/readme.txt
/usr/share/doc/memcached-1.4.15/threads.txt
/usr/share/man/man1/memcached-tool.1.gz
/usr/share/man/man1/memcached.1.gz

能修改的東西也非常少，但是有一點要注意，他默認的CACHESIZE只有64M
我們要把它變大一點，我這裏設置的是1024M,在生產中要看實際的情況來設定。

[ root@B /etc/tomcat ]# vim /etc/sysconfig/memcached 
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="1024"
OPTIONS=""

其他的就基本沒什麼要修改的了。
啓動服務。

[ root@A ~ ]# systemctl start nginx
[ root@B /etc/tomcat ]# systemctl start tomcat
[ root@B /etc/tomcat ]# systemctl start memcached
[ root@C /etc/tomcat ]# systemctl start tomcat
[ root@C /etc/tomcat ]# systemctl start memcached

實驗部分

到這裏實驗就成功了。
對了，
還有如果發現在測試的時候碰見了sessionID不是固定的
可以使用命令查看memcached是否緩存了session信息

寫進去了是有值的，如果沒有值就要排錯嘍.....
[ root@C /etc/tomcat ]# memdump --servers=172.18.25.51:11211
5E1F1A437DA5335CC4A5FC482761F2B0-m1.tcA

最後

這段時間隨着難度的加深，時間越來越少了，
只能抽時間把一些關鍵的東西拿出來寫一些了，寫的不詳細，勿怪…