用Awstats工具來分析httpd和tomcat的訪問日誌
準備工作：
Awstats 是由perl語言編寫的，所以要首先準備好awstats的運行環境。
# yum install –y perl*
一、首先，要安裝apache服務器，並且啓動httpd服務。
我的apache是yum安裝的，配置文件路徑爲：/etc/httpd/conf/httpd.conf
下載awstats工具。
# cd /opt
# tar xvf awstats-7.1.tar.gz
# mv awstats-7.1 /usr/local/awstats
二、配置awstats分析httpd的訪問日誌
# cd /usr/local/awstats
# ls
# cd tools
# perl awstats_configure.pl
----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).
-----> Running OS detected: Linux, BSD or Unix
-----> Check for web server install
Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example:c:\Programfiles\apachegroup\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
>/etc/httpd/conf/httpd.conf 根據自己的httpd服務安裝的具體路徑填寫
-----> Check and complete web server config file
'/etc/httpd/conf/httpd.conf'
Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'
Add 'Alias /awstatscss
"/usr/local/awstats/wwwroot/css/"'
Add 'Alias /awstatsicons
"/usr/local/awstats/wwwroot/icon/"'
Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'
Add '<Directory>' directive
AWStats directives added to Apache config file.
-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
File awstats.model.conf updated.
-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ?y
-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
>lingling 可以是任意的名字，也可以是完整的域名格式，只是爲了區分你要分析的那份日誌的來源的網站，自己注意不要混淆就好。
-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
>
默認的awstats生成的配置文件目錄，根據喜好可以更改。
-----> Create config file '/etc/awstats/awstats.lingling.conf'
Config file /etc/awstats/awstats.lingling.conf created.
-----> Restart Web server with '/sbin/service httpd restart'
Stopping httpd: [OK]
Starting httpd: [OK]
-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue...
A SIMPLE config file has been created: /etc/awstats/awstats.lingling.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'lingling' with command:
> perl awstats.pl -update -config=lingling
You can also read your statistics for 'lingling' with URL:
> http://localhost/awstats/awstats.pl?config=lingling
Press ENTER to finish...
1、由於httpd的log文件默認是/var/log/httpd/access.log，
所以要修改/etc/awstats/awstats.lingling.conf文件裏的LogFile：
把LogFile="/var/log/httpd/mylog.log"改爲LogFile="/var/log/httpd/access_log"
或者LogFile="var/log/access_log.%YYYY-0%MM-0%DD-0.log"
2、然後，手動更新一下：
# cd /usr/local/awstats/wwwroot/cgi-bin/
# perl awstats.pl –update –config=lingling
或者：# /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling
Error: AWStats database directory defined in config file by 'DirData' parameter (/var/lib/awstats) does not exist or is not writable.
Setup ('/etc/awstats/awstats.lingling.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
出錯：忘記了創建awstats的默認數據存放目錄
解決：# mkdir –m 755 /var/lib/awstats
Create/Update database for config "/etc/awstats/awstats.lingling.conf" by AWStats version 7.0 (build 1.976)
From data in log file "/var/log/httpd/access_log"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 33664)
Jumped lines in file: 33664
Found 33664 already parsed records.
Parsed lines in file:7
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 7 old records,
Found 0 new qualified records.
3、打開瀏覽器，用awstats分析日誌：
分析結果如下圖:
4、可以將更新的命令作爲執行計劃，使其每天執行一次，方便分析前一天的日。
# crontab –e
10 1 * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=lingling > /dev/null 2&>1
三、用awstats分析tomcat的訪問日誌
1、要分析tomcat的日誌，就要首先了解其日誌格式。
並比較與httpd的訪問日誌格式有什麼不同之處，然後就可以參照awstats分析httpd日誌的格式來定義awstats分析tomcat的日誌。
我的tomcat服務器上定義的訪問日誌格式如下：
<Valve className="org.apache.catalina.valves.
AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
%...a: 遠程IP地址
%...A: 本地IP地址
%...B: 已發送的字節數，不包含HTTP頭
%...b: CLF格式的已發送字節數量，不包含HTTP頭。
例如當沒有發送數據時，寫入‘-’而不是0。
%e: 環境變量FOOBAR的內容
%...f: 文件名字
%...h: 遠程主機
%...H 請求的協議
%i: Foobar的內容，發送給服務器的請求的標頭行。
%...l: 遠程登錄名字（來自identd，如提供的話）
%...m 請求的方法
%n: 來自另外一個模塊的註解“Foobar”的內容
%o: Foobar的內容，應答的標頭行
%...p: 服務器響應請求時使用的端口
%...P: 響應請求的子進程ID。
%...q 查詢字符串（如果存在查詢字符串，則包含“?”後面的
部分；否則，它是一個空字符串。）
%...r: 請求的第一行
%...s: 狀態。對於進行內部重定向的請求，這是指*原來*請求
的狀態。如果用%...>s，則是指後來的請求。
%...t: 以公共日誌時間格式表示的時間（或稱爲標準英文格式）
%t: 以指定格式format表示的時間
%...T: 爲響應請求而耗費的時間，以秒計
%...u: 遠程用戶（來自auth；如果返回狀態（%s）是401則可能是僞造的）
%...U: 用戶所請求的URL路徑
%...v: 響應請求的服務器的ServerName
%...V: 依照UseCanonicalName設置得到的服務器名字
最後的tomcat的訪問日誌內容如下：
203.156.200.162 - - [29/Aug/2012:11:16:58 +0800] "GET /front/magazine/getContent.htm?contentId=124504 HTTP/1.1" 200 20001
2、由於我的tomcat服務器是在其他機器上，所以我將tomcat的服務日誌copy到本機的/var/log/httpd/下即可。
如copy的文件是：localhost_access_log.2012-08-29.txt
3、配置awstats分析此日誌(tomcat 的域名並不是httpd的虛擬主機，所以沒有寫進httpd.conf文件裏面)
# cd /usr/local/awstats/tools
# perl awstats_configure.pl
----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).
-----> Running OS detected: Linux, BSD or Unix
-----> Check for web server install
Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
>none
Your web server config file(s) could not be found.
You will need to setup your web server manually to declare AWStats
script as a CGI, if you want to build reports dynamically.
See AWStats setup documentation (file docs/index.html)
-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
File awstats.model.conf updated.
-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y
-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
>buoqu.com
-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
>
-----> Create config file '/etc/awstats/awstats.buoqu.com.conf'
Config file /etc/awstats/awstats.buoqu.com.conf created.
-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=buoqu.com
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
Press ENTER to continue...
A SIMPLE config file has been created: /etc/awstats/awstats.buoqu.com.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'buoqu.com' with command:
> perl awstats.pl -update -config=buoqu.com
You can also build static report pages for 'buoqu.com' with command:
> perl awstats.pl -output=pagetype -config=buoqu.com
Press ENTER to finish...
4、修改要分析日誌文件
# vim /etc/awstats/awstats.buoqu.com.conf
將LogFile="/var/log/httpd/mylog.log"
改爲LogFile="/var/log/httpd/localhost_access_log.2012-08-29.txt"
LogFile="/var/log/httpd/localhost_access_log. YYY-0%MM-0%DD-0.txt"都可以。
5、重啓httpd服務，並分析日誌
# service httpd restart
# cd /usr/local/awstats/wwwroot/cgi-bin
# perl awstats.pl -update -config=buoqu.com
Create/Update database for config "/etc/awstats/awstats.buoqu.com.conf" by AWStats version 7.0 (build 1.976)
From data in log file "/var/log/httpd/localhost_access_log.2012-08-29.txt"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.
Your log file /var/log/httpd/localhost_access_log.2012-08-29.txt must have a bad format or LogFormat parameter setup does not match this format.
Your AWStats LogFormat parameter is:
1
This means each line in your web server log file need to have "combined log format" like this:
111.22.33.44 - - [10/Jan/2001:02:14:14 +0200] "GET / HTTP/1.1" 200 1234 "http://www.fromserver.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
And this is an example of records AWStats found in your log file (the record number 50 in your log):
203.208.60.231 - - [29/Aug/2012:00:02:47 +0800] "GET /front/magazine/getContent.htm?contentId=52253 HTTP/1.1" 200 18419
Setup ('/etc/awstats/awstats.buoqu.com.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
出錯：日誌格式不匹配。
解決：這個時候，就知道我爲什麼要先了解怎麼定義tomcat的日誌格式了。
修改文件/etc/awstats/awstats.buoqu.com.conf
# vim /etc/awstats/awstats.buoqu.com.conf
LogFormat = 1
LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot" 這兩行是註釋的，1表示是web日誌，下面是httpd的日誌格式
LogFormat ="%host %other %logname %time1 %methodurl %code"
這是我修改後的匹配tomcat的日誌格式
# perl awstats.pl -update -config=buoqu.com
Create/Update database for config "/etc/awstats/awstats.buoqu.com.conf" by AWStats version 7.0 (build 1.976)
From data in log file "/var/log/httpd/localhost_access_log.2012-08-29.txt"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
Jumped lines in file: 0
Parsed lines in file: 22442
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 22442 new qualified records.
6、打開網址查看分析結果：
7、手動執行命令可寫入crontab。
①、如果，想在分析頁面上直接刷新，可以開啓AllowToUpdateStatsFromBrowser=1，默認情況下是關閉的。
②、若是想每個頁面上都直接有“立即更新”的按鈕，而不想每次都手動的修改配置文件的話，可以再awstats的基本配置文件裏修改。
# cd /usr/local/awstats/wwwroot/cgi-bin
# vim awstats.model.conf
將AllowToUpdateStatsFromBrowser=0改爲AllowToUpdateStatsFromBrowser=1即可。
這樣，以後的網頁都可以直接點擊刷新的。
注意：每次修改配置文件後要重啓httpd服務
③、若是要在瀏覽器上直接刷新，那麼apache用戶就要有對數據文件操作的權限
# chown apache.apache –R /var/lib/awstats
# chmod 755 /var/log/httpd
效果如圖：
四、添加一些插件，使awstats看起來更人性化和直觀化。
1、首先下載所需要的插件：
# cd /opt
# tar xvf GeoIP-1.4.8.tar.gz
# cd GeoIP-1.4.8
# ./configure
# make && make install
# cd ..
# tar xvf Geo-IP-1.40.tar.gz
# cd Geo-IP-1.40
# perl Makeinstall.pl
# make && make install
# cd ..
#tar xvf Geo-IP-PurePerl-1.25.tar.gz
# perl –MCPAN –e shell
cpan[1]>install Geo::IP
cpan[2]>install Geo::IP::PurePerl
# cd /opt
# tar xvf GeoIP.dat.gz
2、定義插件的使用方法：
可在各自的/etc/awstats/awstats.domain.conf的配置文件中定義，也可在awstats的主配置文件定義，然後全局生效。
如：
# vim /usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf
將以下語句的#註釋去掉：
#LoadPlugin="tooltips" 在html報告中增加一些提示信息
#LoadPlugin="decodeutfkeys" 處理搜索引擎UTF8編碼的關鍵字
#LoadPlugin="geoip GEOIP_STANDARD /opt/GeoIP.dat"
從Internet IP-Country數據庫生成訪問者來自國家的統計圖表
並增加一條LoadPlugin="qqhostinfo"
在訪問主機表格中增加一列，顯示ip對應的位置
下載之後，要將這三個文件，放至/usr/local/awstats/wwwroot/cgi-bin/plugins/下面：
# cd /usr/local/awstats/wwwroot/cgi-bin/plugins
# chmod 777 . –R
