Nginx日誌分析統計 goaccess-1.3部署與使用

其實別人已經寫得很好了，但是我將幾個需要注意的點給標註一下。

需求：及時得到線上用戶nginx訪問日誌分析統計結果！我需要知道IP訪問排行、鏈接訪問排行、靜態資源、404錯誤排行等
直接上效果圖：

具體安裝步驟如下：
1、安裝依賴

[root@iZbp1f0xuq9rc41s6gdvfyZ ~]# mkdir access
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# yum install glib2 glib2-devel GeoIP-devel  ncurses-devel zlib zlib-devel -y
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# wget https://github.com/maxmind/geoip-api-c/releases/download/v1.6.11/GeoIP-1.6.11.tar.gz
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# tar -zvxf GeoIP-1.6.11.tar.gz
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# cd GeoIP-1.6.11
[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# ./configure
[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# make && make install

2、安裝goaccess

[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# cd ..
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# wget https://tar.goaccess.io/goaccess-1.3.tar.gz
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# tar -xzvf goaccess-1.3.tar.gz
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# cd goaccess-1.3
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# ./configure --enable-utf8 --enable-geoip=legacy
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# make && make install

3、校對nginx和goaccess日誌格式
查看當前的nginx配置，日誌格式

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# pwd
/usr/local/nginx/conf
[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# cat /usr/local/nginx/conf/nginx.conf

記住這一段，後面有用。每個機器配置可能不一樣哦，使用你自己的

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
		'$status $body_bytes_sent "$http_referer" ' 
		'"$http_user_agent" "$http_x_forwarded_for" "$request_body"';

4、goaccess日誌格式配置

一：使用nginx2goaccess.sh腳本將nginx日誌格式格式化爲goaccess能識別的日誌格式，nginx2goaccess.sh腳本內容在 

[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# vi nginx2goaccess.sh

#!/bin/bash
#
# Convert from this:
#   http://nginx.org/en/docs/http/ngx_http_log_module.html
# To this:
#   https://goaccess.io/man
#
# Conversion table:
#   $time_local         %d:%t %^
#   $host               %v
#   $http_host          %v
#   $remote_addr        %h
#   $request_time       %T
#   $request_method     %m
#   $request_uri        %U
#   $server_protocol    %H
#   $request            %r
#   $status             %s
#   $body_bytes_sent    %b
#   $bytes_sent         %b
#   $http_referer       %R
#   $http_user_agent    %u
#
# Samples:
#
# log_format combined '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $body_bytes_sent '
# '"$http_referer" "$http_user_agent"';
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
#
# log_format compression '$remote_addr - $remote_user [$time_local] '
# '"$request" $status $bytes_sent '
# '"$http_referer" "$http_user_agent" "$gzip_ratio"';
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"'
#
# log_format main
# '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\t'
# 'Local:\t$status\t$body_bytes_sent\t$request_time\t'
# 'Proxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\t'
# 'Agent:\t$http_user_agent\t'
# 'Fwd:\t$http_x_forwarded_for';
#   ./nginx2goaccess.sh '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\tLocal:\t$status\t$body_bytes_sent\t$request_time\tProxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\tAgent:\t$http_user_agent\tFwd:\t$http_x_forwarded_for'
#
# log_format main
# '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t'
# '${http_referer}\t${http_x_mobile_group}\t'
# 'Local:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\t'
# 'Proxy:\t${upstream_status}\t${upstream_cache_status}\t'
# '${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\t'
# 'Agent:\t${http_user_agent}\t'
# 'Fwd:\t${http_x_forwarded_for}';
#   ./nginx2goaccess.sh '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t${http_referer}\t${http_x_mobile_group}\tLocal:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\tProxy:\t${upstream_status}\t${upstream_cache_status}\t${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\tAgent:\t${http_user_agent}\tFwd:\t${http_x_forwarded_for}'
#
# Author: Rogério Carvalho Schneider <[email protected]>

# Params
log_format="$1"

# Usage
if [[ -z "$log_format" ]]; then
    echo "Usage: $0 '<log_format>'"
    exit 1
fi

# Variables map
conversion_table="time_local,%d:%t_%^
host,%v
http_host,%v
remote_addr,%h
request_time,%T
request_method,%m
request_uri,%U
server_protocol,%H
request,%r
status,%s
body_bytes_sent,%b
bytes_sent,%b
http_referer,%R
http_user_agent,%u"

# Conversion
for item in $conversion_table; do
    nginx_var=${item%%,*}
    goaccess_var=${item##*,}
    goaccess_var=${goaccess_var//_/ }
    log_format=${log_format//\$\{$nginx_var\}/$goaccess_var}
    log_format=${log_format//\$$nginx_var/$goaccess_var}
done
log_format=$(echo "$log_format" | sed 's/${[a-z_]*}/%^/g')
log_format=$(echo "$log_format" | sed 's/$[a-z_]*/%^/g')

# Config output
echo "
- Generated goaccess config:
time-format %T
date-format %d/%b/%Y
log_format $log_format
"

# EOF

二：使用nginx2goaccess.sh方式獲取日誌格式

將第二步裏面Nginx日誌的單引號刪除：

刪除前：'$remote_addr - $remote_user [$time_local] "$request" '
		'$status $body_bytes_sent "$http_referer" ' 
		'"$http_user_agent" "$http_x_forwarded_for" "$request_body"'

刪除後：$remote_addr - $remote_user [$time_local] "$request" 
		$status $body_bytes_sent "$http_referer" 
		"$http_user_agent" "$http_x_forwarded_for" "$request_body"

然後執行命令如：
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# sh nginx2goaccess.sh '刪除後'
執行真實命令：
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# sh nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_body"'

得到如下結果：

- Generated goaccess config:
time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%r_body"

三：將上面生成的格式在goaccess-1.3/config下面創建一個nginxlog.conf

[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# cd config

[root@iZbp1f0xuq9rc41s6gdvfyZ config]# vi nginxlog.conf

# Generated goaccess config:
time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%r_body"

5、配置完成，開始見證奇蹟的時刻，生成統計頁面。

第一步：生成結果文件夾的位置   /data/nginx
[root@iZbp1f0xuq9rc41s6gdvfyZ /]# mkdir /data
[root@iZbp1f0xuq9rc41s6gdvfyZ /]# cd data/
[root@iZbp1f0xuq9rc41s6gdvfyZ data]# mkdir nginx

第二步：nginx日誌位置、最好做了每日切割，否則太大了
[root@iZbp1f0xuq9rc41s6gdvfyZ logs]# pwd
/usr/local/nginx/logs

第三步：開始日誌分析，需要回到goaccess-1.3 文件夾
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# ./goaccess -f /usr/local/nginx/logs/access.log -p config/nginxlog.conf -o /data/nginx/report.html

第四步：生成中文的方法
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# LANG="zh_CN.UTF-8" bash -c "./goaccess -f /usr/local/nginx/logs/access.log -p config/nginxlog.conf -o /data/nginx/report.html"

說明：第三步、第四步分別生成的是當前時間的日誌分析結果，五分鐘之後想看新的得再執行一次
下面我們寫個定時器，讓五分鐘分析一次

下面配置定時更新
第五步：創建腳本
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# mkdir scripts
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# cd scripts
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# vi goaccess.sh

#!/bin/bash
LANG="zh_CN.UTF-8"           #頁面轉換爲中文
"/root/goaccess-1.3/goaccess -f /usr/local/nginx/logs/access.log -p /root/goaccess-1.3/config/nginxlog.conf -o /data/nginx/report.html"

第六步：加入定時器
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# chmod + ./goaccess.sh
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# pwd
/root/goaccess-1.3/scripts
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# crontab -e
# 定時goaccess 每五分鐘生成的信息保存爲html
5 * * * * /root/goaccess-1.3/scripts/goaccess.sh

6、配置nginx的配置文件，可訪問

配置nginx訪問剛剛生成的report.html頁面，阿里雲的朋友自行配置即可，沒多大的區別！

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# pwd
/usr/local/nginx/conf

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# vi nginx.conf
加上此段：

server{
	listen 8080;
	server_name localhost;

	location /report.html {
            alias /data/nginx/report.html;
        } 
}

重啓nginx

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf

在瀏覽器就 http://192.168.2.253:8080/report.html 可以瀏覽上上面的截圖了。