Nginx日誌增長過快詳細分析

原博客地址： http://blog.itpub.net/26230597/viewspace-1305133/

前言：

Nginx日誌裏面Mobileweb_access.log增長特別大，一天上百兆，將近100W的訪問記錄，按照我們目前的規模，熱點用戶才500個左右，就算人人用手機app訪問，怎麼可能會有這麼大的url訪問量？以前只是安裝使用nginx，還沒有抽出時間仔細研究，這回需要徹底的去分析nginx日誌了。

1，日誌分類

主要2種，一種是錯誤日誌，一種是訪問日誌，這些配置都在/usr/local/nginx/conf/nginx.conf裏面，默認都是打開的，自己也可以選擇關閉。

1.1，訪問日誌

訪問日誌主要記錄每一個訪問nginx的請求，格式可以自己定義，在nginx.conf文件裏面，通過訪問日誌，你可以看到每一個請求的詳細信息，對於訪問日誌的格式，主要是配置文件中的log_format來限制的。

1.1.1 log_format日誌格式

$request_time：整個請求的總時間。

$time_iso8601：訪問的時間與時區，比如18/Jul/2012:17:00:01 +0800，時間信息最後的"+0800"表示服務器所處時區位於UTC之後的8小時。

$upstream_response_time：請求過程中，upstream的響應時間。

$request_method：客戶端請求的動作，通常爲GET或POST。

$request_uri：是瀏覽器發過來的值。該值是rewrite後的值。例如做了internal redirects後。

$args：這個變量等於請求行中(GET請求)的參數，例如foo=123&bar=blahblah;

$query_string：與$args相同。

$proxy_add_x_forwarded_for：變量包含客戶端請求頭中的"X-Forwarded-For"，與$remote_addr用逗號分開，如果沒有"X-Forwarded-For" 請求頭，則$proxy_add_x_forwarded_for等於$remote_addr。

$upstream_addr：upstream的地址，即真正提供服務的主機地址。

$status：記錄請求返回的http狀態碼，比如成功是200。

$http_user_agent：客戶端瀏覽器信息

$http_range

$sent_http_content_length：發送內容的長度

$body_bytes_sent：發送給客戶端的文件主體內容的大小，比如899，可以將日誌每條記錄中的這個值累加起來以粗略估計服務器吞吐量。

$http_referer：記錄從哪個頁面鏈接訪問過來的。

$host：請求主機頭字段，否則爲服務器名稱。

$http_x_forwarded_for：客戶端的真實ip，通常web服務器放在反向代理的後面，這樣就不能獲取到客戶的IP地址了，通過$remote_add拿到的IP地址是反向代理服務器的iP地址。反向代理服務器在轉發請求的http頭信息中，可以增加x_forwarded_for信息，用以記錄原有客戶端的IP地址和原來客戶端的請求的服務器地址。

$http_user_agent：客戶端瀏覽器信息

$body_bytes_sent：發送給客戶端的文件主體內容的大小，比如899，可以將日誌每條記錄中的這個值累加起來以粗略估計服務器吞吐量。

$ssl_protocol：SSL協議版本，比如TLSv1。

$ssl_cipher：交換數據中的算法，比如RC4-SHA。

生產環境上的範例：

log_format main '$proxy_add_x_forwarded_for $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for" '

'upsteam: $upstream_addr';

access_log logs/access.log main;

log_not_found off;

1.1.2，訪問日誌路徑

access_log logs/access.log main;

Nginx支持爲每個location指定強大的日誌記錄。同樣的連接可以在同一時間輸出到不止一個的日誌中。如果想關閉日誌，可以如下：

access_log off;

能夠使用access_log指令的字段包括：http、server、location。

PS：Nginx進程設置的用戶和組必須對日誌路徑有創建文件的權限，否則，會報錯。

1.2，錯誤日誌

錯誤日誌主要記錄客戶端訪問Nginx出錯時的日誌，格式不支持自定義。通過錯誤日誌，你可以得到系統某個服務或server的性能瓶頸等。因此，將日誌好好利用，你可以得到很多有價值的信息。錯誤日誌由指令error_log來指定，具體格式如下：

error_log path(存放路徑) level(日誌等級)

path含義同access_log，level表示日誌等級，具體如下：

從左至右，日誌詳細程度逐級遞減，即debug最詳細，crit最少，舉例說明如下：

error_log logs/mobileweb_error.log error;

需要注意的是：error_log off並不能關閉錯誤日誌，而是會將錯誤日誌記錄到一個文件名爲off的文件中。正確的關閉錯誤日誌記錄功能的方法如下：

error_log /dev/null;

上面表示將存儲日誌的路徑設置爲“垃圾桶”。

2，爲每一個工程定義特定的日誌

location ~* ^/mobileWeb/.*$ {

client_max_body_size 5m;

include deny.conf;

proxy_pass http://mobilewebbackend;

include proxy.conf;

error_log logs/mobileweb_error.log error;

access_log logs/mobileweb_access.log main;

include gzip.conf;

}

這樣，就會在日誌路徑/usr/local/nginx/logs/下面生成mobileWeb工程的專門日誌mobileweb_error.log 以及mobileweb_access.log 日誌，如果想查詢mobileWeb工程的訪問記錄，就可以單獨去查看這2個日誌。

3，開始分析

根據來源ip進行分組統計分析，看看哪個ip的訪問量最多

[root@wgq_idc_web_1_21 tmp]# cat mobileweb_access.log |grep "14/Oct/2014" |awk '{print $1}'|sort -nr |uniq -c |sort -nr |more

705980 1xx.xx.xx.185,

190273 6x.1x4.1xx.35,

14900 1xx.xxx.xx.xx3,

14670 1xx.xxx.x3.8x,

結果發現，這幾個ip都是我們公司廣場公用的wifi出口ip地址，屬於安全地址，不是私人的IP地址，很大程度上排除了從外部惡意攻擊我們網站的可能性。接下來就需要重點分析，爲什麼會有這麼多的URL記錄。

仔細排查來源爲1xx.xx.xx.185的日誌記錄，發現有很多$http_user_agent爲空的記錄，大概90%的記錄都是如此，看記錄如下：

1xx.xx.xx.185, 10.2xx.xx1.xx0 - [10/Oct/2014:10:52:11 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "1xx.xx.xx.185"upsteam: 110.xx7.1.22:7100

猜測是否不是手機app訪問的記錄？只有自己停掉wifi，用手機的4G網絡，去登錄我們的移動app應用，操作完，點擊了幾下贊，訪問了一些頁面，操作時間2分鐘，然後使用自己的移動4Gip地址“2xx.10x.5.129”去檢索下nginx下的mobileweb的記錄，4臺nginx記錄，每一臺40個左右url訪問，4臺就是160個記錄，下面是一臺的記錄

[root@wgq_idc_web_1_22 logs]# more mobileweb_access.log |grep "2xx.10x.5.129"

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:01 +0800] "POST /mobileWeb/userMobileCenter/queryUserNameAndIconByIds.htm? HTTP/1.1" 200 20 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:37 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:42 +0800] "POST /mobileWeb/square/query.htm? HTTP/1.1" 200 9485 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:49 +0800] "POST /mobileWeb/square/clickSupport.htm? HTTP/1.1" 200 46 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:51 +0800] "POST /mobileWeb/square/clickSupport.htm? HTTP/1.1" 200 46 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:54 +0800] "POST /mobileWeb/square/clickSupport.htm? HTTP/1.1" 200 46 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:55 +0800] "POST /mobileWeb/square/query.htm? HTTP/1.1" 200 4831 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:54:57 +0800] "POST /mobileWeb/userMobileCenter/queryUserNameAndIconByIds.htm? HTTP/1.1" 200 20 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:03 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:04 +0800] "POST /mobileWeb/version/queryVersion.htm? HTTP/1.1" 200 160 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:06 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:06 +0800] "POST /mobileWeb/mobile/loadCart.htm? HTTP/1.1" 200 940 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:07 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:07 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:07 +0800] "POST /mobileWeb/userMobileCenter/findAllinterfaceVersion.htm? HTTP/1.1" 200 411 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:13 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:56 +0800] "POST /mobileWeb/userMobileCenter/queryUserNameAndIconByIds.htm? HTTP/1.1" 200 20 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:57 +0800] "POST /mobileWeb/userMobileCenter/findAllinterfaceVersion.htm? HTTP/1.1" 200 411 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:58 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:58 +0800] "POST /mobileWeb/version/queryVersion.htm? HTTP/1.1" 200 160 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:55:59 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:00 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:06 +0800] "POST /mobileWeb/userMobileCenter/findAllinterfaceVersion.htm? HTTP/1.1" 200 411 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:07 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:07 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:08 +0800] "POST /mobileWeb/version/queryVersion.htm? HTTP/1.1" 200 160 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:08 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:08 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:16 +0800] "POST /mobileWeb/userMobileCenter/queryUserNameAndIconByIds.htm? HTTP/1.1" 200 20 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:19 +0800] "POST /mobileWeb/userMobileCenter/findAllinterfaceVersion.htm? HTTP/1.1" 200 411 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:21 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:21 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:23 +0800] "POST /mobileWeb/userMobileCenter/queryUserNameAndIconByIds.htm? HTTP/1.1" 200 20 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:23 +0800] "POST /mobileWeb/userMobileCenter/findAllinterfaceVersion.htm? HTTP/1.1" 200 411 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:24 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:24 +0800] "POST /mobileWeb/userMobileCenter/queryAdvertisement.htm? HTTP/1.1" 200 5175 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:24 +0800] "POST /mobileWeb/version/queryVersion.htm? HTTP/1.1" 200 160 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:25 +0800] "POST /mobileWeb/userMobileCenter/unReadNumsMobile.htm? HTTP/1.1" 200 239 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:56:25 +0800] "POST /mobileWeb/mobile/getCartItemNum.htm? HTTP/1.1" 200 114 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:14:57:37 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:00:37 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:02:07 +0800] "POST /mobileWeb/userMobileCenter/messageListMobile.htm? HTTP/1.1" 200 106 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:02:08 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:02:37 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:05:07 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:05:08 +0800] "POST /mobileWeb/push/query.htm? HTTP/1.1" 200 97 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:05:37 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.21:7100

2xx.10x.5.129, 10.2xx.xx1.xx0 - [16/Oct/2014:15:11:44 +0800] "POST /mobileWeb/square/queryCounts.htm? HTTP/1.1" 200 82 "-" "-" "2xx.10x.5.129"upsteam: 110.xx7.1.22:7100

[root@wgq_idc_web_1_22 logs]#

看到了我的訪問url記錄，其中$http_user_agent幾乎都是爲”-”空記錄，奇怪，我也是用手機訪問的，詢問andriod開發人員，他說有些低版本的手機在記錄$http_user_agent後退回去會報錯返回空界面，所以後來就不記錄$http_user_agent信息了。

原來如此，而且看到這麼多url全是我訪問過的，移動mobileweb後臺開發人員說，移動app一個頁面裏面有許多url需要加載，所以你訪問1個頁面就會加載N個link連接去取各種數據值。分析道這裏，已經差不多明瞭：就是一個登錄用戶訪問頁面，會加載N（N>10）個link連接url，這些url都被記錄在nginx訪問日誌裏面，短短2分鐘內，我訪問了一些頁面，就有160個左右的記錄，照這麼算下來，一個小時就是5000個左右的記錄，一天平均25分鐘分鐘，500個用戶個就是SELECT 5000*25/60*500=1041667，差不多100W左右了，通常來說nginx日誌的量比較大是正常的。

其中，半夜1點到6點左右，這個公司廣場wifi的ip地址還會不停的訪問mobileweb，經過分析是由於登錄了移動app應用，但是睡覺了沒有退出應用，手機也沒有關係，所以導致移動app依然不停的在訪問mobile應用（因爲1分鐘左右會刷新一次去獲取訪問當前登錄用戶的站內互動消息）。

從此可以看出nginx的訪問日誌記錄了用戶的所有訪問行爲記錄，而且詳細到每一個頁面裏內嵌的url記錄，如果用適當的工具仔細分析nginx日誌，就會大概摸清楚用戶的訪問習慣，這些數據對於市場部門、產品部門來說，是非常有價值的。

Nginx日誌增長過快詳細分析

公司剛入職了一名 Java 中級開發，短短 4 行代碼居然湊齊了 3 個 bug！我哭了~~

公衆號5月C#/.NET熱文一覽

git 下載大陸鏡像地址

流量限制--令牌桶和漏桶算法（流量整形算法）

Nginx日誌增長過快詳細分析

學習geohash算法

如何用LinkedHashMap實現LRU緩存算法

java系統高併發解決方案

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結