e_anchor提取方式是基於字符串下標進行提取的,其優點具有方便、快捷、效率高等特點。但是缺點也很明顯靈活性不夠強,通用性比較差,適用於前後綴標識比較明顯,規律性比較強的字符串提取。
正則提取方式衆所周知是通過正則表達式對字符串提取的,優點是靈活性、邏輯性和功能性非常強,基本上很多字符串提取問題都能使用此方式解決,並且可以迅速地用極簡單的方式達到字符串的複雜控制。但是缺點也是不言而喻的,與e_anchor提取模式相比性能會差一些,並且對與剛接觸的人來說,比較晦澀難懂。從而導致學習成本高、上手難度大、不適合新手快速解決自身遇到的問題等。
本文將通過具體實例,向大家講述使用e_anchor和正則的適用場景
解析自定義日誌文本
e_anchor函數適合解決一些單個,或者多個規律性比較強,有明顯前綴標識的文本字符串,比如遇到以下類型日誌:
# 日誌1:
__source__: 1.1.16.15
__tag__:__client_ip__: 12.1.75.140
__tag__:__receive_time__: 1563443076
content: Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: [email protected]: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down.
# 日誌2:
__source__: 1.1.16.15
__tag__:__client_ip__: 12.1.75.140
__tag__:__receive_time__: 1563443077
content: Feb 5 3:15:09: host=1.1.1.1: local/ssl2 error mcpd[1222]: [email protected]: severity=error: 01070639:6: Pool member 192.168.1.1 monitor status invalid.
從日誌結構上分析可以看出日誌信息都包含以下字段名:其中host、mcpd、User、severity是原始日誌固定字段。
LOG DSL編排
本部分將提供兩種方案,解析上述日誌文本。
方案一:e_anchor解析
# Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: [email protected]: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down.
e_anchor("content", "*: host=*: local/ssl2 * mcpd[*]: User=*: severity=*: * Pool member * monitor status *.", ["time", "host","level", "mcpd", "user_field","severity_field","*","pool_member", "monitor_status"])
預覽處理日誌:
# 日誌1
content: Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: [email protected]: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down.
time: Aug 2 04:06:08
host: 10.1.1.124
level: notice
mcpd: 3772
user_field: [email protected]
severity_field: warning
pool_member: 172.31.51.22:0
monitor_status: down
# 日誌2
content: Feb 5 3:15:09: host=1.1.1.1: local/ssl2 error mcpd[1222]: [email protected]: severity=error: 01070639:6: Pool member 192.168.1.1 monitor status invalid.
time: Feb 5 3:15:09
host: 1.1.1.1
level: error
mcpd: 1222
user_field: [email protected]
severity_field: error
pool_member: 192.168.1.1
monitor_status: invalid
方案二:e_regex正則解析
e_regex("content",r'(?P<time>(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?) (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)): host=(?P<host>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])): local\/ssl2 [a-z]+ mcpd\[(?P<mcpd>[0-9]+)\]: User=(?P<user_field>[a-zA-Z][a-zA-Z0-9_.+-=:]+@\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)): severity=(?P<severity_field>[a-z]+): (?P<temp_field>[0-9]+:[0-9]+:) Pool member (?P<pool_member>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]):[0-9]+) [a-z]+ [a-z]+ (?P<monitor_status>[a-z]+).')
grok函數解析詳見數據加工文檔-grok函數,正則與grok函數對比,詳見解析Nginx日誌方案對比:
e_regex("content",grok('%{DATE_TIME:time}: host=%{IP:host}: local/ssl2 %{USERNAME:level} mcpd\[%{NUMBER:mcpd}\]: User=%{EMAILADDRESS:user_field}: severity=%{USERNAME:severity_field}: %{NUMBER}:%{NUMBER}: %{USERNAME} %{USERNAME} %{IP:pool_member}:%{NUMBER} %{USERNAME} %{USERNAME} %{USERNAME:monitor_status}.',extend={'DATE_TIME': '%{MONTH} %{MONTHDAY} %{TIME}'}))
預覽處理日誌:
# 日誌1
content: Aug 2 04:06:08: host=10.1.1.124: local/ssl2 notice mcpd[3772]: [email protected]: severity=warning: 01070638:5: Pool member 172.31.51.22:0 monitor status down.
time: Aug 2 04:06:08
host: 10.1.1.124
level: notice
mcpd: 3772
user_field: [email protected]
severity_field: warning
pool_member: 172.31.51.22:0
monitor_status: down
# 日誌2
content: Feb 5 3:15:09: host=1.1.1.1: local/ssl2 error mcpd[1222]: [email protected]: severity=error: 01070639:6: Pool member 192.168.1.1 monitor status invalid.
time: Feb 5 3:15:09
host: 1.1.1.1
level: error
mcpd: 1222
user_field: [email protected]
severity_field: error
pool_member: 192.168.1.1
monitor_status: invalid
從加工結果上分析可以看出使用anchor的本質跟據不變的邊界提取變化的值,所以可能會有一部分變化的值被提取出來但不一定會被使用,不需要的值可以在fields中命名爲*
詳細參考e_anchor中的fields參數說明。
從加工規則上看方案一e_anchor加工規則更加容易理解上手,語法更簡單。而方案二正則加工規則卻晦澀難懂,並且容易出錯,並且加工性能方面不如方案一。
對比
e_anchor函數
該函數靈活度不夠強,比如以上日誌形式改成如下:
# 日誌1
content: Aug 2 04:06:08: 10.1.1.124: local/ssl2 notice mcpd[3772]: [email protected]: warning: 01070638:5: 172.31.51.22:0 down.
# 日誌2
content: Feb 5 3:15:09: 1.1.1.1: local/ssl2 error mcpd[1222]: [email protected]: error: 01070639:6: 192.168.1.1:0 invalid.
這種沒有明顯通用前後綴標識形式的日誌文本(因爲時間中也有:
導致標識邊界不明顯),使用e_anchor函數很難將全部的content解析出來,只能對單個的content有效,不具有通用性。
e_regex函數
這種形式的日誌文本,用正則解析是能夠解析出來的,具體操作如下:
e_regex("content","(?P<time>\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?) (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)): (?P<host>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])): local\/ssl2 [a-z]+ mcpd\[(?P<mcpd>[0-9]+\]): (?P<user_field>[a-zA-Z][a-zA-Z0-9_.+-=:]+@\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)): (?P<severity_field>[a-z]+): (?P<temp_field>[0-9]+:[0-9]+:) (?P<pool_member>(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9]):[0-9]+) (?P<monitor_status>[a-z]+).")
預覽處理日誌:
# 日誌1
content: Aug 2 04:06:08: 10.1.1.124: local/ssl2 notice mcpd[3772]: [email protected]: warning: 01070638:5: 172.31.51.22:0 down.
time: Aug 2 04:06:08
host: 10.1.1.124
level: notice
mcpd: 3772
user_field: [email protected]
severity_field: warning
pool_member: 172.31.51.22:0
monitor_status: down
# 日誌2
content: Feb 5 3:15:09: 1.1.1.1: local/ssl2 error mcpd[1222]: [email protected]: error: 01070639:6: 192.168.1.1:0 invalid.
time: Feb 5 3:15:09
host: 1.1.1.1
level: error
mcpd: 1222
user_field: [email protected]
severity_field: error
pool_member: 192.168.1.1
monitor_status: invalid
從加工的結果來看,解析出來的內容,與之前一樣。在上述加工規則中,正則表達式只是把之前的前綴標識去掉,規則改動並不大,也能夠把全部的content解析出來。
總結
1、有通用前後綴標識文本解析
日誌
# 日誌1
content: Time=10/Jun/2020:11:32:16 +0800; Host=m.zf.cn; Method=GET; Url=http://aliyun/zf/11874.html;
# 日誌2
content: Time=11/Feb/2020:12:22:10 +0800; Host=sls.aliyun.cn; Method=POST; Url=http://aliyun/sls/1235.html;
以上日誌形式都有通用的前綴標識如:Time=, Host=, Method=, Url=
後綴標識:;
即每個字段信息結尾都有一個分號。類似這種形式日誌推薦使用e_anchor來做文本解析。
解析規則
# e_anchor提取
e_anchor("content","Time=*; Host=*; Method=*; Url=*;",["time","host","method","url"])
# 正則提取
e_regex("content","Time=(?<time>\b(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\/\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b\/(?>\d\d){1,2}:(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)\s\+[0-9]{4}); Host=(?<host>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)); Method=(?<method>[a-zA-Z]+); Url=(?<url>[a-zA-Z][a-zA-Z0-9_.+-=:]+);")
# e_kv提取
e_kv("content",sep="=", quote='"')
如果考慮加工速率,e_kv雖然語法短但是本質上也是使用正則做的提取,而e_anchor本質是使用字符串下標做解析的,所以推薦使用e_anchor函數解析。
2、有無通用前(後)綴標識混合文本解析
日誌
# 日誌1
content: 10/Jun/2020:11:32:16 +0800; m.zf.cn; Method=GET; Url=http://aliyun/zf/11874.html;
# 日誌2
content: 11/Feb/2020:12:22:10 +0800; sls.aliyun.cn; Method=POST; Url=http://aliyun/sls/1235.html;
以上日誌形式通用的前綴標識如:Method=, Url=
後綴標識:;
即每個字段信息結尾都有一個分號。但是,像time和hostname信息沒有明顯的前綴標識。類似這種形式日誌也可以使用e_anchor來做文本解析。
解析規則
# e_anchor提取
e_anchor("content","*; *; Method=*; Url=*;",["time","host","method","url"])
# 正則提取
e_regex("content","(?<time>\b(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\/\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b\/(?>\d\d){1,2}:(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)\s\+[0-9]{4}); (?<host>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)); Method=(?<method>[a-zA-Z]+); Url=(?<url>[a-zA-Z][a-zA-Z0-9_.+-=:]+);")
此類型的日誌使用e_kv是不能夠完全解析出來。所以這種形式的日誌推薦使用e_anchor函數
3、無明顯通用前或後綴標識文本解析
日誌
# 日誌1
content: Aug 10: 11:12:03: Twiss Programmer Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
# 日誌2
content: Feb 11: 10:00:00: Iran VC This will be 0 if no session key was requested.
無明顯前後綴標識日誌
解析規則
e_regex("content","(?P<time>(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?) (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]): (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)): (?P<name>[a-zA-Z]+) (?P<job>[a-zA-Z]+) (?P<msg>(.*)$)")
如果使用e_anchor規則解析的話解析起來會比較冗餘,需要把month, day, time單獨提取出來,如:
e_anchor("content","* *: *: * * *","month,day,time,user,job,msg")
如果把日期時間作爲整體提取如:
e_anchor("source","*: * * *","time,user,job,msg")
# 提取日誌爲
"""
content : Feb 11: 10:00:00: Iran VC This will be 0 if no session key was requested.
time : Feb 11
user: 10:00:00:
job : Iran
msg : VC This will be 0 if no session key was requested.
"""
明顯看出以上e_anchor把日期時間作爲整體解析出來的日誌是錯誤的。因此,此類型的日誌前綴和後綴標識不明顯推薦e_regex函數解析。
4、*
特殊字符作爲前後綴標識文本
日誌
# 日誌1
content: Aug 10 11:12:03* Twiss* Programmer;
# 日誌2
content: Feb 11 10:00:00* Iran* VC;
此類型日誌無明顯前綴標識,有明顯的*
標做後綴。
解析規則
正則解析:
e_regex("content","(?P<time>(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?) (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))\* (?P<user>[a-zA-Z]+)、* (?P<Job>[a-zA-Z]+);")
以下使用e_anchor解析方式是錯誤的(是一個錯誤示例):
e_anchor("content","** **; *;",["time","user","job"])
e_anchor中的提取規則不支持**
這種形式。
此類型以*
特殊符號作爲明顯前後標識的文本,適合使用e_regex函數,不適合使用e_anchor函數。
5、e_anchor函數和e_regex函數適用場景表
場景 | e_anchor函數 | e_regex函數 |
---|---|---|
有通用前後綴標識文本解析 | 適合(推薦使用) | 適合 |
有無通用前(後)綴標識混合文本解析 | 適合(推薦使用) | 適合 |
無明顯通用前或後綴標識文本解析 | 不適合 | 適合(推薦使用) |
* 特殊字符作爲前後綴標識文本 |
不適合 | 適合(推薦使用) |