前言
EFK可能都不熟悉,實際上EFK是大名鼎鼎的日誌系統ELK的一個變種。
在沒有分佈式日誌的時候,每次出問題了需要查詢日誌的時候,需要登錄到Linux服務器,使用命令cat -n xxxx|grep xxxx 搜索出日誌在哪一行,然後cat -n xxx|tail -n +n行|head -n 顯示多少行,這樣不僅效率低下,而且對於程序異常也不方便查詢,日誌少還好,一旦整合出來的日誌達到幾個G或者幾十G的時候,僅僅是搜索都會搜索很長時間了,當然如果知道是哪天什麼時候發生的問題當然也方便查詢,但是實際上很多時候有問題的時候,是不知道到底什麼時候出的問題,所以就必須要在聚合日誌中去搜索(一般日誌是按照天來分文件的,聚合日誌就是把很多天的日誌合併在一起,這樣方便查詢),而搭建EFK日誌分析系統的目的就是將日誌聚合起來,達到快速查看快速分析的目的,使用EFK不僅可以快速的聚合出每天的日誌,還能將不同項目的日誌聚合起來,對於微服務和分佈式架構來說,查詢日誌尤爲方便,而且因爲日誌保存在Elasticsearch中,所以查詢速度非常之快。
我認爲,日誌數據在以下幾方面具有非常重要的作用:
- 數據查找:通過檢索日誌信息,定位相應的 bug ,找出解決方案
- 服務診斷:通過對日誌信息進行統計、分析,瞭解服務器的負荷和服務運行狀態
- 數據分析:可以做進一步的數據分析,比如根據請求中的課程 id ,找出 TOP10 用戶感興趣課程。
認識EFK
EFK不是一個軟件,而是一套解決方案,並且都是開源軟件,之間互相配合使用,完美銜接,高效的滿足了很多場合的應用,是目前主流的一種日誌系統。EFK是三個開源軟件的縮寫,分別表示:Elasticsearch , FileBeat, Kibana , 其中ELasticsearch負責日誌保存和搜索,FileBeat負責收集日誌,Kibana 負責界面,當然EFK和大名鼎鼎的ELK只有一個區別,那就是EFK把ELK的Logstash替換成了FileBeat,因爲Filebeat相對於Logstash來說有2個好處:
1、侵入低,無需修改程序目前任何代碼和配置
2、相對於Logstash來說性能高,Logstash對於IO佔用很大
Filebeat 是基於 logstash-forwarder 的源碼改造而成,用 Go語言編寫,無需依賴 Java 環境,效率高,佔用內存和 CPU 比較少,非常適合作爲 Agent 跑在服務器上。當然,FileBeat也並不是完全好過Logstash,畢竟Logstash對於日誌的格式化這些相對FileBeat好很多,FileBeat只是將日誌從日誌文件中讀取出來,當然如果你日誌本身是有一定格式的,FileBeat也可以格式化,但是相對於Logstash來說,還是差一點。
Elasticsearch
Elasticsearch是個開源分佈式搜索引擎,提供蒐集、分析、存儲數據三大功能。它的特點有:分佈式,零配置,自動發現,索引自動分片,索引副本機制,restful風格接口,多數據源,自動搜索負載等。
具有高可伸縮、高可靠、易管理等特點。可以用於全文檢索、結構化檢索和分析,並能將這三者結合起來。Elasticsearch 基於 Lucene 開發,現在使用最廣的開源搜索引擎之一,Wikipedia 、StackOverflow、Github 等都基於它來構建自己的搜索引擎。
FileBeat
輕量級數據收集引擎。基於原先 Logstash-fowarder 的源碼改造出來。換句話說:Filebeat就是新版的 Logstash-fowarder,也會是 ELK Stack 在 shipper 端的第一選擇。
Filebeat隸屬於Beats。目前Beats包含六種工具:
Packetbeat(蒐集網絡流量數據)
Metricbeat(蒐集系統、進程和文件系統級別的 CPU 和內存使用情況等數據)
Filebeat(蒐集文件數據)
Winlogbeat(蒐集 Windows 事件日誌數據)
Auditbeat( 輕量型審計日誌採集器)
Heartbeat(輕量級服務器健康採集器)
Kibana
可視化化平臺。它能夠搜索、展示存儲在 Elasticsearch 中索引數據。使用它可以很方便的用圖表、表格、地圖展示和分析數據。
可以爲 Logstash 、Beats和 ElasticSearch 提供的日誌分析友好的 Web 界面,可以幫助彙總、分析和搜索重要數據日誌。
EFK架構圖
常用日誌收集架構及使用場景
1 最簡單架構
在這種架構中,只有一個 Logstash、Elasticsearch 和 Kibana 實例。Logstash 通過輸入插件從多種數據源(比如日誌文件、標準輸入 Stdin 等)獲取數據,再經過濾插件加工數據,然後經 Elasticsearch 輸出插件輸出到 Elasticsearch,通過 Kibana 展示。詳見圖 1。
圖 1. 最簡單架構
這種架構非常簡單,使用場景也有限。初學者可以搭建這個架構,瞭解 ELK 如何工作。
2 Logstash 作爲日誌蒐集器
這種架構是對上面架構的擴展,把一個 Logstash 數據蒐集節點擴展到多個,分佈於多臺機器,將解析好的數據發送到 Elasticsearch server 進行存儲,最後在 Kibana 查詢、生成日誌報表等。詳見圖 2。
圖 2. Logstash 作爲日誌搜索器
這種結構因爲需要在各個服務器上部署 Logstash,而它比較消耗 CPU 和內存資源,所以比較適合計算資源豐富的服務器,否則容易造成服務器性能下降,甚至可能導致無法正常工作。
3 Beats 作爲日誌蒐集器
這種架構引入 Beats 作爲日誌蒐集器。目前 Beats 包括四種:
Packetbeat(蒐集網絡流量數據);
Topbeat(蒐集系統、進程和文件系統級別的 CPU 和內存使用情況等數據);
Filebeat(蒐集文件數據);
Winlogbeat(蒐集 Windows 事件日誌數據)。
Beats 將蒐集到的數據發送到 Logstash,經 Logstash 解析、過濾後,將其發送到 Elasticsearch 存儲,並由 Kibana 呈現給用戶。詳見圖 3。
圖 3. Beats 作爲日誌蒐集器
這種架構解決了 Logstash 在各服務器節點上佔用系統資源高的問題。相比 Logstash,Beats 所佔系統的 CPU 和內存幾乎可以忽略不計。另外,Beats 和 Logstash 之間支持 SSL/TLS 加密傳輸,客戶端和服務器雙向認證,保證了通信安全。
因此這種架構適合對數據安全性要求較高,同時各服務器性能比較敏感的場景。
4 引入消息隊列機制的架構
這種架構使用 Logstash 從各個數據源蒐集數據,然後經消息隊列輸出插件輸出到消息隊列中。目前 Logstash 支持 Kafka、Redis、RabbitMQ 等常見消息隊列。然後 Logstash 通過消息隊列輸入插件從隊列中獲取數據,分析過濾後經輸出插件發送到 Elasticsearch,最後通過 Kibana 展示。詳見圖 4。
圖 4. 引入消息隊列機制的架構
這種架構適合於日誌規模比較龐大的情況。但由於 Logstash 日誌解析節點和 Elasticsearch 的負荷比較重,可將他們配置爲集羣模式,以分擔負荷。引入消息隊列,均衡了網絡傳輸,從而降低了網絡閉塞,尤其是丟失數據的可能性,但依然存在 Logstash 佔用系統資源過多的問題。
5 基於 Filebeat 架構的配置部署詳解
前面提到 Filebeat 已經完全替代了 Logstash-Forwarder 成爲新一代的日誌採集器,同時鑑於它輕量、安全等特點,越來越多人開始使用它。這個章節將詳細講解如何部署基於 Filebeat 的 ELK 集中式日誌解決方案,具體架構見圖 5。
圖 5. 基於 Filebeat 的 ELK 集羣架構
因爲免費的 ELK 沒有任何安全機制,所以這裏使用了 Nginx 作反向代理,避免用戶直接訪問 Kibana 服務器。加上配置 Nginx 實現簡單的用戶認證,一定程度上提高安全性。另外,Nginx 本身具有負載均衡的作用,能夠提高系統訪問性能。
1. FileBeat
(1)概述:
Filebeat是一個日誌文件託運工具,在你的服務器上安裝客戶端後,filebeat會監控日誌目錄或者指定的日誌文件,追蹤讀取這些文件(追蹤文件的變化,不停的讀),並且轉發這些信息到elasticsearch或者logstarsh中存放。
以下是filebeat的工作流程:當你開啓filebeat程序的時候,它會啓動一個或多個探測器(prospectors)去檢測你指定的日誌目錄或文件,對於探測器找出的每一個日誌文件,filebeat啓動收割進程(harvester),每一個收割進程讀取一個日誌文件的新內容,併發送這些新的日誌數據到處理程序(spooler),處理程序會集合這些事件,最後filebeat會發送集合的數據到你指定的地點。
(個人理解,filebeat是一個輕量級的logstash,當你需要收集信息的機器配置或資源並不是特別多時,使用filebeat來收集日誌。日常使用中,filebeat十分穩定,筆者沒遇到過宕機。)
(2)工作原理:
Filebeat由兩個主要組成部分組成:prospector和 harvesters。這些組件一起工作來讀取文件並將事件數據發送到您指定的output。
1)harvester是什麼:
一個harvester負責讀取一個單個文件的內容。
harvester逐行讀取每個文件(一行一行地讀取每個文件),並把這些內容發送到輸出。每個文件啓動一個harvester,harvester負責打開和關閉這個文件,這就意味着在harvester運行時文件描述符保持打開狀態。
在harvester正在讀取文件內容的時候,文件被刪除或者重命名了,那麼Filebeat會續讀這個文件。這就有一個問題了,就是隻要負責這個文件的harvester沒用關閉,那麼磁盤空間就不會釋放。默認情況下,Filebeat保存文件打開直到close_inactive到達。
關閉harvester會產生以下結果:
1)如果在harvester仍在讀取文件時文件被刪除,則關閉文件句柄,釋放底層資源。
2)文件的採集只會在scan_frequency過後重新開始。
3)如果在harvester關閉的情況下移動或移除文件,則不會繼續處理文件。
要控制收割機何時關閉,請使用close_ *配置選項。
2)prospector是什麼:
prospector 負責管理harvester並找到所有要讀取的文件來源。如果輸入類型爲日誌,則查找器將查找路徑匹配的所有文件,併爲每個文件啓動一個harvester;每個prospector都在自己的Go協程中運行。
下面的例子配置Filebeat從所有匹配指定的glob模式的文件中讀取行:
filebeat.inputs:
- type: log
paths:
- /var/log/*.log
- /var/path2/*.log
Filebeat目前支持兩種prospector類型:log和stdin。
每個prospector類型可以定義多次。
log prospector檢查每個文件以查看harvester是否需要啓動,是否已經運行,或者該文件是否可以被忽略(請參閱ignore_older)。如果是在Filebeat運行過程中新創建的文件,只要在Harvster關閉後,文件大小發生了變化,新文件纔會被Prospector選擇到。
注:Filebeat prospector只能讀取本地文件, 沒有功能可以連接到遠程主機來讀取存儲的文件或日誌。
3)Filebeat如何保持文件狀態
Filebeat保存每個文件的狀態,並經常刷新狀態到磁盤上的註冊文件(registry)。狀態用於記住harvester讀取的最後一個偏移量,並確保所有日誌行被髮送(到輸出)。如果輸出,比如Elasticsearch 或者 Logstash等,無法訪問,那麼Filebeat會跟蹤已經發送的最後一行,並只要輸出再次變得可用時繼續讀取文件。當Filebeat運行時,會將每個文件的狀態新保存在內存中。當Filebeat重新啓動時,將使用註冊文件中的數據重新構建狀態,Filebeat將在最後一個已知位置繼續每個harvester。
對於每個輸入,Filebeat保存它找到的每個文件的狀態。因爲文件可以重命名或移動,所以文件名和路徑不足以標識文件。對於每個文件,Filebeat存儲惟一標識符,以檢測文件是否以前讀取過。
如果你的情況涉及每天創建大量的新文件,你可能會發現註冊表文件變得太大了。
(畫外音:Filebeat保存每個文件的狀態,並將狀態保存到registry_file中的磁盤。當重新啓動Filebeat時,文件狀態用於在以前的位置繼續讀取文件。如果每天生成大量新文件,註冊表文件可能會變得太大。爲了減小註冊表文件的大小,有兩個配置選項可用:clean_remove和clean_inactive。對於你不再訪問且被忽略的舊文件,建議您使用clean_inactive。如果想從磁盤上刪除舊文件,那麼使用clean_remove選項。)
4) Filebeat如何確保至少投遞一次(at-least-once)
Filebeat保證事件將被投遞到配置的輸出中至少一次,並且不會丟失數據。Filebeat能夠實現這種行爲,因爲它將每個事件的投遞狀態存儲在註冊表文件中。在定義的輸出被阻塞且沒有確認所有事件的情況下,Filebeat將繼續嘗試發送事件,直到輸出確認收到事件爲止。
如果Filebeat在發送事件的過程中關閉了,則在關閉之前它不會等待輸出確認所有事件。當Filebeat重新啓動時,發送到輸出(但在Filebeat關閉前未確認)的任何事件將再次發送。這確保每個事件至少被髮送一次,但是你最終可能會將重複的事件發送到輸出。你可以通過設置shutdown_timeout選項,將Filebeat配置爲在關閉之前等待特定的時間。
注意:
Filebeat的至少一次交付保證包括日誌輪換和刪除舊文件的限制。如果將日誌文件寫入磁盤並且寫入速度超過Filebeat可以處理的速度,或者在輸出不可用時刪除了文件,則可能會丟失數據。
在Linux上,Filebeat也可能因inode重用而跳過行。
(3)Filebeat簡單使用
第1步:安裝
第2步:配置
配置文件:filebeat.yml
爲了配置Filebeat:
1. 定義日誌文件路徑
對於最基本的Filebeat配置,你可以使用單個路徑。例如:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
在這個例子中,獲取在/var/log/*.log路徑下的所有文件作爲輸入,這就意味着Filebeat將獲取/var/log目錄下所有以.log結尾的文件。
爲了從預定義的子目錄級別下抓取所有文件,可以使用以下模式:/var/log/*/*.log。這將抓取/var/log的子文件夾下所有的以.log結尾的文件。它不會從/var/log文件夾本身抓取。目前,不可能遞歸地抓取這個目錄下的所有子目錄下的所有.log文件。
(畫外音:
假設配置的輸入路徑是/var/log/*/*.log,假設目錄結構是這樣的:
那麼只會抓取到2.log和3.log,而不會抓到1.log和4.log。因爲/var/log/aaa/ccc/1.log和/var/log/4.log不會被抓到。
)
2. 如果你發送輸出目錄到Elasticsearch(並且不用Logstash),那麼設置IP地址和端口以便能夠找到Elasticsearch:
output.elasticsearch:
hosts: ["192.168.1.42:9200"]
3. 如果你打算用Kibana儀表盤,可以這樣配置Kibana端點:
setup.kibana:
host: "localhost:5601"
4. 如果你的Elasticsearch和Kibana配置了安全策略,那麼在你啓動Filebeat之前需要在配置文件中指定訪問憑據。例如:
output.elasticsearch:
hosts: ["myEShost:9200"]
username: "filebeat_internal"
password: "{pwd}"
setup.kibana:
host: "mykibanahost:5601"
username: "my_kibana_user"
password: "{pwd}"
第3步:配置Filebeat以使用Logstash
如果你想使用Logstash對Filebeat收集的數據執行額外的處理,那麼你需要將Filebeat配置爲使用Logstash。
output.logstash:
hosts: ["127.0.0.1:5044"]
第4步:在Elasticsearch中加載索引模板
在Elasticsearch中,索引模板用於定義設置和映射,以確定如何分析字段。(畫外音:相當於定義索引文檔的數據結構,因爲要把採集的數據轉成標準格式輸出)
Filebeat包已經安裝了推薦的索引模板。如果你接受filebeat.yml中的默認配置,那麼Filebeat在成功連接到Elasticsearch以後會自動加載模板。如果模板已經存在,不會覆蓋,除非你配置了必須這樣做。
通過在Filebeat配置文件中配置模板加載選項,你可以禁用自動模板加載,或者自動加載你自己的目標。
配置模板加載:
默認情況下,如果Elasticsearch輸出是啓用的,那麼Filebeat會自動加載推薦的模板文件 ——— fields.yml。
- 加載不同的模板
-
setup.template.name: "your_template_name" setup.template.fields: "path/to/fields.yml"
覆蓋一個已存在的模板
-
setup.template.overwrite: true
禁用自動加載模板
-
setup.template.enabled: false
修改索引名稱
-
# 默認情況下,Filebeat寫事件到名爲filebeat-6.3.2-yyyy.MM.dd的索引,其中yyyy.MM.dd是事件被索引的日期。爲了用一個不同的名字,你可以在Elasticsearch輸出中設置index選項。例如: output.elasticsearch.index: "customname-%{[beat.version]}-%{+yyyy.MM.dd}" setup.template.name: "customname" setup.template.pattern: "customname-*" setup.dashboards.index: "customname-*"
手動加載模板:
./filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
第5步:設置Kibana dashboards
Filebeat附帶了Kibana儀表盤、可視化示例。在你用dashboards之前,你需要創建索引模式,filebeat-*,並且加載dashboards到Kibana中。爲此,你可以運行setup命令或者在filebeat.yml配置文件中配置dashboard加載。
./filebeat setup --dashboards
第6步:啓動Filebeat
./filebeat -e -c filebeat.yml -d "publish"
第7步:查看Kibana儀表板示例
http://127.0.0.1:5601
完整的配置:
#=========================== Filebeat inputs ==============
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
#============================== Dashboards ===============
setup.dashboards.enabled: false
#============================== Kibana ==================
setup.kibana:
host: "192.168.101.5:5601"
#-------------------------- Elasticsearch output ---------
output.elasticsearch:
hosts: ["localhost:9200"]
啓動Elasticsearch:
/usr/local/programs/elasticsearch/elasticsearch-6.3.2/bin/elasticsearch
啓動Kibana:
/usr/local/programs/kibana/kibana-6.3.2-linux-x86_64/bin/kibana
設置dashboard:
./filebeat setup --dashboards
啓動Filebeat:
./filebeat -e -c filebeat.yml -d "publish"
瀏覽器訪問 http://192.168.101.5:5601
查看索引:
請求:
curl -X GET "localhost:9200/_cat/indices?v"
響應:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open bank 59jD3B4FR8iifWWjrdMzUg 5 1 1000 0 475.1kb 475.1kb
green open .kibana DzGTSDo9SHSHcNH6rxYHHA 1 0 153 23 216.8kb 216.8kb
yellow open filebeat-6.3.2-2018.08.08 otgYPvsgR3Ot-2GDcw_Upg 3 1 255 0 63.7kb 63.7kb
yellow open customer DoM-O7QmRk-6f3Iuls7X6Q 5 1 1 0 4.5kb 4.5kb
更加詳細的配置和說明參考:日誌收集系統EFK之Filebeat 模塊與配置
filebeat.yml(中文配置詳解)
################### Filebeat Configuration Example #########################
############################# Filebeat ######################################
filebeat:
# List of prospectors to fetch data.
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
# Paths that should be crawled and fetched. Glob based paths.
# To fetch all ".log" files from a specific level of subdirectories
# /var/log/*/*.log can be used.
# For each file found under this path, a harvester is started.
# Make sure not file is defined twice as this can lead to unexpected behaviour.
# 指定要監控的日誌,可以指定具體得文件或者目錄
paths:
- /var/log/*.log (這是默認的)(自行可以修改)(比如我放在/home/hadoop/app.log裏)
#- c:\programdata\elasticsearch\logs\*
# Configure the file encoding for reading files with international characters
# following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
# Some sample encodings:
# plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
# hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
# 指定被監控的文件的編碼類型,使用plain和utf-8都是可以處理中文日誌的
#encoding: plain
# Type of the files. Based on this the way the file is read is decided.
# The different types cannot be mixed in one prospector
#
# Possible options are:
# * log: Reads every line of the log file (default)
# * stdin: Reads the standard in
# 指定文件的輸入類型log(默認)或者stdin
input_type: log
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list. The include_lines is called before
# 在輸入中排除符合正則表達式列表的那些行。
# exclude_lines. By default, no lines are dropped.
# exclude_lines: ["^DBG"]
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list. The include_lines is called before
# exclude_lines. By default, all the lines are exported.
# 包含輸入中符合正則表達式列表的那些行(默認包含所有行),include_lines執行完畢之後會執行exclude_lines
# include_lines: ["^ERR", "^WARN"]
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
# 忽略掉符合正則表達式列表的文件
# exclude_files: [".gz$"]
# Optional additional fields. These field can be freely picked
# to add additional information to the crawled log files for filtering
# 向輸出的每一條日誌添加額外的信息,比如“level:debug”,方便後續對日誌進行分組統計。
# 默認情況下,會在輸出信息的fields子目錄下以指定的新增fields建立子目錄,例如fields.level
# 這個得意思就是會在es中多添加一個字段,格式爲 "filelds":{"level":"debug"}
#fields:
# level: debug
# review: 1
# Set to true to store the additional fields as top level fields instead
# of under the "fields" sub-dictionary. In case of name conflicts with the
# fields added by Filebeat itself, the custom fields overwrite the default
# fields.
# 如果該選項設置爲true,則新增fields成爲頂級目錄,而不是將其放在fields目錄下。
# 自定義的field會覆蓋filebeat默認的field
# 如果設置爲true,則在es中新增的字段格式爲:"level":"debug"
#fields_under_root: false
# Ignore files which were modified more then the defined timespan in the past.
# In case all files on your system must be read you can set this value very large.
# Time strings like 2h (2 hours), 5m (5 minutes) can be used.
# 可以指定Filebeat忽略指定時間段以外修改的日誌內容,比如2h(兩個小時)或者5m(5分鐘)。
#ignore_older: 0
# Close older closes the file handler for which were not modified
# for longer then close_older
# Time strings like 2h (2 hours), 5m (5 minutes) can be used.
# 如果一個文件在某個時間段內沒有發生過更新,則關閉監控的文件handle。默認1h
#close_older: 1h
# Type to be published in the 'type' field. For Elasticsearch output,
# the type defines the document type these entries should be stored
# in. Default: log
# 設定Elasticsearch輸出時的document的type字段 可以用來給日誌進行分類。Default: log
#document_type: log
# Scan frequency in seconds.
# How often these files should be checked for changes. In case it is set
# to 0s, it is done as often as possible. Default: 10s
# Filebeat以多快的頻率去prospector指定的目錄下面檢測文件更新(比如是否有新增文件)
# 如果設置爲0s,則Filebeat會儘可能快地感知更新(佔用的CPU會變高)。默認是10s
#scan_frequency: 10s
# Defines the buffer size every harvester uses when fetching the file
# 每個harvester監控文件時,使用的buffer的大小
#harvester_buffer_size: 16384
# Maximum number of bytes a single log event can have
# All bytes after max_bytes are discarded and not sent. The default is 10MB.
# This is especially useful for multiline log messages which can get large.
# 日誌文件中增加一行算一個日誌事件,max_bytes限制在一次日誌事件中最多上傳的字節數,多出的字節會被丟棄
#max_bytes: 10485760
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# 適用於日誌中每一條日誌佔據多行的情況,比如各種語言的報錯信息調用棧
#multiline:
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
# 多行日誌開始的那一行匹配的pattern
#pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
# 是否需要對pattern條件轉置使用,不翻轉設爲true,反轉設置爲false。 【建議設置爲true】
#negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
# 匹配pattern後,與前面(before)還是後面(after)的內容合併爲一條日誌
#match: after
# The maximum number of lines that are combined to one event.
# In case there are more the max_lines the additional lines are discarded.
# Default is 500
# 合併的最多行數(包含匹配pattern的那一行)
#max_lines: 500
# After the defined timeout, an multiline event is sent even if no new pattern was found to start a new event
# Default is 5s.
# 到了timeout之後,即使沒有匹配一個新的pattern(發生一個新的事件),也把已經匹配的日誌事件發送出去
#timeout: 5s
# Setting tail_files to true means filebeat starts readding new files at the end
# instead of the beginning. If this is used in combination with log rotation
# this can mean that the first entries of a new file are skipped.
# 如果設置爲true,Filebeat從文件尾開始監控文件新增內容,把新增的每一行文件作爲一個事件依次發送,
# 而不是從文件開始處重新發送所有內容
#tail_files: false
# Backoff values define how agressively filebeat crawls new files for updates
# The default values can be used in most cases. Backoff defines how long it is waited
# to check a file again after EOF is reached. Default is 1s which means the file
# is checked every second if new lines were added. This leads to a near real time crawling.
# Every time a new line appears, backoff is reset to the initial value.
# Filebeat檢測到某個文件到了EOF(文件結尾)之後,每次等待多久再去檢測文件是否有更新,默認爲1s
#backoff: 1s
# Max backoff defines what the maximum backoff time is. After having backed off multiple times
# from checking the files, the waiting time will never exceed max_backoff idenependent of the
# backoff factor. Having it set to 10s means in the worst case a new line can be added to a log
# file after having backed off multiple times, it takes a maximum of 10s to read the new line
# Filebeat檢測到某個文件到了EOF之後,等待檢測文件更新的最大時間,默認是10秒
#max_backoff: 10s
# The backoff factor defines how fast the algorithm backs off. The bigger the backoff factor,
# the faster the max_backoff value is reached. If this value is set to 1, no backoff will happen.
# The backoff value will be multiplied each time with the backoff_factor until max_backoff is reached
# 定義到達max_backoff的速度,默認因子是2,到達max_backoff後,變成每次等待max_backoff那麼長的時間才backoff一次,
# 直到文件有更新纔會重置爲backoff
# 根據現在的默認配置是這樣的,每隔1s檢測一下文件變化,如果連續檢測兩次之後文件還沒有變化,下一次檢測間隔時間變爲10s
#backoff_factor: 2
# This option closes a file, as soon as the file name changes.
# This config option is recommended on windows only. Filebeat keeps the files it's reading open. This can cause
# issues when the file is removed, as the file will not be fully removed until also Filebeat closes
# the reading. Filebeat closes the file handler after ignore_older. During this time no new file with the
# same name can be created. Turning this feature on the other hand can lead to loss of data
# on rotate files. It can happen that after file rotation the beginning of the new
# file is skipped, as the reading starts at the end. We recommend to leave this option on false
# but lower the ignore_older value to release files faster.
# 這個選項關閉一個文件,當文件名稱的變化。#該配置選項建議只在windows
#force_close_files: false
# Additional prospector
#-
# Configuration to use stdin input
#input_type: stdin
# General filebeat configuration options
#
# Event count spool threshold - forces network flush if exceeded
# spooler的大小,spooler中的事件數量超過這個閾值的時候會清空發送出去(不論是否到達超時時間)
#spool_size: 2048
# Enable async publisher pipeline in filebeat (Experimental!)
# 是否採用異步發送模式(實驗功能)
#publish_async: false
# Defines how often the spooler is flushed. After idle_timeout the spooler is
# Flush even though spool_size is not reached.
# spooler的超時時間,如果到了超時時間,spooler也會清空發送出去(不論是否到達容量的閾值)
#idle_timeout: 5s
# Name of the registry file. Per default it is put in the current working
# directory. In case the working directory is changed after when running
# filebeat again, indexing starts from the beginning again.
# 記錄filebeat處理日誌文件的位置的文件,默認是在啓動的根目錄下
#registry_file: .filebeat
# Full Path to directory with additional prospector configuration files. Each file must end with .yml
# These config files must have the full filebeat config part inside, but only
# the prospector part is processed. All global options like spool_size are ignored.
# The config_dir MUST point to a different directory then where the main filebeat config file is in.
# 如果要在本配置文件中引入其他位置的配置文件,可以寫在這裏(需要寫完整路徑),但是隻處理prospector的部分
#config_dir:
###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features
############################# Output ##########################################
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:
### Elasticsearch as output
elasticsearch: (這是默認的,filebeat收集後放到es裏)(自行可以修改,比如我有時候想filebeat收集後,然後到redis,再到es,就可以註銷這行)
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: http://localhost:9200/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"] (這是默認的,filebeat收集後放到es裏)(自行可以修改,比如我有時候想filebeat收集後,然後到redis,再到es,就可以註銷這行)
# Optional protocol and basic auth credentials. #protocol: "https" #username: "admin" #password: "s3cr3t" # Number of workers per Elasticsearch host. #worker: 1 # Optional index name. The default is "filebeat" and generates # [filebeat-]YYYY.MM.DD keys. #index: "filebeat" # A template is used to set the mapping in Elasticsearch # By default template loading is disabled and no template is loaded. # These settings can be adjusted to load your own template or overwrite existing ones #template: # Template name. By default the template name is filebeat. #name: "filebeat" # Path to template file #path: "filebeat.template.json" # Overwrite existing template #overwrite: false # Optional HTTP Path #path: "/elasticsearch" # Proxy server url #proxy_url: http://proxy:3128 # The number of times a particular Elasticsearch index operation is attempted. If # the indexing operation doesn't succeed after this many retries, the events are # dropped. The default is 3. #max_retries: 3 # The maximum number of events to bulk in a single Elasticsearch bulk API index request. # The default is 50. #bulk_max_size: 50 # Configure http request timeout before failing an request to Elasticsearch. #timeout: 90 # The number of seconds to wait for new events between two bulk API index requests. # If `bulk_max_size` is reached before this interval expires, addition bulk index # requests are made. #flush_interval: 1 # Boolean that sets if the topology is kept in Elasticsearch. The default is # false. This option makes sense only for Packetbeat. #save_topology: false # The time to live in seconds for the topology information that is stored in # Elasticsearch. The default is 15 seconds. #topology_expire: 15 # tls configuration. By default is off. #tls: # List of root certificates for HTTPS server verifications #certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for TLS client authentication #certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #certificate_key: "/etc/pki/client/cert.key" # Controls whether the client verifies server certificates and host name. # If insecure is set to true, all server host names and certificates will be # accepted. In this mode TLS based connections are susceptible to # man-in-the-middle attacks. Use only for testing. #insecure: true # Configure cipher suites to be used for TLS connections #cipher_suites: [] # Configure curve types for ECDHE based cipher suites #curve_types: [] # Configure minimum TLS version allowed for connection to logstash #min_version: 1.0 # Configure maximum TLS version allowed for connection to logstash #max_version: 1.2 ### Logstash as output #logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Number of workers per Logstash host. #worker: 1 # The maximum number of events to bulk into a single batch window. The # default is 2048. #bulk_max_size: 2048 # Set gzip compression level. #compression_level: 3 # Optional load balance the events between the Logstash hosts #loadbalance: true # Optional index name. The default index name depends on the each beat. # For Packetbeat, the default is set to packetbeat, for Topbeat # top topbeat and for Filebeat to filebeat. #index: filebeat # Optional TLS. By default is off. #tls: # List of root certificates for HTTPS server verifications #certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for TLS client authentication #certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #certificate_key: "/etc/pki/client/cert.key" # Controls whether the client verifies server certificates and host name. # If insecure is set to true, all server host names and certificates will be # accepted. In this mode TLS based connections are susceptible to # man-in-the-middle attacks. Use only for testing. #insecure: true # Configure cipher suites to be used for TLS connections #cipher_suites: [] # Configure curve types for ECDHE based cipher suites #curve_types: [] ### File as output #file: # Path to the directory where to save the generated files. The option is mandatory. #path: "/tmp/filebeat" # Name of the generated files. The default is `filebeat` and it generates files: `filebeat`, `filebeat.1`, `filebeat.2`, etc. #filename: filebeat # Maximum size in kilobytes of each file. When this size is reached, the files are # rotated. The default value is 10 MB. #rotate_every_kb: 10000 # Maximum number of files under path. When this number of files is reached, the # oldest file is deleted and the rest are shifted from last to first. The default # is 7 files. #number_of_files: 7 ### Console output # console: # Pretty print json event #pretty: false ############################# Shipper ######################################### shipper: # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. # If this options is not defined, the hostname is used. #name: # The tags of the shipper are included in their own field with each # transaction published. Tags make it easy to group servers by different # logical properties. #tags: ["service-X", "web-tier"] # Uncomment the following if you want to ignore transactions created # by the server on which the shipper is installed. This option is useful # to remove duplicates if shippers are installed on multiple servers. #ignore_outgoing: true # How often (in seconds) shippers are publishing their IPs to the topology map. # The default is 10 seconds. #refresh_topology_freq: 10 # Expiration time (in seconds) of the IPs published by a shipper to the topology map. # All the IPs will be deleted afterwards. Note, that the value must be higher than # refresh_topology_freq. The default is 15 seconds. #topology_expire: 15 # Internal queue size for single events in processing pipeline #queue_size: 1000 # Configure local GeoIP database support. # If no paths are not configured geoip is disabled. #geoip: #paths: # - "/usr/share/GeoIP/GeoLiteCity.dat" # - "/usr/local/var/GeoIP/GeoLiteCity.dat" ############################# Logging ######################################### # There are three options for the log ouput: syslog, file, stderr. # Under Windos systems, the log files are per default sent to the file output, # under all other system per default to syslog. # 建議在開發時期開啓日誌並把日誌調整爲debug或者info級別,在生產環境下調整爲error級別 # 開啓日誌 必須設置to_files 屬性爲true logging: # Send all logging output to syslog. On Windows default is false, otherwise # default is true. # 配置beats日誌。日誌可以寫入到syslog也可以是輪滾日誌文件。默認是syslog # tail -f /var/log/messages #to_syslog: true # Write all logging output to files. Beats automatically rotate files if rotateeverybytes # limit is reached. # 日誌發送到輪滾文件 #to_files: false # To enable logging to files, to_files option has to be set to true # to_files設置爲true纔可以開啓輪滾日誌記錄 files: # The directory where the log files will written to. # 指定日誌路徑 #path: /var/log/mybeat # The name of the files where the logs are written to. # 指定日誌名稱 #name: mybeat # Configure log file size limit. If limit is reached, log file will be # automatically rotated # 默認文件達到10M就會滾動生成新文件 rotateeverybytes: 10485760 # = 10MB # Number of rotated log files to keep. Oldest files will be deleted first. # 保留日誌文件週期。 默認 7天。值範圍爲2 到 1024 #keepfiles: 7 # Enable debug output for selected components. To enable all selectors use ["*"] # Other available selectors are beat, publish, service # Multiple selectors can be chained. #selectors: [ ] # Sets log level. The default log level is error. # Available log levels are: critical, error, warning, info, debug # 日誌級別,默認是error #level: error
下一篇介紹Kibana: IT學習筆記--日誌收集系統EFK之Kibana
參考文章: