原理實踐,全面講解Logstash+Kibana+kafka

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面的時候,我因爲後臺粉絲的一些問題,整理了一篇文章,將ELK三個技術進行詳細的講解,從原理到實踐,全面覆蓋,但是因爲篇幅原因,我分成了兩篇進行整理,","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"上篇主講ES","attrs":{}},{"type":"text","text":",文章鏈接在這裏:","attrs":{}},{"type":"link","attrs":{"href":"https://www.toutiao.com/i6898327590717899268/?group_id=6898327590717899268","title":null},"content":[{"type":"text","text":"熬夜不睡覺整理ELK技術文檔,從此擺脫靠百度的工作(附源碼)","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天是其他的個技術:","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Logstash+Kibana,中間穿插着講解Kafka應用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"話不多說,直接上正題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一、 Logstash數據採集工具安裝和使用","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 簡介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash是一款輕量級的日誌蒐集處理框架,可以方便的把分散的、多樣化的日誌蒐集起來,並進行自定義的處理,然後傳輸到指定的位置,比如某個服務器或者文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在官網,對於Logstash的介紹更是完整,我這裏就展示一下官網的介紹","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b4/b44b1dff167105d300665724baecd338.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"輸入:採集各種樣式、大小和來源的數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4d/4d56a8efd5672d724bf8fd2da5cf28ed.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"過濾器:實時解析和轉換數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fa/fab280a39a47d9a52c0a42bf533b8639.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"輸出:選擇你的存儲,導出你的數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/91/9151ba83e690940d43d5c89665e9cce9.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在官網的介紹中,最讓我興奮的就是","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"可擴展性","attrs":{}},{"type":"text","text":",Logstash 採用可插拔框架,擁有 200 多個插件。您可以將不同的輸入選擇、過濾器和輸出選擇混合搭配、精心安排,讓它們在管道中和諧地運行。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"這也就意味着可以用自己的方式創建和配置管道","attrs":{}},{"type":"text","text":",就跟樂高積木一樣,我自己感覺太爽了","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了,理論的東西過一遍就好","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ps:不過這也體現出官網在學習的過程中的重要性,雖然都是英文的,但是,現在可以翻譯的軟件的太多了,這不是問題","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 安裝","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有的技術,不自己實際操作一下是不可以的,安裝上自己動手實踐一下,毛爺爺都說:","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"實踐是檢驗真理的唯一標準","attrs":{}},{"type":"text","text":",不得不誇獎一下Logstash的工程師,真的太人性化了,下載後直接解壓,就可以了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且提供了很多的安裝方式供你選擇,舒服","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/68/68be95b370676f4102311b77435973a3.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. helloword使用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開始我們今天的第一個實踐吧,就像我們剛開始學Java的時候,第一個命令就是helloworld,不知道各位還能不能手寫出來呢?來看一下logstash的第一個運行時怎麼處理的","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過命令行,進入到","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"logstash/bin","attrs":{}},{"type":"text","text":"目錄,執行下面的命令:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n  kafka {\n    type => \"accesslogs\"\n    codec => \"plain\"\n    auto_offset_reset => \"smallest\"\n    group_id => \"elas1\"\n    topic_id => \"accesslogs\"\n    zk_connect => \"172.16.0.11:2181,172.16.0.12:2181,172.16.0.13:2181\"\n  }\n \n  kafka {\n    type => \"gamelogs\"\n    auto_offset_reset => \"smallest\"\n    codec => \"plain\"\n    group_id => \"elas2\"\n    topic_id => \"gamelogs\"\n    zk_connect => \"172.16.0.11:2181,172.16.0.12:2181,172.16.0.13:2181\"\n  }\n}\n \nfilter {\n  if [type] == \"accesslogs\" {\n    json {\n      source => \"message\"\n  remove_field => [ \"message\" ]\n  target => \"access\"\n    }\n  }\n \n  if [type] == \"gamelogs\" {\n    mutate {\n      split => { \"message\" => \" \" }\n      add_field => {\n        \"event_type\" => \"%{message[3]}\"\n        \"current_map\" => \"%{message[4]}\"\n        \"current_X\" => \"%{message[5]}\"\n        \"current_y\" => \"%{message[6]}\"\n        \"user\" => \"%{message[7]}\"\n        \"item\" => \"%{message[8]}\"\n        \"item_id\" => \"%{message[9]}\"\n        \"current_time\" => \"%{message[12]}\"\n     }\n     remove_field => [ \"message\" ]\n   }\n  }\n}\n \noutput {\n \n  if [type] == \"accesslogs\" {\n    elasticsearch {\n      index => \"accesslogs\"\n  codec => \"json\"\n      hosts => [\"172.16.0.14:9200\", \"172.16.0.15:9200\", \"172.16.0.16:9200\"]\n    }\n  }\n \n  if [type] == \"gamelogs\" {\n    elasticsearch {\n      index => \"gamelogs\"\n      codec => plain {\n        charset => \"UTF-16BE\"\n      }\n      hosts => [\"172.16.0.14:9200\", \"172.16.0.15:9200\", \"172.16.0.16:9200\"]\n    }\n  }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  可以看到提示下面信息(這個命令稍後介紹),","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"輸入hello world!","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/48a50ad4df1ce707263118fc3747085b.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  可以看到logstash爲我們自動添加了幾個字段:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時間戳:@ timestamp","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"版本:@ version","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輸入的類型:type","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主機名:host。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4.1. 簡單的工作原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  Logstash使用管道方式進行日誌的蒐集處理和輸出。有點類似*NIX系統的管道命令 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"xxx | ccc | ddd","attrs":{}},{"type":"text","text":",xxx執行完了會執行ccc,然後執行ddd。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  在logstash中,包括了三個階段:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"  輸入input --> 處理filter(不是必須的) --> 輸出output","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d9/d9a26af4fbcfe6d2db1a229b71c32fa9.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  每個階段都有很多的插件配合工作,比如file、elasticsearch、redis等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  每個階段也可以指定多種方式,比如輸出既可以輸出到elasticsearch中,也可以指定到stdout在控制檯打印。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  由於這種插件式的組織方式,使得logstash變得易於擴展和定製。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4.2. 命令行中常用的命令","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" -f:","attrs":{}},{"type":"text","text":"通過這個命令可以指定Logstash的配置文件,根據配置文件配置logstash","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bd/bdf00f3b095af2fdee57583fad255b45.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" -e:","attrs":{}},{"type":"text","text":"後面跟着字符串,該字符串可以被當做logstash的配置(如果是“” 則默認使用stdin作爲輸入,stdout作爲輸出)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/97f598d1eabc6a5e4679c035b6cd271f.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":" -l:","attrs":{}},{"type":"text","text":"日誌輸出的地址(默認就是stdout直接在控制檯中輸出)","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"-t:","attrs":{}},{"type":"text","text":"測試配置文件是否正確,然後退出。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/42/42499d694339c58c1c97d29cd0926306.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4.3. 配置文件說明","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  前面介紹過logstash基本上由三部分組成,input、output以及用戶需要才添加的filter,因此標準的配置文件格式如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {...}\nfilter {...}\noutput {...}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/79/79ee35083c836d760704d63c8b824423.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  在每個部分中,也可以指定多個訪問方式,例如我想要指定兩個日誌來源文件,則可以這樣寫:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n file { path =>\"/var/log/messages\" type =>\"syslog\"}\n file { path =>\"/var/log/apache/access.log\" type =>\"apache\"}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  類似的,如果在filter中添加了多種處理規則,則按照它的順序一一處理,但是有一些插件並不是線程安全的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  比如在filter中指定了兩個一樣的的插件,這兩個任務並不能保證準確的按順序執行,因此官方也推薦避免在filter中重複使用插件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說完這些,簡單的創建一個配置文件的小例子看看:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\nfile {\n   #指定監聽的文件路徑,注意必須是絕對路徑\n        path => \"E:/software/logstash-1.5.4/logstash-1.5.4/data/test.log\"\n        start_position => beginning\n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌大致如下:注意最後有一個空行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1 hello,this is first line in test.log!\n2 hello,my name is xingoo!\n3 goodbye.this is last line in test.log!\n4","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 執行命令得到如下信息:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ec8e68dd37be9432eda62c579953d0b6.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5. 最常用的input插件——file。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 這個插件可以從指定的目錄或者文件讀取內容,輸入到管道處理,也算是logstash的核心插件了,大多數的使用場景都會用到這個插件,因此這裏詳細講述下各個參數的含義與使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5.1. 最小化的配置文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Logstash中可以在 input{} 裏面添加file配置,默認的最小化配置如下:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n    file {\n        path => \"E:/software/logstash-1.5.4/logstash-1.5.4/data/*\"\n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然也可以監聽多個目標文件:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n    file {\n        path => [\"E:/software/logstash-1.5.4/logstash-1.5.4/data/*\",\"F:/test.txt\"]\n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5.2. 其他的配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外,處理path這個必須的項外,file還提供了很多其他的屬性:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n    file {\n        #監聽文件的路徑\n        path => [\"E:/software/logstash-1.5.4/logstash-1.5.4/data/*\",\"F:/test.txt\"]\n \n        #排除不想監聽的文件\n        exclude => \"1.log\"\n        \n        #添加自定義的字段\n        add_field => {\"test\"=>\"test\"}\n \n        #增加標籤\n        tags => \"tag1\"\n \n        #設置新事件的標誌\n        delimiter => \"\\n\"\n \n        #設置多長時間掃描目錄,發現新文件\n        discover_interval => 15\n \n        #設置多長時間檢測文件是否修改\n        stat_interval => 1\n \n         #監聽文件的起始位置,默認是end\n        start_position => beginning\n \n        #監聽文件讀取信息記錄的位置\n        sincedb_path => \"E:/software/logstash-1.5.4/logstash-1.5.4/test.txt\"\n \n        #設置多長時間會寫入讀取的位置信息\n        sincedb_write_interval => 15\n        \n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"其中值得注意的是:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  1 path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  是必須的選項,每一個file配置,都至少有一個path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  2 exclude","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  是不想監聽的文件,logstash會自動忽略該文件的監聽。配置的規則與path類似,支持字符串或者數組,但是要求必須是絕對路徑。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  3 start_position","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  是監聽的位置,默認是end,即一個文件如果沒有記錄它的讀取信息,則從文件的末尾開始讀取,也就是說,僅僅讀取新添加的內容。對於一些更新的日誌類型的監聽,通常直接使用end就可以了;相反,beginning就會從一個文件的頭開始讀取。但是如果記錄過文件的讀取信息,這個配置也就失去作用了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  4 sincedb_path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  這個選項配置了默認的讀取文件信息記錄在哪個文件中,默認是按照文件的inode等信息自動生成。其中記錄了inode、主設備號、次設備號以及讀取的位置。因此,如果一個文件僅僅是重命名,那麼它的inode以及其他信息就不會改變,因此也不會重新讀取文件的任何信息。類似的,如果複製了一個文件,就相當於創建了一個新的inode,如果監聽的是一個目錄,就會讀取該文件的所有信息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  5 其他的關於掃描和檢測的時間,按照默認的來就好了,如果頻繁創建新的文件,想要快速監聽,那麼可以考慮縮短檢測的時間。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"  //6 add_field\n  #這個技術感覺挺六的,但是其實就是增加一個字段,例如:\nfile {\n     add_field => {\"test\"=>\"test\"}\n        path => \"D:/tools/logstash/path/to/groksample.log\"\n        start_position => beginning\n    }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6.  Kafka與Logstash的數據採集對接","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於Logstash跑通Kafka還是需要注意很多東西,最重要的就是理解Kafka的原理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.1. Logstash工作原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Kafka採用解耦的設計思想,並非原始的發佈訂閱,生產者負責產生消息,直接推送給消費者。而是在中間加入持久化層——broker,生產者把數據存放在broker中,消費者從broker中取數據。這樣就帶來了幾個好處:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1 生產者的負載與消費者的負載解耦","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2 消費者按照自己的能力fetch數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3 消費者可以自定義消費的數量","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外,由於broker採用了主題topic-->分區的思想,使得某個分區內部的順序可以保證有序性,但是分區間的數據不保證有序性。這樣,消費者可以以分區爲單位,自定義讀取的位置——offset。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka採用zookeeper作爲管理,記錄了producer到broker的信息,以及consumer與broker中partition的對應關係。因此,生產者可以直接把數據傳遞給broker,broker通過zookeeper進行leader-->followers的選舉管理;消費者通過zookeeper保存讀取的位置offset以及讀取的topic的partition分區信息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/22/22f513b585e5e9dd07408a3293132dc3.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於上面的架構設計,使得生產者與broker相連;消費者與zookeeper相連。有了這樣的對應關係,就容易部署logstash-->kafka-->logstash的方案了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來,按照下面的步驟就可以實現logstash與kafka的對接了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/acbf36bd3e51ba43a6168c8492ac8edd.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.2. 啓動kafka","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"##啓動zookeeper:\n$zookeeper/bin/zkServer.sh start\n\n##啓動kafka:\n$kafka/bin/kafka-server-start.sh $kafka/config/server.properties &","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.3. 創建主題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"#創建主題:\n$kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --create --topic hello --replication-factor 1 --partitions 1\n\n#查看主題:\n$kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.4. 測試環境","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"#執行生產者腳本:\n$kafka/bin/kafka-console-producer.sh --broker-list 10.0.67.101:9092 --topic hello\n\n#執行消費者腳本,查看是否寫入:\n$kafka/bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --from-beginning --topic hello","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.5. 向kafka中輸出數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input{\n       stdin{}\n      }\noutput{\n       kafka{\n       topic_id => \"hello\" \n       bootstrap_servers => \"192.168.0.4:9092,172.16.0.12:9092\" \n       # kafka的地址 \n       batch_size => 5\n  codec => plain {\nformat => \"%{message}\"\ncharset => \"UTF-8\"\n  }\n      }\nstdout{\n       codec => rubydebug\n      }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.6. 從kafka中讀取數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"logstash配置文件:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input{\n      kafka {\n              codec => \"plain\" \n              group_id => \"logstash1\" \n              auto_offset_reset => \"smallest\" \n              reset_beginning => true \n              topic_id => \"hello\" \n              zk_connect => \"192.168.0.5:2181\" \n              }\n       }\noutput{\n       stdout{\n               codec => rubydebug\n               }\n       }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7. Filter","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"7.1. 過濾插件grok組件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"#日誌\n55.3.244.1 GET /index.html 15824 0.043\n \nbin/logstash -e '\ninput { stdin {} }\nfilter {\n  grok {\n    match => { \"message\" => \"%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}\" }\n  }\n}\noutput { stdout {codec => rubydebug} }'\n ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"7.2. 分割插件split","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"filter {\n  mutate {\n    split => { \"message\" => \" \" }\n      add_field => {\n        \"event_type\" => \"%{message[3]}\"\n        \"current_map\" => \"%{message[4]}\"\n        \"current_X\" => \"%{message[5]}\"\n        \"current_y\" => \"%{message[6]}\"\n        \"user\" => \"%{message[7]}\"\n        \"item\" => \"%{message[8]}\"\n        \"item_id\" => \"%{message[9]}\"\n        \"current_time\" => \"%{message[12]}\"\n     }\n     remove_field => [ \"message\" ]\n  }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"四、 Kibana報表工具的安裝和使用","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1. 簡介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash 早期曾經自帶了一個特別簡單的 logstash-web 用來查看 ES 中的數據。其功能太過簡單,於是產生了Kibana。不過是用PHP編寫,後來爲了滿足更多的使用需求,懶人推動科技的進步嘛,並且Logstash使用ruby進行編寫,所以重新編寫Kibana,直到現在,Kibana因爲重構,導致3,4某些情況下不兼容,所以出現了一山容二虎的情況,具體怎麼選擇,可以根據業務場景進行實際分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d3ce5d416ee9c81ff23c753506a6b75f.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Kibana衆多的優秀特性中,我個人最喜歡的是這一個特性,我起名叫包容性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4ebba47ac0054217657077bc4a00f932.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  因爲在官網介紹中,Kibana可以非常方便地把來自Logstash、ES-Hadoop、Beats或第三方技術的數據整合到Elasticsearch,支持的第三方技術包括Apache Flume、Fluentd等。這也就表明我在日常的開發工作中,對於技術選型和操作的時候,我可以有更多的選擇,在開發時也能找到相應的開發實例,節省了大量的開發時間","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ps:有一次體現了官網的重要性,真的,有時候官網可以幫你解決大多數的問題,有時間可以去看一下官網啊,好了,話不多說,看正題","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2. 安裝","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下載安裝包後解壓","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"編輯文件config/kibana.yml ,配置屬性:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"[[email protected] ~]# cd kibana/config/\n [[email protected] config]# vim kibana.yml\n //添加:\n server.host: \"192.168.80.32\"\nelasticsearch.url: \"http://172.16.0.14:9200\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"先啓動ES,然後再啓動","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"cd /usr/local/kibana530bin/kibana","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、kibana必須是在root下運行,否則會報錯,啓動失敗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、下載解壓安裝包,一定要裝與ES相同的版本","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3. 導入數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將使用莎士比亞全集作爲我們的示例數據。要更好的使用 Kibana,你需要爲自己的新索引應用一個映射集(mapping)。我們用下面這個映射集創建\"莎士比亞全集\"索引。實際數據的字段比這要多,但是我們只需要指定下面這些字段的映射就可以了。注意到我們設置了對 speaker 和 play_name 不分析。原因會在稍後講明。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在終端運行下面命令:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"curl -XPUT http://localhost:9200/shakespeare -d '\n{\n \"mappings\" : {\n  \"_default_\" : {\n   \"properties\" : {\n    \"speaker\" : {\"type\": \"string\", \"index\" : \"not_analyzed\" },\n    \"play_name\" : {\"type\": \"string\", \"index\" : \"not_analyzed\" },\n    \"line_id\" : { \"type\" : \"integer\" },\n    \"speech_number\" : { \"type\" : \"integer\" }\n   }\n  }\n }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們這就創建好了索引。現在需要做的時導入數據。莎士比亞全集的內容我們已經整理成了 elasticsearch 批量 導入所需要的格式,你可以通過","attrs":{}},{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"shakeseare.json","attrs":{}},{"type":"text","text":"下載。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用如下命令導入數據到你本地的 elasticsearch 進程中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4. 訪問 Kibana 界面","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"打開瀏覽器,訪問已經發布了 Kibana 的本地服務器。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ed/ed0a735f3f4b848dd0d133707eb0bb57.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你解壓路徑無誤(譯者注:使用 github 源碼的讀者記住發佈目錄應該是 kibana/src/ 裏面),你已經就可以看到上面這個可愛的歡迎頁面。點擊 Sample Dashboard 鏈接","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8d599f10225db946129f7554a0966944.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了,現在顯示的就是你的 sample dashboard!如果你是用新的 elasticsearch 進程開始本教程的,你會看到一個百分比佔比很重的餅圖。這裏顯示的是你的索引中,文檔類型的情況。如你所見,99% 都是 lines,只有少量的 acts 和scenes。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在下面,你會看到一長段 JSON 格式的莎士比亞詩文。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5. 第一次搜索","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kibana 允許使用者採用 Lucene Query String 語法搜索 Elasticsearch 中的數據。請求可以在頁面頂部的請求輸入框中書寫。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8ece525726c07bd12766dc9d88c784e7.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在請求框中輸入如下內容。然後查看錶格中的前幾行內容。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"friends, romans, countrymen","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7b77d41033317278fa4fccf667bb81c5.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6. 配置另一個索引","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前 Kibana 指向的是 Elasticsearch 一個特殊的索引叫 _all。 _all 可以理解爲全部索引的大集合。目前你只有一個索引, shakespeare,但未來你會有更多其他方面的索引,你肯定不希望 Kibana 在你只想搜《麥克白》裏心愛的句子的時候還要搜索全部內容。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置索引,點擊右上角的配置按鈕:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33a7ac272839b9cdc00aafca7f44fe06.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這裏,你可以設置你的索引爲 shakespeare ,這樣 Kibana 就只會搜索 shakespeare 索引的內容了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a6/a6b99b9e268563b33867a0c037726929.jpeg","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/85ade52ad4983ac4371a22ee20f92f50.png","alt":"原理實踐,全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是因爲 ES1.4 增強了權限管理。你需要在 ES 配置文件 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"elasticsearch.yml","attrs":{}},{"type":"text","text":" 中添加下列配置並重啓服務後才能正常訪問:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"http.cors.enabled: true\nhttp.cors.allow-origin: \"*\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"記住 kibana3 頁面也要刷新緩存纔行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外,如果你可以很明確自己 kibana 以外沒有其他 http 訪問,可以把 kibana 的網址寫在http.cors.allow-origin 參數的值中。比如:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"http.cors.allow-origin: \"/https?:\\/\\/kbndomain/\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了,到這裏就結束了,不知道有沒有收穫呀,有收穫的朋友給我點個贊吧~","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"希望可以對大家有幫助,喜歡的小夥伴可以關注公衆號:小遷不禿頭,每天不定時更新~","attrs":{}}]}]}
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章