原理實踐，全面講解Logstash+Kibana+kafka

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"前面的時候，我因爲後臺粉絲的一些問題，整理了一篇文章，將ELK三個技術進行詳細的講解，從原理到實踐，全面覆蓋，但是因爲篇幅原因，我分成了兩篇進行整理，","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"上篇主講ES","attrs":{}},{"type":"text","text":"，文章鏈接在這裏：","attrs":{}},{"type":"link","attrs":{"href":"https://www.toutiao.com/i6898327590717899268/?group_id=6898327590717899268","title":null},"content":[{"type":"text","text":"熬夜不睡覺整理ELK技術文檔，從此擺脫靠百度的工作（附源碼）","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"今天是其他的個技術：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"Logstash+Kibana，中間穿插着講解Kafka應用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"話不多說，直接上正題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"一、 Logstash數據採集工具安裝和使用","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1． 簡介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash是一款輕量級的日誌蒐集處理框架，可以方便的把分散的、多樣化的日誌蒐集起來，並進行自定義的處理，然後傳輸到指定的位置，比如某個服務器或者文件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在官網，對於Logstash的介紹更是完整，我這裏就展示一下官網的介紹","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/b4/b44b1dff167105d300665724baecd338.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"輸入：採集各種樣式、大小和來源的數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4d/4d56a8efd5672d724bf8fd2da5cf28ed.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"過濾器：實時解析和轉換數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/fa/fab280a39a47d9a52c0a42bf533b8639.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"輸出：選擇你的存儲，導出你的數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/91/9151ba83e690940d43d5c89665e9cce9.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而在官網的介紹中，最讓我興奮的就是","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"可擴展性","attrs":{}},{"type":"text","text":"，Logstash 採用可插拔框架，擁有 200 多個插件。您可以將不同的輸入選擇、過濾器和輸出選擇混合搭配、精心安排，讓它們在管道中和諧地運行。","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"這也就意味着可以用自己的方式創建和配置管道","attrs":{}},{"type":"text","text":"，就跟樂高積木一樣，我自己感覺太爽了","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了，理論的東西過一遍就好","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ps：不過這也體現出官網在學習的過程中的重要性，雖然都是英文的，但是，現在可以翻譯的軟件的太多了，這不是問題","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2． 安裝","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有的技術，不自己實際操作一下是不可以的，安裝上自己動手實踐一下，毛爺爺都說：","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"實踐是檢驗真理的唯一標準","attrs":{}},{"type":"text","text":"，不得不誇獎一下Logstash的工程師，真的太人性化了，下載後直接解壓，就可以了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"而且提供了很多的安裝方式供你選擇，舒服","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/68/68be95b370676f4102311b77435973a3.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3． helloword使用","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開始我們今天的第一個實踐吧，就像我們剛開始學Java的時候，第一個命令就是helloworld，不知道各位還能不能手寫出來呢？來看一下logstash的第一個運行時怎麼處理的","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過命令行，進入到","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"logstash/bin","attrs":{}},{"type":"text","text":"目錄，執行下面的命令：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n  kafka {\n    type => \"accesslogs\"\n    codec => \"plain\"\n    auto_offset_reset => \"smallest\"\n    group_id => \"elas1\"\n    topic_id => \"accesslogs\"\n    zk_connect => \"172.16.0.11:2181,172.16.0.12:2181,172.16.0.13:2181\"\n  }\n \n  kafka {\n    type => \"gamelogs\"\n    auto_offset_reset => \"smallest\"\n    codec => \"plain\"\n    group_id => \"elas2\"\n    topic_id => \"gamelogs\"\n    zk_connect => \"172.16.0.11:2181,172.16.0.12:2181,172.16.0.13:2181\"\n  }\n}\n \nfilter {\n  if [type] == \"accesslogs\" {\n    json {\n      source => \"message\"\n  remove_field => [ \"message\" ]\n  target => \"access\"\n    }\n  }\n \n  if [type] == \"gamelogs\" {\n    mutate {\n      split => { \"message\" => \" \" }\n      add_field => {\n        \"event_type\" => \"%{message[3]}\"\n        \"current_map\" => \"%{message[4]}\"\n        \"current_X\" => \"%{message[5]}\"\n        \"current_y\" => \"%{message[6]}\"\n        \"user\" => \"%{message[7]}\"\n        \"item\" => \"%{message[8]}\"\n        \"item_id\" => \"%{message[9]}\"\n        \"current_time\" => \"%{message[12]}\"\n     }\n     remove_field => [ \"message\" ]\n   }\n  }\n}\n \noutput {\n \n  if [type] == \"accesslogs\" {\n    elasticsearch {\n      index => \"accesslogs\"\n  codec => \"json\"\n      hosts => [\"172.16.0.14:9200\", \"172.16.0.15:9200\", \"172.16.0.16:9200\"]\n    }\n  }\n \n  if [type] == \"gamelogs\" {\n    elasticsearch {\n      index => \"gamelogs\"\n      codec => plain {\n        charset => \"UTF-16BE\"\n      }\n      hosts => [\"172.16.0.14:9200\", \"172.16.0.15:9200\", \"172.16.0.16:9200\"]\n    }\n  }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　可以看到提示下面信息（這個命令稍後介紹），","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"輸入hello world!","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/48/48a50ad4df1ce707263118fc3747085b.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　可以看到logstash爲我們自動添加了幾個字段：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"時間戳：@ timestamp","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"版本：@ version","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"輸入的類型：type","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主機名：host。","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4.1． 簡單的工作原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　Logstash使用管道方式進行日誌的蒐集處理和輸出。有點類似*NIX系統的管道命令 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"xxx | ccc | ddd","attrs":{}},{"type":"text","text":"，xxx執行完了會執行ccc，然後執行ddd。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　在logstash中，包括了三個階段:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"　　輸入input --> 處理filter（不是必須的） --> 輸出output","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d9/d9a26af4fbcfe6d2db1a229b71c32fa9.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　每個階段都有很多的插件配合工作，比如file、elasticsearch、redis等等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　每個階段也可以指定多種方式，比如輸出既可以輸出到elasticsearch中，也可以指定到stdout在控制檯打印。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　由於這種插件式的組織方式，使得logstash變得易於擴展和定製。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4.2． 命令行中常用的命令","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"　-f：","attrs":{}},{"type":"text","text":"通過這個命令可以指定Logstash的配置文件，根據配置文件配置logstash","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/bd/bdf00f3b095af2fdee57583fad255b45.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"　-e：","attrs":{}},{"type":"text","text":"後面跟着字符串，該字符串可以被當做logstash的配置（如果是“” 則默認使用stdin作爲輸入，stdout作爲輸出）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/97/97f598d1eabc6a5e4679c035b6cd271f.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"　-l：","attrs":{}},{"type":"text","text":"日誌輸出的地址（默認就是stdout直接在控制檯中輸出）","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"-t：","attrs":{}},{"type":"text","text":"測試配置文件是否正確，然後退出。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/42/42499d694339c58c1c97d29cd0926306.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4.3． 配置文件說明","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　前面介紹過logstash基本上由三部分組成，input、output以及用戶需要才添加的filter，因此標準的配置文件格式如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {...}\nfilter {...}\noutput {...}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/79/79ee35083c836d760704d63c8b824423.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　在每個部分中，也可以指定多個訪問方式，例如我想要指定兩個日誌來源文件，則可以這樣寫：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n file { path =>\"/var/log/messages\" type =>\"syslog\"}\n file { path =>\"/var/log/apache/access.log\" type =>\"apache\"}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　類似的，如果在filter中添加了多種處理規則，則按照它的順序一一處理，但是有一些插件並不是線程安全的。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　比如在filter中指定了兩個一樣的的插件，這兩個任務並不能保證準確的按順序執行，因此官方也推薦避免在filter中重複使用插件。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"說完這些，簡單的創建一個配置文件的小例子看看：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\nfile {\n　　 #指定監聽的文件路徑，注意必須是絕對路徑\n        path => \"E:/software/logstash-1.5.4/logstash-1.5.4/data/test.log\"\n        start_position => beginning\n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌大致如下：注意最後有一個空行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"1 hello,this is first line in test.log!\n2 hello,my name is xingoo!\n3 goodbye.this is last line in test.log!\n4","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　執行命令得到如下信息：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ec/ec8e68dd37be9432eda62c579953d0b6.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5． 最常用的input插件——file。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　這個插件可以從指定的目錄或者文件讀取內容，輸入到管道處理，也算是logstash的核心插件了，大多數的使用場景都會用到這個插件，因此這裏詳細講述下各個參數的含義與使用。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5.1． 最小化的配置文件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Logstash中可以在 input{} 裏面添加file配置，默認的最小化配置如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n    file {\n        path => \"E:/software/logstash-1.5.4/logstash-1.5.4/data/*\"\n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然也可以監聽多個目標文件：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n    file {\n        path => [\"E:/software/logstash-1.5.4/logstash-1.5.4/data/*\",\"F:/test.txt\"]\n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5.2． 其他的配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，處理path這個必須的項外，file還提供了很多其他的屬性：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input {\n    file {\n        #監聽文件的路徑\n        path => [\"E:/software/logstash-1.5.4/logstash-1.5.4/data/*\",\"F:/test.txt\"]\n \n        #排除不想監聽的文件\n        exclude => \"1.log\"\n        \n        #添加自定義的字段\n        add_field => {\"test\"=>\"test\"}\n \n        #增加標籤\n        tags => \"tag1\"\n \n        #設置新事件的標誌\n        delimiter => \"\\n\"\n \n        #設置多長時間掃描目錄，發現新文件\n        discover_interval => 15\n \n        #設置多長時間檢測文件是否修改\n        stat_interval => 1\n \n         #監聽文件的起始位置，默認是end\n        start_position => beginning\n \n        #監聽文件讀取信息記錄的位置\n        sincedb_path => \"E:/software/logstash-1.5.4/logstash-1.5.4/test.txt\"\n \n        #設置多長時間會寫入讀取的位置信息\n        sincedb_write_interval => 15\n        \n    }\n}\nfilter {\n    \n}\noutput {\n    stdout {}\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"其中值得注意的是：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　1 path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　是必須的選項，每一個file配置，都至少有一個path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　2 exclude","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　是不想監聽的文件，logstash會自動忽略該文件的監聽。配置的規則與path類似，支持字符串或者數組，但是要求必須是絕對路徑。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　3 start_position","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　是監聽的位置，默認是end，即一個文件如果沒有記錄它的讀取信息，則從文件的末尾開始讀取，也就是說，僅僅讀取新添加的內容。對於一些更新的日誌類型的監聽，通常直接使用end就可以了；相反，beginning就會從一個文件的頭開始讀取。但是如果記錄過文件的讀取信息，這個配置也就失去作用了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　4 sincedb_path","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　這個選項配置了默認的讀取文件信息記錄在哪個文件中，默認是按照文件的inode等信息自動生成。其中記錄了inode、主設備號、次設備號以及讀取的位置。因此，如果一個文件僅僅是重命名，那麼它的inode以及其他信息就不會改變，因此也不會重新讀取文件的任何信息。類似的，如果複製了一個文件，就相當於創建了一個新的inode，如果監聽的是一個目錄，就會讀取該文件的所有信息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　5 其他的關於掃描和檢測的時間，按照默認的來就好了，如果頻繁創建新的文件，想要快速監聽，那麼可以考慮縮短檢測的時間。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"　　//6 add_field\n　　#這個技術感覺挺六的，但是其實就是增加一個字段，例如：\nfile {\n     add_field => {\"test\"=>\"test\"}\n        path => \"D:/tools/logstash/path/to/groksample.log\"\n        start_position => beginning\n    }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6． 　Kafka與Logstash的數據採集對接","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於Logstash跑通Kafka還是需要注意很多東西，最重要的就是理解Kafka的原理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.1． Logstash工作原理","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於Kafka採用解耦的設計思想，並非原始的發佈訂閱，生產者負責產生消息，直接推送給消費者。而是在中間加入持久化層——broker,生產者把數據存放在broker中，消費者從broker中取數據。這樣就帶來了幾個好處:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1 生產者的負載與消費者的負載解耦","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2 消費者按照自己的能力fetch數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"3 消費者可以自定義消費的數量","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，由於broker採用了主題topic-->分區的思想，使得某個分區內部的順序可以保證有序性，但是分區間的數據不保證有序性。這樣，消費者可以以分區爲單位，自定義讀取的位置——offset。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kafka採用zookeeper作爲管理，記錄了producer到broker的信息，以及consumer與broker中partition的對應關係。因此，生產者可以直接把數據傳遞給broker，broker通過zookeeper進行leader-->followers的選舉管理；消費者通過zookeeper保存讀取的位置offset以及讀取的topic的partition分區信息。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/22/22f513b585e5e9dd07408a3293132dc3.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於上面的架構設計，使得生產者與broker相連；消費者與zookeeper相連。有了這樣的對應關係，就容易部署logstash-->kafka-->logstash的方案了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"接下來，按照下面的步驟就可以實現logstash與kafka的對接了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ac/acbf36bd3e51ba43a6168c8492ac8edd.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.2． 啓動kafka","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"##啓動zookeeper：\n$zookeeper/bin/zkServer.sh start\n\n##啓動kafka：\n$kafka/bin/kafka-server-start.sh $kafka/config/server.properties &","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.3． 創建主題","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"#創建主題:\n$kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --create --topic hello --replication-factor 1 --partitions 1\n\n#查看主題:\n$kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.4． 測試環境","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"#執行生產者腳本:\n$kafka/bin/kafka-console-producer.sh --broker-list 10.0.67.101:9092 --topic hello\n\n#執行消費者腳本，查看是否寫入:\n$kafka/bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --from-beginning --topic hello","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.5． 向kafka中輸出數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input{\n       stdin{}\n      }\noutput{\n       kafka{\n       topic_id => \"hello\" \n       bootstrap_servers => \"192.168.0.4:9092,172.16.0.12:9092\" \n       # kafka的地址 \n       batch_size => 5\n  codec => plain {\nformat => \"%{message}\"\ncharset => \"UTF-8\"\n  }\n      }\nstdout{\n       codec => rubydebug\n      }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"6.6． 從kafka中讀取數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"logstash配置文件:","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"input{\n      kafka {\n              codec => \"plain\" \n              group_id => \"logstash1\" \n              auto_offset_reset => \"smallest\" \n              reset_beginning => true \n              topic_id => \"hello\" \n              zk_connect => \"192.168.0.5:2181\" \n              }\n       }\noutput{\n       stdout{\n               codec => rubydebug\n               }\n       }","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"7． Filter","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"7.1． 過濾插件grok組件","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"#日誌\n55.3.244.1 GET /index.html 15824 0.043\n \nbin/logstash -e '\ninput { stdin {} }\nfilter {\n  grok {\n    match => { \"message\" => \"%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}\" }\n  }\n}\noutput { stdout {codec => rubydebug} }'\n ","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"7.2． 分割插件split","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"filter {\n  mutate {\n    split => { \"message\" => \" \" }\n      add_field => {\n        \"event_type\" => \"%{message[3]}\"\n        \"current_map\" => \"%{message[4]}\"\n        \"current_X\" => \"%{message[5]}\"\n        \"current_y\" => \"%{message[6]}\"\n        \"user\" => \"%{message[7]}\"\n        \"item\" => \"%{message[8]}\"\n        \"item_id\" => \"%{message[9]}\"\n        \"current_time\" => \"%{message[12]}\"\n     }\n     remove_field => [ \"message\" ]\n  }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"四、 Kibana報表工具的安裝和使用","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"1． 簡介","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash 早期曾經自帶了一個特別簡單的 logstash-web 用來查看 ES 中的數據。其功能太過簡單，於是產生了Kibana。不過是用PHP編寫，後來爲了滿足更多的使用需求，懶人推動科技的進步嘛，並且Logstash使用ruby進行編寫，所以重新編寫Kibana，直到現在，Kibana因爲重構，導致3，4某些情況下不兼容，所以出現了一山容二虎的情況，具體怎麼選擇，可以根據業務場景進行實際分析","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/d3/d3ce5d416ee9c81ff23c753506a6b75f.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在Kibana衆多的優秀特性中，我個人最喜歡的是這一個特性，我起名叫包容性","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/4e/4ebba47ac0054217657077bc4a00f932.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"　　因爲在官網介紹中，Kibana可以非常方便地把來自Logstash、ES-Hadoop、Beats或第三方技術的數據整合到Elasticsearch，支持的第三方技術包括Apache Flume、Fluentd等。這也就表明我在日常的開發工作中，對於技術選型和操作的時候，我可以有更多的選擇，在開發時也能找到相應的開發實例，節省了大量的開發時間","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ps：有一次體現了官網的重要性，真的，有時候官網可以幫你解決大多數的問題，有時間可以去看一下官網啊，好了，話不多說，看正題","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"2． 安裝","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下載安裝包後解壓","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"編輯文件config/kibana.yml ，配置屬性：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"[root@H32 ~]# cd kibana/config/\n [root@H32 config]# vim kibana.yml\n //添加：\n server.host: \"192.168.80.32\"\nelasticsearch.url: \"http://172.16.0.14:9200\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"先啓動ES，然後再啓動","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"cd /usr/local/kibana530bin/kibana","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1、kibana必須是在root下運行，否則會報錯，啓動失敗","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2、下載解壓安裝包，一定要裝與ES相同的版本","attrs":{}}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"3． 導入數據","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們將使用莎士比亞全集作爲我們的示例數據。要更好的使用 Kibana，你需要爲自己的新索引應用一個映射集(mapping)。我們用下面這個映射集創建\"莎士比亞全集\"索引。實際數據的字段比這要多，但是我們只需要指定下面這些字段的映射就可以了。注意到我們設置了對 speaker 和 play_name 不分析。原因會在稍後講明。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在終端運行下面命令：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"curl -XPUT http://localhost:9200/shakespeare -d '\n{\n \"mappings\" : {\n  \"_default_\" : {\n   \"properties\" : {\n    \"speaker\" : {\"type\": \"string\", \"index\" : \"not_analyzed\" },\n    \"play_name\" : {\"type\": \"string\", \"index\" : \"not_analyzed\" },\n    \"line_id\" : { \"type\" : \"integer\" },\n    \"speech_number\" : { \"type\" : \"integer\" }\n   }\n  }\n }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們這就創建好了索引。現在需要做的時導入數據。莎士比亞全集的內容我們已經整理成了 elasticsearch 批量 導入所需要的格式，你可以通過","attrs":{}},{"type":"text","marks":[{"type":"underline","attrs":{}}],"text":"shakeseare.json","attrs":{}},{"type":"text","text":"下載。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用如下命令導入數據到你本地的 elasticsearch 進程中。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"4． 訪問 Kibana 界面","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"打開瀏覽器，訪問已經發布了 Kibana 的本地服務器。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/ed/ed0a735f3f4b848dd0d133707eb0bb57.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果你解壓路徑無誤(譯者注：使用 github 源碼的讀者記住發佈目錄應該是 kibana/src/ 裏面)，你已經就可以看到上面這個可愛的歡迎頁面。點擊 Sample Dashboard 鏈接","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8d/8d599f10225db946129f7554a0966944.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了，現在顯示的就是你的 sample dashboard！如果你是用新的 elasticsearch 進程開始本教程的，你會看到一個百分比佔比很重的餅圖。這裏顯示的是你的索引中，文檔類型的情況。如你所見，99% 都是 lines，只有少量的 acts 和scenes。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在下面，你會看到一長段 JSON 格式的莎士比亞詩文。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"5． 第一次搜索","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kibana 允許使用者採用 Lucene Query String 語法搜索 Elasticsearch 中的數據。請求可以在頁面頂部的請求輸入框中書寫。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/8e/8ece525726c07bd12766dc9d88c784e7.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在請求框中輸入如下內容。然後查看錶格中的前幾行內容。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"friends, romans, countrymen","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7b/7b77d41033317278fa4fccf667bb81c5.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"heading","attrs":{"align":null,"level":1},"content":[{"type":"text","text":"6． 配置另一個索引","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"目前 Kibana 指向的是 Elasticsearch 一個特殊的索引叫 _all。 _all 可以理解爲全部索引的大集合。目前你只有一個索引， shakespeare，但未來你會有更多其他方面的索引，你肯定不希望 Kibana 在你只想搜《麥克白》裏心愛的句子的時候還要搜索全部內容。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置索引，點擊右上角的配置按鈕：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33a7ac272839b9cdc00aafca7f44fe06.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在這裏，你可以設置你的索引爲 shakespeare ，這樣 Kibana 就只會搜索 shakespeare 索引的內容了。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/a6/a6b99b9e268563b33867a0c037726929.jpeg","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/85/85ade52ad4983ac4371a22ee20f92f50.png","alt":"原理實踐，全面講解Logstash+Kibana+kafka","title":null,"style":null,"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是因爲 ES1.4 增強了權限管理。你需要在 ES 配置文件 ","attrs":{}},{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"elasticsearch.yml","attrs":{}},{"type":"text","text":" 中添加下列配置並重啓服務後才能正常訪問：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"http.cors.enabled: true\nhttp.cors.allow-origin: \"*\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"記住 kibana3 頁面也要刷新緩存纔行。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"此外，如果你可以很明確自己 kibana 以外沒有其他 http 訪問，可以把 kibana 的網址寫在http.cors.allow-origin 參數的值中。比如：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"http.cors.allow-origin: \"/https?:\\/\\/kbndomain/\"","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"好了，到這裏就結束了，不知道有沒有收穫呀，有收穫的朋友給我點個贊吧~","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"希望可以對大家有幫助，喜歡的小夥伴可以關注公衆號：小遷不禿頭，每天不定時更新~","attrs":{}}]}]}