{"type":"doc","content":[{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1.背景"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在公司有兩套日誌分析平臺，一套是最初基於開源ELK棧（OSS版本）自行搭建的微服務系統的日誌分析平臺，該平臺主要用於自研系統中微服務的日誌集中管理，日誌的搜索與問題分析查找。由於微服務的監控告警，有另外一套監控告警平臺Prometheus進行處理，因此日誌監控告警並未納入系統建設考量中。另外一套是後續引入的基於國內某商用日誌產品（以下簡稱R產品）搭建的運維日誌分析平臺，該日誌平臺主要用於非自主開發的各業務應用系統，以及各基礎設施，操作系統等日誌的收集，日誌的集中存儲。一是滿足金融監管的需要，二是對第三方日誌數據進行分析，監控與告警。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"R產品與其他廣泛使用的商用日誌平臺如商用ELK、Splunk的收費模式不一樣，它基於日誌流量來計費。以前我們只接入了挑選的核心日誌數據，隨着公司需接入的日誌種類越來越多，現有的流量計劃已經遠不能滿足運維日誌增長的需求了，超出的流量需要購買額外的流量包，花費也還不小。而自研系統中日誌監控也在納入建設計劃中，並且ELK平臺不僅僅是一個日誌分析平臺，它更是一個完善的大數據分析平臺，爲統一日誌分析平臺，同時也爲今後的大數據分析打下基礎，我們有了自建平臺並替換現有的R產品的構想。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面我們就將這次日誌平臺的選型、部署架構設計、配置管理、相關插件應用、性能分析、服務監控的改進與展示等實踐經驗進行分享。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.1適用對象"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該日誌平臺重構全部基於開源軟件，經過近半年的生產上線實踐，平臺運行穩定，可擴展性好，可用性高，可以很好滿足公司對於金融業務不斷髮展的需要，也對中小型企業的日誌平臺，大數據分析平臺選型和部署有較好的參考作用。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.2術語與縮寫"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES：Elasticsearch"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ELK：Elasticsearch，LogStash，Kibana棧"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ODFE：Open Distro for Elasticsearch"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2.日誌平臺選型"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"原有的R產品相對開源的ELK的額外功能，對於我們的系統應用來說主要體現在以下幾個方面："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用戶認證及權限管理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"告警功能，R產品提供了郵件、短信等告警功能。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"提供了統一的部署、管理、監控功能，界面配置更方便。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於權限管理與告警功能，Elastic增值版（增值模塊部分開源，比如加密與SQL，帶開源增值模塊的稱爲Basic版，但License不同，我們需要的告警和認證權限管理爲收費版本，它按數據節點收費）和商用的Splunk也涵蓋，但按我們的需求兩者每年的授權費用也都不低。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"再看看開源方案，亞馬遜有一個利用開源的Elasticsearch和Kibana的代碼開發的一個開源日誌產品Open Disto for ElasticSearch，許可方式與開源的ELK一致，它以插件的方式提供了Elastic的增值功能，如告警與安全權限管理，多租戶管理等，涵蓋了我們需要的所有增值功能，重要的是它100%開源。該產品也比較成熟，亞馬遜自身就基於Open Distro提供日誌雲服務，與Elastic展開競爭，Elastic爲此還修訂了開源平臺許可協議。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下列表爲幾個Elasticsearch產品增值功能的橫向對比。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":"模塊 | 開源版ELK | 增值版ELK | Open Distro 告警 ✘ ✅ ✅ 用戶身份驗證與權限控制 ✘ ✅ ✅ 數據加密 ✘ ✅ ✅ LDAP，AD，SAML， Kerberos集成 ✘ ✅ ✅ SQL ✘ ✅ ✅ JDBC ✘ ✅ ✅ 性能分析 ✘ ✅ ✅"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Amazon也正在將Open Distro遷往opensearch.org，讓該產品成爲一個獨立的競品，產品將基於Elasticsearch 7.10.2，並繼續保持開源，Amazon同時也表達了團隊對該產品的長期持續的支持。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上增值功能數據表明，Open Distro for Elasticsearch可以涵蓋現有公司運維R產品平臺的所有功能，同時有著名企業Amazon長期的產品支持，而我們現有的兩個日誌平臺底層都是基於Elasticsearch，可以進行平滑遷移，遷移成本很低，因此它也成爲我們這次平臺遷移的不二選擇。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3.部署架構設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於數據類服務，內存和磁盤I\/O非常重要，而對於數據集羣來說，由於涉及節點之間的通信和數據傳輸，快速可靠的網絡顯然對分佈式系統的性能也是很重要的。對於硬件配置，Elasticsearch官方有一些指引。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.1硬件配置指引"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.1.1內存"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch官方對於內存的使用，32G是一條重要的分界線，一般建議一個節點內存不要超過64G，標準的建議是把 50％ 的可用內存作爲 Elasticsearch 的堆內存，因爲排序和聚合都很耗內存，剩餘的留給Lucene做緩存。Lucene段存放在單獨的文件中，這些文件不會改動，非常適合在內存中做緩存以提升存取速度。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於超過64G內存的大內存機器，可以考慮在同一臺機器中劃分出多個Elasticsearch節點，每個節點使用64G內存。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.1.2磁盤"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"建議SSD存儲或速度較快的硬盤，Elasticsearch中內部使用Replica來保證數據的高可用，因此沒有必要考慮使用硬盤鏡像如RAID 1，RAID 0可提供更高的磁盤容量，存取速率更高。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch可以通過設置節點類型熱，溫，冷的方式來實現不同存儲介質的混合存儲，以取得存儲速度和存儲成本之間的平衡。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.1.3網絡"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在的千兆和萬兆網絡對於Elasticsearch集羣來說已經足夠，應該避免集羣跨越多個數據中心，更大的延時會加重分佈式系統中的問題而且使得調試和排錯更困難。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.1.4CPU"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"大多數 Elasticsearch 部署往往對 CPU 要求不高，當然更多的核心數更好，因爲elasticsearch的thread pool和這個配置直接相關。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.2軟硬件配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下是我們這次重構使用的軟硬件方面的配置信息。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.2.1主機硬件配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" 主機名 配置 描述 elk1\/elk2\/elk3 40*2 Cores CPU, 3.1GHz, 512G memory, 44T hard disk，4 Nic 共計3臺，配置相同"}}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"沒有配置SSD，硬盤轉速10K，存儲介質沒有差異，我們暫不考慮Hot-swarm節點。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.2.2軟件配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" 系統\/軟件名稱 版本 描述 CentOS 7.6 操作系統 Docker Swarm CE 20.10.5 容器集羣及編排 Open Distro for Elasticsearch 1.13.1 AWS開源ELK棧，支持安全告警等ELK開源版本不支持的功能（注：該鏡像自己構建，email模塊有改動，並對配置文件針對我們的定製信息進行了修改） Pace Maker 0.9.169 Linux資源HA方案"}}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.3Elasticsearch節點設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"由於物理機器配置比較高，內存大，CPU核心數多，單臺物理機部署單個ES節點並使用全部內存的方式並不能充分發揮ES的性能。按以上內存配置指引，我們可以在一臺物理機上部署多個ES節點，在我們的節點類型設計中，每個ES節點不採用複合類型而採用專用節點類型，也就是各個節點只能有一種節點類型，不能既是主節點，又是數據節點。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.3.1ES節點類型與設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"按Open Distro官方支持的類型，我們選定部署以下節點類型："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主節點，主節點負責整個集羣的元數據管理，包括全局的配置信息、索引信息和節點信息，負責索引的創建或刪除，分配分片等操作。爲實現高可用集羣將設置專用的3個主節點。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據節點，用於數據的存儲，CRUD，搜索，聚合等，數據節點保存包含已建立索引的文檔的分片，集羣中各物理主機設計了4個數據節點。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"預處理節點，在索引數據之前通過Pipeline執行數據的預處理，類似Logstash，對數據進行解析轉換，集羣中設計了3個專用預處理節點，各物理主機1個。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"協調節點，協調節點將請求轉發給持有數據的數據節點。每個數據節點在本地執行請求，並將其結果返回給協調節點。在收集階段，協調節點協調節點將這些結果彙總(reduce)成一個單一的全局結果集(gather階段) 。集羣中設計了3個協調節點，各物理主機1個。 "}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下表是各節點類型對硬件資源的需求情況："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" 節點類型 存儲 內存 CPU 網絡 主節點 低 低 低 低 協調節點 低 高\/中 高\/中 高 數據處理節點 低 中 高 中 數據節點 極高 高 高 中"}}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.3.2ES節點設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單臺物理機部署多個ES節點，可以採用多種方式："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虛擬機"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"容器"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"單機啓動多個ES進程"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"虛擬機相對容器來說比較重，對資源消耗大，而採用多ES進程方式卻難以對資源進行隔離和控制，容器則結合了兩者的優點，ES容器節點可做到相互隔離，比虛擬機輕量，也更能充分發揮物理主機的性能，容器方式是一個不錯的選擇。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於以上原因，整個ES集羣及相關服務軟件將採用容器方式進行配置與管理，爲便於管理，容器集羣的編排方案採用公司現有標準Docker Swarm Mode，配置文件則採用docker-compose.yml格式。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲保證數據的高可用，我們設置ES副本數 Replica爲1，按設計同一臺物理機將部署多個數據節點，爲避免主分片和副本分片分配到同一臺物理機上，我們可以通過Shard Allocation Awareness的設計，將ES集羣中的節點按不同的物理主機分配不同的rack id，實現方式如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"節點命名規範化，比如elk1機器上的節點名後綴爲x1。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Lable將所有節點名後綴爲x1限定部署在elk1上。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有節點名後綴爲x1的設置爲同一個rack id，比如rack_01。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch應用配置文件中配置相應的awareness。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"整個ES相關節點的配置如下表所示（生產環境）："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"embedcomp","attrs":{"type":"table","data":{"content":" 節點名稱 節點類型 資源分配 rack ID LableS master-node-1 Master 16G memory, 1-4 Cores rack_01 master-node=true master-node-2 Master 16G memory, 1-4 Cores rack_02 master-node=true master-node-3 Master 16G memory, 1-4 Cores rack_03 master-node=true coordinating-node coordinating 64G memory, 4-16 Cores global coordinating-node=true ingest-node-1 Ingest 64G memory, 2-8 Cores rack_01 ingest-node=true ingest-node-2 Ingest 64G memory, 2-8 Cores rack_02 ingest-node=true ingest-node-3 Ingest 64G memory, 2-8 Cores rack_03 ingest-node=true data-node-11 Data 64G memory, 2-8 Cores rack_01 data-node=true data-node-12 Data 64G memory, 2-8 Cores rack_01 data-node=true data-node-13 Data 64G memory, 2-8 Cores rack_01 data-node=true data-node-14 Data 64G memory, 2-8 Cores rack_01 data-node=true data-node-21 Data 64G memory, 2-8 Cores rack_02 data-node=true data-node-22 Data 64G memory, 2-8 Cores rack_02 data-node=true data-node-23 Data 64G memory, 2-8 Cores rack_02 data-node=true data-node-24 Data 64G memory, 2-8 Cores rack_02 data-node=true data-node-31 Data 64G memory, 2-8 Cores rack_03 data-node=true data-node-32 Data 64G memory, 2-8 Cores rack_03 data-node=true data-node-33 Data 64G memory, 2-8 Cores rack_03 data-node=true data-node-34 Data 64G memory, 2-8 Cores rack_03 data-node=true logstash Logstash 64G memory, 4-16 Cores global logstash-node=true"}}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.4邏輯架構設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"日誌按數據源分爲兩大類，一類是落地的日誌文件，這些日誌文件需要通過ES客戶端比如FileBeat，WinLogBeat直接讀取併發送到ElasticSearch由ingest節點進行解析然後通過數據節點進行存儲；一類是日誌格式的網絡包，它們通過網絡協議（TCP\/UDP）發送，比如網絡設備通過TCP\/UDP或者微服務直接通過TCP發送的日誌信息，這些通過網絡傳輸日誌一般先通過LogStash進行日誌解析，然後再通過Logstash寫入ElasticSearch。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" 以下是ES集羣的邏輯架構圖："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/20\/53\/20a3f87bf83ab934dbb0188702d8ee53.png","alt":null,"title":"Figure 1: 邏輯架構","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"圖中，對於直接通過網絡比如網絡設備傳輸的日誌我們引入了Nginx對數據進行負載均衡。其實Docker Swarm集羣內部本身就提供負載均衡機制，但由於該負載均衡不能保持網絡設備發送端的源IP（經Swarm變成了Docker Ingress的內部IP），因此我們引入了外部的Nginx分別爲TCP和UDP包做負載均衡並配置了相應的源IP保留設置，Logstash容器運行也採用了host網絡模式以保留源IP。對於流量比較小的日誌數據包，可以不需要通過Nginx做負載均衡，直接發送到Logstash進行處理即可。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"架構中並沒有設計消息隊列，這是基於以下原因："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現有通過網絡協議傳輸的日誌整體流量不大，不到M\/s級別，而Logstash本身內部就具有緩存機制，可有效應對波峯的數據緩衝，該緩存大小可以自行配置。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES文件類傳輸客戶端如Filebeat，在將數據發送到Elasticsearch 時，Filebeat 使用背壓敏感協議，以應對更多的數據量。如果 Elasticsearch正在忙於處理數據時，則會告訴 Filebeat 減慢讀取速度。一旦擁堵得到解決，Filebeat 就會恢復到原來的步伐並繼續傳輸數據。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此在我們架構設計中沒有采用消息隊列，避免進行過度設計。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"3.5ES集羣高可用部署架構設計"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"金融系統向來對高可用性有着較高的要求，在ES集羣的高可用設計中，我們主要是通過以下幾個層面進行。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.5.1硬件設施"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"硬件上主要是保證主機和網絡設備的高可用："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"物理主機三臺，通過部署架構設計，允許在一臺主機出現問題時，另外兩臺主機還可以支撐整個ES集羣的運行。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"主機上4個網卡兩兩綁定，綁定的兩組網卡分別連接到一主一備的交換機，允許在一臺交換機或兩個網卡出現問題時，不影響ES集羣節點之間的通信。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.5.2基礎服務軟件"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"3.5.2.1系統軟件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於ES集羣的訪問都限定爲通過設定的域名進行訪問，爲防止主機故障時導致的不可用，在系統層面，我們配置了虛IP（VIP），該虛IP對應設定的域名。對於虛IP資源的創建和管理我們採用了Linux系統提供的資源管理軟件PaceMaker&CoroSync，該軟件通過heartbeat方式檢測主機故障，在主機故障時會自動進行IP地址的切換，從故障主機切換到其他運行正常的主機。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"紅帽的OpenShift套件底層就是使用PaceMaker&CoroSync進行的資源高可用性管理，它的穩定性毋庸置疑。"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"3.5.2.2Docker集羣"}]},{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES所有相關的服務都採用Docker方式部署在Docker集羣，在Docker集羣的節點編排上，我們採用的是公司現有實施標準Docker Swarm。三臺物理主機都配置爲Swarm Manager，三個Manager可保證單臺Docker節點故障時依然保證Docker集羣的高可用。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"添加更多的物理主機時可以將Docker節點以Worker節點方式加入Swarm集羣。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.5.3ES核心訪問軟件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES集羣本身本來就是按高可用模式設計，可以簡單通過加入更多的ES節點進行橫向擴展，ES在存儲數據時，會盡量把一個索引的分片（Shard）存儲在不同的節點上，同樣分片的副本也儘可能存在不同的節點上，這樣可以提高容錯率，在節點設計一節中我們描述瞭如何配置節點rack_id以支持在一個物理節點部署了多個ES節點的情況下仍就保持ES集羣的高可用（ES會根據rack_id避免將主分片和副本分片分配到同一臺物理機上）。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.5.4ES集羣高可用部署架構"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如果將ES集羣所用到的所有系統軟件和應用軟件劃分開來，ES集羣高可用軟件部署架構如下圖所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/4d\/2a\/4dcd5b86a4fea91a627f48e84407eb2a.png","alt":null,"title":"Figure 2: 高可用軟件部署拓撲架構","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通過Linux PaceMaker服務，在elk1和elk2主機創建一個虛IP資源，並在DNS服務器創建一個域名指向該VIP，客戶端對ES集羣的訪問都通過該域名進行。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在elk1和elk2上分別部署了一個Nginx和Kibana容器實例，這樣在其中一臺主機發生故障VIP漂移到另外一臺主機時，客戶端可以以最快的速度切換連接到該正常主機上啓動的Nginx\/Kibana服務。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash和Ingest節點需要通過Pipeline對數據進行預處理，計算資源消耗較高，因此在每個主機分別部署一個實例。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有對ES的請求（包括Logstash和Filebeat客戶端）都是通過Coordinating節點進行，因此也在各個主機部署了一個實例，這個採用Swarm的global選項來配置，Swarn的ingress內部機制保證訪問的負載均衡，當然也可以直接在客戶端通過指定ES節點進行直接訪問。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":" 3.6Elasticsearch服務治理"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.6.1ES集羣監控"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch本身可以蒐集系統，應用或服務的數據（包括使用Filebeat收集日誌數據，Metricbeat蒐集系統級如CPU，磁盤等統計數據，或經常使用的服務如Nginx，MySQL，Prometheus等的指標數據），通過分析這些日誌，指標\/度量數據，對相應的系統，服務和應用的性能，健康狀態進行監控和告警。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"但Elasticsearch自身出現問題不能由自己來監控，除非準備另外一套ES集羣，ES集羣自身的問題可以通過其他的工具來監控。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"公司現有的運營微服務平臺有一套基於Prometheus的業務監控系統，Prometheus有着豐富的監控插件，其中就提供了對Elasticsearch集羣的度量數據進行導出的Exporter；結合Grafana，還可以對ES集羣的度量數據進行圖形展示。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.6.2ES集羣監控管理工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下兩個開源工具都是使用比較廣的ES集羣監控管理工具，比較直觀，可以提供實時監控、全集羣管理、搜索和查詢服務，兩者對監控數據展示略有差異，比如Cerebro使用圖形化方式對於節點資源使用情況進行展示，同時也對索引的分片進行了展示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Cerebro"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/24\/a0\/248296224ba2ae09ac91a447ecfcffa0.png","alt":null,"title":"Figure 3: Cerebro界面","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ElasticHQ"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/44\/4b\/443ebeca52c4bf85091d5016ae3ff94b.png","alt":null,"title":"Figure 4: ElastiHQ界面","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因此，兩者互有補充，可以結合使用。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4.OpenDistro告警功能改進"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OpenDistro主要針對國外用戶，告警中的有的功能並不能滿足我們的需求，比如本地化時間，微信告警，郵件內容HTML格式支持等。這裏我們就體會到使用開源軟件帶來的好處了，我們可以從GitHub獲取源碼，然後根據我們的需求對源碼做相應地改進。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.1通用"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PeriodStart和PeriodEnd變量支持本地時區，缺省採用的是UTC時間格式，看上去不是很直觀，我們對相應的模塊TriggerExecutionContext.kt進行了修改，修改後顯示時區以UTC+800北京時間爲準"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.2郵件"}]},{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於OpenDistro郵件告警，因爲它現有的功能不能滿足我們郵件服務器配置的需求，以及對郵件內容格式化的需要，因此我們對原有代碼進行了改進，改進的功能包括："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"郵件SMTP連接認證，原告警安全模塊只有在郵件選擇TLS\/HTTPS加密協議的情況下，才從祕鑰庫（Key Store）中讀取用戶名和密碼，而我們的郵件服務採用的是非加密協議，但是發送郵件的時候又需要使用用戶名和密碼進行認證，因此我們需要修改源文件DestinationContextFactory.kt，註釋掉對應的emailAccount.method判斷語句。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"郵件內容支持HTML，原郵件發送模塊DestinationEmailClient.java中採用的是mailmsg.setText，這個我們修改爲先檢測郵件內容中是否有HTML標記，如果是調用mailmsg.setContent 以HTML格式進行發送，否則採用文本格式發送。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"4.3微信"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OpenDistro告警方式中支持的即時通是Slack和Chime，主要針對的是國外用戶；對於國內主流的微信和釘釘，我們可以通過OpenDistro提供的custom webhook進行，該方式需要先建羣，然後通過羣機器人進行告警，但這種方式顯得不夠靈活。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"企業微信可以通過API對特定的用戶或組發送信息，這樣在配置告警推送時更加方便。但該API需要兩次調用，一次獲取Token，另一次再跟進獲取的Token進行信息推送，而webhook只能進行一次調用。對此，有兩種實現方式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考Prometheus，Prometheus原生支持微信，只需在配置文件中填寫api_corp_id和api_secret，Prometheus內部根據配置信息，調用企業微信API完成信息推送。但是OpenDistro中的告警配置在Kibana界面中進行，如果採取這種方式，需要改動Kibana，工作量比較大。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將API的兩次調用放到另外一個服務中進行處理，比如我們的企業微信微服務，該微服務扮演一個企業微信代理或網關的角色，這樣我們就可以webhook方式調用該微服務了。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們採用了第二種方式並已成功對接。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以上改進的源碼已上載到Github，參考"},{"type":"link","attrs":{"href":"https:\/\/github.com\/marshell0\/alerting","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/marshell0\/alerting"}]}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5.數據備份與恢復"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於日誌數據，由於ES我們配置了副本，而且日誌數據量大，因此對於日誌數據我們並未進行數據的備份。但是對於集羣中通過界面配置的配置信息，由於經常有更新，我們需要定時進行備份，比如安全，告警等配置數據，主要關係到以下幾個索引："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":".kibana*，該索引包括Kibana租戶配置數據，包括各租戶配置的搜索，index pattern、Dashboard。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":".opendistro-alerting-config，該索引包含所有的告警配置相關信息，如Monitor、Destinations。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":".opendistro_security，包括安全配置信息，如角色，權限等。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於這些重要數據的備份，可以通過以下幾種方式進行數據備份："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於索引，可以使用快照（snapshot）的方式對數據進行備份。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於安全配置，可以使用OpenDistro內置的工具securityadmin.sh進行導出和導入，對於其他索引或全局信息配置如ingress pipeline，可以通過相應的查詢語句將數據以json格式存儲。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"上述操作的數據恢復可以通過相應的API調用進行。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"6.配置管理與安裝"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"基於Docker的安裝，OpenDistro官方有基於docker-compose的集羣例子，但該例中集羣的配置是基於單臺主機的簡單配置，因爲有的配置對於Docker集羣來說只能配置在Docker Engine中，以下是基於Docker Swarm的配置與安裝管理，ELK核心配置仍舊採用docker-compose.yml，只不過需要使用docker stack deploy運行。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下配置的大部分配置文件和腳本已經上載到："},{"type":"link","attrs":{"href":"https:\/\/github.com\/marshell0\/opendistro-deployment","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/marshell0\/opendistro-deployment"}]}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.1操作系統配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"作爲數據密集處理平臺，ES對操作系統配置有一些要求，在生產環境中，對於Linux系統Open Distro建議以下配置："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"vm.max_map_count 至少設置爲262144，Open Distro Docker鏡像中缺省是該值，這個可以通過sysctl -p檢測，sysctl.conf其他配置項可以參考其他大數據中心的配置進行參數調整。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文件打開數nofile 65536，禁止內存交換memlock"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Docker Swarm Mode運行的Docker配置修改這些參數比較方便的地方是在Docker Engine的配置文件daemon.json中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" \"default-ulimits\": {\n \"nofile\": {\n \"Name\": \"nofile\",\n \"Hard\": 65536,\n \"Soft\": 65536\n },\n \"memlock\" : {\n \"Name\": \"memlock\",\n \"Hard\": -1,\n \"Soft\": -1\n }\n }"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意對於Docker Swarm這些配置在docker-compose.yml中將不會起任何作用。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.2Elasticsearch配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES節點的配置我們將按節點類型進行描述。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.1節點通用配置描述"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所有的節點共有的配置選項如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES節點的Docker image爲自建的鏡像，在官方鏡像的基礎上添加了網絡工具，經過修改的配置文件，如elasticsearch.yml，config.yml，用於初始化配置的自定義腳本，擴展的plugin，修改後的認證配置文件，改進後的告警模塊編譯後的jar包文件等。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES節點的集羣名爲udic-odfe，表示所有的節點在一個集羣中。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES節點都接入外部創建的Swarm Overlay 網絡esnet。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲區分每個ES節點都定義了自己特有的端口。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外，所有的節點共享elasticsearch.yml，config.yml配置文件（打包在鏡像文件中），因此，個性化的參數配置都通過配置環境變量來覆蓋。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.2Master節點配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置如下圖所示，描述如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"node.master爲true，其他兩個屬性爲false表明這是一個專職的Master節點。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"discovery.seed_hosts，cluster.initial_master_nodes配置Master節點列表。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"bootstrap.memory_lock配合memlock禁止內存交換。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES_JAVA_OPTS配置ES的虛擬內存爲最大內存的一半。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"config項加載認證文件到指定目錄，所有節點使用相同的指定目錄，以便於配置文件中對認證的統一處理。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"deploy中設定了節點使用CPU和內存的保留值和最大值。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"constrain定義了該節點只能運行在elk1物理機 。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"master-1:\n image: nexus.my-domain.com:9443\/opendistro-for-elasticsearch:1.13.1\n hostname: master-1\n environment:\n - cluster.name=udic-odfe\n - node.name=master-1\n - node.attr.rack_id=rack_01\n - node.master=true\n - node.data=false\n - node.ingest=false \n - http.port=9202 \n - transport.port=9302\n - discovery.seed_hosts=master-1:9302,master-2:9303,master-3:9304\n - cluster.initial_master_nodes=master-1,master-2,master-3\n - bootstrap.memory_lock=true # disables swapping\n - \"ES_JAVA_OPTS=-Xms4g -Xmx4g -Des.enforce.bootstrap.checks=true\"\n volumes:\n - \/esdata\/master-node\/data:\/usr\/share\/elasticsearch\/data\n configs:\n - source: master-node.pem\n target: \/usr\/share\/elasticsearch\/config\/node.pem\n - source: master-node-key.pem\n target: \/usr\/share\/elasticsearch\/config\/node-key.pem\n networks:\n - esnet\n deploy:\n replicas: 1\n resources:\n limits:\n cpus: '4'\n memory: 8G\n reservations:\n cpus: '1'\n memory: 2G\n placement:\n constraints:\n - node.labels.master-node==true\n - node.hostname==elk1.my-domain.com"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Master節點2和3配置類似。 "}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.3Coodinating節點配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"協調節點作爲ES的入口，其配置如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"node.master等三個屬性都設爲false表明這是一個專職的協調節點。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲是入口節點，所以它的配置端口爲標準端口9200\/9300\/9600。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"涉及計算和內存使用比較大，因此CPU和內存的保留值和最大值都設置得相對比較大。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"部署模式設爲global，也就是在各個配置了node.labels.coordinating-node標籤的物理主機都會啓動一個實例，自動實現負載均衡。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"coordinating:\n image: nexus.my-domain.com:9443\/opendistro-for-elasticsearch:1.13.1\n hostname: coordinating\n environment:\n - cluster.name=udic-odfe\n - node.name=coordinating\n - node.master=false\n - node.data=false\n - node.ingest=false \n - http.port=9200 # custom port for HTTP\n - transport.port=9300\n - discovery.seed_hosts=master-1:9302,master-2:9303,master-3:9304\n - cluster.initial_master_nodes=master-1,master-2,master-3\n - bootstrap.memory_lock=true \n - \"ES_JAVA_OPTS=-Xms31g -Xmx31g -Des.enforce.bootstrap.checks=true -Des.transport.cname_in_publish_address=true\"\n volumes:\n - \/esdata\/coordinating-node\/data:\/usr\/share\/elasticsearch\/data\n configs:\n - source: coordinating-node.pem\n target: \/usr\/share\/elasticsearch\/config\/node.pem\n - source: coordinating-node-key.pem\n target: \/usr\/share\/elasticsearch\/config\/node-key.pem\n ports:\n - 9200:9200\n - 9300:9300\n - 9600:9600 # required for Performance Analyzer\n networks:\n - esnet\n deploy:\n mode: global\n resources:\n limits:\n cpus: '16'\n memory: 64G\n reservations:\n cpus: '4'\n memory: 4G\n placement:\n constraints:\n - node.labels.coordinating-node==true"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.4數據節點配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據節點配置項這裏不完整貼出，重點配置項描述如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"node.data屬性設爲true，其餘爲false，表明這是一個專職的數據節點。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據存儲映射關係爲：data-11節點 -> \/esdata\/data-node-x1，data-12節點 -> \/esdata\/data-node-x2，以此類推，保證同一臺物理機上ES節點數據相互獨立。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"注意node.attr.rack_id=rack_01，數據節點data-1x（x=[1,2,3,4]）的rack_id都相同，並且他們的constrain都限定了node.hostname==elk1.my-domain.com，也就是綁定到elk1主機上，這樣Elasticsearch知道這些節點處於同一個rack，不會將主分片和副本分片分配到同一個物理主機，保證高可用。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"data-11:\n hostname: data-11\n environment:\n - cluster.name=udic-odfe\n - node.name=data-11\n - node.attr.rack_id=rack_01\n - node.master=false\n - node.data=true\n - node.ingest=false \n ……\n - \"ES_JAVA_OPTS=-Xms31g -Xmx31g -Des.enforce.bootstrap.checks=true -Des.transport.cname_in_publish_address=true\" \n volumes:\n - \/esdata\/data-node-x1\/data:\/usr\/share\/elasticsearch\/data\n deploy:\n replicas: 1\n resources:\n limits:\n cpus: '8'\n memory: 64G\n reservations:\n cpus: '2'\n memory: 4G\n placement:\n constraints:\n - node.labels.data-node==true\n - node.hostname==elk1.my-domain.com"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.5Ingest節點配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Ingest節點配置與數據節點類似，只不過node屬性有所不同，不復贅述。"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.6Kibana配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kibana節點使用官方鏡像，配置如下所示："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因爲Kibana服務與ES節點同處一個Docker Overlay網絡，因此Kibana可以使用內部服務名訪問ES 協調節點，coodinating服務名對應一個虛IP，可以實現負載均衡。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kibana連接ES節點需要配置用戶名，密碼和CA證書。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kibana配置了認證證書，連接到Kibana需通過https。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" kibana:\n image: nexus.my-domain.com:8445\/amazon\/opendistro-for-elasticsearch-kibana:1.13.1\n hostname: kibana\n ports:\n - 5601:5601\n environment:\n ELASTICSEARCH_URL: https:\/\/coordinating:9200\n ELASTICSEARCH_HOSTS: https:\/\/coordinating:9200\n ELASTICSEARCH_USERNAME: kibanaserver\n ELASTICSEARCH_PASSWORD: xxxxxx\n SERVER_SSL_ENABLED: \"true\"\n ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: \/usr\/share\/kibana\/config\/root-ca.pem\n SERVER_SSL_CERTIFICATE: \/usr\/share\/kibana\/config\/node.pem\n SERVER_SSL_KEY: \/usr\/share\/kibana\/config\/node-key.pem\n networks:\n - esnet\n deploy:\n mode: global"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.2.7Logstash配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"因需要保留源IP地址，因此Logstash Docker運行在host網絡，不採用Docker service方式部署，而採用本地docker直接運行方式，配置文件如下，需要在三臺主機上分別運行，Logstash會定期從pipeline目錄掃描更新，因此配置了pipeline的本地映射，這樣可以方便從本地映射目錄更新pipeline腳本，不過需要注意在三臺機器中同步。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash使用Elastic官方的oss版本，版本號需要與Open Distro內部的ES版本保持一致（7.12.0）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"docker run -d \\\n --restart=unless-stopped \\\n --name logstash \\\n --mount type=bind,source=\/esdata\/logstash\/data,target=\/usr\/share\/logstash\/data \\\n --mount type=bind,source=\/home\/elasticsearch\/ssl-key\/root-ca.pem, target=\/usr\/share\/logstash\/config\/root-ca.pem,readonly \\\n --mount type=bind,source=\/home\/elasticsearch\/ssl-key\/logstash.pem, target=\/usr\/share\/logstash\/config\/logstash.pem,readonly \\\n --mount type=bind,source=\/home\/elasticsearch\/ssl-key\/logstash-key.pem, target=\/usr\/share\/logstash\/config\/logstash-key.pem,readonly \\\n --mount type=bind,source=\/home\/elasticsearch\/logstash\/pipeline\/, target=\/usr\/share\/logstash\/pipeline\/ \\\n --mount type=bind,source=\/home\/elasticsearch\/logstash\/config\/pipelines.yml, target=\/usr\/share\/logstash\/config\/pipelines.yml \\\n --mount type=bind,source=\/home\/elasticsearch\/logstash\/config\/logstash.yml, target=\/usr\/share\/logstash\/config\/logstash.yml \\\n --network host \\\n -e 'LS_HEAP_SIZE=8G' \\\n nexus.my-domain.com:9443\/logstash\/logstash-oss:7.12.1"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"6.2.7.1 pipelines.yml"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該文檔配置所有要加載的Pipeline配置文件："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"- pipeline.id: ufos-pipeline\n path.config: \"\/usr\/share\/logstash\/pipeline\/ufos.conf\"\n queue.max_bytes: 4gb\n pipeline.workers: 6\n- pipeline.id: hsfw-pipeline\n path.config: \"\/usr\/share\/logstash\/pipeline\/hsfw.conf\"\n queue.max_bytes: 4gb\n pipeline.workers: 6"}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"6.2.7.2 ufos.conf"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該文件存放在本地~\/logstash\/pipeline目錄，由logstash docker映射到docker的\/usr\/share\/logstash\/pipeline\/目錄，logstash會定時掃描該目錄，並加載更新，因此可以在不重啓logstash docker的情況下修改Pipeline配置文件。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置中proxy_protocol是配合nginx對TCP傳輸源地址保持的配置。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES連接配置了用戶名，密碼和CA證書。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ilm_enable必須設爲false，它是xpack選型，我們這裏使用的是OpenDistro安全配置。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"input {\n tcp {\n port => 5100\n type => ufoslog\n codec => json\n proxy_protocol => true\n }\n}\n……\noutput {\n elasticsearch {\n hosts => [\"https:\/\/elk.my-domain.com:9200\"]\n index => \"ufos-%{+YYYY.MM}\"\n ssl => true\n ssl_certificate_verification => true\n cacert => '\/usr\/share\/logstash\/config\/root-ca.pem'\n user => eslog\n password => xxxxxxx\n ilm_enabled => false\n }"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.3Elasticsearch Snapshot配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要對ES進行快照，需要進行以下配置："}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在ES的配置文件elasticsearch.yml中登記快照倉庫配置，例如：path.repo: [\"\/mnt\/snapshots\"]。這個配置項我們已經放進了自建的Docker images中。快照倉庫其實就是一個存儲路徑，注意該存儲路徑必須是以共享性質的路徑，比如NFS，HDFS，Amazon S3等，不能配置爲本地路徑，因此在docker-compose.yml中，我們配置了NFS共享目錄用於snapshot綁定。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建快照的倉庫。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"volumes:\n snapshots:\n driver: local\n driver_opts:\n type: nfs\n o: \"addr=eagle.my-domain.com,nolock,soft,rw\"\n device: \":\/var\/nfsshare\/elasticsearch\/snapshots\""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"創建倉庫後，就可以使用相關的REST API進行ES的快照和從快照中恢復。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"PUT \/_snapshot\/config_repository\n{\n \"type\": \"fs\",\n \"settings\": {\n\"location\": \"\/mnt\/snapshot\",\n \"compress\": true\n}\n}"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.4Elasticsearch擴展配置"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.4.1中文分詞插件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch模糊搜索快是因爲底層的開源庫Lucene使用了倒排索引，倒排索引需要對文檔進行分詞，英文語句分詞非常簡單，語句本身就有空格分詞，但是中文語句因爲漢字都是連在一起的，所以Elasticsearch內置的分詞器對於中文語句分詞只能分成一個個的漢字，顯然這樣我們不能查詢出我們想要的結果，因此我們需要安裝專門的中文分詞器來對語句進行解析。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch中最常用的分詞器是IK分詞器，其地址在 "},{"type":"link","attrs":{"href":"https:\/\/github.com\/medcl\/elasticsearch-analysis-ik","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/github.com\/medcl\/elasticsearch-analysis-ik"}]},{"type":"text","text":"，可以直接下載軟件包，並解壓到Elasticsearch的plugins目錄中即可。我們在Elasticsearch鏡像的腳本Dockerfile中已經將該軟件包直接打入了Elasticsearch的plugins目錄。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在索引模板中，就可以配置索引使用ik進行分詞了，配置analyzer，支持兩種模式："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ik_smart：最粗粒度的拆分，一般這種模式就夠用了。"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ik_max_word：會將文本做最細粒度的拆分，窮盡各種組合。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.4.2性能分析插件（Performance Analyzer）"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OpenDistro官方提供一個插件生成性能分析數據並提供相應REST API進行查詢，該插件缺省已經安裝到官方提供的容器鏡像中，我們只需要打開相應開關和設置相關配置，就可以獲取這些性能數據了。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置和啓用性能分析功能的步驟如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"性能分析器缺省使用\/dev\/shm做數據緩存，在數據處理量大的集羣中，可能會生成高達1G的度量數據，而容器缺省的\/dev\/shm容量是64M，容器修改該參數比較方便的地方是在Docker Engine的配置daemon.json中設置："}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\"default-shm-size\": \"1G\""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然也可以選擇進行動態綁定，但是相對配置起來比較麻煩。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"修改plugins\/opendistro_performance_analyzer\/pa_config\/目錄下的配置文件performance-analyzer.properties"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲配置方便，我們把該配置文件打包到基於官方的自定義容器鏡像中。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"# WebService bind host; default to all interfaces\nwebservice-bind-host = 0.0.0.0\n#Setup the correct path for certificates\ncertificate-file-path = \/usr\/share\/elasticsearch\/config\/node.pem\nprivate-key-file-path = \/usr\/share\/elasticsearch\/config\/node-key.pem"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"開啓Performance Analyzer插件。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"curl -XPOST https:\/\/elk.my-domain.com:9200\/_opendistro\/_performanceanalyzer\/rca\/cluster\/config -u admin: passwd --insecure -H 'Content-Type: application\/json' -d '{\"enabled\": true}'"}]},{"type":"paragraph","attrs":{"indent":0,"number":2,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"4.開啓Root Cause Analyzer (RCA) 框架。"}]}]}]},{"type":"heading","attrs":{"align":null,"level":5},"content":[{"type":"text","text":"6.4.2.1客戶端工具"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"PerfTop是展示性能分析數據的缺省工具，可以根據官方提供的地址進行下載，使用方式如下： "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":".\/perf-top-linux --dashboard NodeAnalysis --endpoint https:\/\/elk.my-domain.com:9600"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"curl -XPOST https:\/\/elk.my-domain.com:9200\/_opendistro\/_performanceanalyzer\/cluster\/config -u admin:passwd --insecure -H 'Content-Type: application\/json' -d '{\"enabled\": true}'"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Dashboard可以參考官方的文檔進行創建，簡單的方式是使用官方提供的四種標準Dashboard："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClusterOverview"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClusterNetworkMemoryAnalysis"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ClusterThreadAnalysis"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"NodeAnalysis"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"截屏如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/f6\/b9\/f6ceba56045fyyd80edf57bb1b6242b9.png","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"數據圖形展示並不是很好，通過配置stats度量數據拉取到Prometheus，然後通過Grafana展示，不管是數據還是圖形細節都相對更好。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.5自定義鏡像及相關配置文件"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"爲了簡化啓動配置文件docker-compose.yml，我們將一些公共的配置文件打包到了基礎鏡像中（該文件還可以通過yml的alias進行重複配置的簡化），各腳本和配置文件列表如下："}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.5.1Dockerfile"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在定製鏡像Dockerfile中，包含的主要指令有："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"鏡像本地化配置"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用定製的配置文件，keystore覆蓋缺省的配置文件"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"拷貝公共的認證文件，如admin和root-ca"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用定製的腳本在啓動的時候對容器進行初始化"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"打包擴展的插件"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"使用修改後的告警的插件覆蓋原有的插件"}]}]}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.5.2elasticsearch.yml"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該配置文件爲Elasticsearch節點的配置信息，配置信息可以通過環境變量進行覆蓋，比如docker-compose.yml中的cluster.name，node.master等。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面描述一下里邊幾段重要的配置信息："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Shard Allocation Awareness，上面講的利用rack_id進行高可用"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"安全認證配置文件，因此各個節點的安全認證配置文件的路徑需要保持一致"}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"集羣中節點的DN配置，必須對集羣中所有節點的DN（對應認證文件中的DN）進行配置，不然節點之間訪問會報安全錯誤；定義管理員DN，也就是擁有admin_dn的具有管理員權限"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"opendistro_security.nodes_dn:\n - \"CN=*.xxxxx.com,OU=ITDEPT,O=XXXXXXX,L=SHENZHEN,ST=GD,C=CN\"\n - \"CN=master-*,OU=ITDEPT,O=XXXXXX,L=SHENZHEN,ST=GD,C=CN\"\nopendistro_security.authcz.admin_dn:\n - \"CN=admin,OU=ITDEPT,O=XXXXXX,L=SHENZHEN,ST=GD,C=CN\""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"ENV TZ=Asia\/Shanghai\nENV LANG=zh_CN.UTF-8 \\\n LC_ALL=zh_CN.UTF-8 \\\n LC_CTYPE=zh_CN.UTF-8 \\\n LC_MESSAGES=zh_CN.UTF-8\nCOPY --chown=elasticsearch:elasticsearch .\/config\/config.yml .\/config\/internal_users.yml .\/config\/roles_mapping.yml \/usr\/share\/elasticsearch\/plugins\/opendistro_security\/securityconfig\/\nCOPY --chown=elasticsearch:elasticsearch .\/config\/elasticsearch.yml .\/config\/elasticsearch.keystore .\/config\/log4j2.properties .\/ssl-key\/root-ca.pem .\/ssl-key\/admin.pem .\/ssl-key\/admin-key.pem \/usr\/share\/elasticsearch\/config\/\nCOPY --chown=elasticsearch:elasticsearch .\/config\/default.policy \/opt\/jdk\/lib\/security\/default.policy\nCOPY --chown=elasticsearch .\/config\/coordinating.sh .\/config\/docker-entrypoint.sh \/usr\/local\/bin\/\nCOPY --chown=elasticsearch .\/config\/performance-analyzer.properties \/usr\/share\/elasticsearch\/plugins\/opendistro-performance-analyzer\/pa_config\/"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置審計auditlog索引格式，缺省是按日生成，這樣創建的索引過多。"}]}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置快照的存儲路徑，必須配置該選項才能進行索引的快照"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"opendistro_security.audit.config.index: \"'security-auditlog-'YYYY\"\npath.repo: [\"\/mnt\/snapshots\"]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":" ldap:\n description: \"Authenticate via LDAP or Active Directory\"\n http_enabled: true\n transport_enabled: true\n order: 1\n http_authenticator:\n type: basic\n challenge: true\n authentication_backend:\n type: ldap\n config:\n enable_ssl: false\n enable_start_tls: false\n enable_ssl_client_auth: false\n verify_hostnames: false\n hosts:\n - xxx.0.0.1:xxxx\n bind_dn: 'CN=xxx,CN=xxxx,DC=xxxx,DC=com'\n password: 'xxxxxxxxxxx'\n userbase: 'DC=xxxxx,DC=com'\n usersearch: '(sAMAccountName={0})'\n username_attribute: uid"}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"6.5.3 config.yml"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"該配置文件主要是配置認證與授權信息。認證中我們使用了兩種認證方式，一種是基礎的內部用戶名口令方式（初始從internal_user.yml裝載，該文件裏配置的hash配置項可以通過plugins\/opendistro_security\/tools\/hash.sh產生）；另外認證方式是通過公司辦公網AD域進行認證，因此配置了ldap選項。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"其他詳細配置信息請參考項目Repository中的配置文件以及腳本。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"6.6 第三方軟件配置"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考項目Repository腳本，略。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"7.告警配置與管理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對於國內用戶只有兩種選擇，郵件或Webhook，由於大多數通信軟件都支持Webhook，因此可以通過Webhook支持微信，釘釘等國內流行通信軟件平臺告警。"}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"7.1郵件告警"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"OpenDistro郵件告警配置中，與郵件服務器通信的方式支持三種，TLS，SSL，明文，如果選擇前面兩種加密通信方式，則必須在節點上添加郵件用戶名與密碼到keystore。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":".\/bin\/elasticsearch-keystore add opendistro.alerting.destination.email. |