請警惕 ES 的三大坑

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文主要內容如下：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/73/730e3b8e996f2759031dc791832427f9.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"搜索引擎現在是用得越來越多了，比如 日誌系統用到的 ELK 中的 E 就是 搜索引擎 Elasticsearch（簡稱 ES）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那對於搜索這種技術來說，最看重的是搜索的結果的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"準確性","attrs":{}}],"attrs":{}},{"type":"text","text":"和搜索的","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"響應時間","attrs":{}}],"attrs":{}},{"type":"text","text":"。ES 的準確性可以通過 倒排索引算法來保證，那響應時間就需要磁盤或緩存來支持了，那麼磁盤和緩存會帶來哪些坑呢？ ( 其實不論是分佈式的，還是單機模式下的搜索引擎都會遇到這個問題。 )","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"一、ES 慢查詢之坑","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Elasticsearch 是現如今用的最廣泛的搜索引擎。它是一個分佈式的開源搜索和分析引擎，適用於所有類型的數據，包括文本、數字、地理空間、結構化和非結構化數據。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.1 工作原理：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"ES 的工作原理","attrs":{}},{"type":"text","text":"：往 ES 裏寫數據時，實際是寫到磁盤文件，查詢時，操作系統會將磁盤文件裏的數據自動緩存 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"filesystem cache","attrs":{}}],"attrs":{}},{"type":"text","text":" 裏面。如果給 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"filesystem cache","attrs":{}}],"attrs":{}},{"type":"text","text":" 更多的內存，儘量讓內存可以容納所有的 ","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"idx segment file","attrs":{}}],"attrs":{}},{"type":"text","text":" 索引數據文件，則搜索的時候走內存，性能較好。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/83/830c25310f94c5611d73b3c26b8e6416.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"codeinline","content":[{"type":"text","text":" 坑：","attrs":{}}],"attrs":{}},{"type":"text","text":" 首先如果訪問磁盤那一定很慢，而走緩存會快很多。但如果很多沒用的字段數據都丟到緩存裏面，則會浪費緩存的空間，所以很多數據還是存在磁盤裏面的，那麼大部分查詢走的數據庫，則會帶來性能問題。","attrs":{}}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.2 案例","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES 節點 3 臺機器，每臺機器 32 G 內存，總內存 96 G，給 ES JVM 堆內存是 16 G，那麼剩下來給 cache 的是 16 G，總共 ES 集羣的的 cache 佔用內存 48 G ，如果所有的數據佔據磁盤空間 600 G，那麼每臺機器的數據量是 200 G，而查詢時，有 150 G 左右的的數據是走磁盤查詢的，那麼走 cache 的概率是 48 G/ 600 G = 8%，也就是說大量查詢是走磁盤的。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"1.3 避坑指南：","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1.3.1 存儲關鍵信息","attrs":{}}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/dc/dccdf24e1ea379acfb82ca0852960e1c.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將數據中索引字段存到 cache，比如 一行數據有 name、gender、age、city、job 字段，而檢索這條數據只需要 name 和 gender 就可以查詢出數據，那麼 cache 就只需要存 id、name 和 gender 字段。別把所有字段都丟到 cache 裏面，純屬浪費空間，資源是有限的。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那剩下的字段怎麼檢索出來？可以把其他字段存到 mysql/hbase 裏面。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"hbase 特點：適用於海量數據的在線存儲。缺點是不能進行復雜的搜索。根據 name 和 gender 字段從 ES 中拿到 100 條數據 ( 包含 doc id ) ，然後根據 doc id 再去 hbase 中查詢每個 doc id 對應的完整數據，將結果組裝後返回給前端 ( 需要考慮分頁的情況 ) 。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1.3.2 數據預熱","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"將訪問量高的數據或者即將訪問量高的數據放到 filesystem cache 裏面。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"每隔一段時間就從數據庫訪問下數據，然後同步到 filesystem cache 裏面。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1.3.3 冷熱分離","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"不常訪問的數據和經常訪問的數據進行隔離。比如 3 臺機器存放冷數據的索引，另外 3 臺存放熱數據的索引。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"1.3.4 避免使用關聯查詢","attrs":{}}]},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES 中的關聯查詢是比較慢的，性能不佳，儘量避免使用。","attrs":{}}]}]}],"attrs":{}},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"二、ES 架構之坑","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"通常情況下，我們會使用 ES 的集羣模式，在集羣規模不大的情況下，性能還算可以，但如果集羣規模變得很大，則會遇到集羣瓶頸，也就是說集羣擴大，性能提升甚微，甚至不增反降。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES 的集羣也是採用中心化的分佈式架構，整個集羣只有一個是 Master 節點。而它的職責非常重要：負責整個集羣的元數據管理，元數據包含全局的配置信息、索引信息、節點信息，如果元數據發生改變，則需要 master 節點將變更信息發佈到集羣的其他節點。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"另外因爲 master 節點的任務處理是單線程的，所以每次處理任務時，需要等待全部節點接收到變更信息，並處理完變更的任務後，纔算完成了變更任務。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/60/60ba1c11263bbb8c347a744d12fbd98d.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"那麼這樣的架構會帶來什麼問題：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"響應時間問題","attrs":{}},{"type":"text","text":"。如果元數據發生了改變，但某節點","attrs":{}},{"type":"codeinline","content":[{"type":"text","text":"假死","attrs":{}}],"attrs":{}},{"type":"text","text":"，比如 JVM 的內存爆了，但是進程還活着，那麼響應 master 節點的時間會非常長，今而影響單個同步信息任務的完成時間。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"任務恢復問題","attrs":{}},{"type":"text","text":"。有大量恢復任務的時候，任務需要排隊，恢復時間變長。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong","attrs":{}}],"text":"任務回調問題","attrs":{}},{"type":"text","text":"。任務執行完成後，需要回調大量 listener 處理元數據變更。如果分片的數據很大，則處理時間會到 10 秒級，嚴重影響了集羣的恢復能力。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決方案：採用 ES 的 tribe node 特性實現 ES 多集羣。文中後面會介紹下 tribe node 的原理。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"三、業務場景的坑","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES 的被廣泛應用到多個場景，比如查詢日誌、查詢商品資料、數據聚合等。而這些場景的需求又有非常大的差異，這也是一個坑。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"**場景一：前端首頁搜索功能。**比如搜索商品，數據實時寫入的頻率不高，但是讀的頻率很高。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c4/c4d935332722fc08c4ae48fcd6502918.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"**場景二：日誌檢索的功能。**日誌系統中，我們一般都是 ELK 這種架構模式，對實時寫入要求很高，而查詢的次數其實不多，畢竟查詢日誌多工作還是開發和運維人員來做。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/33/33a73e49397392604a70c835875491f3.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"場景三：監控、分析的功能。ES 也會被運用到需要監控數據和分析數據的場景中，而這種場景又是對 ES 的內存要求比較高，因爲這些分析功能是在內存中完成的，若內存出現太大的壓力，則會造成系統的垃圾回收，可能出現短暫的服務抖動。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"解決方案：按業務場景劃分 ES 集羣，同樣採用 ES tribe node 功能。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"四、ES Tribe Node 方案","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES tribe node 功能原理圖如下所示：","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/7f/7f721f9ee337c3529686a6b9fd18fbea.png","alt":null,"title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"bulletedlist","content":[{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"有兩個 ES 集羣，每個集羣都有多個 ES 節點。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Logstash 負責日誌蒐集。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Kibana 負責客戶端查詢，將查詢命令傳送到 ES Tribe Node。","attrs":{}}]}]},{"type":"listitem","attrs":{"listStyle":null},"content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"ES Tribe Node 還承擔了集羣管理的職責。","attrs":{}}]}]}],"attrs":{}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"參考資料：https://www.infoq.cn/article/SbfS6uOcF_gW6FEpQlLKhttps://www.elastic.co/guide/en/elasticsearch/reference/2.0/modules-tribe.htmladvance-java","attrs":{}}]}]}