在Linux系統發行版（以CentOS7爲例）上部署ElasticSearch集羣並啓動Kibana和Logstash

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前分別寫過關於使用ES來做日誌統一管理，以及在CentOS上部署Redis哨兵集羣的博客，今兒的話題還是要圍繞他們，這次是CentOS和ELK的交集。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前我們的ES日誌集羣是在Windows系統下運行的，這兩天上頭給批了兩臺CentOS的主機，就把日誌的集羣遷移到了CentOS服務器上。遷移的過程總體還是比較順利，但過程中還是遇到了一些問題，或者說我個人感覺應該要記錄一下的東西，在這裏簡單總結一下。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/11/05/%E5%9C%A8Linux%E7%B3%BB%E7%BB%9F%E5%8F%91%E8%A1%8C%E7%89%88%EF%BC%88%E4%BB%A5CentOS7%E4%B8%BA%E4%BE%8B%EF%BC%89%E4%B8%8A%E9%83%A8%E7%BD%B2ElasticSearch%E9%9B%86%E7%BE%A4%E5%B9%B6%E5%90%AF%E5%8A%A8Kibana%E5%92%8CLogstash/#%E4%B8%80%E3%80%81Elasticsearch%E7%89%88%E6%9C%AC%E9%80%89%E6%8B%A9","title":"一、Elasticsearch版本選擇","type":null}},{"type":"text","text":"一、Elasticsearch版本選擇","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方的建議是，選擇最新的版本~，這其實也是我在選擇開源組件的基本原則，大多數情況下，越新的產品，性能，功能完整性也會越好，尤其是ES這種級別的軟件。但因爲我是從Windows服務器上遷移過來，所以我選擇了和之前Windows一致的版本，7.14.1（當前最新的是7.15.1）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏需要提一句的是，如果是es的老用戶，都應該清楚，es在發佈產品的時候，都是捆綁發佈的，不如es發佈了7.14.1，對應的kibana，logstash，beats等等，也會有相同的版本跟着發佈，所以如果es選了7.14.1，用其餘產品的時候，也要選同樣的版本！","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/11/05/%E5%9C%A8Linux%E7%B3%BB%E7%BB%9F%E5%8F%91%E8%A1%8C%E7%89%88%EF%BC%88%E4%BB%A5CentOS7%E4%B8%BA%E4%BE%8B%EF%BC%89%E4%B8%8A%E9%83%A8%E7%BD%B2ElasticSearch%E9%9B%86%E7%BE%A4%E5%B9%B6%E5%90%AF%E5%8A%A8Kibana%E5%92%8CLogstash/#%E4%BA%8C%E3%80%81%E4%B8%8B%E8%BD%BD%E5%AE%89%E8%A3%85","title":"二、下載安裝","type":null}},{"type":"text","text":"二、下載安裝","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方的下載提供了很多種類型，比如爲Windows提供了msi，爲linux提供了deb，rpm等，還有壓縮包的類型。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏我個人建議是統一用壓縮包的類型，因爲可能會涉及到遷移或者集羣部署，用壓縮包的形式，解壓後可以很方便的修改配置文件，對文件目錄授權，以及設定服務啓動方式等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方的下載指導頁如下：","attrs":{}},{"type":"link","attrs":{"href":"https://www.elastic.co/cn/downloads/elasticsearch","title":null,"type":null},"content":[{"type":"text","text":"https://www.elastic.co/cn/downloads/elasticsearch","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"對應的Kibana，logstash就是把鏈接最後的內容改成對應的名字即可","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"壓縮包下載後，對應安裝即可，我這裏是建立了一個上層目錄，然後把es，kibana和logstash都分別安裝到了裏面，方便管理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e9/e9ccce890b6db611d8688d1806b7394c.png","alt":"image-20211108091227345","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三、配置集羣","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/11/05/%E5%9C%A8Linux%E7%B3%BB%E7%BB%9F%E5%8F%91%E8%A1%8C%E7%89%88%EF%BC%88%E4%BB%A5CentOS7%E4%B8%BA%E4%BE%8B%EF%BC%89%E4%B8%8A%E9%83%A8%E7%BD%B2ElasticSearch%E9%9B%86%E7%BE%A4%E5%B9%B6%E5%90%AF%E5%8A%A8Kibana%E5%92%8CLogstash/#3-1-ElastercSearch%E9%85%8D%E7%BD%AE","title":"3.1 ElastercSearch配置","type":null}},{"type":"text","text":"3.1 ElastercSearch配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這裏先給出官方的配置指導頁：","attrs":{}},{"type":"link","attrs":{"href":"https://www.elastic.co/guide/en/elasticsearch/reference/7.14/settings.html%EF%BC%8C%E6%88%91%E8%BF%99%E4%B8%AA%E6%98%AF7.14%EF%BC%8C%E5%A6%82%E6%9E%9C%E4%BD%A0%E7%94%A8%E5%88%AB%E7%9A%84%E7%89%88%E6%9C%AC%EF%BC%8C%E5%B0%B1%E6%8A%8A%E9%93%BE%E6%8E%A5%E9%87%8C%E7%9A%84%E7%89%88%E6%9C%AC%E5%8F%B7%E6%94%B9%E4%B8%80%E4%B8%8B%E5%B0%B1%E5%A5%BD%E3%80%82","title":null,"type":null},"content":[{"type":"text","text":"https://www.elastic.co/guide/en/elasticsearch/reference/7.14/settings.html，","attrs":{}}]},{"type":"text","text":"我這個是7.14，如果你用別的版本，就把鏈接裏的版本號改一下就好。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方指導裏，分別給出了源於elasticsearch.yml，jvm.options，log4j2.propertites三給關鍵配置文件的配置說明。我這裏只用到了第一個，官方指導了也提到了，對於後兩項，分別是調整Java虛擬機和日誌的選項，一般情況下不需要修改，按默認配置即可。如果是二般情況，就再根據說明按需調整。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"# 節點1\n# ---------------------------------- Cluster -----------------------------------\n# 集羣名字，每個集羣內的節點應該保持一致\ncluster.name: magicloud-cluster\n# ------------------------------------ Node ------------------------------------\n# 節點名稱\nnode.name: es-node1\n# 是否可以被選舉爲主節點\nnode.master: true\n# 是否爲數據節點（存儲空間足夠的情況下，我個人建議把每個節點都設定成可存儲數據的節點，這樣會最大限度保證數據分片不會丟失）\nnode.data: true\n# ----------------------------------- Paths ------------------------------------\n# 數據和日誌存放地址，根據官方建議，在生產環境下，這裏把數據放到了$ES_HOME之外的位置。\n# https://www.elastic.co/guide/en/elasticsearch/reference/7.14/important-settings.html\npath.data: /usr/local/elasticsearch/data\npath.logs: /usr/local/elasticsearch/logs\n# ---------------------------------- Network -----------------------------------\n# 網絡配置，默認是127.0.0.1的迴環地址，修改成要暴露的ip或者直接0.0.0.0\nnetwork.host: 0.0.0.0\n# 端口，默認9200，個人不建議修改，不爲別的，主要是其他開發或者運維人員一看到9200就知道es在跑着，辨識度較高\nhttp.port: 9200\n# --------------------------------- Discovery ----------------------------------\n# 集羣發現，發現集羣內的其他節點，把集羣內的節點地址寫在這裏，官方說也可以寫節點名字，但我試了名字不好使，沒細研究~~\ndiscovery.seed_hosts: [\"10.185.1.176:9300\", \"10.185.1.178:9300\", \"10.185.1.75:9301\"]\n# 初始的主節點，也可以不設置\ncluster.initial_master_nodes: [\"es-node1\"]\n# ---------------------------------- Various -----------------------------------\n# 爲了防止誤刪除，禁止使用通配符或_all刪除索引\naction.destructive_requires_name: true\n# 節點的通信地址，也就是discovery裏配置的節點ip和port，建議host就是本機ip，port就是9300，也是辨識度較高\ntransport.host: 10.185.1.178\ntransport.tcp.port: 9300","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"#節點2和其他節點\n# ---------------------------------- Cluster -----------------------------------\ncluster.name: magicloud-cluster\n# ------------------------------------ Node ------------------------------------\nnode.name: es-node2\nnode.master: true\nnode.data: true\n# ----------------------------------- Paths ------------------------------------\npath.data: /usr/local/elasticsearch/data\npath.logs: /usr/local/elasticsearch/logs\n# ---------------------------------- Network -----------------------------------\nnetwork.host: 0.0.0.0\nhttp.port: 9200\n# --------------------------------- Discovery ----------------------------------\ndiscovery.seed_hosts: [\"10.185.1.176:9300\", \"10.185.1.178:9300\", \"10.185.1.75:9301\"]\n# ---------------------------------- Various -----------------------------------\naction.destructive_requires_name: true\ntransport.host: 10.185.1.178\ntransport.tcp.port: 9300","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置文件的關鍵配置就是這些，關於es的配置，還有幾個關鍵的點","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.修改 /etc/security/limits.conf，在配置文件最下面添加","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"* soft nofile 65535\n* hard nofile 65535\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"否則啓動es的時候會報錯，其實按報錯的提示修改即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"錯誤大概是這樣max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.在es 7.x的版本，不可以通過root用戶啓動es，這個可以強制修改，但還是按官方的建議來比較好，操作也比較簡單","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"groupadd elsearch #1.添加新的管理組\nuseradd -m -g elsearch elsearch- #2.添加新管理組下的用戶\npasswd elsearch #3.回車後輸入密碼\nchown -R elsearch:elsearch /opt/es/elasticsearch7.14.1/ #4.給elsearch用戶授予es根目錄的管理權限\nchown -R elsearch:elsearch /usr/local/elasticsearch/data/ #5.給elsearch用戶授予es的存儲數據目錄的管理權限\nchown -R elsearch:elsearch /usr/local/elasticsearch/logs/ #6.給elsearch用戶授予es的存儲日誌目錄的管理權限","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"授權完成後，就可以以elsearch用戶分別啓動es集羣了","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"su elsearch #1.切換到elsearch用戶\nbin/elsticsearch -d #2.後臺啓動es","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分別啓動各個es節點後，可以在當前終端查看相應進程，也可以通過es的心跳接口查看， 或者通過head或者其他es管理客戶端來查看運行情況，","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0b/0b4405eb391ac9f27a85a6b61ccf392d.png","alt":"image-20211108101358307","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8537dd13a7cdbcd95f461e54caaf147.png","alt":"image-20211108101630629","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5f/5f924b6e171e29421cac0578ae83f669.png","alt":"image-20211108101717038","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，es的配置就基本完成了.","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.2 Kibana配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置好es集羣和，Kibana的配置就十分簡單了，官方配置地址：","attrs":{}},{"type":"link","attrs":{"href":"https://www.elastic.co/guide/en/kibana/7.14/index.html","title":null,"type":null},"content":[{"type":"text","text":"https://www.elastic.co/guide/en/kibana/7.14/index.html","attrs":{}}]}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"server.port: 5601 #端口號，建議保持5601的默認配置，依舊是爲了高辨識度\nserver.host: \"10.185.1.178\" #要暴露的ip地址\nserver.publicBaseUrl: \"http://10.185.1.178:5601\" #默認訪問地址，如果掛了域名指向，這裏可以寫域名地址\nelasticsearch.hosts: [\"http://10.185.1.178:9200\"] #es集羣地址，我這暫時就寫了一個\ni18n.locale: \"zh-CN\" #kibana的默認語言配置，默認是英文，我這裏改成了中文，英文好的同學可以忽略","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7.x版本的es同樣是不建議使用root用戶啓動，所以還是按照給es目錄授權的步驟，給Kibana目錄也進行一下普通用戶的授權","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"su root #1.如果當前不是root用戶，就先切回到root用戶，區分標誌就是看命令輸入行是$符號還是#符號，#就是root\nchown -R elsearch:elsearch /opt/es/kibana/ #2.給elsearch用戶授權管理kibana目錄\nsu elsearch #3.切換用戶\nnohup /opt/es/kibana/bin/kibana & #4.後臺啓動kibana","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"啓動後，就可以在本地瀏覽器打開kibana管理界面了","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/32/326b6e05accf1e1c701f8863d0bf9674.png","alt":"image-20211108102920006","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.3 Logstash配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"logstash主要是做數據傳輸管道的，這個沒有用戶限制，配置也是按需進行，比如我們的業務是用logstash來傳輸日誌到es進行集中管理。所以我的配置文件就長這樣","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"input {\n redis {\n codec => plain\n host => \"10.185.1.176\"\n port => 6379\n password => \"不給看，按需配置\"\n data_type => list\n key => \"eslog\"\n db => 7\n }\n}\n\nfilter {\n\t# grok是logstash自帶的數據解析插件，我這裏用的是正則匹配，注意這裏配置好的格式，要和代碼裏傳輸日誌內容的格式一致\n grok {\n \n match=>{\"message\"=>\"%{DATA:system} %{DATA:level} %{DATA:time} %{DATA:user} %{DATA:method} %{DATA:path} \\\"%{DATA:param}\\\" %{DATA:ip} %{DATA:webbrower} \\\"%{DATA:remark}\\\"\" } \n remove_field => [\"message\"]\n }\n geoip {\n source => \"ip\" #啓用自帶的geoip插件，定位ip地址，按需啓用，不用就去掉\n }\n}\n\noutput {\n elasticsearch {\n hosts => [\"http://10.185.1.176:9200\"]\n index => \"cloudlog-%{+YYYY.MM.dd}\" \n }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個就不多說了，之前寫過一篇相關的同步的內容：","attrs":{}},{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/08/17/%E6%97%A5%E5%BF%97%E5%BD%92%E4%B8%80%E7%AE%A1%E7%90%86%E7%9A%84%E4%B8%80%E7%A7%8D%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/","title":null,"type":null},"content":[{"type":"text","text":"https://www.tonydf.top/2021/08/17/%E6%97%A5%E5%BF%97%E5%BD%92%E4%B8%80%E7%AE%A1%E7%90%86%E7%9A%84%E4%B8%80%E7%A7%8D%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置好後，啓動logstash即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，在CentOS7環境下elk的基本配置就完成了。","attrs":{}}]}]}