在Linux系统发行版（以CentOS7为例）上部署ElasticSearch集群并启动Kibana和Logstash

{"type":"doc","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前分别写过关于使用ES来做日志统一管理，以及在CentOS上部署Redis哨兵集群的博客，今儿的话题还是要围绕他们，这次是CentOS和ELK的交集。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"之前我们的ES日志集群是在Windows系统下运行的，这两天上头给批了两台CentOS的主机，就把日志的集群迁移到了CentOS服务器上。迁移的过程总体还是比较顺利，但过程中还是遇到了一些问题，或者说我个人感觉应该要记录一下的东西，在这里简单总结一下。","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/11/05/%E5%9C%A8Linux%E7%B3%BB%E7%BB%9F%E5%8F%91%E8%A1%8C%E7%89%88%EF%BC%88%E4%BB%A5CentOS7%E4%B8%BA%E4%BE%8B%EF%BC%89%E4%B8%8A%E9%83%A8%E7%BD%B2ElasticSearch%E9%9B%86%E7%BE%A4%E5%B9%B6%E5%90%AF%E5%8A%A8Kibana%E5%92%8CLogstash/#%E4%B8%80%E3%80%81Elasticsearch%E7%89%88%E6%9C%AC%E9%80%89%E6%8B%A9","title":"一、Elasticsearch版本选择","type":null}},{"type":"text","text":"一、Elasticsearch版本选择","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方的建议是，选择最新的版本~，这其实也是我在选择开源组件的基本原则，大多数情况下，越新的产品，性能，功能完整性也会越好，尤其是ES这种级别的软件。但因为我是从Windows服务器上迁移过来，所以我选择了和之前Windows一致的版本，7.14.1（当前最新的是7.15.1）。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里需要提一句的是，如果是es的老用户，都应该清楚，es在发布产品的时候，都是捆绑发布的，不如es发布了7.14.1，对应的kibana，logstash，beats等等，也会有相同的版本跟着发布，所以如果es选了7.14.1，用其余产品的时候，也要选同样的版本！","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/11/05/%E5%9C%A8Linux%E7%B3%BB%E7%BB%9F%E5%8F%91%E8%A1%8C%E7%89%88%EF%BC%88%E4%BB%A5CentOS7%E4%B8%BA%E4%BE%8B%EF%BC%89%E4%B8%8A%E9%83%A8%E7%BD%B2ElasticSearch%E9%9B%86%E7%BE%A4%E5%B9%B6%E5%90%AF%E5%8A%A8Kibana%E5%92%8CLogstash/#%E4%BA%8C%E3%80%81%E4%B8%8B%E8%BD%BD%E5%AE%89%E8%A3%85","title":"二、下载安装","type":null}},{"type":"text","text":"二、下载安装","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方的下载提供了很多种类型，比如为Windows提供了msi，为linux提供了deb，rpm等，还有压缩包的类型。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里我个人建议是统一用压缩包的类型，因为可能会涉及到迁移或者集群部署，用压缩包的形式，解压后可以很方便的修改配置文件，对文件目录授权，以及设定服务启动方式等。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方的下载指导页如下：","attrs":{}},{"type":"link","attrs":{"href":"https://www.elastic.co/cn/downloads/elasticsearch","title":null,"type":null},"content":[{"type":"text","text":"https://www.elastic.co/cn/downloads/elasticsearch","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"对应的Kibana，logstash就是把链接最后的内容改成对应的名字即可","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"压缩包下载后，对应安装即可，我这里是建立了一个上层目录，然后把es，kibana和logstash都分别安装到了里面，方便管理。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/e9/e9ccce890b6db611d8688d1806b7394c.png","alt":"image-20211108091227345","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":3},"content":[{"type":"text","text":"三、配置集群","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/11/05/%E5%9C%A8Linux%E7%B3%BB%E7%BB%9F%E5%8F%91%E8%A1%8C%E7%89%88%EF%BC%88%E4%BB%A5CentOS7%E4%B8%BA%E4%BE%8B%EF%BC%89%E4%B8%8A%E9%83%A8%E7%BD%B2ElasticSearch%E9%9B%86%E7%BE%A4%E5%B9%B6%E5%90%AF%E5%8A%A8Kibana%E5%92%8CLogstash/#3-1-ElastercSearch%E9%85%8D%E7%BD%AE","title":"3.1 ElastercSearch配置","type":null}},{"type":"text","text":"3.1 ElastercSearch配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这里先给出官方的配置指导页：","attrs":{}},{"type":"link","attrs":{"href":"https://www.elastic.co/guide/en/elasticsearch/reference/7.14/settings.html%EF%BC%8C%E6%88%91%E8%BF%99%E4%B8%AA%E6%98%AF7.14%EF%BC%8C%E5%A6%82%E6%9E%9C%E4%BD%A0%E7%94%A8%E5%88%AB%E7%9A%84%E7%89%88%E6%9C%AC%EF%BC%8C%E5%B0%B1%E6%8A%8A%E9%93%BE%E6%8E%A5%E9%87%8C%E7%9A%84%E7%89%88%E6%9C%AC%E5%8F%B7%E6%94%B9%E4%B8%80%E4%B8%8B%E5%B0%B1%E5%A5%BD%E3%80%82","title":null,"type":null},"content":[{"type":"text","text":"https://www.elastic.co/guide/en/elasticsearch/reference/7.14/settings.html，","attrs":{}}]},{"type":"text","text":"我这个是7.14，如果你用别的版本，就把链接里的版本号改一下就好。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"官方指导里，分别给出了源于elasticsearch.yml，jvm.options，log4j2.propertites三给关键配置文件的配置说明。我这里只用到了第一个，官方指导了也提到了，对于后两项，分别是调整Java虚拟机和日志的选项，一般情况下不需要修改，按默认配置即可。如果是二般情况，就再根据说明按需调整。","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"# 节点1\n# ---------------------------------- Cluster -----------------------------------\n# 集群名字，每个集群内的节点应该保持一致\ncluster.name: magicloud-cluster\n# ------------------------------------ Node ------------------------------------\n# 节点名称\nnode.name: es-node1\n# 是否可以被选举为主节点\nnode.master: true\n# 是否为数据节点（存储空间足够的情况下，我个人建议把每个节点都设定成可存储数据的节点，这样会最大限度保证数据分片不会丢失）\nnode.data: true\n# ----------------------------------- Paths ------------------------------------\n# 数据和日志存放地址，根据官方建议，在生产环境下，这里把数据放到了$ES_HOME之外的位置。\n# https://www.elastic.co/guide/en/elasticsearch/reference/7.14/important-settings.html\npath.data: /usr/local/elasticsearch/data\npath.logs: /usr/local/elasticsearch/logs\n# ---------------------------------- Network -----------------------------------\n# 网络配置，默认是127.0.0.1的回环地址，修改成要暴露的ip或者直接0.0.0.0\nnetwork.host: 0.0.0.0\n# 端口，默认9200，个人不建议修改，不为别的，主要是其他开发或者运维人员一看到9200就知道es在跑着，辨识度较高\nhttp.port: 9200\n# --------------------------------- Discovery ----------------------------------\n# 集群发现，发现集群内的其他节点，把集群内的节点地址写在这里，官方说也可以写节点名字，但我试了名字不好使，没细研究~~\ndiscovery.seed_hosts: [\"10.185.1.176:9300\", \"10.185.1.178:9300\", \"10.185.1.75:9301\"]\n# 初始的主节点，也可以不设置\ncluster.initial_master_nodes: [\"es-node1\"]\n# ---------------------------------- Various -----------------------------------\n# 为了防止误删除，禁止使用通配符或_all删除索引\naction.destructive_requires_name: true\n# 节点的通信地址，也就是discovery里配置的节点ip和port，建议host就是本机ip，port就是9300，也是辨识度较高\ntransport.host: 10.185.1.178\ntransport.tcp.port: 9300","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"#节点2和其他节点\n# ---------------------------------- Cluster -----------------------------------\ncluster.name: magicloud-cluster\n# ------------------------------------ Node ------------------------------------\nnode.name: es-node2\nnode.master: true\nnode.data: true\n# ----------------------------------- Paths ------------------------------------\npath.data: /usr/local/elasticsearch/data\npath.logs: /usr/local/elasticsearch/logs\n# ---------------------------------- Network -----------------------------------\nnetwork.host: 0.0.0.0\nhttp.port: 9200\n# --------------------------------- Discovery ----------------------------------\ndiscovery.seed_hosts: [\"10.185.1.176:9300\", \"10.185.1.178:9300\", \"10.185.1.75:9301\"]\n# ---------------------------------- Various -----------------------------------\naction.destructive_requires_name: true\ntransport.host: 10.185.1.178\ntransport.tcp.port: 9300","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置文件的关键配置就是这些，关于es的配置，还有几个关键的点","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"1.修改 /etc/security/limits.conf，在配置文件最下面添加","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"* soft nofile 65535\n* hard nofile 65535\n","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"否则启动es的时候会报错，其实按报错的提示修改即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"错误大概是这样max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2.在es 7.x的版本，不可以通过root用户启动es，这个可以强制修改，但还是按官方的建议来比较好，操作也比较简单","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"groupadd elsearch #1.添加新的管理组\nuseradd -m -g elsearch elsearch- #2.添加新管理组下的用户\npasswd elsearch #3.回车后输入密码\nchown -R elsearch:elsearch /opt/es/elasticsearch7.14.1/ #4.给elsearch用户授予es根目录的管理权限\nchown -R elsearch:elsearch /usr/local/elasticsearch/data/ #5.给elsearch用户授予es的存储数据目录的管理权限\nchown -R elsearch:elsearch /usr/local/elasticsearch/logs/ #6.给elsearch用户授予es的存储日志目录的管理权限","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"授权完成后，就可以以elsearch用户分别启动es集群了","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"su elsearch #1.切换到elsearch用户\nbin/elsticsearch -d #2.后台启动es","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"分别启动各个es节点后，可以在当前终端查看相应进程，也可以通过es的心跳接口查看， 或者通过head或者其他es管理客户端来查看运行情况，","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/0b/0b4405eb391ac9f27a85a6b61ccf392d.png","alt":"image-20211108101358307","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/c8/c8537dd13a7cdbcd95f461e54caaf147.png","alt":"image-20211108101630629","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/5f/5f924b6e171e29421cac0578ae83f669.png","alt":"image-20211108101717038","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，es的配置就基本完成了.","attrs":{}}]},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.2 Kibana配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置好es集群和，Kibana的配置就十分简单了，官方配置地址：","attrs":{}},{"type":"link","attrs":{"href":"https://www.elastic.co/guide/en/kibana/7.14/index.html","title":null,"type":null},"content":[{"type":"text","text":"https://www.elastic.co/guide/en/kibana/7.14/index.html","attrs":{}}]}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"server.port: 5601 #端口号，建议保持5601的默认配置，依旧是为了高辨识度\nserver.host: \"10.185.1.178\" #要暴露的ip地址\nserver.publicBaseUrl: \"http://10.185.1.178:5601\" #默认访问地址，如果挂了域名指向，这里可以写域名地址\nelasticsearch.hosts: [\"http://10.185.1.178:9200\"] #es集群地址，我这暂时就写了一个\ni18n.locale: \"zh-CN\" #kibana的默认语言配置，默认是英文，我这里改成了中文，英文好的同学可以忽略","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"7.x版本的es同样是不建议使用root用户启动，所以还是按照给es目录授权的步骤，给Kibana目录也进行一下普通用户的授权","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"su root #1.如果当前不是root用户，就先切回到root用户，区分标志就是看命令输入行是$符号还是#符号，#就是root\nchown -R elsearch:elsearch /opt/es/kibana/ #2.给elsearch用户授权管理kibana目录\nsu elsearch #3.切换用户\nnohup /opt/es/kibana/bin/kibana & #4.后台启动kibana","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"启动后，就可以在本地浏览器打开kibana管理界面了","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/32/326b6e05accf1e1c701f8863d0bf9674.png","alt":"image-20211108102920006","title":null,"style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":null,"fromPaste":true,"pastePass":true}},{"type":"heading","attrs":{"align":null,"level":4},"content":[{"type":"text","text":"3.3 Logstash配置","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"logstash主要是做数据传输管道的，这个没有用户限制，配置也是按需进行，比如我们的业务是用logstash来传输日志到es进行集中管理。所以我的配置文件就长这样","attrs":{}}]},{"type":"codeblock","attrs":{"lang":"shell"},"content":[{"type":"text","text":"input {\n redis {\n codec => plain\n host => \"10.185.1.176\"\n port => 6379\n password => \"不给看，按需配置\"\n data_type => list\n key => \"eslog\"\n db => 7\n }\n}\n\nfilter {\n\t# grok是logstash自带的数据解析插件，我这里用的是正则匹配，注意这里配置好的格式，要和代码里传输日志内容的格式一致\n grok {\n \n match=>{\"message\"=>\"%{DATA:system} %{DATA:level} %{DATA:time} %{DATA:user} %{DATA:method} %{DATA:path} \\\"%{DATA:param}\\\" %{DATA:ip} %{DATA:webbrower} \\\"%{DATA:remark}\\\"\" } \n remove_field => [\"message\"]\n }\n geoip {\n source => \"ip\" #启用自带的geoip插件，定位ip地址，按需启用，不用就去掉\n }\n}\n\noutput {\n elasticsearch {\n hosts => [\"http://10.185.1.176:9200\"]\n index => \"cloudlog-%{+YYYY.MM.dd}\" \n }\n}","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"这个就不多说了，之前写过一篇相关的同步的内容：","attrs":{}},{"type":"link","attrs":{"href":"https://www.tonydf.top/2021/08/17/%E6%97%A5%E5%BF%97%E5%BD%92%E4%B8%80%E7%AE%A1%E7%90%86%E7%9A%84%E4%B8%80%E7%A7%8D%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/","title":null,"type":null},"content":[{"type":"text","text":"https://www.tonydf.top/2021/08/17/%E6%97%A5%E5%BF%97%E5%BD%92%E4%B8%80%E7%AE%A1%E7%90%86%E7%9A%84%E4%B8%80%E7%A7%8D%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88/","attrs":{}}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"配置好后，启动logstash即可。","attrs":{}}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"至此，在CentOS7环境下elk的基本配置就完成了。","attrs":{}}]}]}