ES version: 7.10.0
這裏我們使用elk集成鏡像,docker hub 地址:https://hub.docker.com/r/sebp/elk/
前提條件
- Docker至少得分配3GB的內存;
- Elasticsearch至少需要單獨2G的內存;
- 防火牆開放相關端口; 9200,5404,56001
如果配置太差了,可以指定使用的配置參數
vm.max_map_count至少需要262144
echo "vm.max_map_count=262144" > /etc/sysctl.conf
sysctl -p
docker-compose
version: '2'
networks:
backend:
driver: bridge
services:
elk:
container_name: elk
image: sebp/elk:7.10.0
stdin_open: true
tty: true
restart: always
volumes:
- /opt/elk-data:/var/lib/elasticsearch
- /etc/localtime:/etc/localtime
ports:
- 5601:5601
- 9200:9200
- 5044:5044
environment:
- ES_MIN_MEM=128m
- ES_MAX_MEM=2048m
networks:
- backend
或者:
sudo docker run -dit --name elk \
-p 5601:5601 \
-p 9200:9200 \
-p 5044:5044 \
-e ES_MIN_MEM=128m \
-e ES_MAX_MEM=2048m \
-v /opt/elk-data:/var/lib/elasticsearch \
-v /etc/localtime:/etc/localtime \
sebp/elk:7.10.0
filebeat
https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html
filebeat的版本與es的版本一致
docker pull docker.elastic.co/beats/filebeat:7.10.0
Example configuration fileedit
Download this example configuration file as a starting point:
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.10/deploy/docker/filebeat.docker.yml
filebeat.docker.yml 文件的內容
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
processors:
- add_cloud_metadata: ~
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
username: '${ELASTICSEARCH_USERNAME:}'
password: '${ELASTICSEARCH_PASSWORD:}'
docker-composer.yml
version: '2'
networks:
backend:
driver: bridge
services:
filebeat:
container_name: filebeat
image: docker.elastic.co/beats/filebeat:7.10.0
stdin_open: true
tty: true
restart: always
volumes:
- $(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
- /var/lib/docker/containers:/var/lib/docker/containers
- /var/run/docker.sock:/var/run/docker.sock
- /etc/localtime:/etc/localtime
- /www/wwwroot/PHP/thinkphp/runtime/log/:/logs/thinkphp/
networks:
- backend
filebeat.docker.yml
# 日誌輸入配置
filebeat.inputs:
- type: log
enabled: true
paths:
- /logs/thinkphp/*/*.log
include_lines: ['error']
multiline:
pattern: '^\['
negate: true
match: after
output.elasticsearch:
hosts: ["elasticsearch:9200"]
#日誌輸出配置(採用 logstash 收集日誌,5044爲logstash端口)
#output.logstash:
# hosts: ['elasticsearch:5044']
只提取 error 日誌
kibana 設置顯示中文
修改 kibana 安裝目錄下的 config/kibana.yml
, 在文件末尾添加:
i18n.locale: "zh-CN"
重啓 kibana
- 使用docker安裝ELK
- ELK搭建(docker環境)
- Docker 部署ELK
- Filebeat 關鍵字多行匹配日誌採集(multiline與include_lines) filebeat 正則表達式匹配日誌
ES 設置密碼
ES 版本: 7.10.0
- 修改 elasticsearch 安裝目錄下的
config/elasticsearch.yml
,在文件末尾添加,修改完之後,重啓elasticsearch
服務
# discovery.type: single-node
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
- 執行設置用戶名和密碼的命令,這裏會爲4個左右用戶分別設置密碼,elastic, kibana, logstash_system,beats_system 等
D:\devops\elasticsearch-7.8.0\bin> elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y
Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]
- 修改密碼命令如下
curl -H "Content-Type:application/json" -XPOST -u elastic 'http://127.0.0.1:9200/_xpack/security/user/elastic/_password' -d '{ "password" : "123456" }'
- 修改 Kibana 配置文件
config/kibana.yml
文件
elasticsearch.username: "elastic"
elasticsearch.password: "123456"
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- D:/workspace/lamp/thinkphp/runtime/v1/log/*/*.log
# include_lines: ['error']
multiline:
pattern: '^\[\s\d{4}-\d{2}(-\d{2})?'
negate: true
match: after
tags: ["thinkphp-api"]
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# username: "elastic"
# password: "wakasann"
# indices:
# - index: "thinkphp-api-error"
# when.contains:
# tags: "thinkphp-api"
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
index: "thinkphp-api-error"
filebeat 運行
filebeat.exe -c filebeat.yml -e
logstash
thinkphpapi.conf
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "wakasann"
}
}
通過powershell 運行 logstash
.\bin\logstash.bat -f .\config\thinkphpapi.conf
- ElasticSearch設置用戶名密碼訪問
- 設置elasticsearch 7.x用戶名和密碼
- 使用Filebeat採集日誌結合logstash過濾出你想要的日誌
- filebeat收集json格式的日誌 可以自定義索引
- 使用Filebeat採集日誌結合logstash過濾出特定格式的日誌至Elasticsearch