日誌收集流程爲：fluentd - kafka - logstash - elasticsearch - kibana
使用鏡像如下：
| registry.cn-hangzhou.aliyuncs.com/xiao_bai/fluentd-kafka:v1 version:1.4 |
| registry.cn-hangzhou.aliyuncs.com/xiao_bai/kafka:v1 version:2.1 |
| docker.io/zookeeper:3.5 |
| docker.elastic.co/elasticsearch/elasticsearch:6.5.4 |
| docker.elastic.co/logstash/logstash:6.5.4 |
| docker.elastic.co/kibana/kibana:6.5.4 |
| docker.io/mobz/elasticsearch-head:5 |
| 192.168.2.244 | fluentd、zk、kafka、es、kibana |
| 192.168.2.245 | logstash、es-head、zk、kafka、es |
| 192.168.2.246 | es-balance、zk、kafka、es |
創建數據持久化目錄，配置文件目錄
mkdir -p /efk/{es,es-balance,kafka,zk,fluentd,logstash}/conf && mkdir /efk/{es,kafka,zk}/data系統環境配置
echo "vm.max_map_count=262144" >> /etc/sysctl.conf && sysctl -p echo -e "* soft nofile 65536\n* hard nofile 65536" >> /etc/security/limits.conf
zk集羣安裝
配置文件(三臺機器使用相同配置)
cat > /efk/zk/conf/zoo.cfg << EOF tickTime=2000 dataDir=/data/zookeeper/ dataLogDir=/data/zookeeper/logs clientPort=2181 initLimit=5 syncLimit=2 server.1=192.168.2.244:2888:3888 server.2=192.168.2.245:2888:3888 server.3=192.168.2.246:2888:3888 EOF
爲每臺機器設置myid
echo 1 > /efk/zk/data/myid #244 跟配置文件server.id設置一致 echo 2 > /efk/zk/data/myid #245 echo 3 > /efk/zk/data/myid #246
啓動容器(三臺機器執行相同命令)
docker run -d --restart=always --name zk --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/zk/conf/zoo.cfg:/conf/zoo.cfg -v /efk/zk/data/:/data/zookeeper zookeeper:3.5
zk集羣啓動後查看各節點身份
docker exec -it zk zkServer.sh status
kafka集羣安裝
配置文件(每臺機器的broker.id不能相同)
cat > /efk/kafka/conf/server.properties << EOF broker.id=1 listeners=PLAINTEXT://192.168.2.246:9092 port=9092 logs.dirs=/tmp/kafka-logs message.max.byte=5242880 default.replication.factor=2 replica.fetch.max.bytes=5242880 zookeeper.connect=192.168.2.244:2181,192.168.2.245:2181,192.168.2.246:2181 EOF
啓動容器(三臺機器執行相同命令)
docker run -d --restart=always --name=kafka --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/kafka/conf/server.properties:/usr/local/kafka_2.11-2.1.0/config/server.properties -v /efk/kafka/data:/tmp/kafka-logs registry.cn-hangzhou.aliyuncs.com/xiao_bai/kafka:v1
es集羣安裝
配置文件(修改node.name、network.publish_host)
cat > /efk/es/conf/elasticsearch.yml << EOF cluster.name: es-cluster node.name: es-node1 network.bind_host: 0.0.0.0 network.publish_host: 192.168.2.244 http.port: 9200 transport.tcp.port: 9300 http.cors.enabled: true http.cors.allow-origin: "*" node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["192.168.2.244:9300","192.168.2.245:9300","192.168.2.246:9300"] discovery.zen.minimum_master_nodes: 2 EOF
balance節點配置文件
cat > /efk/es-balance/conf/elasticsearch.yml << EOF cluster.name: es-cluster node.name: es-node4 network.bind_host: 0.0.0.0 network.publish_host: 192.168.2.246 http.port: 9201 transport.tcp.port: 9301 http.cors.enabled: true http.cors.allow-origin: "*" node.master: false node.data: false node.ingest: false discovery.zen.ping.unicast.hosts: ["192.168.2.244:9300","192.168.2.245:9300","192.168.2.246:9300"] discovery.zen.minimum_master_nodes: 2 EOF
啓動容器(三臺機器執行相同命令，數據節點)
docker run -d --restart=always --name=es --net=host -e ES_JAVA_OPTS="-Xms2g -Xmx2g" -v /etc/localtime:/etc/localtime:ro -v /efk/es/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /efk/es/data:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:6.5.4
balance節點
docker run -d --restart=always --name=es-balance --net=host -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -v /etc/localtime:/etc/localtime:ro -v /efk/es-balance/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.5.4
啓動es-head插件
docker run -d --name=es-head -p 9100:9100 mobz/elasticsearch-head:5
kibana安裝
啓動容器
docker run -d --name kibana --restart=always -e ELASTICSEARCH_URL=http://192.168.2.246:9201 -v /etc/localtime:/etc/localtime:ro -p 5601:5601 docker.elastic.co/kibana/kibana:6.5.4
logstash安裝
配置文件
consumer_threads與kafka設置的分區數保持一致
touch /efk/logstash/conf/logstash.yml
cat > /efk/logstash/conf/kafka-logstash-es.conf << EOF
input {
kafka {
bootstrap_servers => "192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092"
group_id => "kafka-logstash"
topics => ["docker_log","sys_log"]
consumer_threads => 1
codec => "json"
}
}
output {
elasticsearch {
hosts => ["192.168.2.244:9200","192.168.2.245:9200","192.168.2.246:9200"]
index => "logstash-%{+YYYY.MM.dd}"
codec => "json"
}
stdout {
codec => "rubydebug"
}
}
EOF啓動logstash容器
docker run -d --name=logstash --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/logstash/conf/logstash.yml:/usr/share/logstash/config/logstash.yml -v /efk/logstash/conf/kafka-logstash-es.conf:/usr/share/logstash/pipeline/kafka-logstash-es.conf docker.elastic.co/logstash/logstash:6.5.4
Fluentd安裝
配置文件(收集容器日誌以及系統日誌)
最後一個match需要設置爲**，否則會出現warn信息
cat > /efk/fluentd/conf/fluent.conf << EOF <source> @type tail path /var/log/containers/*/*.log pos_file /var/log/containers/containers.log.pos tag docker.log <parse> @type json </parse> read_from_head true </source> <source> @type syslog port 5140 bind 0.0.0.0 <parse> message_format rfc3164 </parse> tag system.log </source> <filter **> @type record_transformer <record> hostname "192.168.2.244" </record> </filter> <match docker.*> @type kafka_buffered brokers 192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092 default_topic docker_log output_data_type json </match> <match **> @type kafka_buffered brokers 192.168.2.244:9092,192.168.2.245:9092,192.168.2.246:9092 default_topic sys_log output_data_type json </match> EOF
收集系統日誌需要開啓rsyslog服務(根據服務器IP進行配置)
echo '*.* @192.168.2.244:5140' >> /etc/rsyslog.conf && systemctl restart rsyslog
啓動fluentd容器
docker run -d --name=fluentd --net=host -v /etc/localtime:/etc/localtime:ro -v /efk/fluentd/conf/fluent.conf:/fluentd/etc/fluent.conf -v /var/lib/docker/containers/:/var/log/containers registry.cn-hangzhou.aliyuncs.com/xiao_bai/fluentd-kafka:v1
參考文檔：
https://github.com/mobz/elasticsearch-head
http://kafka.apache.org/21/documentation.html
http://zookeeper.apache.org/doc/r3.5.4-beta/zookeeperAdmin.html