kafka+zookeeper+filebeat集羣

環境

ElasticSearch集羣的高可用和自平衡方案會在節點掛掉（重啓）後自動在別的結點上覆制該結點的分片，這將導致了大量的IO和網絡開銷。
如果離開的節點重新加入集羣，elasticsearch爲了對數據分片(shard)進行再平衡，會爲重新加入的節點再次分配數據分片(Shard), 當一臺es因爲壓力過大而掛掉以後，其他的es服務會備份本應那臺es保存的數據，造成更大壓力，於是整個集羣會發生雪崩。
生產環境下建議關閉自動平衡。

三臺服務器
10.21.2.224
10.21.2.225
10.21.2.226

1.安裝zookeeper源碼包.配三臺，每臺/tmp/zookeeper/myid不同

配置文件 conf/zoo.cfg

dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
tickTime=2000
initLimit=10
syncLimit=5
server.0=10.21.2.224:2887:3887
server.1=10.21.2.225:2887:3887
server.2=10.21.2.226:2887:3887

啓動

./zkServer.sh start

2.安裝kafka源碼包

10.21.2.224 >> server.properties

broker.id=0
listeners=PLAINTEXT://0.0.0.0:9092
port=9092
advertised.host.name=10.21.2.224
advertised.listeners=PLAINTEXT://10.21.2.224:9092
num.network.threads=30
num.io.threads=80
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/data/kafka_data
num.partitions=15
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.21.2.224:2181,10.21.2.225:2181,10.21.2.226:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

10.21.2.225 >> server.properties

broker.id=1
listeners=PLAINTEXT://0.0.0.0:9092
port=9092
advertised.host.name=10.21.2.225
advertised.listeners=PLAINTEXT://10.21.2.225:9092
num.network.threads=30
num.io.threads=80
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/data/kafka_data
num.partitions=15
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.21.2.224:2181,10.21.2.225:2181,10.21.2.226:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

10.21.2.226 >> server.properties

broker.id=2
listeners=PLAINTEXT://0.0.0.0:9092
port=9092
advertised.host.name=10.21.2.226
advertised.listeners=PLAINTEXT://10.21.2.226:9092
num.network.threads=30
num.io.threads=80
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/data/kafka_data
num.partitions=15
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.21.2.224:2181,10.21.2.225:2181,10.21.2.226:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

啓動

nohup ./bin/kafka-server-start.sh config/server.properties &

測試

選擇一個節點上創建一個新的Topic
./bin/kafka-topics.sh --create --zookeeper 10.21.2.224:2181,10.21.2.225:2181,10.21.2.226:2181 --topic test  --replication-factor 3 --partitions 3

查看topic副本信息
./bin/kafka-topics.sh --describe --zookeeper 10.21.2.224:2181,10.21.2.225:2181,10.21.2.226:2181 --topic test

查看已經創建的topic信息
./bin/kafka-topics.sh --list --zookeeper 10.21.2.224:2181,10.21.2.225:2181,10.21.2.226:2181

 測試生產者發送消息
 ./bin/kafka-console-producer.sh --broker-list 10.21.2.224:9092,10.21.2.225:9092,10.21.2.226:9092 --topic test
 
 測試消費者消費消息
 ./bin/kafka-console-consumer.sh --bootstrap-server 10.21.2.224:9092,10.21.2.225:9092,10.21.2.226:9092  --from-beginning --topic test
 

壓測

https://www.cnblogs.com/xiao987334176/p/10075659.html

3.filebeat

filebeat.modules:
filebeat.prospectors:
- input_type: log
 paths:
   - /tmp/nohup1.log

 fields:
   log_source: asc_batch
   log_topics: asc

- input_type: log
 paths:
   - /tmp/nohup2.log
 fields:
   log_topics: test2

#------------------------------- Kafka output ----------------------------------
output.kafka:
 enabled: true
 hosts: ["10.21.2.224:9092","10.21.2.225:9092","10.21.2.226:9092"]
 topic: '%{[fields][log_topics]}'
 compression: gzip
 max_message_bytes: 1000000

4.logstash配置

input {
        kafka{
        bootstrap_servers => "10.21.2.224:9092,10.21.2.225:9092,10.21.2.226:9092"
        topics => ["ad","novel","mdm","yuqing","ir-ygx","mini-game","azkaban","ir-feihuo","ykusercenter","yksplus","ykforgetpwd","ykpaycenter","yksocialite","hps"]
        codec => "json"
        consumer_threads => 5
        decorate_events => true
        auto_offset_reset => "latest"
        }
}

output {
   if [fields][log_topics] == "ad" {
        elasticsearch {
        hosts => ["10.21.2.219:9200"]
        index => "ad-%{+YYYY.MM.dd}"
        user => elastic
        password => changeme
        }
      }
   else if [fields][log_topics] == 'nove' {
        elasticsearch {
        hosts => ["10.21.2.219:9200"]
        index => "novel-%{+YYYY.MM.dd}"
        user => elastic
        password => changeme
        }
      }
}