（三）使用logstash-input-kafka消費信息並根據topic寫入不同的es索引

一、logstash-input-kafka介紹

[Logstash 參考指南（Kafka輸入插件）

二、舉例

分析：利用 decorate_events 參數，將 kafka的元信息（如：topic，group，offset等）全部輸出，根據不同的topic即可輸出到不同的es索引。

input{
	kafka{
		bootstrap_servers => ["55.0.10.19:9092,55.0.10.20:9092,55.0.10.21:9092"]
		group_id => "kafkaToes5" 
		auto_offset_reset => "earliest"
		consumer_threads => 5 
		decorate_events => true
		topics => ["a1","a2"]
		//security_protocol => "SASL_PLAINTEXT"     #sasl認證使用
      	//sasl_mechanism => "PLAIN"                 #sasl認證使用
      	//jaas_path => "/var/vcap/jobs/logstash/bin/kafka.conf" #sasl認證使用-認證填寫的密碼文件
	}
}
filter{
	if [@metadata][kafka][topic] == "a1" {
		json{
			source => "message"
			skip_on_invalid_json => true
		}
		mutate { 
            remove_field => ["@version","@timestamp"]
        }
	}
	if [@metadata][kafka][topic] == "a2" {
		json{
			source => "message"
			skip_on_invalid_json => true
		}
		mutate { 
            remove_field => ["@version","@timestamp"]
        }
	}
}
output {
	stdout{
		codec => json_lines
	}
	if [@metadata][kafka][topic] == "a1" {
		elasticsearch { 
			index => "test123"
			hosts => ["55.0.10.10:9280","55.0.10.11:9280"]
			user => "admin"
			password => "123456"
			document_type=> "doc"
		}
	}
	if [@metadata][kafka][topic] == "a2" {
		elasticsearch { 
			index => "test456"
			hosts => ["55.0.10.10:9280","55.0.10.11:9280"]
			user => "admin"
			password => "123456"
			document_type=> "doc"
		}
	}
}

補充：
//認證填寫的內容例子
// cat /var/vcap/jobs/logstash/bin/kafka.conf
KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="kafkaUser"
    password="kafkaPwd";
};

參考：

使用logstash的logstash-input-kafka插件讀取kafka中的數據