packetbeat發送數據到pulsar

默認情況下，packetbeat不支持將數據發送到pulsar，因此使用了streamnative開發的kop（kafka on pulsar）來解決這個問題，以下是單機版安裝步驟。

一、安裝pulsar與kop

1. 下載並安裝

wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/apache-pulsar-2.9.3-bin.tar.gz
tar -zxf apache-pulsar-2.9.3-bin.tar.gz
cd apache-pulsar-2.9.3/
mkdir connectors
cd connectors
wget https://github.91chi.fun/https://github.com//streamnative/kop/releases/download/v2.9.3.7/pulsar-protocol-handler-kafka-2.9.3.7.nar #下載kop，並放到特定目錄下。kop版本要與pulsar版本保持一致。
cd ..

2. 修改pulsar配置

vim conf/standalone.conf （集羣版就修改conf/broker.conf）

# 加入以下配置
messagingProtocols=kafka
protocolHandlerDirectory=/home/apache-pulsar-2.9.3/connectors
allowAutoTopicCreationType=partitioned  # 此處默認爲non-partitioned，修改爲partitioned
# By default, allowAutoTopicCreationType is set to non-partitioned. Since topics are partitioned by default in Kafka,
# it's better to avoid creating non-partitioned topics for Kafka clients unless Kafka clients need to interact
# with existing non-partitioned topics.

kafkaListeners=PLAINTEXT://127.0.0.1:9092
kafkaAdvertisedListeners=PLAINTEXT://127.0.0.1:9092
brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
brokerDeleteInactiveTopicsEnabled=false    #默認爲true，此處修改爲false

3. 啓動pulsar

bin/pulsar standalone

此時，pulsar就可以接收kafka協議的數據。

二、配置packetbeat將數據發送到kafka

vim packetbeat/packetbeat.yml

# -------------------------------- Kafka Output --------------------------------
output.kafka:
#   Boolean flag to enable or disable the output module.
   enabled: true
   hosts: ["localhost:9092"]
   topic: dns_topic
   version: 2.0.0   # 注意版本指定爲2.0.0
   codec.json:
     pretty: false

此時，啓動packetbeat後，數據就可以發送到pulsar了，可以通過pulsar消費者消費數據了