packetbeat发送数据到pulsar

默认情况下,packetbeat不支持将数据发送到pulsar,因此使用了streamnative开发的kop(kafka on pulsar)来解决这个问题,以下是单机版安装步骤。

一、安装pulsar与kop

1. 下载并安装

wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/apache-pulsar-2.9.3-bin.tar.gz
tar -zxf apache-pulsar-2.9.3-bin.tar.gz
cd apache-pulsar-2.9.3/
mkdir connectors
cd connectors
wget https://github.91chi.fun/https://github.com//streamnative/kop/releases/download/v2.9.3.7/pulsar-protocol-handler-kafka-2.9.3.7.nar #下载kop,并放到特定目录下。kop版本要与pulsar版本保持一致。
cd ..

2. 修改pulsar配置

vim conf/standalone.conf (集群版就修改conf/broker.conf)

# 加入以下配置
messagingProtocols=kafka
protocolHandlerDirectory=/home/apache-pulsar-2.9.3/connectors
allowAutoTopicCreationType=partitioned  # 此处默认为non-partitioned,修改为partitioned
# By default, allowAutoTopicCreationType is set to non-partitioned. Since topics are partitioned by default in Kafka,
# it's better to avoid creating non-partitioned topics for Kafka clients unless Kafka clients need to interact
# with existing non-partitioned topics.

kafkaListeners=PLAINTEXT://127.0.0.1:9092
kafkaAdvertisedListeners=PLAINTEXT://127.0.0.1:9092
brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
brokerDeleteInactiveTopicsEnabled=false    #默认为true,此处修改为false

3. 启动pulsar

bin/pulsar standalone

此时,pulsar就可以接收kafka协议的数据。

二、配置packetbeat将数据发送到kafka

vim packetbeat/packetbeat.yml

# -------------------------------- Kafka Output --------------------------------
output.kafka:
#   Boolean flag to enable or disable the output module.
   enabled: true
   hosts: ["localhost:9092"]
   topic: dns_topic
   version: 2.0.0   # 注意版本指定为2.0.0
   codec.json:
     pretty: false

此时,启动packetbeat后,数据就可以发送到pulsar了,可以通过pulsar消费者消费数据了

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章