今天我在cetos上安装好了开启了kafka,网上找了几个例子,想用c++写一个生产者和消费者模型的例子.
然后踩了几个坑,公布出来,希望大家以后不要再踩
我是用它自带的sh工作作为生产者,配置好了主题和partition等信息,如下所示
[root@localhost bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test11959
Created topic "test11959".
[root@localhost bin]# ./kafka-console-producer.sh --broker-list ${IP ADDRESS}:9092 --topic test11959
> 此时等待你的输入
然后我启动了我的消费者程序(用的是librdkafka这个库),但是出现了两个问题
首先出现的现象是这样的:
(这一条是网上的:Connecting to IPv6 addresses: Connect to ipv6#[::1]:9092 failed: Connection refused)
%3|1567166247.579|FAIL|rdkafka#consumer-1| [thrd:${IP address}:9092/bootstrap]: ${IP address}:9092/0: Connect to ipv4#${IP address}:9092 failed: Connection refused (after 1ms in state CONNECT)
%3|1567166247.579|ERROR|rdkafka#consumer-1| [thrd:${IP address}:9092/bootstrap]: ${IP address}:9092/0: Connect to ipv4#${IP address}:9092 failed: Connection refused (after 1ms in state CONNECT)
解决方法是 :
cd到kafka的安装路径,然后到config目录下.在文件server.properties 将下面这一项给打开,他的说明中解释的很清楚了
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://172.16.100.121:9092
然后出现的问题是:
%3|1567165624.868|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: testTopic [1]: desired partition does not exist in cluster (Local: Unknown partition)
最终我找到了这个链接:https://github.com/edenhill/librdkafka/issues/327 他里面有一层楼是这样说的
If you create a topic with 4 partitions then those partitions are numbered 0..3,
you seem to try to consume partition 4 which is indeed an Unknown partition.
看懂了吗?假如partitions 是4 则可用的partition是从0 开始计数的:0,1,2,3 所以,我的生产者的设置的partition=1 ,我缩写的客户端用的也是1 就导致客户端找不到partition了
希望大家少走弯路!!!