目錄
1.使用默認的PartitionAssignor -> RangeAssignor
2.使用PartitionAssignor -> RoundRobinAssignor
一、前言
在官網學習spring-kafka的過程中,MessageListenerContainer不易理解,實踐出真知,故通過敲代碼實測來加深理解。
官網鏈接:https://docs.spring.io/spring-kafka/docs/2.2.13.RELEASE/reference/html/#message-listener-container
通過配置MessageListenerContainer和提供消息偵聽器或使用@KafkaListener批註來接收消息。
MessageListenerContainer提供了兩種實現:
KafkaMessageListenerContainer
ConcurrentMessageListenerContainer
該KafkaMessageListenerContainer接收在單個線程從所有的主題或分區上的所有消息。所述ConcurrentMessageListenerContainer的一個或多個代表KafkaMessageListenerContainer實例,以提供多線程消耗。
ConcurrentMessageListenerContainer構造函數
public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
ContainerProperties containerProperties)
對於這個構造函數,Kafka使用其組管理功能在用戶之間分配分區。
@KafkaListener可以配置明確的主題和分區(本文示例中不要配置分區,因爲要測試分區分配策略)
@KafkaListener(id = "thing2", topicPartitions =
{ @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
@TopicPartition(topic = "topic2", partitions = "0",
partitionOffsets = @PartitionOffset(partition = "1", initialOffset = "100"))
})
public void listen(ConsumerRecord<?, ?> record) {
...
}
官方教程中有如下內容:
When listening to multiple topics, the default partition distribution may not be what you expect. For example, if you have three topics with five partitions each and you want to use concurrency=15
, you see only five active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. This is because the default Kafka PartitionAssignor
is the RangeAssignor
(see its Javadoc). For this scenario, you may want to consider using the RoundRobinAssignor
instead, which distributes the partitions across all of the consumers. Then, each consumer is assigned one topic or partition. To change the PartitionAssignor
, you can set the partition.assignment.strategy
consumer property (ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG
) in the properties provided to the DefaultKafkaConsumerFactory
.
注意:本文針對這段內容進行測試
二、測試準備
1.kafka客戶端配置
修改kafka的配置文件server.properties(目錄D:\ProgramFiles\kafka_2.12\config),配置五個分區
num.partitions=5
重啓kafka
注意:如果啓動報錯可以清空這2個目錄
2.SpringBoot配置
本文代碼在上一篇博客《spring-kafka入門學習(三):使用SpringBoot發送並接收消息》的基礎上更改。
配置線程數是15
@Bean
ConcurrentKafkaListenerContainerFactory<Integer, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(15);//例如,container.setConcurrency(3)創建三個KafkaMessageListenerContainer實例。
return factory;
}
配置監聽器
@KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
public void listen1(String foo) throws Exception {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
Thread.sleep(1000 * 2);
}
消息發送接口
@RequestMapping(value = "test2")
public String test2() {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
//每個topic發送5條數據,每條數據在一個分區
for (int i = 0; i < 5; i++) {
//send方法的參數(String topic, Integer partition, K key, @Nullable V data)
kafkaTemplate.send("topic_1", i, 0, "topic_1_" + i);
kafkaTemplate.send("topic_2", i, 0, "topic_2_" + i);
kafkaTemplate.send("topic_3", i, 0, "topic_3_" + i);
kafkaTemplate.flush();
}
return "test";
}
三、消費者分配情況測試
1.使用默認的PartitionAssignor -> RangeAssignor
啓動項目,整理出如下日誌:
只會看到五個活動的使用者,每個消費者都爲每個主題分配了一個分區,而其他10個消費者處於空閒狀態。
2020-05-12 14:30:24.917 INFO 13152 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2, topic_2-2, topic_3-2]
2020-05-12 14:30:24.917 INFO 13152 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917 INFO 13152 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917 INFO 13152 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1, topic_2-1, topic_3-1]
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0, topic_2-0, topic_3-0]
2020-05-12 14:30:24.918 INFO 13152 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4, topic_2-4, topic_3-4]
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919 INFO 13152 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919 INFO 13152 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3, topic_2-3, topic_3-3]
2020-05-12 14:30:24.920 INFO 13152 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.920 INFO 13152 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions []
2.使用PartitionAssignor ->
RoundRobinAssignor
爲每個消費者分配一個主題或分區。要更改PartitionAssignor
,可以在提供給的屬性中設置partition.assignment.strategy
使用者屬性(ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG
)DefaultKafkaConsumerFactory
。
Kafka默認的 PartitionAssignor
是RangeAssignor,需要設置成
RoundRobinAssignor。
只需要修改一個地方
重啓項目,整理出如下日誌:
可以看出沒有空閒的消費者
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-2]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-4]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-3]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1]
2020-05-12 15:02:57.306 INFO 13720 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-3]
2020-05-12 15:02:57.306 INFO 13720 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-1]
2020-05-12 15:02:57.307 INFO 13720 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-1]
2020-05-12 15:02:57.308 INFO 13720 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4]
2020-05-12 15:02:57.308 INFO 13720 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-0]
2020-05-12 15:02:57.310 INFO 13720 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2]
2020-05-12 15:02:57.312 INFO 13720 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-4]
2020-05-12 15:02:57.313 INFO 13720 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-0]
2020-05-12 15:02:57.315 INFO 13720 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-2]
四、性能測試
啓動SpringBoot,訪問接口:http://127.0.0.1:8080/test2
監聽器睡眠2s模擬業務耗時,方便日誌分析
@KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
public void listen1(String foo) throws Exception {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
Thread.sleep(1000 * 2);
}
1.情景一 單線程消費
factory.setConcurrency(1);
消費結果:消費性能比較差(每2秒消費一條消息)
2020-05-12 15:15:15.223 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:15 - Listener-接收消息:topic_1_0
2020-05-12 15:15:17.224 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:17 - Listener-接收消息:topic_1_4
2020-05-12 15:15:19.224 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:19 - Listener-接收消息:topic_1_3
2020-05-12 15:15:21.224 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:21 - Listener-接收消息:topic_2_4
2020-05-12 15:15:23.225 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:23 - Listener-接收消息:topic_1_2
2020-05-12 15:15:25.225 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:25 - Listener-接收消息:topic_2_3
2020-05-12 15:15:27.226 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:27 - Listener-接收消息:topic_3_4
2020-05-12 15:15:29.227 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:29 - Listener-接收消息:topic_1_1
2020-05-12 15:15:31.227 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:31 - Listener-接收消息:topic_2_2
2020-05-12 15:15:33.228 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:33 - Listener-接收消息:topic_3_3
2020-05-12 15:15:35.229 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:35 - Listener-接收消息:topic_2_1
2020-05-12 15:15:37.230 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:37 - Listener-接收消息:topic_3_2
2020-05-12 15:15:39.230 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:39 - Listener-接收消息:topic_2_0
2020-05-12 15:15:41.231 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:41 - Listener-接收消息:topic_3_1
2020-05-12 15:15:43.232 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:43 - Listener-接收消息:topic_3_0
2.情景二 併發消費
factory.setConcurrency(15);
消費結果:消費性能很好
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_2
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-10-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_2
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-12-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_4
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-2-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_4
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-8-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_0
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-5-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_2
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-6-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_3
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-4-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_1
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-7-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_4
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-1-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_3
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-9-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_1
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-3-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_0
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-11-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_3
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-14-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_1
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-13-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_0
五、總結
使用併發消費可以提高消息的消費速率,但是要注意以下問題:
1.使用併發消息偵聽器容器時,將在所有消費者線程上調用一個偵聽器實例。因此,偵聽器必須是線程安全的。
2.不在同一個分區的消息不能做到順序讀取。