spring-kafka入門學習(四):消息監聽器容器ConcurrentMessageListenerContainer測試示例

目錄

一、前言

二、測試準備

1.kafka客戶端配置

2.SpringBoot配置

三、消費者分配情況測試

1.使用默認的PartitionAssignor -> RangeAssignor

2.使用PartitionAssignor -> RoundRobinAssignor

四、性能測試

1.情景一 單線程消費

2.情景二 併發消費

五、總結


一、前言

在官網學習spring-kafka的過程中,MessageListenerContainer不易理解,實踐出真知,故通過敲代碼實測來加深理解。

官網鏈接:https://docs.spring.io/spring-kafka/docs/2.2.13.RELEASE/reference/html/#message-listener-container

通過配置MessageListenerContainer和提供消息偵聽器或使用@KafkaListener批註來接收消息。

MessageListenerContainer提供了兩種實現:
KafkaMessageListenerContainer
ConcurrentMessageListenerContainer

該KafkaMessageListenerContainer接收在單個線程從所有的主題或分區上的所有消息。所述ConcurrentMessageListenerContainer的一個或多個代表KafkaMessageListenerContainer實例,以提供多線程消耗。

ConcurrentMessageListenerContainer構造函數

public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
                            ContainerProperties containerProperties)

對於這個構造函數,Kafka使用其組管理功能在用戶之間分配分區。

@KafkaListener可以配置明確的主題和分區(本文示例中不要配置分區,因爲要測試分區分配策略

@KafkaListener(id = "thing2", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
          @TopicPartition(topic = "topic2", partitions = "0",
             partitionOffsets = @PartitionOffset(partition = "1", initialOffset = "100"))
        })
public void listen(ConsumerRecord<?, ?> record) {
    ...
}

官方教程中有如下內容:

When listening to multiple topics, the default partition distribution may not be what you expect. For example, if you have three topics with five partitions each and you want to use concurrency=15, you see only five active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. This is because the default Kafka PartitionAssignor is the RangeAssignor (see its Javadoc). For this scenario, you may want to consider using the RoundRobinAssignor instead, which distributes the partitions across all of the consumers. Then, each consumer is assigned one topic or partition. To change the PartitionAssignor, you can set the partition.assignment.strategyconsumer property (ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG) in the properties provided to the DefaultKafkaConsumerFactory.

注意:本文針對這段內容進行測試

二、測試準備

1.kafka客戶端配置

修改kafka的配置文件server.properties(目錄D:\ProgramFiles\kafka_2.12\config),配置五個分區

num.partitions=5

重啓kafka

注意:如果啓動報錯可以清空這2個目錄

2.SpringBoot配置

本文代碼在上一篇博客《spring-kafka入門學習(三):使用SpringBoot發送並接收消息》的基礎上更改。

配置線程數是15

  @Bean
    ConcurrentKafkaListenerContainerFactory<Integer, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(15);//例如,container.setConcurrency(3)創建三個KafkaMessageListenerContainer實例。
        return factory;
    }

配置監聽器

    @KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
    public void listen1(String foo) throws Exception {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
        Thread.sleep(1000 * 2);
    }

消息發送接口

    @RequestMapping(value = "test2")
    public String test2() {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        //每個topic發送5條數據,每條數據在一個分區
        for (int i = 0; i < 5; i++) {
            //send方法的參數(String topic, Integer partition, K key, @Nullable V data)
            kafkaTemplate.send("topic_1", i, 0, "topic_1_" + i);
            kafkaTemplate.send("topic_2", i, 0, "topic_2_" + i);
            kafkaTemplate.send("topic_3", i, 0, "topic_3_" + i);
            kafkaTemplate.flush();
        }
        return "test";
    }

三、消費者分配情況測試

1.使用默認的PartitionAssignor -> RangeAssignor

啓動項目,整理出如下日誌:

只會看到五個活動的使用者,每個消費者都爲每個主題分配了一個分區,而其他10個消費者處於空閒狀態。

2020-05-12 14:30:24.917  INFO 13152 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2, topic_2-2, topic_3-2]
2020-05-12 14:30:24.917  INFO 13152 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917  INFO 13152 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917  INFO 13152 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1, topic_2-1, topic_3-1]
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0, topic_2-0, topic_3-0]
2020-05-12 14:30:24.918  INFO 13152 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4, topic_2-4, topic_3-4]
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919  INFO 13152 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919  INFO 13152 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3, topic_2-3, topic_3-3]
2020-05-12 14:30:24.920  INFO 13152 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.920  INFO 13152 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions []

2.使用PartitionAssignor -> RoundRobinAssignor

爲每個消費者分配一個主題或分區。要更改PartitionAssignor,可以在提供給的屬性中設置partition.assignment.strategy使用者屬性(ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIGDefaultKafkaConsumerFactory

Kafka默認的 PartitionAssignorRangeAssignor,需要設置成RoundRobinAssignor。

只需要修改一個地方

重啓項目,整理出如下日誌:

可以看出沒有空閒的消費者

2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-2]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-4]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-3]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1]
2020-05-12 15:02:57.306  INFO 13720 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-3]
2020-05-12 15:02:57.306  INFO 13720 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-1]
2020-05-12 15:02:57.307  INFO 13720 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-1]
2020-05-12 15:02:57.308  INFO 13720 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4]
2020-05-12 15:02:57.308  INFO 13720 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-0]
2020-05-12 15:02:57.310  INFO 13720 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2]
2020-05-12 15:02:57.312  INFO 13720 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-4]
2020-05-12 15:02:57.313  INFO 13720 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-0]
2020-05-12 15:02:57.315  INFO 13720 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-2]

四、性能測試

啓動SpringBoot,訪問接口:http://127.0.0.1:8080/test2

監聽器睡眠2s模擬業務耗時,方便日誌分析

 @KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
    public void listen1(String foo) throws Exception {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
        Thread.sleep(1000 * 2);
    }

1.情景一 單線程消費

factory.setConcurrency(1);

消費結果:消費性能比較差(每2秒消費一條消息)

2020-05-12 15:15:15.223  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:15 - Listener-接收消息:topic_1_0
2020-05-12 15:15:17.224  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:17 - Listener-接收消息:topic_1_4
2020-05-12 15:15:19.224  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:19 - Listener-接收消息:topic_1_3
2020-05-12 15:15:21.224  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:21 - Listener-接收消息:topic_2_4
2020-05-12 15:15:23.225  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:23 - Listener-接收消息:topic_1_2
2020-05-12 15:15:25.225  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:25 - Listener-接收消息:topic_2_3
2020-05-12 15:15:27.226  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:27 - Listener-接收消息:topic_3_4
2020-05-12 15:15:29.227  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:29 - Listener-接收消息:topic_1_1
2020-05-12 15:15:31.227  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:31 - Listener-接收消息:topic_2_2
2020-05-12 15:15:33.228  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:33 - Listener-接收消息:topic_3_3
2020-05-12 15:15:35.229  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:35 - Listener-接收消息:topic_2_1
2020-05-12 15:15:37.230  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:37 - Listener-接收消息:topic_3_2
2020-05-12 15:15:39.230  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:39 - Listener-接收消息:topic_2_0
2020-05-12 15:15:41.231  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:41 - Listener-接收消息:topic_3_1
2020-05-12 15:15:43.232  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:43 - Listener-接收消息:topic_3_0

2.情景二 併發消費

factory.setConcurrency(15);

消費結果:消費性能很好

2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_2
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-10-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_2
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-12-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_4
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-2-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_4
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-8-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_0
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-5-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_2
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-6-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_3
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-4-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_1
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-7-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_4
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-1-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_3
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-9-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_1
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-3-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_0
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-11-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_3
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-14-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_1
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-13-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_0

五、總結

使用併發消費可以提高消息的消費速率,但是要注意以下問題:

1.使用併發消息偵聽器容器時,將在所有消費者線程上調用一個偵聽器實例。因此,偵聽器必須是線程安全的。

2.不在同一個分區的消息不能做到順序讀取。

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章