rocketmq client-啓動及消息獲取流程分析

文章基於rocket-mq4.0 代碼分析

主要分析消息拉取流程

Client端啓動入口

以Push模式爲例

org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#start
-->org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#start

  ------------------------------------------------------ 

public void start() throws MQClientException {
        System.out.println(this.getSubscriptionInner());
        switch (this.serviceState) {
            case CREATE_JUST:
                
              //....省略代碼....

                mQClientFactory.start();
                log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The PushConsumer service state not OK, maybe started once, "//
                    + this.serviceState//
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }

        this.updateTopicSubscribeInfoWhenSubscriptionChanged();

        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();

        this.mQClientFactory.rebalanceImmediately();
    }

主要是下面標註代碼

mQClientFactory 是 MQClientInstance的實例，主要通過

this.pullMessageService = new PullMessageService(this);
this.rebalanceService = new RebalanceService(this);

兩個service配合完成消息的拉取

PullMessageService 和 RebalanceService 都繼承了 ServiceThread

PullMessageService 啓動後會阻塞在

PullRequest pullRequest = this.pullRequestQueue.take();

獲取PullRequest對象，再根據這個對象請求broker獲得消息體

而PullRequest對象是RebalanceService啓動後根據topic定時到broker查詢該topic是否變更了（定時調 org.apache.rocketmq.client.impl.consumer.RebalanceImpl#doRebalance方法

），如果變更了纔會生成該對象並放入到

org.apache.rocketmq.client.impl.consumer.PullMessageService#pullRequestQueue 中去

 

RebalanceService

在run方法啓動後，通過一些列調用（類相互關係複雜）；以push模式爲例

MQClientInstance.doRebalance()
-->DefaultMQPushConsumerImpl.doRebalance()
-->RebalanceImpl.doRebalance(final boolean isOrder)
-->RebalanceImpl.rebalanceByTopic(final String topic, final boolean isOrder)
最後更消息模式[BROADCASTING,CLUSTERING]去獲取是否改拉取消息了

以 CLUSTERING 爲例

case CLUSTERING: {

//mqSet 爲該topic在broker端邏輯隊列comsumequeue集合的映射，該值會有單獨線程定時更新；
//例如：org.apache.rocketmq.client.impl.factory.MQClientInstance#startScheduledTask，還有其他地方也會更新該值

                Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
                List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);

                //省略代碼.....

                if (mqSet != null && cidAll != null) {
                    List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
                    mqAll.addAll(mqSet);

                    Collections.sort(mqAll);
                    Collections.sort(cidAll);

                    //通過消費端負載均衡策略，client獲得自己的MessageQueue

                    AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
                    List<MessageQueue> allocateResult = null;
                    try {
                        allocateResult = strategy.allocate(
                            this.consumerGroup,
                            this.mQClientFactory.getClientId(),
                            mqAll,
                            cidAll);
                    } catch (Throwable e) {
                        log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
                            e);
                        return;
                    }

                    Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
                    if (allocateResult != null) {
                        allocateResultSet.addAll(allocateResult);
                    }

                    //判斷改topic是否有新消息可以拉取了                    

                    boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
                    if (changed) {
                       
                        //省略代碼.....

                        //組裝PullMessageService拉取消息所需的PullRequest對象並放入隊列中
                           
                        this.messageQueueChanged(topic, mqSet, allocateResultSet);
                    }
                }
                break;
            }

 

 

PullMessageService

PullMessageService 獲取到 PullRequest後最終執行方法到：

org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage

然後是一個很長的 PullCallback，會在拉取消息成功後調用 ConsumeMessageService執行（說實話，這代碼不是一般的繞）

只有結果爲  pullResult.getPullStatus()爲FOUND纔會執行 submitConsumeRequest 請求（用戶自定義註冊監聽執行邏輯），其他的貌似都會放入重試

最終是我們註冊的監聽執行了自定義邏輯（DefaultMQPushConsumer 即push模式下我們需要實現  MessageListenerOrderly 或者 MessageListenerConcurrently  ）

 

消費進度更新

ConsumeMessageService   目前框架提供了兩種實現，

ConsumeMessageOrderlyService 和 ConsumeMessageConcurrentlyService，框架會根據我們實現的自定義監聽類型注入對應的實現類；

我理解的ConsumeMessageConcurrentlyService 和 ConsumeMessageOrderlyService關鍵區別在於

其內部類ConsumeRequest在執行run方法時後者通過加鎖的方式，旨在保證獲得鎖的時候框架才處理對應的消息；

 

如果從comsumequeue裏獲取到了消息，

會執行用戶自定義監聽：

 

這裏會根據自定義程序返回的status做一個處理，如果監聽程序返回的是null，則會被賦值成 ：ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT

 

continueConsume = ConsumeMessageOrderlyService.this.processConsumeResult(msgs, status, context, this);

 這個方法會根據status的值做不同的邏輯處理，如果是  SUSPEND_CURRENT_QUEUE_A_MOMENT，則會在校驗是否可以重複消費後，另外提交一個線程池任務處理該請求

是否可重複消費校驗：

private boolean checkReconsumeTimes(List<MessageExt> msgs) {
        boolean suspend = false;
        if (msgs != null && !msgs.isEmpty()) {
            for (MessageExt msg : msgs) {

//********這裏的 getMaxReconsumeTimes() 默認是 Integer.MAX_VALUE

                if (msg.getReconsumeTimes() >= getMaxReconsumeTimes()) {
                    MessageAccessor.setReconsumeTime(msg, String.valueOf(msg.getReconsumeTimes()));

//*******如果客戶端重試次數大於了設置的次數，則構建 RETRY 消息重新發送至broker端

                    if (!sendMessageBack(msg)) {
                        suspend = true;
                        msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                    }
                } else {
                    suspend = true;
                    msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                }
            }
        }
        return suspend;
    }

提交線程池新任務：

private void submitConsumeRequestLater(
        final ProcessQueue processQueue,
        final MessageQueue messageQueue,
        final long suspendTimeMillis
    ) {
        long timeMillis = suspendTimeMillis;
        if (timeMillis == -1) {
            timeMillis = this.defaultMQPushConsumer.getSuspendCurrentQueueTimeMillis();
        }

        if (timeMillis < 10) {
            timeMillis = 10;
        } else if (timeMillis > 30000) {
            timeMillis = 30000;
        }


//*****重新提交線程池處理*****

        this.scheduledExecutorService.schedule(new Runnable() {

            @Override
            public void run() {
                ConsumeMessageOrderlyService.this.submitConsumeRequest(null, processQueue, messageQueue, true);
            }
        }, timeMillis, TimeUnit.MILLISECONDS);
    }

 

 在執行完自定義邏輯後就會向broker發起更新消費offset的請求

 

****to continue

 

附客戶端代碼：

public class Consumer {

    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("ConsumerGroupNamecc4");
        consumer.setNamesrvAddr("127.0.0.1:9876");

        consumer.subscribe("topicaaa","TagA");

        consumer.registerMessageListener(new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                ConsumeConcurrentlyContext context) {
                System.out.printf(Thread.currentThread().getName() + " Receive1 New Messages: " + msgs + "%n");
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();

        System.out.printf("Consumer Started.%n");
    }
}