RocketMQ——消費過程

PullMessageService負責拉取消息，從遠端服務器拉取消息後存儲到ProcessQueue中，然後調用ConsumeMessageService#submitConsumeRequest 方法進行消費，適應小城池來消費消息，確保消息拉取與消息消費的解耦。

 

1.jpg

消息消費

ConsumeMessageConcurrentlyService#submitConsumeRequest 負責提交消費請求

 @Override
    public void submitConsumeRequest(
        final List<MessageExt> msgs,
        final ProcessQueue processQueue,
        final MessageQueue messageQueue,
        final boolean dispatchToConsume) {
        //獲取消費批次
        final int consumeBatchSize = this.defaultMQPushConsumer.getConsumeMessageBatchMaxSize();
        //小於最大消息數則直接提交
        if (msgs.size() <= consumeBatchSize) {
            ConsumeRequest consumeRequest = new ConsumeRequest(msgs, processQueue, messageQueue);
                this.consumeExecutor.submit(consumeRequest);
        } else {
        //消息切割提交
            for (int total = 0; total < msgs.size(); ) {
                ConsumeRequest consumeRequest = new ConsumeRequest(msgThis, processQueue, messageQueue);
                try {
                    this.consumeExecutor.submit(consumeRequest);
                } catch (RejectedExecutionException e) {
                //失敗後重試
                    this.submitConsumeRequestLater(consumeRequest);
                }
            }
        }
    }

消息消費只是先提交到線程池中，每一個ConsumeRequest 爲一個線程。
ConsumeMessageConcurrentlyService$ConsumeRequest#run 整體流程如下

 

2.jpg

消費進度管理

• 集羣模式：消費進度保存在broker，需要每個消費端都可以訪問到
• 廣播模式：消息進度存儲在本地，只需要本機訪問即可。

核心類圖

 

 

3.jpg

 

廣播模式消息進度存儲

org.apache.rocketmq.client.consumer.store.LocalFileOffsetStore
存儲位置

 public final static String LOCAL_OFFSET_STORE_DIR = System.getProperty(
        "rocketmq.client.localOffsetStoreDir",
        System.getProperty("user.home") + File.separator + ".rocketmq_offsets");

MQClientInstance#startScheduledTask

       this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                try {
                    MQClientInstance.this.persistAllConsumerOffset();
                } catch (Exception e) {
                    log.error("ScheduledTask persistAllConsumerOffset exception", e);
                }
            }
        }, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

每隔十秒持久化一次。

集羣模式消息進度存儲

org.apache.rocketmq.client.consumer.store.RemoteBrokerOffsetStore

    private long fetchConsumeOffsetFromBroker(MessageQueue mq) throws RemotingException, MQBrokerException,
        InterruptedException, MQClientException {
        FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
        if (null == findBrokerResult) {

            this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
            findBrokerResult = this.mQClientFactory.findBrokerAddressInAdmin(mq.getBrokerName());
        }

        if (findBrokerResult != null) {
            QueryConsumerOffsetRequestHeader requestHeader = new QueryConsumerOffsetRequestHeader();
            requestHeader.setTopic(mq.getTopic());
            requestHeader.setConsumerGroup(this.groupName);
            requestHeader.setQueueId(mq.getQueueId());

            return this.mQClientFactory.getMQClientAPIImpl().queryConsumerOffset(
                findBrokerResult.getBrokerAddr(), requestHeader, 1000 * 5);
        } else {
            throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
        }
    }

所以mq一個隊列同一時刻也只允許一個消費者消費，也是避免這裏更新的併發問題了。