rocketMQ-consumer源碼

RocketMQ 服務器與消費之之前的消息傳送方式分爲拉模式和推模式，其中推模式是基於拉模式實現的，一個拉取任務完成之後立刻開始下一個拉取任務。

消費模式分爲集羣消費（clustering）和廣播消費（broadcasting）。

集羣模式，同一個主題下的同一條消息只能被一個集羣中的一個消費者消費；
廣播模式，同一個主題下的同一條消息要被集羣中所有消費者消費。

啓動

以DefaultMQPushConsumerImpl的start方法爲例

獲取訂閱信息SubscriptionData，並放入RebalanceImp的訂閱信息中
訂閱信息來源於：

調用DefaultMQPushConsumerImpl。subscribe方法訂閱
訂閱重試主題：%RETRY%+消費組名，消費者在啓動時會自動訂閱這個主題

private void copySubscription() throws MQClientException {
    try {
        Map<String, String> sub = this.defaultMQPushConsumer.getSubscription();
        if (sub != null) {
            for (final Map.Entry<String, String> entry : sub.entrySet()) {
                final String topic = entry.getKey();
                final String subString = entry.getValue();
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    topic, subString);
                this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
            }
        }

        if (null == this.messageListenerInner) {
            this.messageListenerInner = this.defaultMQPushConsumer.getMessageListener();
        }

        switch (this.defaultMQPushConsumer.getMessageModel()) {
            case /BROADCASTING/:
                break;
            case /CLUSTERING/:
                final String retryTopic = MixAll./getRetryTopic/(this.defaultMQPushConsumer.getConsumerGroup());
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    retryTopic, SubscriptionData./SUB_ALL/);
                this.rebalanceImpl.getSubscriptionInner().put(retryTopic, subscriptionData);
                break;
            default:
                break;
        }
    } catch (Exception e) {
        throw new MQClientException(“subscription exception”, e);
    }
}

初始化MqClientInstance和rebalanceImpl

if (this.defaultMQPushConsumer.getMessageModel() == MessageModel./CLUSTERING/) {
    this.defaultMQPushConsumer.changeInstanceNameToPID();
}

this.mQClientFactory = MQClientManager./getInstance/().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);

this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

this.pullAPIWrapper = new PullAPIWrapper(
    mQClientFactory,
    this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

初始化消息進度，如是集羣消費，則消息進度存儲在broker，如是廣播消費則在consumer

if (this.defaultMQPushConsumer.getOffsetStore() != null) {
    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case /BROADCASTING/:
            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        case /CLUSTERING/:
            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        default:
            break;
    }
    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();

根據是否是順序消費啓動ConsumeMessageOrderlyService或ConsumeMessageConcurrentlyService服務線程負責消息消費，內部維護一個線程池

if (this.getMessageListenerInner() instance MessageListenerOrderly) {
    this.consumeOrderly = true;
    this.consumeMessageService =
        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instance MessageListenerConcurrently) {
    this.consumeOrderly = false;
    this.consumeMessageService =
        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}

this.consumeMessageService.start();

向MQClientInstance註冊消費者，並啓動MQClientInstance。一個JVM中所有生產者和消費者公用一個MQClientInstance，MQClientInstance只會啓動一次。

boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
    this.serviceState = ServiceState./CREATE_JUST/;
    this.consumeMessageService.shutdown();
    throw new MQClientException(“The consumer group[“ + this.defaultMQPushConsumer.getConsumerGroup()
        + “] has been created before, specify another name please.” + FAQUrl./suggestTodo/(FAQUrl./GROUP_NAME_DUPLICATE_URL/),
        null);
}

mQClientFactory.start();

消息拉取

在MQClientINstance啓動的時候，啓動了pullMessageService服務。pull消費模式不同，pull方法是客戶端自己手動調用，不會在隊列中放pullRequset對象
在它的run方法中可以看到

public void run() {
    log.info(this.getServiceName() + “ service started”);

    while (!this.isStopped()) {
        try {
            PullRequest pullRequest = this.pullRequestQueue.take();
            this.pullMessage(pullRequest);
        } catch (InterruptedException ignored) {
        } catch (Exception e) {
            log.error(“Pull Message Service Run Method exception”, e);
        }
    }

    log.info(this.getServiceName() + “ service end”);
}

只要服務沒有停止，就不停的從pullRequestQueue中take出pullRequest任務。pullRequestQueue是一個阻塞隊列

LinkedBlockingQueue<PullRequest> pullRequestQueue

pullRequest是每一個消息隊列一個pullRequest，放進去的時機：

執行完一次拉取任務之後又將pullRequest放入到pullRequestQueue
真正創建是在RebalanceImpl中

PullRequest

private String consumerGroup; //消費組
private MessageQueue messageQueue; //待拉取消息隊列
private ProcessQueue processQueue; //消息處理隊列，從broker拉取的消息先存放在這裏再提交到消費者消費線程池消費，是messageQueue在消費者的快照
private long nextOffset; //待拉取的messageQueue偏移量
private boolean lockedFirst = false; //是否被鎖定

ProcessQueue

processQueueTable是map結構，key是MessageQueue，value是processQueue，Rebalance做負載均衡時會把consumer分配到的messageQueue和processTable做對比，將匹配上的processQueue構造一個PullRequest，獲取下一個消息的offset設置到pullRequset中，並把pullrequest放入PullrequestQueue。

ProcessQueue是messageQueue的快照，保存了從broker拉取了但是還沒有消費的消息。有一個讀寫鎖控制併發，一個TreeMap保存拉取的消息

PullMessageService每次從消息服務器默認拉取32條消息，按照消息隊列偏移量順序存放在processQueue，然後PullMessageService將消息提交到消費者消費線程池，消費成功之後從processQueue中移除。

public final static long /REBALANCE_LOCK_MAX_LIVE_TIME/=
    Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockMaxLiveTime", "30000"));
public final static long /REBALANCE_LOCK_INTERVAL/= Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockInterval", "20000"));
private final static long /PULL_MAX_IDLE_TIME/= Long./parseLong/(System./getProperty/("rocketmq.client.pull.pullMaxIdleTime", "120000"));
private final InternalLogger log = ClientLogger./getLog/();
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();//消息存儲容器，鍵爲消息在consumerQueue中的偏移量，value爲消息實體
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
//**/
/ * A subset of msgTreeMap, will only be used when orderly consume/
/ *//
private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>();//消息臨時存儲容器，用於處理順序消息，從processQueue中的msgTressMap中取出消息前，先臨時存儲在這裏
private final AtomicLong tryUnlockTimes = new AtomicLong(0);
private volatile long queueOffsetMax = 0L;
private volatile boolean dropped = false;
private volatile long lastPullTimestamp = System./currentTimeMillis/();
private volatile long lastConsumeTimestamp = System./currentTimeMillis/();
private volatile boolean locked = false;
private volatile long lastLockTimestamp = System./currentTimeMillis/();
private volatile boolean consuming = false;
private volatile long msgAccCnt = 0;

消息拉取流程

consumer端

主要入口在DefaultMQPushConsumerImpl的pullMessage
方法。

檢查processQueue的狀態
判斷是否觸發流控，如果出發流量控制，則延遲50ms執行，流控的觸發點有：
- processQueue當前處理的消息條數超過了pullThresholdForQueue = 1000
- processQueue當前處理的消息大小超過了pullThresholdSizeForQueue = 100M
- 順序消費時，processQueue中隊列最大偏移量與最小偏移量間距超過consumeConcurrentlyMaxSpan = 2000，主要擔心一條消息堵塞，消息進度無法向前推進，從而可能造成大量消息重複消費
根據主題獲取訂閱信息,如果爲空，則延遲3s執行
構建消息拉取系統標記PullSysFlag

private final static int /FLAG_COMMIT_OFFSET/= 0x1 << 0;//從內存中讀取的消費進度大於0
private final static int /FLAG_SUSPEND/= 0x1 << 1;//消息拉取時支持掛起
private final static int /FLAG_SUBSCRIPTION/= 0x1 << 2;//消息過濾機制爲表達式，這位置1
private final static int /FLAG_CLASS_FILTER/= 0x1 << 3;//消息過濾機制爲類過濾模式，這位置1

this.pullAPIWrapper.pullKernelImpl
1. 從MQClientInstance中獲取Broker地址。相同名稱的broker構成主從結構，brokerId不同。
2. 如果過濾模式位類過濾，從filterServer上拉取消息，否則從broker上拉取
3. 調用this.mQClientFactory.getMQClientAPIImpl().pullMessage異步向broker拉取消息

public PullResult pullKernelImpl(
    final MessageQueue mq,
    final String subExpression,
    final String expressionType,
    final long subVersion,
    final long offset,
    final int maxNums,
    final int sysFlag,
    final long commitOffset,
    final long brokerSuspendMaxTimeMillis,
    final long timeoutMillis,
    final CommunicationMode communicationMode,
    final PullCallback pullCallback
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    FindBrokerResult findBrokerResult =
        this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
            this.recalculatePullFromWhichNode(mq), false);
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult =
            this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
                this.recalculatePullFromWhichNode(mq), false);
    }

    if (findBrokerResult != null) {
        {
            // check version
            if (!ExpressionType./isTagType/(expressionType)
                && findBrokerResult.getBrokerVersion() < MQVersion.Version./V4_1_0_SNAPSHOT/.ordinal()) {
                throw new MQClientException("The broker[" + mq.getBrokerName() + ", "
                    + findBrokerResult.getBrokerVersion() + "] does not upgrade to support for filter message by " + expressionType, null);
            }
        }
        int sysFlagInner = sysFlag;

        if (findBrokerResult.isSlave()) {
            sysFlagInner = PullSysFlag./clearCommitOffsetFlag/(sysFlagInner);
        }

        PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
        requestHeader.setConsumerGroup(this.consumerGroup);
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setQueueOffset(offset);
        requestHeader.setMaxMsgNums(maxNums);
        requestHeader.setSysFlag(sysFlagInner);
        requestHeader.setCommitOffset(commitOffset);
        requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
        requestHeader.setSubscription(subExpression);
        requestHeader.setSubVersion(subVersion);
        requestHeader.setExpressionType(expressionType);

        String brokerAddr = findBrokerResult.getBrokerAddr();
        if (PullSysFlag./hasClassFilterFlag/(sysFlagInner)) {
            brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
        }

        PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
            brokerAddr,
            requestHeader,
            timeoutMillis,
            communicationMode,
            pullCallback);

        return pullResult;
    }

    throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}

pullMessage方法：

public void pullMessage(final PullRequest pullRequest) {
    final ProcessQueue processQueue = pullRequest.getProcessQueue();
/**1. 檢查processQueue的狀態*/
    if (processQueue.isDropped()) {
        log.info("the pull request[{}] is dropped.", pullRequest.toString());
        return;
    }

    pullRequest.getProcessQueue().setLastPullTimestamp(System./currentTimeMillis/());

    try {
        this.makeSureStateOK();
    } catch (MQClientException e) {
        log.warn("pullMessage exception, consumer state not ok", e);
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        return;
    }

    if (this.isPause()) {
        log.warn("consumer was paused, execute pull request later. instanceName={}, group={}", this.defaultMQPushConsumer.getInstanceName(), this.defaultMQPushConsumer.getConsumerGroup());
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_SUSPEND/);
        return;
    }
/**2. 判斷是否觸發流控，如果出發流量控制，則延遲50ms執行*/
    long cachedMessageCount = processQueue.getMsgCount().get();
    long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);

    if (cachedMessageCount > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message count exceeds the threshold {}, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (cachedMessageSizeInMiB > this.defaultMQPushConsumer.getPullThresholdSizeForQueue()) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message size exceeds the threshold {} MiB, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdSizeForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (!this.consumeOrderly) {
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
            if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
                log.warn(
                    "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
                    processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
                    pullRequest, queueMaxSpanFlowControlTimes);
            }
            return;
        }
    } else {
        if (processQueue.isLocked()) {
            if (!pullRequest.isLockedFirst()) {
                final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
                boolean brokerBusy = offset < pullRequest.getNextOffset();
                log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
                    pullRequest, offset, brokerBusy);
                if (brokerBusy) {
                    log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
                        pullRequest, offset);
                }

                pullRequest.setLockedFirst(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
            this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
            log.info("pull message later because not locked in broker, {}", pullRequest);
            return;
        }
    }
/**3. 根據主題獲取訂閱信息*/
    final SubscriptionData subscriptionData = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (null == subscriptionData) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        log.warn("find the consumer's subscription failed, {}", pullRequest);
        return;
    }

    final long beginTimestamp = System./currentTimeMillis/();

    PullCallback pullCallback = new PullCallback() {
        @Override
        public void onSuccess(PullResult pullResult) {
            if (pullResult != null) {
                pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                    subscriptionData);

                switch (pullResult.getPullStatus()) {
                    case /FOUND/:
                        long prevRequestOffset = pullRequest.getNextOffset();
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                        long pullRT = System./currentTimeMillis/() - beginTimestamp;
                        DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                            pullRequest.getMessageQueue().getTopic(), pullRT);

                        long firstMsgOffset = Long./MAX_VALUE/;
                        if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        } else {
                            firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

                            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                            boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                            DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                pullResult.getMsgFoundList(),
                                processQueue,
                                pullRequest.getMessageQueue(),
                                dispatchToConsume);

                            if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                    DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                            } else {
                                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                            }
                        }

                        if (pullResult.getNextBeginOffset() < prevRequestOffset
                            || firstMsgOffset < prevRequestOffset) {
                            log.warn(
                                "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                                pullResult.getNextBeginOffset(),
                                firstMsgOffset,
                                prevRequestOffset);
                        }

                        break;
                    case /NO_NEW_MSG/:
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        break;
                    case /NO_MATCHED_MSG/:
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        break;
                    case /OFFSET_ILLEGAL/:
                        log.warn("the pull request offset illegal, {} {}",
                            pullRequest.toString(), pullResult.toString());
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        pullRequest.getProcessQueue().setDropped(true);
                        DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

                            @Override
                            public void run() {
                                try {
                                    DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                        pullRequest.getNextOffset(), false);

                                    DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                                    DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                                    log.warn("fix the pull request offset, {}", pullRequest);
                                } catch (Throwable e) {
                                    log.error("executeTaskLater Exception", e);
                                }
                            }
                        }, 10000);
                        break;
                    default:
                        break;
                }
            }
        }

        @Override
        public void onException(Throwable e) {
            if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll./RETRY_GROUP_TOPIC_PREFIX/)) {
                log.warn("execute the pull request exception", e);
            }

            DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        }
    };

    boolean commitOffsetEnable = false;
    long commitOffsetValue = 0L;
    if (MessageModel./CLUSTERING/== this.defaultMQPushConsumer.getMessageModel()) {
        commitOffsetValue = this.offsetStore.readOffset(pullRequest.getMessageQueue(), ReadOffsetType./READ_FROM_MEMORY/);
        if (commitOffsetValue > 0) {
            commitOffsetEnable = true;
        }
    }

    String subExpression = null;
    boolean classFilter = false;
    SubscriptionData sd = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (sd != null) {
        if (this.defaultMQPushConsumer.isPostSubscriptionWhenPull() && !sd.isClassFilterMode()) {
            subExpression = sd.getSubString();
        }

        classFilter = sd.isClassFilterMode();
    }
/**4. 構建消息拉取系統標記*/
    int sysFlag = PullSysFlag./buildSysFlag/(
        commitOffsetEnable, // commitOffset
        true, // suspend
        subExpression != null, // subscription
        classFilter // class filter
    );
    try {
        this.pullAPIWrapper.pullKernelImpl(
            pullRequest.getMessageQueue(),
            subExpression,
            subscriptionData.getExpressionType(),
            subscriptionData.getSubVersion(),
            pullRequest.getNextOffset(),
            this.defaultMQPushConsumer.getPullBatchSize(),
            sysFlag,
            commitOffsetValue,
            /BROKER_SUSPEND_MAX_TIME_MILLIS/,
            /CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND/,
            CommunicationMode./ASYNC/,
            pullCallback
        );
    } catch (Exception e) {
        log.error("pullKernelImpl exception", e);
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
    }
}

broker端

根據命定Code RequestCode.PULL_MESSAGE 找到broker端的處理消息拉取的類爲PullMessageProcessor

final GetMessageResult getMessageResult =
    this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
        requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);

拉取成功回調

拉取消息成功之後會回調註冊的回調函數

這個方法會處理返回回來的消息，如果有tag，再進行一次tag的過濾,把二進制的消息內容轉爲MessageExt對象

DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
            subscriptionData);

如果是FOUND狀態，但是消息列表爲空，則DefaultMQPushConsumerImpl.this.executePullRequestImmediately方法把當前pullRequest再次放入pullRequestQueue中
消息列表不爲空時，將消息列表放入processQueue的msgTreeMap中，並且新建一個ConsumeRequest對象，submit到consumeExecutor線程池。
ConsumeRequest是一個Runnable，run()方法執行實際的消費邏輯

@Override
public void onSuccess(PullResult pullResult) {
    if (pullResult != null) {
        pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
            subscriptionData);

        switch (pullResult.getPullStatus()) {
            case *FOUND*:
                long prevRequestOffset = pullRequest.getNextOffset();
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                long pullRT = System.*currentTimeMillis*() - beginTimestamp;
                DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                    pullRequest.getMessageQueue().getTopic(), pullRT);

                long firstMsgOffset = Long.*MAX_VALUE*;
                if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                    DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                } else {
                    firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

                    DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                        pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                    boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                    DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                        pullResult.getMsgFoundList(),
                        processQueue,
                        pullRequest.getMessageQueue(),
                        dispatchToConsume);

                    if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                            DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                    } else {
                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                    }
                }

                if (pullResult.getNextBeginOffset() < prevRequestOffset
                    || firstMsgOffset < prevRequestOffset) {
                    log.warn(
                        "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                        pullResult.getNextBeginOffset(),
                        firstMsgOffset,
                        prevRequestOffset);
                }

                break;
            case *NO_NEW_MSG*:
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                break;
            case *NO_MATCHED_MSG*:
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                break;
            case *OFFSET_ILLEGAL*:
                log.warn("the pull request offset illegal, {} {}",
                    pullRequest.toString(), pullResult.toString());
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                pullRequest.getProcessQueue().setDropped(true);
                DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

                    @Override
                    public void run() {
                        try {
                            DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                pullRequest.getNextOffset(), false);

                            DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                            DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                            log.warn("fix the pull request offset, {}", pullRequest);
                        } catch (Throwable e) {
                            log.error("executeTaskLater Exception", e);
                        }
                    }
                }, 10000);
                break;
            default:
                break;
        }
    }
}

消息隊列負載，重新分佈機制

由RebalanceService實現，每個MQClientInstance持有一個RebalanceService對象.這個線程每隔20s執行一次

this.mqClientFactory.doRebalance();

遍歷註冊的消費者，對每個消費者依次執行doRebalance() 方法。實際執行的是RebalanceImpl的方法。
每個DefaultMQPushConsumerImpl都持有一個RebalanceImpl對象，該方法遍歷訂閱信息對每個主題的隊列都進行重新負載。

Map<String, SubscriptionData> subTable在消費者調用subscribe方法時填充。

public void doRebalance(final boolean isOrder) {
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
                if (!topic.startsWith(MixAll./RETRY_GROUP_TOPIC_PREFIX/)) {
                    /log/.warn(“rebalanceByTopic Exception”, e);
                }
            }
        }
    }

    this.truncateMessageQueueNotMyTopic();
}

對每個topic進行負載均衡

從broker中獲取所有訂閱該topic的consumer的客戶端cid，這是MQClientInstance給Broker發心跳包時發送過去的
對所有的cid和消息隊列排序，保證所有消費者的視角一致
調用AllocateMessageQueueStrategy的allocate方法，獲取當前消費者可消費的消息隊列列表，負載均衡策略有：
1. AllocateMessageQueueAveragely，平均分配，推薦使用
2. AllocateMessageQueueAveragelyByCircle，輪訓平均分配，推薦使用
3. AllocateMessageQueueConsistentHash，一致性hash，不推薦，消息隊列負載信息不容易跟蹤
4. AllocateMessageQueueByMachineRoom，根據broker部署機房名，對每個消費者負責不同的broker上的隊列
5. AllocateMessageQueueByConfig，根據配置，爲每個消費者制定固定消息隊列
更新當前的負載隊列集合，如果隊列不在新負載均衡中，則停止消費，並保存消費進度。如果新隊列不在當前隊列中，則創建這個隊列的拉取任務pullRequest，添加到pullMessageService線程的pullRequestQueue中。在一個JVM中，同一個消費組同一個消息隊列只有一個pullRequest對象

  case *CLUSTERING*: {
        Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
        List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
        if (null == mqSet) {
            if (!topic.startsWith(MixAll.*RETRY_GROUP_TOPIC_PREFIX*)) {
                *log*.warn(“doRebalance, {}, but the topic[{}] not exist.”, consumerGroup, topic);
            }
        }

        if (null == cidAll) {
            *log*.warn(“doRebalance, {} {}, get consumer id list failed”, consumerGroup, topic);
        }

        if (mqSet != null && cidAll != null) {
            List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
            mqAll.addAll(mqSet);

            Collections.*sort*(mqAll);
            Collections.*sort*(cidAll);

            AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;

            List<MessageQueue> allocateResult = null;
            try {
                allocateResult = strategy.allocate(
                    this.consumerGroup,
                    this.mQClientFactory.getClientId(),
                    mqAll,
                    cidAll);
            } catch (Throwable e) {
                *log*.error(“AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}”, strategy.getName(),
                    e);
                return;
            }

            Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
            if (allocateResult != null) {
                allocateResultSet.addAll(allocateResult);
            }

            boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
            if (changed) {
                *log*.info(
                    “rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}”,
                    strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
                    allocateResultSet.size(), allocateResultSet);
                this.messageQueueChanged(topic, mqSet, allocateResultSet);
            }
        }
        break;
    }
    default:
        break;
}

updateProcessQueueTableInRebalance方法更新當前消費的消費隊列

private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
    final boolean isOrder) {
    boolean changed = false;

    Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<MessageQueue, ProcessQueue> next = it.next();
        MessageQueue mq = next.getKey();
        ProcessQueue pq = next.getValue();

        if (mq.getTopic().equals(topic)) {
            if (!mqSet.contains(mq)) {
                pq.setDropped(true);
                if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                    it.remove();
                    changed = true;
                    *log*.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                }
            } else if (pq.isPullExpired()) {
                switch (this.consumeType()) {
                    case *CONSUME_ACTIVELY*:
                        break;
                    case *CONSUME_PASSIVELY*:
                        pq.setDropped(true);
                        if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                            it.remove();
                            changed = true;
                            *log*.error("[BUG]doRebalance, {}, remove unnecessary mq, {}, because pull is pause, so try to fixed it",
                                consumerGroup, mq);
                        }
                        break;
                    default:
                        break;
                }
            }
        }
    }

    List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
    for (MessageQueue mq : mqSet) {
        if (!this.processQueueTable.containsKey(mq)) {
            if (isOrder && !this.lock(mq)) {
                *log*.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
                continue;
            }

            this.removeDirtyOffset(mq);
            ProcessQueue pq = new ProcessQueue();
            long nextOffset = this.computePullFromWhere(mq);
            if (nextOffset >= 0) {
                ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                if (pre != null) {
                    *log*.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
                } else {
                    *log*.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                    PullRequest pullRequest = new PullRequest();
                    pullRequest.setConsumerGroup(consumerGroup);
                    pullRequest.setNextOffset(nextOffset);
                    pullRequest.setMessageQueue(mq);
                    pullRequest.setProcessQueue(pq);
                    pullRequestList.add(pullRequest);
                    changed = true;
                }
            } else {
                *log*.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
            }
        }
    }

    this.dispatchPullRequest(pullRequestList);

    return changed;
}

消息消費

pullMessageService在拉取到消息之後，把消息存儲到processQueue消息處理隊列中，然後調用consumeMeaasgeService.submitConsumeRequest提交消息消費請求。由ConsumerMessageService來處理消息的消費.

ConsumerMessageService的實現類ConsumeMessageConcurrentlyService和ConsumeMessageOrderlyService，分別對應併發消費和順序消費。

併發消費：

private static final InternalLogger *log*= ClientLogger.*getLog*();
private final DefaultMQPushConsumerImpl defaultMQPushConsumerImpl;//消息推模式實現類
private final DefaultMQPushConsumer defaultMQPushConsumer;//消費者對象
private final MessageListenerConcurrently messageListener;//併發消息業務事件類
private final BlockingQueue<Runnable> consumeRequestQueue;//消息消費任務對了
private final ThreadPoolExecutor consumeExecutor;//消息消費線程池
private final String consumerGroup;//消費組

private final ScheduledExecutorService scheduledExecutorService;//添加消費任務到consumeExecutor延遲調度器
private final ScheduledExecutorService cleanExpireMsgExecutors;//定時刪除過期消息線程池

消息消費流程爲：

提交消費任務

public void submitConsumeRequest(
    final List<MessageExt> msgs,
    final ProcessQueue processQueue,
    final MessageQueue messageQueue,
    final boolean dispatchToConsume)

如果消息數量小於最大批次消息量，直接將消息放入consumeRequest中，調到consumeExecutor線程池，如果大於，則分批次提及
2. 進入consumeRequest的run方法，首先檢查processQueue的isDropped是否標記，如果標記則停止消費
3. 執行消費前鉤子函數

ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookBefore(consumeMessageContext);

恢復重試消息主題名,由於消息存入commitLog文件時，如果消息delayTimeLevel不爲0，會首先將充實主題存入在消息的屬性中，以便時間到了之後重新消費

ConsumeMessageConcurrentlyService.this.resetRetryTopic(msgs);

執行具體的消息消費，註冊消費者的時候注入的消息監聽器consumeMessage方法。

status = listener.consumeMessage(Collections.*unmodifiableList*(msgs), context);

執行鉤子函數

ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookAfter(consumeMessageContext);

再次檢查processQueue的isDropped狀態，如果爲true則不處理，也就是說如果消息消費過程中processQueue的isDropped狀態由false變爲true，消息會被重複消費
根據status處理消息消費結果，如果是集羣消費，並且狀態爲consume_later，向broker發送ACK，如果發送失敗，記錄消息消費失敗的消息和失敗消息的已經消費了的次數，並添加到延遲消費，5s中之後添加到線程池

this.scheduledExecutorService.schedule(new Runnable() {

    @Override
    public void run() {
        ConsumeMessageConcurrentlyService.this.submitConsumeRequest(msgs, processQueue, messageQueue, true);
    }
}, 5000, TimeUnit.*MILLISECONDS*);

然後從processQueue移除這一批消費的消息。

public void processConsumeResult(
    final ConsumeConcurrentlyStatus status,
    final ConsumeConcurrentlyContext context,
    final ConsumeRequest consumeRequest
) {
    int ackIndex = context.getAckIndex();

    if (consumeRequest.getMsgs().isEmpty())
        return;

    switch (status) {
        case *CONSUME_SUCCESS*:
            if (ackIndex >= consumeRequest.getMsgs().size()) {
                ackIndex = consumeRequest.getMsgs().size() - 1;
            }
            int ok = ackIndex + 1;
            int failed = consumeRequest.getMsgs().size() - ok;
            this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
            this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
            break;
        case *RECONSUME_LATER*:
            ackIndex = -1;
            this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
                consumeRequest.getMsgs().size());
            break;
        default:
            break;
    }

    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case *BROADCASTING*:
            for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                MessageExt msg = consumeRequest.getMsgs().get(i);
                *log*.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
            }
            break;
        case *CLUSTERING*:
            List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
            for (int I = ackIndex + 1; i < consumeRequest.getMsgs().size(); I++) {
                MessageExt msg = consumeRequest.getMsgs().get(i);
                boolean result = this.sendMessageBack(msg, context);
                if (!result) {
                    msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                    msgBackFailed.add(msg);
                }
            }

            if (!msgBackFailed.isEmpty()) {
                consumeRequest.getMsgs().removeAll(msgBackFailed);

                this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
            }
            break;
        default:
            break;
    }

    long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
    if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
        this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
    }
}

消息確認ACK

對於集羣消費，消費結果爲CONSUME_LATER的消息，會向broker發送消息確認

命令爲：RequestCode.CONSUMER_SEND_MSG_BACK

public boolean sendMessageBack(final MessageExt msg, final ConsumeConcurrentlyContext context) {
    int delayLevel = context.getDelayLevelWhenNextConsume();

    try {
        this.defaultMQPushConsumerImpl.sendMessageBack(msg, delayLevel, context.getMessageQueue().getBrokerName());
        return true;
    } catch (Exception e) {
        *log*.error(“sendMessageBack exception, group: “ + this.consumerGroup + “ msg: “ + msg.toString(), e);
    }

    return false;
}

Broker端處理類爲SendMessageProcessor

獲取SubscriptionGroupConfig，消費組訂閱信息配置信息存儲在broker的${Rocket_Home}/store/config/subscriptionGroup.json。
如果重試隊列數量小於1，說明不支持重試，直接返回成功。
創建重試主題，主題名稱%RETRY%+消費組名，並從重試隊列中隨機選擇一個隊列，構建TopicConfig主題配置信息
從commitLog獲取消息，並設置消息的retry主題
設置消息的重試次數，如果此時的重試次數已經超過了最大重試次數，則將新的主題改爲%DLQ%,這個主題的權限爲只寫，說明消息一旦進入這個隊列，mq不再進行消費
根據之前的消息重新建一個新的msgId的消息，別的類容相同，並存入commitlog。如果delayTimeLevel>0，替換消息主題與隊列定時任務主題“SCHEDULE_TOPIC_XXXX”，隊列ID爲延遲級別-1.再將消息主題，隊列存入消息屬性中，鍵爲PROPERTY_REAL_TOPIC,PROPERTY_REAL_QUEUE_ID

if (msg.getDelayTimeLevel() > 0) {
    if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
        msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
    }

    topic = ScheduleMessageService.*SCHEDULE_TOPIC*;
    queueId = ScheduleMessageService.*delayLevel2QueueId*(msg.getDelayTimeLevel());

    // Backup real topic, queueId
    MessageAccessor.*putProperty*(msg, MessageConst.*PROPERTY_REAL_TOPIC*, msg.getTopic());
    MessageAccessor.*putProperty*(msg, MessageConst.*PROPERTY_REAL_QUEUE_ID*, String.*valueOf*(msg.getQueueId()));
    msg.setPropertiesString(MessageDecoder.*messageProperties2String*(msg.getProperties()));

    msg.setTopic(topic);
    msg.setQueueId(queueId);
}

消費進度管理

管理類爲OffsetStore。具體的實現有

LocalFileOffsetStore這個是廣播消費模式的消費進度管理，進度保存在消費者本地
RemoteBrokerOffsetStore，由於集羣消費模式，一條消息只能被一個消費者消費，因此進度保存在broker中

定時消息

由於任意時間間隔的定時消息需要排序，效率低，rocketmq只支持固定延遲級別的定時消息。
在消息存入commitlog時，如果消息的delayLevel>0，會設置新的主題SCHEDULE_TOPIC_XXXX和消息隊列id=delayLevel-1

rocketMQ給每個延遲級別創建一個定時任務，按照延遲時間定時去對應的延遲消息隊列拉取消息，然後從commitlog中加載完整的消息，清楚延遲屬性，恢復原先的主題和隊列，再次創建一條新的消息放入commitlog中供消費者消費。

實現類是ScheduleMessageService，在start方法中啓動這個service和定時任務。

rocketMQ-consumer源碼

啓動

消息拉取

PullRequest

ProcessQueue

消息拉取流程

consumer端

broker端

拉取成功回調

消息隊列負載，重新分佈機制

消息消費

消息確認ACK

消費進度管理

定時消息

測試人員都是畫畫大神，讓我看看誰還不會用代碼圖？

Object.values()對象遍歷

網絡現代化通向雲原生應用的高速公路

面試官：說說你對序列化的理解

rocketMQ-consumer源碼

rocketMQ-consumer

rockeMQ-producer源碼

rocketMQ消息存儲源碼

sublime text3 markdown

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結