rocketMQ-consumer

RocketMQ 服務器與消費之之前的消息傳送方式分爲拉模式和推模式，其中推模式是基於拉模式實現的，一個拉取任務完成之後立刻開始下一個拉取任務。

消費模式分爲集羣消費（clustering）和廣播消費（broadcasting）。

集羣模式，同一個主題下的同一條消息只能被一個集羣中的一個消費者消費；
廣播模式，同一個主題下的同一條消息要被集羣中所有消費者消費。

啓動

以DefaultMQPushConsumerImpl的start方法爲例

獲取訂閱信息SubscriptionData，並放入RebalanceImp的訂閱信息中
訂閱信息來源於：

調用DefaultMQPushConsumerImpl。subscribe方法訂閱
訂閱重試主題：%RETRY%+消費組名，消費者在啓動時會自動訂閱這個主題

private void copySubscription() throws MQClientException {
    try {
        Map<String, String> sub = this.defaultMQPushConsumer.getSubscription();
        if (sub != null) {
            for (final Map.Entry<String, String> entry : sub.entrySet()) {
                final String topic = entry.getKey();
                final String subString = entry.getValue();
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    topic, subString);
                this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
            }
        }

        if (null == this.messageListenerInner) {
            this.messageListenerInner = this.defaultMQPushConsumer.getMessageListener();
        }

        switch (this.defaultMQPushConsumer.getMessageModel()) {
            case /BROADCASTING/:
                break;
            case /CLUSTERING/:
                final String retryTopic = MixAll./getRetryTopic/(this.defaultMQPushConsumer.getConsumerGroup());
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    retryTopic, SubscriptionData./SUB_ALL/);
                this.rebalanceImpl.getSubscriptionInner().put(retryTopic, subscriptionData);
                break;
            default:
                break;
        }
    } catch (Exception e) {
        throw new MQClientException(“subscription exception”, e);
    }
}

初始化MqClientInstance和rebalanceImpl

if (this.defaultMQPushConsumer.getMessageModel() == MessageModel./CLUSTERING/) {
    this.defaultMQPushConsumer.changeInstanceNameToPID();
}

this.mQClientFactory = MQClientManager./getInstance/().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);

this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

this.pullAPIWrapper = new PullAPIWrapper(
    mQClientFactory,
    this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

初始化消息進度，如是集羣消費，則消息進度存儲在broker，如是廣播消費則在consumer

if (this.defaultMQPushConsumer.getOffsetStore() != null) {
    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case /BROADCASTING/:
            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        case /CLUSTERING/:
            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        default:
            break;
    }
    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();

根據是否是順序消費啓動ConsumeMessageOrderlyService或ConsumeMessageConcurrentlyService服務線程負責消息消費，內部維護一個線程池

if (this.getMessageListenerInner() instance MessageListenerOrderly) {
    this.consumeOrderly = true;
    this.consumeMessageService =
        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instance MessageListenerConcurrently) {
    this.consumeOrderly = false;
    this.consumeMessageService =
        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}

this.consumeMessageService.start();

向MQClientInstance註冊消費者，並啓動MQClientInstance。一個JVM中所有生產者和消費者公用一個MQClientInstance，MQClientInstance只會啓動一次。

boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
    this.serviceState = ServiceState./CREATE_JUST/;
    this.consumeMessageService.shutdown();
    throw new MQClientException(“The consumer group[“ + this.defaultMQPushConsumer.getConsumerGroup()
        + “] has been created before, specify another name please.” + FAQUrl./suggestTodo/(FAQUrl./GROUP_NAME_DUPLICATE_URL/),
        null);
}

mQClientFactory.start();

消息拉取

在MQClientINstance啓動的時候，啓動了pullMessageService服務。
在它的run方法中可以看到

public void run() {
    log.info(this.getServiceName() + “ service started”);

    while (!this.isStopped()) {
        try {
            PullRequest pullRequest = this.pullRequestQueue.take();
            this.pullMessage(pullRequest);
        } catch (InterruptedException ignored) {
        } catch (Exception e) {
            log.error(“Pull Message Service Run Method exception”, e);
        }
    }

    log.info(this.getServiceName() + “ service end”);
}

只要服務沒有停止，就不停的從pullRequestQueue中take出pullRequest任務。pullRequestQueue是一個阻塞隊列

LinkedBlockingQueue<PullRequest> pullRequestQueue

pullRequest放進去的時機：

執行完一次拉取任務之後又將pullRequest放入到pullRequestQueue
真正創建是在RebalanceImpl中

PullRequest

private String consumerGroup; //消費組
private MessageQueue messageQueue; //待拉取消息隊列
private ProcessQueue processQueue; //消息處理隊列，從broker拉取的消息先存放在這裏再提交到消費者消費線程池消費，是messageQueue在消費者的快照
private long nextOffset; //待拉取的messageQueue偏移量
private boolean lockedFirst = false; //是否被鎖定

ProcessQueue

PullMessageService每次從消息服務器默認拉取32條消息，按照消息隊列偏移量順序存放在processQueue，然後PullMessageService將消息提交到消費者消費線程池，消費成功之後從processQueue中移除。

public final static long /REBALANCE_LOCK_MAX_LIVE_TIME/=
    Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockMaxLiveTime", "30000"));
public final static long /REBALANCE_LOCK_INTERVAL/= Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockInterval", "20000"));
private final static long /PULL_MAX_IDLE_TIME/= Long./parseLong/(System./getProperty/("rocketmq.client.pull.pullMaxIdleTime", "120000"));
private final InternalLogger log = ClientLogger./getLog/();
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();//消息存儲容器，鍵爲消息在consumerQueue中的偏移量，value爲消息實體
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
//**/
/ * A subset of msgTreeMap, will only be used when orderly consume/
/ *//
private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>();//消息臨時存儲容器，用於處理順序消息，從processQueue中的msgTressMap中取出消息前，先臨時存儲在這裏
private final AtomicLong tryUnlockTimes = new AtomicLong(0);
private volatile long queueOffsetMax = 0L;
private volatile boolean dropped = false;
private volatile long lastPullTimestamp = System./currentTimeMillis/();
private volatile long lastConsumeTimestamp = System./currentTimeMillis/();
private volatile boolean locked = false;
private volatile long lastLockTimestamp = System./currentTimeMillis/();
private volatile boolean consuming = false;
private volatile long msgAccCnt = 0;

消息拉取流程

consumer端

主要入口在DefaultMQPushConsumerImpl的pullMessage
方法。

檢查processQueue的狀態
判斷是否觸發流控，如果出發流量控制，則延遲50ms執行，流控的觸發點有：
- processQueue當前處理的消息條數超過了pullThresholdForQueue = 1000
- processQueue當前處理的消息大小超過了pullThresholdSizeForQueue = 100M
- 非順序消費時，processQueue中隊列最大偏移量與最小偏移量間距超過consumeConcurrentlyMaxSpan = 2000，主要擔心一條消息堵塞，消息進度無法向前推進，從而可能造成大量消息重複消費
根據主題獲取訂閱信息,如果爲空，則延遲3s執行
構建消息拉取系統標記PullSysFlag

private final static int /FLAG_COMMIT_OFFSET/= 0x1 << 0;//從內存中讀取的消費進度大於0
private final static int /FLAG_SUSPEND/= 0x1 << 1;//消息拉取時支持掛起
private final static int /FLAG_SUBSCRIPTION/= 0x1 << 2;//消息過濾機制爲表達式，這位置1
private final static int /FLAG_CLASS_FILTER/= 0x1 << 3;//消息過濾機制爲類過濾模式，這位置1

this.pullAPIWrapper.pullKernelImpl
1. 從MQClientInstance中獲取Broker地址。相同名稱的broker構成主從結構，brokerId不同。
2. 如果過濾模式位類過濾，從filterServer上拉取消息，否則從broker上拉取
3. 調用this.mQClientFactory.getMQClientAPIImpl().pullMessage異步向broker拉取消息

public PullResult pullKernelImpl(
    final MessageQueue mq,
    final String subExpression,
    final String expressionType,
    final long subVersion,
    final long offset,
    final int maxNums,
    final int sysFlag,
    final long commitOffset,
    final long brokerSuspendMaxTimeMillis,
    final long timeoutMillis,
    final CommunicationMode communicationMode,
    final PullCallback pullCallback
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    FindBrokerResult findBrokerResult =
        this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
            this.recalculatePullFromWhichNode(mq), false);
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult =
            this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
                this.recalculatePullFromWhichNode(mq), false);
    }

    if (findBrokerResult != null) {
        {
            // check version
            if (!ExpressionType./isTagType/(expressionType)
                && findBrokerResult.getBrokerVersion() < MQVersion.Version./V4_1_0_SNAPSHOT/.ordinal()) {
                throw new MQClientException("The broker[" + mq.getBrokerName() + ", "
                    + findBrokerResult.getBrokerVersion() + "] does not upgrade to support for filter message by " + expressionType, null);
            }
        }
        int sysFlagInner = sysFlag;

        if (findBrokerResult.isSlave()) {
            sysFlagInner = PullSysFlag./clearCommitOffsetFlag/(sysFlagInner);
        }

        PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
        requestHeader.setConsumerGroup(this.consumerGroup);
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setQueueOffset(offset);
        requestHeader.setMaxMsgNums(maxNums);
        requestHeader.setSysFlag(sysFlagInner);
        requestHeader.setCommitOffset(commitOffset);
        requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
        requestHeader.setSubscription(subExpression);
        requestHeader.setSubVersion(subVersion);
        requestHeader.setExpressionType(expressionType);

        String brokerAddr = findBrokerResult.getBrokerAddr();
        if (PullSysFlag./hasClassFilterFlag/(sysFlagInner)) {
            brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
        }

        PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
            brokerAddr,
            requestHeader,
            timeoutMillis,
            communicationMode,
            pullCallback);

        return pullResult;
    }

    throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}

pullMessage方法：

public void pullMessage(final PullRequest pullRequest) {
    final ProcessQueue processQueue = pullRequest.getProcessQueue();
/**1. 檢查processQueue的狀態*/
    if (processQueue.isDropped()) {
        log.info("the pull request[{}] is dropped.", pullRequest.toString());
        return;
    }

    pullRequest.getProcessQueue().setLastPullTimestamp(System./currentTimeMillis/());

    try {
        this.makeSureStateOK();
    } catch (MQClientException e) {
        log.warn("pullMessage exception, consumer state not ok", e);
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        return;
    }

    if (this.isPause()) {
        log.warn("consumer was paused, execute pull request later. instanceName={}, group={}", this.defaultMQPushConsumer.getInstanceName(), this.defaultMQPushConsumer.getConsumerGroup());
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_SUSPEND/);
        return;
    }
/**2. 判斷是否觸發流控，如果出發流量控制，則延遲50ms執行*/
    long cachedMessageCount = processQueue.getMsgCount().get();
    long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);

    if (cachedMessageCount > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message count exceeds the threshold {}, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (cachedMessageSizeInMiB > this.defaultMQPushConsumer.getPullThresholdSizeForQueue()) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message size exceeds the threshold {} MiB, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdSizeForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (!this.consumeOrderly) {
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
            if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
                log.warn(
                    "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
                    processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
                    pullRequest, queueMaxSpanFlowControlTimes);
            }
            return;
        }
    } else {
        if (processQueue.isLocked()) {
            if (!pullRequest.isLockedFirst()) {
                final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
                boolean brokerBusy = offset < pullRequest.getNextOffset();
                log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
                    pullRequest, offset, brokerBusy);
                if (brokerBusy) {
                    log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
                        pullRequest, offset);
                }

                pullRequest.setLockedFirst(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
            this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
            log.info("pull message later because not locked in broker, {}", pullRequest);
            return;
        }
    }
/**3. 根據主題獲取訂閱信息*/
    final SubscriptionData subscriptionData = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (null == subscriptionData) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        log.warn("find the consumer's subscription failed, {}", pullRequest);
        return;
    }

    final long beginTimestamp = System./currentTimeMillis/();

    PullCallback pullCallback = new PullCallback() {
        @Override
        public void onSuccess(PullResult pullResult) {
            if (pullResult != null) {
                pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                    subscriptionData);

                switch (pullResult.getPullStatus()) {
                    case /FOUND/:
                        long prevRequestOffset = pullRequest.getNextOffset();
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                        long pullRT = System./currentTimeMillis/() - beginTimestamp;
                        DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                            pullRequest.getMessageQueue().getTopic(), pullRT);

                        long firstMsgOffset = Long./MAX_VALUE/;
                        if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        } else {
                            firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

                            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                            boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                            DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                pullResult.getMsgFoundList(),
                                processQueue,
                                pullRequest.getMessageQueue(),
                                dispatchToConsume);

                            if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                    DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                            } else {
                                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                            }
                        }

                        if (pullResult.getNextBeginOffset() < prevRequestOffset
                            || firstMsgOffset < prevRequestOffset) {
                            log.warn(
                                "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                                pullResult.getNextBeginOffset(),
                                firstMsgOffset,
                                prevRequestOffset);
                        }

                        break;
                    case /NO_NEW_MSG/:
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        break;
                    case /NO_MATCHED_MSG/:
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        break;
                    case /OFFSET_ILLEGAL/:
                        log.warn("the pull request offset illegal, {} {}",
                            pullRequest.toString(), pullResult.toString());
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        pullRequest.getProcessQueue().setDropped(true);
                        DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

                            @Override
                            public void run() {
                                try {
                                    DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                        pullRequest.getNextOffset(), false);

                                    DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                                    DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                                    log.warn("fix the pull request offset, {}", pullRequest);
                                } catch (Throwable e) {
                                    log.error("executeTaskLater Exception", e);
                                }
                            }
                        }, 10000);
                        break;
                    default:
                        break;
                }
            }
        }

        @Override
        public void onException(Throwable e) {
            if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll./RETRY_GROUP_TOPIC_PREFIX/)) {
                log.warn("execute the pull request exception", e);
            }

            DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        }
    };

    boolean commitOffsetEnable = false;
    long commitOffsetValue = 0L;
    if (MessageModel./CLUSTERING/== this.defaultMQPushConsumer.getMessageModel()) {
        commitOffsetValue = this.offsetStore.readOffset(pullRequest.getMessageQueue(), ReadOffsetType./READ_FROM_MEMORY/);
        if (commitOffsetValue > 0) {
            commitOffsetEnable = true;
        }
    }

    String subExpression = null;
    boolean classFilter = false;
    SubscriptionData sd = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (sd != null) {
        if (this.defaultMQPushConsumer.isPostSubscriptionWhenPull() && !sd.isClassFilterMode()) {
            subExpression = sd.getSubString();
        }

        classFilter = sd.isClassFilterMode();
    }
/**4. 構建消息拉取系統標記*/
    int sysFlag = PullSysFlag./buildSysFlag/(
        commitOffsetEnable, // commitOffset
        true, // suspend
        subExpression != null, // subscription
        classFilter // class filter
    );
    try {
        this.pullAPIWrapper.pullKernelImpl(
            pullRequest.getMessageQueue(),
            subExpression,
            subscriptionData.getExpressionType(),
            subscriptionData.getSubVersion(),
            pullRequest.getNextOffset(),
            this.defaultMQPushConsumer.getPullBatchSize(),
            sysFlag,
            commitOffsetValue,
            /BROKER_SUSPEND_MAX_TIME_MILLIS/,
            /CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND/,
            CommunicationMode./ASYNC/,
            pullCallback
        );
    } catch (Exception e) {
        log.error("pullKernelImpl exception", e);
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
    }
}

broker端

根據命定Code RequestCode.PULL_MESSAGE 找到broker端的處理消息拉取的類爲PullMessageProcessor

final GetMessageResult getMessageResult =
    this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
        requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);

消息隊列負載，重新分佈機制

由RebalanceService實現，每個MQClientInstance持有一個RebalanceService對象.這個線程每隔20s執行一次

this.mqClientFactory.doRebalance();

遍歷註冊的消費者，對每個消費者依次執行doRebalance() 方法。實際執行的是RebalanceImpl的方法。
每個DefaultMQPushConsumerImpl都持有一個RebalanceImpl對象，該方法遍歷訂閱信息對每個主題的隊列都進行重新負載。

Map<String, SubscriptionData> subTable在消費者調用subscribe方法時填充。

public void doRebalance(final boolean isOrder) {
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
                if (!topic.startsWith(MixAll./RETRY_GROUP_TOPIC_PREFIX/)) {
                    /log/.warn(“rebalanceByTopic Exception”, e);
                }
            }
        }
    }

    this.truncateMessageQueueNotMyTopic();
}

啓動

消息拉取

PullRequest

ProcessQueue

消息拉取流程

consumer端

broker端

消息隊列負載，重新分佈機制

rocketMQ-consumer源碼

rocketMQ-consumer

rockeMQ-producer源碼

rocketMQ消息存儲源碼

sublime text3 markdown

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結