rocketMQ-consumer源码

RocketMQ 服务器与消费之之前的消息传送方式分为拉模式和推模式，其中推模式是基于拉模式实现的，一个拉取任务完成之后立刻开始下一个拉取任务。

消费模式分为集群消费（clustering）和广播消费（broadcasting）。

集群模式，同一个主题下的同一条消息只能被一个集群中的一个消费者消费；
广播模式，同一个主题下的同一条消息要被集群中所有消费者消费。

启动

以DefaultMQPushConsumerImpl的start方法为例

获取订阅信息SubscriptionData，并放入RebalanceImp的订阅信息中
订阅信息来源于：

调用DefaultMQPushConsumerImpl。subscribe方法订阅
订阅重试主题：%RETRY%+消费组名，消费者在启动时会自动订阅这个主题

private void copySubscription() throws MQClientException {
    try {
        Map<String, String> sub = this.defaultMQPushConsumer.getSubscription();
        if (sub != null) {
            for (final Map.Entry<String, String> entry : sub.entrySet()) {
                final String topic = entry.getKey();
                final String subString = entry.getValue();
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    topic, subString);
                this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
            }
        }

        if (null == this.messageListenerInner) {
            this.messageListenerInner = this.defaultMQPushConsumer.getMessageListener();
        }

        switch (this.defaultMQPushConsumer.getMessageModel()) {
            case /BROADCASTING/:
                break;
            case /CLUSTERING/:
                final String retryTopic = MixAll./getRetryTopic/(this.defaultMQPushConsumer.getConsumerGroup());
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    retryTopic, SubscriptionData./SUB_ALL/);
                this.rebalanceImpl.getSubscriptionInner().put(retryTopic, subscriptionData);
                break;
            default:
                break;
        }
    } catch (Exception e) {
        throw new MQClientException(“subscription exception”, e);
    }
}

初始化MqClientInstance和rebalanceImpl

if (this.defaultMQPushConsumer.getMessageModel() == MessageModel./CLUSTERING/) {
    this.defaultMQPushConsumer.changeInstanceNameToPID();
}

this.mQClientFactory = MQClientManager./getInstance/().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);

this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

this.pullAPIWrapper = new PullAPIWrapper(
    mQClientFactory,
    this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

初始化消息进度，如是集群消费，则消息进度存储在broker，如是广播消费则在consumer

if (this.defaultMQPushConsumer.getOffsetStore() != null) {
    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case /BROADCASTING/:
            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        case /CLUSTERING/:
            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        default:
            break;
    }
    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();

根据是否是顺序消费启动ConsumeMessageOrderlyService或ConsumeMessageConcurrentlyService服务线程负责消息消费，内部维护一个线程池

if (this.getMessageListenerInner() instance MessageListenerOrderly) {
    this.consumeOrderly = true;
    this.consumeMessageService =
        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instance MessageListenerConcurrently) {
    this.consumeOrderly = false;
    this.consumeMessageService =
        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}

this.consumeMessageService.start();

向MQClientInstance注册消费者，并启动MQClientInstance。一个JVM中所有生产者和消费者公用一个MQClientInstance，MQClientInstance只会启动一次。

boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
    this.serviceState = ServiceState./CREATE_JUST/;
    this.consumeMessageService.shutdown();
    throw new MQClientException(“The consumer group[“ + this.defaultMQPushConsumer.getConsumerGroup()
        + “] has been created before, specify another name please.” + FAQUrl./suggestTodo/(FAQUrl./GROUP_NAME_DUPLICATE_URL/),
        null);
}

mQClientFactory.start();

消息拉取

在MQClientINstance启动的时候，启动了pullMessageService服务。pull消费模式不同，pull方法是客户端自己手动调用，不会在队列中放pullRequset对象
在它的run方法中可以看到

public void run() {
    log.info(this.getServiceName() + “ service started”);

    while (!this.isStopped()) {
        try {
            PullRequest pullRequest = this.pullRequestQueue.take();
            this.pullMessage(pullRequest);
        } catch (InterruptedException ignored) {
        } catch (Exception e) {
            log.error(“Pull Message Service Run Method exception”, e);
        }
    }

    log.info(this.getServiceName() + “ service end”);
}

只要服务没有停止，就不停的从pullRequestQueue中take出pullRequest任务。pullRequestQueue是一个阻塞队列

LinkedBlockingQueue<PullRequest> pullRequestQueue

pullRequest是每一个消息队列一个pullRequest，放进去的时机：

执行完一次拉取任务之后又将pullRequest放入到pullRequestQueue
真正创建是在RebalanceImpl中

PullRequest

private String consumerGroup; //消费组
private MessageQueue messageQueue; //待拉取消息队列
private ProcessQueue processQueue; //消息处理队列，从broker拉取的消息先存放在这里再提交到消费者消费线程池消费，是messageQueue在消费者的快照
private long nextOffset; //待拉取的messageQueue偏移量
private boolean lockedFirst = false; //是否被锁定

ProcessQueue

processQueueTable是map结构，key是MessageQueue，value是processQueue，Rebalance做负载均衡时会把consumer分配到的messageQueue和processTable做对比，将匹配上的processQueue构造一个PullRequest，获取下一个消息的offset设置到pullRequset中，并把pullrequest放入PullrequestQueue。

ProcessQueue是messageQueue的快照，保存了从broker拉取了但是还没有消费的消息。有一个读写锁控制并发，一个TreeMap保存拉取的消息

PullMessageService每次从消息服务器默认拉取32条消息，按照消息队列偏移量顺序存放在processQueue，然后PullMessageService将消息提交到消费者消费线程池，消费成功之后从processQueue中移除。

public final static long /REBALANCE_LOCK_MAX_LIVE_TIME/=
    Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockMaxLiveTime", "30000"));
public final static long /REBALANCE_LOCK_INTERVAL/= Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockInterval", "20000"));
private final static long /PULL_MAX_IDLE_TIME/= Long./parseLong/(System./getProperty/("rocketmq.client.pull.pullMaxIdleTime", "120000"));
private final InternalLogger log = ClientLogger./getLog/();
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();//消息存储容器，键为消息在consumerQueue中的偏移量，value为消息实体
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
//**/
/ * A subset of msgTreeMap, will only be used when orderly consume/
/ *//
private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>();//消息临时存储容器，用于处理顺序消息，从processQueue中的msgTressMap中取出消息前，先临时存储在这里
private final AtomicLong tryUnlockTimes = new AtomicLong(0);
private volatile long queueOffsetMax = 0L;
private volatile boolean dropped = false;
private volatile long lastPullTimestamp = System./currentTimeMillis/();
private volatile long lastConsumeTimestamp = System./currentTimeMillis/();
private volatile boolean locked = false;
private volatile long lastLockTimestamp = System./currentTimeMillis/();
private volatile boolean consuming = false;
private volatile long msgAccCnt = 0;

消息拉取流程

consumer端

主要入口在DefaultMQPushConsumerImpl的pullMessage
方法。

检查processQueue的状态
判断是否触发流控，如果出发流量控制，则延迟50ms执行，流控的触发点有：
- processQueue当前处理的消息条数超过了pullThresholdForQueue = 1000
- processQueue当前处理的消息大小超过了pullThresholdSizeForQueue = 100M
- 顺序消费时，processQueue中队列最大偏移量与最小偏移量间距超过consumeConcurrentlyMaxSpan = 2000，主要担心一条消息堵塞，消息进度无法向前推进，从而可能造成大量消息重复消费
根据主题获取订阅信息,如果为空，则延迟3s执行
构建消息拉取系统标记PullSysFlag

private final static int /FLAG_COMMIT_OFFSET/= 0x1 << 0;//从内存中读取的消费进度大于0
private final static int /FLAG_SUSPEND/= 0x1 << 1;//消息拉取时支持挂起
private final static int /FLAG_SUBSCRIPTION/= 0x1 << 2;//消息过滤机制为表达式，这位置1
private final static int /FLAG_CLASS_FILTER/= 0x1 << 3;//消息过滤机制为类过滤模式，这位置1

this.pullAPIWrapper.pullKernelImpl
1. 从MQClientInstance中获取Broker地址。相同名称的broker构成主从结构，brokerId不同。
2. 如果过滤模式位类过滤，从filterServer上拉取消息，否则从broker上拉取
3. 调用this.mQClientFactory.getMQClientAPIImpl().pullMessage异步向broker拉取消息

public PullResult pullKernelImpl(
    final MessageQueue mq,
    final String subExpression,
    final String expressionType,
    final long subVersion,
    final long offset,
    final int maxNums,
    final int sysFlag,
    final long commitOffset,
    final long brokerSuspendMaxTimeMillis,
    final long timeoutMillis,
    final CommunicationMode communicationMode,
    final PullCallback pullCallback
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    FindBrokerResult findBrokerResult =
        this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
            this.recalculatePullFromWhichNode(mq), false);
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult =
            this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
                this.recalculatePullFromWhichNode(mq), false);
    }

    if (findBrokerResult != null) {
        {
            // check version
            if (!ExpressionType./isTagType/(expressionType)
                && findBrokerResult.getBrokerVersion() < MQVersion.Version./V4_1_0_SNAPSHOT/.ordinal()) {
                throw new MQClientException("The broker[" + mq.getBrokerName() + ", "
                    + findBrokerResult.getBrokerVersion() + "] does not upgrade to support for filter message by " + expressionType, null);
            }
        }
        int sysFlagInner = sysFlag;

        if (findBrokerResult.isSlave()) {
            sysFlagInner = PullSysFlag./clearCommitOffsetFlag/(sysFlagInner);
        }

        PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
        requestHeader.setConsumerGroup(this.consumerGroup);
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setQueueOffset(offset);
        requestHeader.setMaxMsgNums(maxNums);
        requestHeader.setSysFlag(sysFlagInner);
        requestHeader.setCommitOffset(commitOffset);
        requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
        requestHeader.setSubscription(subExpression);
        requestHeader.setSubVersion(subVersion);
        requestHeader.setExpressionType(expressionType);

        String brokerAddr = findBrokerResult.getBrokerAddr();
        if (PullSysFlag./hasClassFilterFlag/(sysFlagInner)) {
            brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
        }

        PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
            brokerAddr,
            requestHeader,
            timeoutMillis,
            communicationMode,
            pullCallback);

        return pullResult;
    }

    throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}

pullMessage方法：

public void pullMessage(final PullRequest pullRequest) {
    final ProcessQueue processQueue = pullRequest.getProcessQueue();
/**1. 检查processQueue的状态*/
    if (processQueue.isDropped()) {
        log.info("the pull request[{}] is dropped.", pullRequest.toString());
        return;
    }

    pullRequest.getProcessQueue().setLastPullTimestamp(System./currentTimeMillis/());

    try {
        this.makeSureStateOK();
    } catch (MQClientException e) {
        log.warn("pullMessage exception, consumer state not ok", e);
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        return;
    }

    if (this.isPause()) {
        log.warn("consumer was paused, execute pull request later. instanceName={}, group={}", this.defaultMQPushConsumer.getInstanceName(), this.defaultMQPushConsumer.getConsumerGroup());
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_SUSPEND/);
        return;
    }
/**2. 判断是否触发流控，如果出发流量控制，则延迟50ms执行*/
    long cachedMessageCount = processQueue.getMsgCount().get();
    long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);

    if (cachedMessageCount > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message count exceeds the threshold {}, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (cachedMessageSizeInMiB > this.defaultMQPushConsumer.getPullThresholdSizeForQueue()) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
        if ((queueFlowControlTimes++ % 1000) == 0) {
            log.warn(
                "the cached message size exceeds the threshold {} MiB, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, pullRequest={}, flowControlTimes={}",
                this.defaultMQPushConsumer.getPullThresholdSizeForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, pullRequest, queueFlowControlTimes);
        }
        return;
    }

    if (!this.consumeOrderly) {
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL/);
            if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
                log.warn(
                    "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
                    processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
                    pullRequest, queueMaxSpanFlowControlTimes);
            }
            return;
        }
    } else {
        if (processQueue.isLocked()) {
            if (!pullRequest.isLockedFirst()) {
                final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
                boolean brokerBusy = offset < pullRequest.getNextOffset();
                log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
                    pullRequest, offset, brokerBusy);
                if (brokerBusy) {
                    log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
                        pullRequest, offset);
                }

                pullRequest.setLockedFirst(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
            this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
            log.info("pull message later because not locked in broker, {}", pullRequest);
            return;
        }
    }
/**3. 根据主题获取订阅信息*/
    final SubscriptionData subscriptionData = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (null == subscriptionData) {
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        log.warn("find the consumer's subscription failed, {}", pullRequest);
        return;
    }

    final long beginTimestamp = System./currentTimeMillis/();

    PullCallback pullCallback = new PullCallback() {
        @Override
        public void onSuccess(PullResult pullResult) {
            if (pullResult != null) {
                pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                    subscriptionData);

                switch (pullResult.getPullStatus()) {
                    case /FOUND/:
                        long prevRequestOffset = pullRequest.getNextOffset();
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                        long pullRT = System./currentTimeMillis/() - beginTimestamp;
                        DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                            pullRequest.getMessageQueue().getTopic(), pullRT);

                        long firstMsgOffset = Long./MAX_VALUE/;
                        if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        } else {
                            firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

                            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                            boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                            DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                pullResult.getMsgFoundList(),
                                processQueue,
                                pullRequest.getMessageQueue(),
                                dispatchToConsume);

                            if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                    DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                            } else {
                                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                            }
                        }

                        if (pullResult.getNextBeginOffset() < prevRequestOffset
                            || firstMsgOffset < prevRequestOffset) {
                            log.warn(
                                "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                                pullResult.getNextBeginOffset(),
                                firstMsgOffset,
                                prevRequestOffset);
                        }

                        break;
                    case /NO_NEW_MSG/:
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        break;
                    case /NO_MATCHED_MSG/:
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        break;
                    case /OFFSET_ILLEGAL/:
                        log.warn("the pull request offset illegal, {} {}",
                            pullRequest.toString(), pullResult.toString());
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                        pullRequest.getProcessQueue().setDropped(true);
                        DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

                            @Override
                            public void run() {
                                try {
                                    DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                        pullRequest.getNextOffset(), false);

                                    DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                                    DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                                    log.warn("fix the pull request offset, {}", pullRequest);
                                } catch (Throwable e) {
                                    log.error("executeTaskLater Exception", e);
                                }
                            }
                        }, 10000);
                        break;
                    default:
                        break;
                }
            }
        }

        @Override
        public void onException(Throwable e) {
            if (!pullRequest.getMessageQueue().getTopic().startsWith(MixAll./RETRY_GROUP_TOPIC_PREFIX/)) {
                log.warn("execute the pull request exception", e);
            }

            DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
        }
    };

    boolean commitOffsetEnable = false;
    long commitOffsetValue = 0L;
    if (MessageModel./CLUSTERING/== this.defaultMQPushConsumer.getMessageModel()) {
        commitOffsetValue = this.offsetStore.readOffset(pullRequest.getMessageQueue(), ReadOffsetType./READ_FROM_MEMORY/);
        if (commitOffsetValue > 0) {
            commitOffsetEnable = true;
        }
    }

    String subExpression = null;
    boolean classFilter = false;
    SubscriptionData sd = this.rebalanceImpl.getSubscriptionInner().get(pullRequest.getMessageQueue().getTopic());
    if (sd != null) {
        if (this.defaultMQPushConsumer.isPostSubscriptionWhenPull() && !sd.isClassFilterMode()) {
            subExpression = sd.getSubString();
        }

        classFilter = sd.isClassFilterMode();
    }
/**4. 构建消息拉取系统标记*/
    int sysFlag = PullSysFlag./buildSysFlag/(
        commitOffsetEnable, // commitOffset
        true, // suspend
        subExpression != null, // subscription
        classFilter // class filter
    );
    try {
        this.pullAPIWrapper.pullKernelImpl(
            pullRequest.getMessageQueue(),
            subExpression,
            subscriptionData.getExpressionType(),
            subscriptionData.getSubVersion(),
            pullRequest.getNextOffset(),
            this.defaultMQPushConsumer.getPullBatchSize(),
            sysFlag,
            commitOffsetValue,
            /BROKER_SUSPEND_MAX_TIME_MILLIS/,
            /CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND/,
            CommunicationMode./ASYNC/,
            pullCallback
        );
    } catch (Exception e) {
        log.error("pullKernelImpl exception", e);
        this.executePullRequestLater(pullRequest, /PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION/);
    }
}

broker端

根据命定Code RequestCode.PULL_MESSAGE 找到broker端的处理消息拉取的类为PullMessageProcessor

final GetMessageResult getMessageResult =
    this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
        requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);

拉取成功回调

拉取消息成功之后会回调注册的回调函数

这个方法会处理返回回来的消息，如果有tag，再进行一次tag的过滤,把二进制的消息内容转为MessageExt对象

DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
            subscriptionData);

如果是FOUND状态，但是消息列表为空，则DefaultMQPushConsumerImpl.this.executePullRequestImmediately方法把当前pullRequest再次放入pullRequestQueue中
消息列表不为空时，将消息列表放入processQueue的msgTreeMap中，并且新建一个ConsumeRequest对象，submit到consumeExecutor线程池。
ConsumeRequest是一个Runnable，run()方法执行实际的消费逻辑

@Override
public void onSuccess(PullResult pullResult) {
    if (pullResult != null) {
        pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
            subscriptionData);

        switch (pullResult.getPullStatus()) {
            case *FOUND*:
                long prevRequestOffset = pullRequest.getNextOffset();
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                long pullRT = System.*currentTimeMillis*() - beginTimestamp;
                DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                    pullRequest.getMessageQueue().getTopic(), pullRT);

                long firstMsgOffset = Long.*MAX_VALUE*;
                if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                    DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                } else {
                    firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

                    DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                        pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                    boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                    DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                        pullResult.getMsgFoundList(),
                        processQueue,
                        pullRequest.getMessageQueue(),
                        dispatchToConsume);

                    if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                        DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                            DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                    } else {
                        DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                    }
                }

                if (pullResult.getNextBeginOffset() < prevRequestOffset
                    || firstMsgOffset < prevRequestOffset) {
                    log.warn(
                        "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                        pullResult.getNextBeginOffset(),
                        firstMsgOffset,
                        prevRequestOffset);
                }

                break;
            case *NO_NEW_MSG*:
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                break;
            case *NO_MATCHED_MSG*:
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);

                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                break;
            case *OFFSET_ILLEGAL*:
                log.warn("the pull request offset illegal, {} {}",
                    pullRequest.toString(), pullResult.toString());
                pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                pullRequest.getProcessQueue().setDropped(true);
                DefaultMQPushConsumerImpl.this.executeTaskLater(new Runnable() {

                    @Override
                    public void run() {
                        try {
                            DefaultMQPushConsumerImpl.this.offsetStore.updateOffset(pullRequest.getMessageQueue(),
                                pullRequest.getNextOffset(), false);

                            DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                            DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());

                            log.warn("fix the pull request offset, {}", pullRequest);
                        } catch (Throwable e) {
                            log.error("executeTaskLater Exception", e);
                        }
                    }
                }, 10000);
                break;
            default:
                break;
        }
    }
}

消息队列负载，重新分布机制

由RebalanceService实现，每个MQClientInstance持有一个RebalanceService对象.这个线程每隔20s执行一次

this.mqClientFactory.doRebalance();

遍历注册的消费者，对每个消费者依次执行doRebalance() 方法。实际执行的是RebalanceImpl的方法。
每个DefaultMQPushConsumerImpl都持有一个RebalanceImpl对象，该方法遍历订阅信息对每个主题的队列都进行重新负载。

Map<String, SubscriptionData> subTable在消费者调用subscribe方法时填充。

public void doRebalance(final boolean isOrder) {
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
                if (!topic.startsWith(MixAll./RETRY_GROUP_TOPIC_PREFIX/)) {
                    /log/.warn(“rebalanceByTopic Exception”, e);
                }
            }
        }
    }

    this.truncateMessageQueueNotMyTopic();
}

对每个topic进行负载均衡

从broker中获取所有订阅该topic的consumer的客户端cid，这是MQClientInstance给Broker发心跳包时发送过去的
对所有的cid和消息队列排序，保证所有消费者的视角一致
调用AllocateMessageQueueStrategy的allocate方法，获取当前消费者可消费的消息队列列表，负载均衡策略有：
1. AllocateMessageQueueAveragely，平均分配，推荐使用
2. AllocateMessageQueueAveragelyByCircle，轮训平均分配，推荐使用
3. AllocateMessageQueueConsistentHash，一致性hash，不推荐，消息队列负载信息不容易跟踪
4. AllocateMessageQueueByMachineRoom，根据broker部署机房名，对每个消费者负责不同的broker上的队列
5. AllocateMessageQueueByConfig，根据配置，为每个消费者制定固定消息队列
更新当前的负载队列集合，如果队列不在新负载均衡中，则停止消费，并保存消费进度。如果新队列不在当前队列中，则创建这个队列的拉取任务pullRequest，添加到pullMessageService线程的pullRequestQueue中。在一个JVM中，同一个消费组同一个消息队列只有一个pullRequest对象

  case *CLUSTERING*: {
        Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
        List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
        if (null == mqSet) {
            if (!topic.startsWith(MixAll.*RETRY_GROUP_TOPIC_PREFIX*)) {
                *log*.warn(“doRebalance, {}, but the topic[{}] not exist.”, consumerGroup, topic);
            }
        }

        if (null == cidAll) {
            *log*.warn(“doRebalance, {} {}, get consumer id list failed”, consumerGroup, topic);
        }

        if (mqSet != null && cidAll != null) {
            List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
            mqAll.addAll(mqSet);

            Collections.*sort*(mqAll);
            Collections.*sort*(cidAll);

            AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;

            List<MessageQueue> allocateResult = null;
            try {
                allocateResult = strategy.allocate(
                    this.consumerGroup,
                    this.mQClientFactory.getClientId(),
                    mqAll,
                    cidAll);
            } catch (Throwable e) {
                *log*.error(“AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}”, strategy.getName(),
                    e);
                return;
            }

            Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
            if (allocateResult != null) {
                allocateResultSet.addAll(allocateResult);
            }

            boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
            if (changed) {
                *log*.info(
                    “rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}”,
                    strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
                    allocateResultSet.size(), allocateResultSet);
                this.messageQueueChanged(topic, mqSet, allocateResultSet);
            }
        }
        break;
    }
    default:
        break;
}

updateProcessQueueTableInRebalance方法更新当前消费的消费队列

private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
    final boolean isOrder) {
    boolean changed = false;

    Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<MessageQueue, ProcessQueue> next = it.next();
        MessageQueue mq = next.getKey();
        ProcessQueue pq = next.getValue();

        if (mq.getTopic().equals(topic)) {
            if (!mqSet.contains(mq)) {
                pq.setDropped(true);
                if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                    it.remove();
                    changed = true;
                    *log*.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                }
            } else if (pq.isPullExpired()) {
                switch (this.consumeType()) {
                    case *CONSUME_ACTIVELY*:
                        break;
                    case *CONSUME_PASSIVELY*:
                        pq.setDropped(true);
                        if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                            it.remove();
                            changed = true;
                            *log*.error("[BUG]doRebalance, {}, remove unnecessary mq, {}, because pull is pause, so try to fixed it",
                                consumerGroup, mq);
                        }
                        break;
                    default:
                        break;
                }
            }
        }
    }

    List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
    for (MessageQueue mq : mqSet) {
        if (!this.processQueueTable.containsKey(mq)) {
            if (isOrder && !this.lock(mq)) {
                *log*.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
                continue;
            }

            this.removeDirtyOffset(mq);
            ProcessQueue pq = new ProcessQueue();
            long nextOffset = this.computePullFromWhere(mq);
            if (nextOffset >= 0) {
                ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                if (pre != null) {
                    *log*.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
                } else {
                    *log*.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                    PullRequest pullRequest = new PullRequest();
                    pullRequest.setConsumerGroup(consumerGroup);
                    pullRequest.setNextOffset(nextOffset);
                    pullRequest.setMessageQueue(mq);
                    pullRequest.setProcessQueue(pq);
                    pullRequestList.add(pullRequest);
                    changed = true;
                }
            } else {
                *log*.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
            }
        }
    }

    this.dispatchPullRequest(pullRequestList);

    return changed;
}

消息消费

pullMessageService在拉取到消息之后，把消息存储到processQueue消息处理队列中，然后调用consumeMeaasgeService.submitConsumeRequest提交消息消费请求。由ConsumerMessageService来处理消息的消费.

ConsumerMessageService的实现类ConsumeMessageConcurrentlyService和ConsumeMessageOrderlyService，分别对应并发消费和顺序消费。

并发消费：

private static final InternalLogger *log*= ClientLogger.*getLog*();
private final DefaultMQPushConsumerImpl defaultMQPushConsumerImpl;//消息推模式实现类
private final DefaultMQPushConsumer defaultMQPushConsumer;//消费者对象
private final MessageListenerConcurrently messageListener;//并发消息业务事件类
private final BlockingQueue<Runnable> consumeRequestQueue;//消息消费任务对了
private final ThreadPoolExecutor consumeExecutor;//消息消费线程池
private final String consumerGroup;//消费组

private final ScheduledExecutorService scheduledExecutorService;//添加消费任务到consumeExecutor延迟调度器
private final ScheduledExecutorService cleanExpireMsgExecutors;//定时删除过期消息线程池

消息消费流程为：

提交消费任务

public void submitConsumeRequest(
    final List<MessageExt> msgs,
    final ProcessQueue processQueue,
    final MessageQueue messageQueue,
    final boolean dispatchToConsume)

如果消息数量小于最大批次消息量，直接将消息放入consumeRequest中，调到consumeExecutor线程池，如果大于，则分批次提及
2. 进入consumeRequest的run方法，首先检查processQueue的isDropped是否标记，如果标记则停止消费
3. 执行消费前钩子函数

ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookBefore(consumeMessageContext);

恢复重试消息主题名,由于消息存入commitLog文件时，如果消息delayTimeLevel不为0，会首先将充实主题存入在消息的属性中，以便时间到了之后重新消费

ConsumeMessageConcurrentlyService.this.resetRetryTopic(msgs);

执行具体的消息消费，注册消费者的时候注入的消息监听器consumeMessage方法。

status = listener.consumeMessage(Collections.*unmodifiableList*(msgs), context);

执行钩子函数

ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookAfter(consumeMessageContext);

再次检查processQueue的isDropped状态，如果为true则不处理，也就是说如果消息消费过程中processQueue的isDropped状态由false变为true，消息会被重复消费
根据status处理消息消费结果，如果是集群消费，并且状态为consume_later，向broker发送ACK，如果发送失败，记录消息消费失败的消息和失败消息的已经消费了的次数，并添加到延迟消费，5s中之后添加到线程池

this.scheduledExecutorService.schedule(new Runnable() {

    @Override
    public void run() {
        ConsumeMessageConcurrentlyService.this.submitConsumeRequest(msgs, processQueue, messageQueue, true);
    }
}, 5000, TimeUnit.*MILLISECONDS*);

然后从processQueue移除这一批消费的消息。

public void processConsumeResult(
    final ConsumeConcurrentlyStatus status,
    final ConsumeConcurrentlyContext context,
    final ConsumeRequest consumeRequest
) {
    int ackIndex = context.getAckIndex();

    if (consumeRequest.getMsgs().isEmpty())
        return;

    switch (status) {
        case *CONSUME_SUCCESS*:
            if (ackIndex >= consumeRequest.getMsgs().size()) {
                ackIndex = consumeRequest.getMsgs().size() - 1;
            }
            int ok = ackIndex + 1;
            int failed = consumeRequest.getMsgs().size() - ok;
            this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), ok);
            this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), failed);
            break;
        case *RECONSUME_LATER*:
            ackIndex = -1;
            this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(),
                consumeRequest.getMsgs().size());
            break;
        default:
            break;
    }

    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case *BROADCASTING*:
            for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                MessageExt msg = consumeRequest.getMsgs().get(i);
                *log*.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
            }
            break;
        case *CLUSTERING*:
            List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
            for (int I = ackIndex + 1; i < consumeRequest.getMsgs().size(); I++) {
                MessageExt msg = consumeRequest.getMsgs().get(i);
                boolean result = this.sendMessageBack(msg, context);
                if (!result) {
                    msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                    msgBackFailed.add(msg);
                }
            }

            if (!msgBackFailed.isEmpty()) {
                consumeRequest.getMsgs().removeAll(msgBackFailed);

                this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
            }
            break;
        default:
            break;
    }

    long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
    if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
        this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
    }
}

消息确认ACK

对于集群消费，消费结果为CONSUME_LATER的消息，会向broker发送消息确认

命令为：RequestCode.CONSUMER_SEND_MSG_BACK

public boolean sendMessageBack(final MessageExt msg, final ConsumeConcurrentlyContext context) {
    int delayLevel = context.getDelayLevelWhenNextConsume();

    try {
        this.defaultMQPushConsumerImpl.sendMessageBack(msg, delayLevel, context.getMessageQueue().getBrokerName());
        return true;
    } catch (Exception e) {
        *log*.error(“sendMessageBack exception, group: “ + this.consumerGroup + “ msg: “ + msg.toString(), e);
    }

    return false;
}

Broker端处理类为SendMessageProcessor

获取SubscriptionGroupConfig，消费组订阅信息配置信息存储在broker的${Rocket_Home}/store/config/subscriptionGroup.json。
如果重试队列数量小于1，说明不支持重试，直接返回成功。
创建重试主题，主题名称%RETRY%+消费组名，并从重试队列中随机选择一个队列，构建TopicConfig主题配置信息
从commitLog获取消息，并设置消息的retry主题
设置消息的重试次数，如果此时的重试次数已经超过了最大重试次数，则将新的主题改为%DLQ%,这个主题的权限为只写，说明消息一旦进入这个队列，mq不再进行消费
根据之前的消息重新建一个新的msgId的消息，别的类容相同，并存入commitlog。如果delayTimeLevel>0，替换消息主题与队列定时任务主题“SCHEDULE_TOPIC_XXXX”，队列ID为延迟级别-1.再将消息主题，队列存入消息属性中，键为PROPERTY_REAL_TOPIC,PROPERTY_REAL_QUEUE_ID

if (msg.getDelayTimeLevel() > 0) {
    if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
        msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
    }

    topic = ScheduleMessageService.*SCHEDULE_TOPIC*;
    queueId = ScheduleMessageService.*delayLevel2QueueId*(msg.getDelayTimeLevel());

    // Backup real topic, queueId
    MessageAccessor.*putProperty*(msg, MessageConst.*PROPERTY_REAL_TOPIC*, msg.getTopic());
    MessageAccessor.*putProperty*(msg, MessageConst.*PROPERTY_REAL_QUEUE_ID*, String.*valueOf*(msg.getQueueId()));
    msg.setPropertiesString(MessageDecoder.*messageProperties2String*(msg.getProperties()));

    msg.setTopic(topic);
    msg.setQueueId(queueId);
}

消费进度管理

管理类为OffsetStore。具体的实现有

LocalFileOffsetStore这个是广播消费模式的消费进度管理，进度保存在消费者本地
RemoteBrokerOffsetStore，由于集群消费模式，一条消息只能被一个消费者消费，因此进度保存在broker中

定时消息

由于任意时间间隔的定时消息需要排序，效率低，rocketmq只支持固定延迟级别的定时消息。
在消息存入commitlog时，如果消息的delayLevel>0，会设置新的主题SCHEDULE_TOPIC_XXXX和消息队列id=delayLevel-1

rocketMQ给每个延迟级别创建一个定时任务，按照延迟时间定时去对应的延迟消息队列拉取消息，然后从commitlog中加载完整的消息，清楚延迟属性，恢复原先的主题和队列，再次创建一条新的消息放入commitlog中供消费者消费。

实现类是ScheduleMessageService，在start方法中启动这个service和定时任务。

rocketMQ-consumer源码

启动

消息拉取

PullRequest

ProcessQueue

消息拉取流程

consumer端

broker端

拉取成功回调

消息队列负载，重新分布机制

消息消费

消息确认ACK

消费进度管理

定时消息

这种嵌套字典类型的数据，我想把它读取到df里，如何操作？

微调真的能让LLM学到新东西吗:引入新知识可能让模型产生更多的幻觉

iNeuOS工业互联网操作系统，增加电力IEC104协议

微服务实践k8s&dapr开发部署实验（3）订阅发布

kbgressdb之数据结构V0.2

rocketMQ-consumer源碼

rocketMQ-consumer

rockeMQ-producer源碼

rocketMQ消息存儲源碼

sublime text3 markdown

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結