RocketMQ源码分析——消息的拉取

创建消息消费者

实例化消费者、设置NameServer的地址、设置消费起始位置（获取消费进度失败时有效）、订阅一个或者多个 Topic 以及 Tag 来过滤需要消费的消息、注册回调实现类来处理从broker拉取回来的消息、启动消费者。
org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#start

...
// 创建订阅消息 SubscriptionData 到负载服务中
this.copySubscription();
// 广播模式消息消费进度由消费者控制，集群模式由Broker控制
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
    switch (this.defaultMQPushConsumer.getMessageModel()) {
        case BROADCASTING:
            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        case CLUSTERING:
            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        default:
            break;
    }
    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
// 加载消息进度
this.offsetStore.load();
// 区分是并发消费还是顺序消费
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
    this.consumeOrderly = true;
    this.consumeMessageService =
        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
    this.consumeOrderly = false;
    this.consumeMessageService =
        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
...
// 更新所有主题订阅信息
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
// 校验所有的订阅配置是否正确
this.mQClientFactory.checkClientInBroker();
// 向所有Broker发送心跳
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
// 立即进行一次消费者负载
this.mQClientFactory.rebalanceImmediately();

消息拉取请求

客户端实例 MQClientInstance 中有一个单独的线程 PullMessageService 来定时拉取消息。

public void run() {
    while (!this.isStopped()) {
        try {
        	// 从 LinkedBlockingQueue 中获取一个拉取请求，没有就阻塞
        	// 拉取请求在客户端做负载均衡时会创建，一个拉取任务完成后还会继续拉取。
            PullRequest pullRequest = this.pullRequestQueue.take();
            this.pullMessage(pullRequest);
        } catch (InterruptedException ignored) {
        } catch (Exception e) {
        }
    }
}

调用拉取消息方法，先校验消费队列状态，进行流控
org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage

public void pullMessage(final PullRequest pullRequest) {
	final ProcessQueue processQueue = pullRequest.getProcessQueue();
	// 队列被删除，禁止消费
	if (processQueue.isDropped()) {
	    return;
	}
	pullRequest.getProcessQueue().setLastPullTimestamp(System.currentTimeMillis());
	try {
		// 客户端服务服务是否正常
	    this.makeSureStateOK();
	} catch (MQClientException e) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
	    return;
	}
	// 队列流控，默认延迟1s之后再拉取消息
	if (this.isPause()) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_SUSPEND);
	    return;
	}
	// 待消费的消息总数量
	long cachedMessageCount = processQueue.getMsgCount().get();
	// 待消费的消息总大小
	long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);
	// 默认超过1000条进行流控
	if (cachedMessageCount > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
	    return;
	}
	// 默认超过100M进行流控
	if (cachedMessageSizeInMiB > this.defaultMQPushConsumer.getPullThresholdSizeForQueue()) {
	    this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
	    return;
	}
    if (!this.consumeOrderly) {
    	// 非顺序消费，默认消息的偏移量间隔超过2000，进行流控
        if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
            return;
        }
    } else {
    	// 顺序消费，需要先判断是否有锁定
        if (processQueue.isLocked()) {
            if (!pullRequest.isLockedFirst()) {
            	// 第一次锁定记录下次拉取的偏移量
                final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
                boolean brokerBusy = offset < pullRequest.getNextOffset();

                pullRequest.setLockedFirst(true);
                pullRequest.setNextOffset(offset);
            }
        } else {
        	// 默认延迟3s再拉取
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
            return;
        }
    }
}

构造请求参数向Broker发送拉取请求
org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#pullKernelImpl

public PullResult pullKernelImpl(
    final MessageQueue mq, //消费队列
    final String subExpression, //过滤表达式
    final String expressionType, //表达式类型 TAG/SQL
    final long subVersion, //过滤信息版本
    final long offset, //拉取偏移量位置
    final int maxNums, //最大拉取数量
    final int sysFlag, //系统标识
    final long commitOffset, //当前消费进度
    final long brokerSuspendMaxTimeMillis, //默认挂起时间15s
    final long timeoutMillis, //默认超时时间30s
    final CommunicationMode communicationMode, //默认异步
    final PullCallback pullCallback //回调函数
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
	...
	String brokerAddr = findBrokerResult.getBrokerAddr();
	// 如果是类过滤，获取到类过滤服务器的地址
	if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
	    brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
	}
	...
}

服务端查找消息

org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest

private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend)
        throws RemotingCommandException {
	...
	// 校验Broker是否允许拉取消息、过滤主题信息是否存在和允许拉取、主题是否存在和允许拉取、队列ID是否正确、过滤表达式或者是否正确是否存在
	// 根据偏移量查找消息结果，消息过滤的逻辑之后分析
	final GetMessageResult getMessageResult =
            this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
                requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);
	// 根据结果，返回各种状态码，此处忽略
	switch (response.getCode()) {
	   case ResponseCode.SUCCESS:
			if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) {
				// 通过内存处理所有的消息
                final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());
                response.setBody(r);
            } else {
            	...
            	//使用netty的异步通信
            }
		case ResponseCode.PULL_NOT_FOUND:
			// 没有找到消息
		    if (brokerAllowSuspend && hasSuspendFlag) {
		    	// Push模式实现的关键点就是这里
		        long pollingTimeMills = suspendTimeoutMillisLong;
		        if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
		            pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
		        }
		
		        String topic = requestHeader.getTopic();
		        long offset = requestHeader.getQueueOffset();
		        int queueId = requestHeader.getQueueId();
		        // 创建 PullRequest 对象
		        PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
		            this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);
		        // 交给 PullRequestHoldService 异步处理拉取请求
		        this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
		        // 返回null，不向 channel 写入数据，相当于挂起当前请求
		        response = null;
		        break;
		    }
}

RocketMQ 的 Consumer 都是从 Broker 拉消息来消费，但是为了能做到实时收消息，RocketMQ 使用长轮询方式，可以保证消息实时性同 Push 方式一致。Broker 单独开启一个线程来 hold 客户端拉取请求。
org.apache.rocketmq.broker.longpolling.PullRequestHoldService#run

public void run() {
    while (!this.isStopped()) {
        try {
            if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
            	// 开启长轮询，每5秒尝试一次
                this.waitForRunning(5 * 1000);
            } else {
            	// 未开启长轮询，默认1s尝试一次
                this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
            }
            // 有满足条件的消息就返回
            this.checkHoldRequest();
        } catch (Throwable e) {
        }
    }
}

checkHoldRequest 会遍历所有的拉取请求，执行 notifyMessageArriving
org.apache.rocketmq.broker.longpolling.PullRequestHoldService#notifyMessageArriving

	...
	// 获取 CommitLog 中最大的物理偏移量 newestOffset，比本次拉取值大代表有新消息存储进来了
	if (newestOffset > request.getPullFromThisOffset()) {
		// 过滤校验消息通过后，唤醒被挂起的拉取线程
		brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
        	request.getRequestCommand());
	}
	if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
		// 超时了，也需要唤醒
		...
	}
	...

长轮询机制：每5s检查一次是否有满足过滤条件的新消息，超过15s仍然没有就返回超时。Broker 在有新消息进入 CommitLog 后，分发任务 ReputMessageService 也会执行 notifyMessageArriving 唤醒拉取任务。
org.apache.rocketmq.store.DefaultMessageStore.ReputMessageService#doReput

	...
	if (BrokerRole.SLAVE != DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole()
	    && DefaultMessageStore.this.brokerConfig.isLongPollingEnable()) {
	    DefaultMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
	        dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
	        dispatchRequest.getTagsCode(), dispatchRequest.getStoreTimestamp(),
	        dispatchRequest.getBitMap(), dispatchRequest.getPropertiesMap());
	}

消息队列负载

负载原则：一个消息消费队列在同一时间只允许被同一消费组内的一个消费者消费，一个消息消费者能同时消费多个消息队列。
MQClientInstance 中有一个单独的 RebalanceService 线程来定时做负载

public void run() {
    while (!this.isStopped()) {
    	// 默认间隔20s
        this.waitForRunning(waitInterval);
        this.mqClientFactory.doRebalance();
    }
}

public void doRebalance(final boolean isOrder) {
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
    	// 遍历每个主题，做负载
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
            }
        }
    }
	// 清除没有订阅的主题下的消息队列
    this.truncateMessageQueueNotMyTopic();
}

集群模式下，消息由所有消费者共同消费，怎么确定消费队列由那个消费者消费？
org.apache.rocketmq.client.impl.consumer.RebalanceImpl#rebalanceByTopic

private void rebalanceByTopic(final String topic, final boolean isOrder) {
	...
	// 从Broker获取主题下有哪些消费者
	List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
	// 排序，保证所有消费者客户端看到的数据一致
    Collections.sort(mqAll);
    Collections.sort(cidAll);
    AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
    List<MessageQueue> allocateResult = null;
    try {
    	// 执行负载算法，计算当前消费者应该消费的队列
        allocateResult = strategy.allocate(
            this.consumerGroup,
            this.mQClientFactory.getClientId(),
            mqAll,
            cidAll);
    } catch (Throwable e) {
        return;
    }
}

负载算法，默认为平均分配 AllocateMessageQueueAveragely

清除旧队列，新增新加入的队列请求

    private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
        final boolean isOrder) {
        boolean changed = false;
		// 清除没有分配给自己的队列
        Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<MessageQueue, ProcessQueue> next = it.next();
            MessageQueue mq = next.getKey();
            ProcessQueue pq = next.getValue();

            if (mq.getTopic().equals(topic)) {
                if (!mqSet.contains(mq)) {
                    pq.setDropped(true);
                    if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                        it.remove();
                        changed = true;
                        log.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                    }
                } else if (pq.isPullExpired()) {
                    // 默认120s过期，push模式才有效，清除消费队列数据
                }
            }
        }
		// 新分配队列给自己，新建 ProcessQueue 
        List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
        for (MessageQueue mq : mqSet) {
            if (!this.processQueueTable.containsKey(mq)) {
            	// 顺序消息需要判断是否有锁
                if (isOrder && !this.lock(mq)) {
                    continue;
                }
				// 移除缓存中此队列的消费进度
                this.removeDirtyOffset(mq);
                ProcessQueue pq = new ProcessQueue();
                // 新队列需要计算从哪个位置开始拉取
                long nextOffset = this.computePullFromWhere(mq);
                if (nextOffset >= 0) {
                    ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                    if (pre != null) {
                    } else {
                        PullRequest pullRequest = new PullRequest();
                        pullRequest.setConsumerGroup(consumerGroup);
                        pullRequest.setNextOffset(nextOffset);
                        pullRequest.setMessageQueue(mq);
                        pullRequest.setProcessQueue(pq);
                        pullRequestList.add(pullRequest);
                        changed = true;
                    }
                } else {
                }
            }
        }
		// 将请求添加到 pullRequestQueue
        this.dispatchPullRequest(pullRequestList);
        return changed;
    }

consumeFromWhere 参数的作用

新增加的消费队列，移除缓存中此队列的消费进度后，需要从磁盘中加载此队列的消费进度，然后创建 PullRequest 请求。
consumeFromWhere 只在从磁盘获取消费进度失败才生效。
偏移量计算逻辑，以PUSH模式为例，消费进度存储在Broker。

    public long computePullFromWhere(MessageQueue mq) {
        long result = -1;
        final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
        final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
        switch (consumeFromWhere) {
            case CONSUME_FROM_LAST_OFFSET: {
            	// 读取Broker存储的进度
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                }
                // First start,no offset
                else if (-1 == lastOffset) {
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        result = 0L;
                    } else {
                        try {
                        	// 读取队列的最大偏移量
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1;
                }
                break;
            }
            case CONSUME_FROM_FIRST_OFFSET: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                } else if (-1 == lastOffset) {
                    result = 0L;
                } else {
                    result = -1;
                }
                break;
            }
            case CONSUME_FROM_TIMESTAMP: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                } else if (-1 == lastOffset) {
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        try {
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    } else {
                        try {
                        	// 以消费者的启动时间来获取消费队列的偏移量
                            long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
                                UtilAll.YYYYMMDDHHMMSS).getTime();
                            // 从 commitLog 读取消息的存储时间，取存储时间和 timestamp 最接近的消息的偏移量，使用了二分法
                            result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1;
                }
                break;
            }

            default:
                break;
        }

        return result;
    }

消息消费

拉取消息获取到结果后，按照服务端返回的状态封装拉取结果 PullResult
org.apache.rocketmq.client.impl.MQClientAPIImpl#processPullResponse

        switch (response.getCode()) {
            case ResponseCode.SUCCESS:
                pullStatus = PullStatus.FOUND;
                break;
            case ResponseCode.PULL_NOT_FOUND:
                pullStatus = PullStatus.NO_NEW_MSG;
                break;
            case ResponseCode.PULL_RETRY_IMMEDIATELY:
                pullStatus = PullStatus.NO_MATCHED_MSG;
                break;
            case ResponseCode.PULL_OFFSET_MOVED:
                pullStatus = PullStatus.OFFSET_ILLEGAL;
                break;

            default:
                throw new MQBrokerException(response.getCode(), response.getRemark());
        }

再回到 org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage
执行拉取任务的回调函数 PullCallback

        PullCallback pullCallback = new PullCallback() {
            @Override
            public void onSuccess(PullResult pullResult) {
                if (pullResult != null) {
                	// tag过滤，还需要再校验一次原始的tag值，Broker只校验了hashcode，不读消息内容保证堆积也能高效过滤
                    pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                        subscriptionData);

                    switch (pullResult.getPullStatus()) {
                        case FOUND:
                            // 记录拉取统计信息
                            	// 将消息数据放入到 processQueue
                                boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                                // 提交一个消息消费任务，区分并行还是顺序
                                DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                    pullResult.getMsgFoundList(),
                                    processQueue,
                                    pullRequest.getMessageQueue(),
                                    dispatchToConsume);

                                // 设置偏移量，再次拉取
                            }
                            break;
                        case NO_NEW_MSG:
                        	// 修正偏移量，再次拉取
                        case NO_MATCHED_MSG:
                        	// 修正偏移量，再次拉取
                        case OFFSET_ILLEGAL:
                            // 删除此消费队列，更新并持久化消费进度
                            break;
                        default:
                            break;
                    }
                ...

消息拉取总结：启动消费者，消费者按照负载均衡策略生成拉取消息请求，设置拉取模式，读取消费进度，设置拉取起始偏移量，然后定时向Broker拉取消息。Broker接受拉取请求，从CommitLog查找消息，按照过滤策略处理消息后返回给消费者。若拉取任务失败，消息消费者修正偏移量再次拉取；若拉取成功，消费者进行消费，并更新消费进度，再次拉取。

消息消费详情、消息确认、消息进度管理见后续文章。

RocketMQ源码分析——消息的拉取

创建消息消费者

消息拉取请求

服务端查找消息

消息队列负载

consumeFromWhere 参数的作用

消息消费

今天！通义灵码在北京、成都、杭州三城开讲啦

【BI 可视化插件】怎么做？手把手教你实现

Mybatis 緩存實現原理——案例實踐

Mysql 函數 STR_TO_DATE

RocketMQ源碼分析——事務消息

RocketMQ源碼分析——消息的拉取

RocketMQ源碼分析——消息索引文件存儲

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結