rocketmq 消費端未拉取到消息處理機制

一句話概述

我們知道在消費端向broker拉取消息的時候是以PullRequest爲驅動的，如果拉取消息成功，那麼這個PullRequest會被再次放回消費端的隊列中，進而不斷的進行拉取；

那在該PullRequest沒有拉取到消息的情況下呢？這種情況下broker並不會立即響應consumer，而是會執行一種叫suspend的機制。
該機制會將該拉取任務對象暫時緩存起來，待一定時間後再次執行消息拉取的動作並將拉取的結果返回給客戶端；

暫停的時間有兩種。A:broker的longPollingEnable = true 時取 requestHeader.getSuspendTimeoutMillis()即消費端傳過來的暫停時間；B : broker的longPollingEnable = false 時取的是broker端的配置即this.brokerController.getBrokerConfig().getShortPollingTimeMills()。

關鍵入口

org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest(io.netty.channel.Channel, org.apache.rocketmq.remoting.protocol.RemotingCommand, boolean){

//省略 lot code

case ResponseCode.PULL_NOT_FOUND:

                    if (brokerAllowSuspend && hasSuspendFlag) {
                        long pollingTimeMills = suspendTimeoutMillisLong;
                        if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
                            pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
                        }

                        String topic = requestHeader.getTopic();
                        long offset = requestHeader.getQueueOffset();
                        int queueId = requestHeader.getQueueId();
                        PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills, this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);

//這裏是關鍵代碼，broker會將組裝一個PullRequest緩存起來放到PullRequestHoldService中，待處理
                        this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
                        response = null;
                        break;
                    }

//省略 lot code

}

PullRequestHoldService

PullRequestHoldService類繼承了ServiceThread，是一個線程類；
看看run方法

    @Override
    public void run() {
        log.info("{} service started", this.getServiceName());
        while (!this.isStopped()) {
            try {
//如果broker開啓了longPollingEnable則每一個循環裏都wait 5秒
                if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
                    this.waitForRunning(5 * 1000);
                } else {
//如果broker沒有開啓longPollingEnable,則每個循環wait時間是ShortPollingTimeMills                    this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
                }

                long beginLockTimestamp = this.systemClock.now();
//處理現所有緩存的Pullrequest
                this.checkHoldRequest();
                long costTime = this.systemClock.now() - beginLockTimestamp;
                if (costTime > 5 * 1000) {
                    log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
                }
            } catch (Throwable e) {
                log.warn(this.getServiceName() + " service has exception. ", e);
            }
        }

        log.info("{} service end", this.getServiceName());
    }

接下來看看處理PullRequest的方法

 

    private void checkHoldRequest() {
        for (String key : this.pullRequestTable.keySet()) {
            String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
            if (2 == kArray.length) {
                String topic = kArray[0];
                int queueId = Integer.parseInt(kArray[1]);
                final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
                try {
                    this.notifyMessageArriving(topic, queueId, offset);
                } catch (Throwable e) {
                    log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
                }
            }
        }
    }

pullRequestTable 的數據結構是topic和queueId組合爲可以，ManyPullRequest爲value的ConcurrentHashMap

private ConcurrentMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable = new ConcurrentHashMap<String, ManyPullRequest>(1024);

ManyPullRequest會有多個PullRequest；這裏會找到符合條件的數據 topic@queueId，以此驅動消息的再次查詢並通知消費端。

 

後續流程：

org.apache.rocketmq.broker.longpolling.PullRequestHoldService#notifyMessageArriving(java.lang.String, int, long, java.lang.Long, long, byte[], java.util.Map<java.lang.String,java.lang.String>)
↓
org.apache.rocketmq.broker.processor.PullMessageProcessor#executeRequestWhenWakeup

PullRequestHoldService.notifyMessageArriving方法主要是

public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
        long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
        String key = this.buildKey(topic, queueId);
        ManyPullRequest mpr = this.pullRequestTable.get(key);
        if (mpr != null) {
            List<PullRequest> requestList = mpr.cloneListAndClear();
            if (requestList != null) {
                List<PullRequest> replayList = new ArrayList<PullRequest>();

                for (PullRequest request : requestList) {
                    long newestOffset = maxOffset;
                    if (newestOffset <= request.getPullFromThisOffset()) {
                        newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
                    }

                    if (newestOffset > request.getPullFromThisOffset()) {
                        boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
                            new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
                        // match by bit map, need eval again when properties is not null.
                        if (match && properties != null) {
                            match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
                        }

                        if (match) {
                            try {
                                this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
                                    request.getRequestCommand());
                            } catch (Throwable e) {
                                log.error("execute request when wakeup failed.", e);
                            }
                            continue;
                        }
                    }

                    if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
                        try {
                            this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
                                request.getRequestCommand());
                        } catch (Throwable e) {
                            log.error("execute request when wakeup failed.", e);
                        }
                        continue;
                    }

                    replayList.add(request);
                }

                if (!replayList.isEmpty()) {
                    mpr.addPullRequest(replayList);
                }
            }
        }
    }

1、獲取主題與隊列的所有 PullRequest 並清除內部 pullRequest 集合，避免重複拉取。
2：如果待拉取偏移量(pullFromThisOffset)大於消息隊列的最大有效偏移量，則再次獲取消息隊列的最大有效偏移量，再給一次機會。
3：如果隊列最大偏移量大於 pullFromThisOffset 說明有新的消息到達，先簡單對消息根據 tag,屬性進行一次消息過濾，如果 tag,屬性爲空，則消息過濾器會返回true,然後 executeRequestWhenWakeup進行消息拉取，結束長輪詢。
4：如果掛起時間超過 suspendTimeoutMillisLong，則超時，結束長輪詢，調用executeRequestWhenWakeup 進行消息拉取，並返回結果到客戶端。
5：如果待拉取偏移量大於消息消費隊列最大偏移量，並且未超時，調用 mpr.addPullRequest(replayList) 將拉取任務重新放入，待下一次檢測。

 

PullMessageProcessor.executeRequestWhenWakeup 方法主要是再次執行消息的查詢，並通知consumer端

public void executeRequestWhenWakeup(final Channel channel,final RemotingCommand request) throws RemotingCommandException {
        Runnable run = new Runnable() {
            @Override
            public void run() {
                try {
//重新調用PullMessageProcessor處理請求（即消息的查找）
                    final RemotingCommand response = PullMessageProcessor.this.processRequest(channel, request, false);

                    if (response != null) {
                        response.setOpaque(request.getOpaque());
                        response.markResponseType();
                        try {

//向與消費端相連的channel中寫入數據
                            channel.writeAndFlush(response).addListener(new ChannelFutureListener() {
                                @Override
                                public void operationComplete(ChannelFuture future) throws Exception {
                                    if (!future.isSuccess()) {
                                        log.error("processRequestWrapper response to {} failed",
                                            future.channel().remoteAddress(), future.cause());
                                        log.error(request.toString());
                                        log.error(response.toString());
                                    }
                                }
                            });
                        } catch (Throwable e) {
                            log.error("processRequestWrapper process request over, but response failed", e);
                            log.error(request.toString());
                            log.error(response.toString());
                        }
                    }
                } catch (RemotingCommandException e1) {
                    log.error("excuteRequestWhenWakeup run", e1);
                }
            }
        };
        this.brokerController.getPullMessageExecutor().submit(new RequestTask(run, channel, request));
    }

 

待續...