1. 簡介
消費消息可以分成pull和push方式，push消息使用比較簡單，因爲RocketMQ已經幫助我們封裝了大部分流程，我們只要重寫回調函數即可。
下面我們就以push消費方式爲例，分析下這部分源代碼流程。
2. 消費者啓動流程圖
3.消費者類圖
4. 消費者源代碼流程
4.1 消費客戶端啓動
根據官方(https://github.com/apache/rocketmq)提供的例子，Consumer.java裏面使用DefaultMQPushConsumer啓動消息消費者，如下：
//初始化DefaultMQPushConsumer
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");
//設置命名服務，參考namesrv的啓動
consumer.setNamesrvAddr("localhost:9876");
//設置消費起始位置
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
//訂閱消費的主題和過濾符
consumer.subscribe("TopicTest", "*");
//設置消息回調函數
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf(Thread.currentThread().getName() + " Receive New Messages: " + msgs + "%n");
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//啓動消費者
consumer.start();4.2 消息者啓動
我們接着看consumer.start()方法
@Override
public void start() throws MQClientException {
this.defaultMQPushConsumerImpl.start();
}DefaultMQPushConsumerImpl.java
public synchronized void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
...
this.checkConfig();//檢查參數
...
this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
...
this.pullAPIWrapper = new PullAPIWrapper(
mQClientFactory,
this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
...
this.offsetStore.load();
...
this.consumeMessageService.start();
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
...
mQClientFactory.start();
this.serviceState = ServiceState.RUNNING;
...
}
...
}在初始化一堆參數之後，然後調用mQClientFactory.start();
private MQClientInstance mQClientFactory;其實這個命名有點奇怪啊(阿里程序員手抖了？)，爲什麼MQClientInstance類型的變量名稱叫mQClientFactory ...
那繼續看MQClientInstance的start
4.3 MQClientInstance
public void start() throws MQClientException {
synchronized (this) {
switch (this.serviceState) {
case CREATE_JUST:
...
// Start request-response channel
this.mQClientAPIImpl.start();
// Start various schedule tasks
this.startScheduledTask();
// Start pull service
this.pullMessageService.start();
// Start rebalance service
this.rebalanceService.start();
// Start push service
this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
log.info("the client factory [{}] start OK", this.clientId);
this.serviceState = ServiceState.RUNNING;
break;
...
}
}
}各行代碼的作用就像源代碼裏面的註釋一樣，重點看下pullMessageService.start和rebalanceService.start
pullMessageService.start作用是不斷從一個阻塞隊列裏面獲取pullRequest請求，然後去RocketMQ broker裏面獲取消息。
如果沒有pullRequest的話，那麼它將阻塞。
那麼，pullRequest請求是怎麼放進去的呢？這個就要看rebalanceService了。
4.4 pullMessageService.start
private final LinkedBlockingQueue<PullRequest> pullRequestQueue = new LinkedBlockingQueue<PullRequest>();
@Override
public void run() {
while (!this.isStopped()) {
try {
PullRequest pullRequest = this.pullRequestQueue.take();
if (pullRequest != null) {
this.pullMessage(pullRequest);
}
} catch (InterruptedException e) {
} catch (Exception e) {
..
}
}
}順便說一句，pullMessageService和rebalanceService都是繼承自ServiceThread
public class PullMessageService extends ServiceThread {}ServiceThread簡單封裝了線程的啓動，調用start方法，就會調用它的run方法。
public ServiceThread() {
this.thread = new Thread(this, this.getServiceName()); //把當前對象作爲runnable傳入線程構造函數
}
public void start() {
this.thread.start();
}這樣啓動線程就要方便一點，看起來舒服一點。
嗯，繼續分析之前的分析。
從pullMessageService的run方法可以看出它是從阻塞隊列pullRequestQueue裏面獲取pullRequest，如果沒有那麼將阻塞。(如果不清楚java阻塞的使用，清百度)
執行完一次pullReqeust之後，再繼續下一次獲取阻塞隊列，因爲它是個while循環。
所以，我們需要分析下pullRequest放進隊列的流程，也就是rebalanceService.
4.5 rebalanceService
public class RebalanceService extends ServiceThread {
@Override
public void run() {
while (!this.isStopped()) {
this.waitForRunning(waitInterval);
this.mqClientFactory.doRebalance();
}
}
} MQClientInstance.java
public void doRebalance() {
for (Map.Entry<String, MQConsumerInner> entry : this.consumerTable.entrySet()) {
MQConsumerInner impl = entry.getValue();
if (impl != null) {
try {
impl.doRebalance();
} catch (Throwable e) {
log.error("doRebalance exception", e);
}
}
}
}DefaultMQPushConsumerImpl.java
@Override
public void doRebalance() {
if (!this.pause) {
this.rebalanceImpl.doRebalance(this.isConsumeOrderly());
}
}RebalanceImpl.java
public void doRebalance(final boolean isOrder) {
Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
if (subTable != null) {
for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
final String topic = entry.getKey();
try {
this.rebalanceByTopic(topic, isOrder);
} catch (Throwable e) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
log.warn("rebalanceByTopic Exception", e);
}
}
}
}
this.truncateMessageQueueNotMyTopic();
}private void rebalanceByTopic(final String topic, final boolean isOrder) {
switch (messageModel) {
case BROADCASTING: {
....
case CLUSTERING: {
Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
if (mqSet != null && cidAll != null) {
List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
mqAll.addAll(mqSet);
Collections.sort(mqAll);
Collections.sort(cidAll);
AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
List<MessageQueue> allocateResult = null;
try {
allocateResult = strategy.allocate(
this.consumerGroup,
this.mQClientFactory.getClientId(),
mqAll,
cidAll);
} catch (Throwable e) {
log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
e);
return;
}
Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
if (allocateResult != null) {
allocateResultSet.addAll(allocateResult);
}
boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
if (changed) {
this.messageQueueChanged(topic, mqSet, allocateResultSet);
}
}
break;
}
default:
break;
}
}一路跟下來，來到了RebalanceImpl.java的rebalanceByTopic方法，這個方法裏面有兩個case(Broadcasting和Clustering)也就是消息消費的兩個模式，廣播和集羣消息。
廣播的話，所有的監聽者都會收到消息，集羣的話，只有一個消費者可以收到，我們以集羣消息爲例。
先大概解釋下在rebalanceByTopic裏面要做什麼。
- 從namesrv獲取broker裏面這個topic的消費者數量
- 從namesrv獲取broker這個topic的消息隊列數量
- 根據前兩部獲取的數據進行負載均衡計算，計算出當前消費者客戶端分配到的消息隊列。
- 按照分配到的消息隊列，去broker請求這個消息隊列裏面的消息。
上面代碼釐米mqset就是這個topic的消費隊列，一般是4個，但是這個值是可以修改的，存儲的位置在~/store/config/topics.json裏面，比如：
"TopicTest":{
"order":false,
"perm":6,
"readQueueNums":4,
"topicFilterType":"SINGLE_TAG",
"topicName":"TopicTest",
"topicSysFlag":0,
"writeQueueNums":4
} 可以修改readQueueNums和writeQueueNums爲其他值
try {
allocateResult = strategy.allocate(
this.consumerGroup,
this.mQClientFactory.getClientId(),
mqAll,
cidAll);
} catch (Throwable e) {
return;
}這段代碼就是客戶端根據獲取到的這個topic消費者數量和消息隊列數量，使用負載均衡策略計算出當前客戶端能夠使用的消息隊列。
負載均衡策略代碼在這個位置。
那我們繼續4.4 pullMessageService.start分析，因爲rebalanceService已經把pullRequest放到了阻塞隊列。
4.6 PullMessageService.run
@Override
public void run() {
while (!this.isStopped()) {
try {
PullRequest pullRequest = this.pullRequestQueue.take();
if (pullRequest != null) {
this.pullMessage(pullRequest);
}
} catch (InterruptedException e) {
} catch (Exception e) {
}
}
}private void pullMessage(final PullRequest pullRequest) {
final MQConsumerInner consumer = this.mQClientFactory.selectConsumer(pullRequest.getConsumerGroup());
if (consumer != null) {
DefaultMQPushConsumerImpl impl = (DefaultMQPushConsumerImpl) consumer;
impl.pullMessage(pullRequest);
} else {
}
}調用到DefaultMQPushConsumerImpl.pullMessage(pullRequest)這個方法裏面。
4.6.1
public void pullMessage(final PullRequest pullRequest) {
...
final long beginTimestamp = System.currentTimeMillis();
PullCallback pullCallback = new PullCallback() {
@Override
public void onSuccess(PullResult pullResult) {
System.out.printf("pullcallback onsuccess: " + pullResult + " %n");
if (pullResult != null) {
pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
subscriptionData);
switch (pullResult.getPullStatus()) {
case FOUND:
long firstMsgOffset = Long.MAX_VALUE;
if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
} else {
DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
pullResult.getMsgFoundList(),
processQueue,
pullRequest.getMessageQueue(),
dispathToConsume);
}
break;
}
}
}
@Override
public void onException(Throwable e) {
DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
}
};
try {
this.pullAPIWrapper.pullKernelImpl(
pullRequest.getMessageQueue(),
subExpression,
subscriptionData.getExpressionType(),
subscriptionData.getSubVersion(),
pullRequest.getNextOffset(),
this.defaultMQPushConsumer.getPullBatchSize(),
sysFlag,
commitOffsetValue,
BROKER_SUSPEND_MAX_TIME_MILLIS,
CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
CommunicationMode.ASYNC,
pullCallback
);
} catch (Exception e) {
this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
}
}上面這段代碼主要就是設置消息獲取後的回調函數PullCallback pullCallback，然後調用pullAPIWrapper.pullKernelImpl去Broker裏面獲取消息。
獲取成功後，就會回調pullCallback的onSuccess方法的FOUND case分支。
在pullCallback的onSucess方法的FOUND case分支，會根據回調是同步還是異步，分爲兩種情況，如下：
同步消息和異步消息區別的源代碼實現以後再講。