一、問題思考
消息拉取在實踐過程中,有以下幾個問題需要考慮:
1、如何全量拉取消息?
2、如何指定MessageQueue從指定offset處拉取消息?
3、如何更新MessageQueue的Offset標誌位?
4、Pull模式下如何實現負載均衡?
二、Pull模式下常用Demo
1、更新MessageQueue的Offset標誌位
consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
2、Pull模式簡單Demo
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("PullConsumer");
consumer.setNamesrvAddr("10.72.86.241:9876;10.72.86.229:9876");
consumer.start();
// 從指定topic中拉取所有消息隊列
Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("user-info");
for (MessageQueue mq : mqs) {
// 獲取消息的offset,指定從store中獲取
long offset = consumer.fetchConsumeOffset(mq, true);
System.out.println("consumer from the queue:" + mq + ":" + offset);
try {
while (true) {
PullResult pullResult = consumer.pullBlockIfNotFound(mq,
null, consumer.fetchConsumeOffset(mq, false), 1);
consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
switch (pullResult.getPullStatus()) {
case FOUND:
List<MessageExt> messageExtList = pullResult.getMsgFoundList();
for (MessageExt m : messageExtList) {
System.out.println(m.toString());
}
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break;
case OFFSET_ILLEGAL:
break;
default:
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
consumer.shutdown();
}
3、指定MessageQueue拉取消息
public class PullMQConsumerMain {
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("PullConsumer");
consumer.setNamesrvAddr("10.72.86.241:9876;10.72.86.229:9876");
consumer.start();
try {
//從指定topic的MessageQueue中拉取消息
MessageQueue mq = new MessageQueue();
mq.setQueueId(0);
mq.setTopic("user-info");
mq.setBrokerName("broker-a");
//指定獲取消息的offset
long offset = 26;
PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, offset, 32);
//更新offset
consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
System.out.println("最新 offset :" + consumer.fetchConsumeOffset(mq, false));
switch (pullResult.getPullStatus()) {
case FOUND:
List<MessageExt> messageExtList = pullResult
.getMsgFoundList();
for (MessageExt m : messageExtList) {
System.out.println(m.toString());
}
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break;
case OFFSET_ILLEGAL:
break;
default:
break;
}
} catch (Exception e) {
e.printStackTrace();
}
consumer.shutdown();
}
}
4、Pull模式下負載均衡(MQPullConsumerScheduleService)
public class PullScheduleMain {
public static void main(String[] args) throws MQClientException {
final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService("PullSchedule");
scheduleService.getDefaultMQPullConsumer().setNamesrvAddr("10.72.86.241:9876;10.72.86.229:9876");
scheduleService.setMessageModel(MessageModel.CLUSTERING);
scheduleService.registerPullTaskCallback("user-info", new PullTaskCallback() {
@Override
public void doPullTask(MessageQueue mq, PullTaskContext context) {
// TODO Auto-generated method stub
MQPullConsumer consumer = context.getPullConsumer();
try {
long offset = consumer.fetchConsumeOffset(mq, false);
if(offset < 0)
offset = 0;
PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, offset, 4);
switch (pullResult.getPullStatus()) {
case FOUND:
List<MessageExt> messageExtList = pullResult
.getMsgFoundList();
for (MessageExt m : messageExtList) {
System.out.println(m.toString());
}
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break;
case OFFSET_ILLEGAL:
break;
default:
break;
}
consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
context.setPullNextDelayTimeMillis(10000);
} catch (Exception e) {
e.printStackTrace();
}
}
});
scheduleService.start();
}
}
三、總結
分別用四個例子回答了上面的4個問題,後續會通過源碼解析的方式描述Pull模式的實現機制。
遺留問題:
調用fetchMessageQueuesInBalance函數時,返回MessageQueue爲空