背景
架構部署
32C 128G 1.8T 4主4從架構部署
rocketmq延遲服務生產者生產數據，消費者無法正常消費到數據，查看日誌有重大發現，所有的主節點均有這報錯
| #vim /data2/logs/rocketmqlogs/storeerror.log |
| 020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1 |
發現消費的位點全部錯亂，導致消費者無法常消費到數據
解決方法
先將broker節點的流量清除，2表示只寫權限，4表示只讀權限，6表示讀寫權限
| #bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 4 |
查看broker節點流量是否爲0,需要等待broker完全沒有流量了纔可以進行關閉broker節點操作，以防止丟數據
| #bin/mqadmin clusterList -n nameserver:9876 |
關閉broker節點進程
| #bin/mqshutdown broker |
將delayOffset.json 和delayOffset.json.bak文件拷走 此文件用於記錄消費位點的，拷走後，重啓應用進程會自動生產
#mv /data1/rocketmq/store/config/delayOffset.json* /data2/ |
將SCHEDULE_TOPIC_XXXX 文件拷走，用於調度任務，拷走後，重啓應用進程會自動生產
| #mv /data1/rocketmq/store/consumequeue/SCHEDULE_TOPIC_XXXX/ /data2/ |
將rocketmq的日誌文件全部刪除
#rm -rf /data2/logs/rocketmqlogs/* |
然後將rocketmq進程啓動
#cd /usr/local/rocketmq/bin/ #nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-async/broker-a.properties & |
把權限重新賦值
| #bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 6 |
等節點流量恢復重，按上面的操作，將集羣所有主節點都操作一遍，就能正常消費數據了，且storeerror.log日誌也沒在報錯了