一、集羣搭建可選方式
RocketMQ的物理部署結構圖如下:
Producer和Consumer對應的是我們的應用程序,多個NameServer實例組成集羣,但相互獨立,沒有信息交換,所以對於NameServer來說部署兩個或兩個以上即可保證高可用,對於Broker來說,我們可以選擇以下幾種集羣部署方式:
1.單Master模式
這種方式風險較大,一旦Broker重啓或者宕機時,會導致整個服務不可用。不建議線上環境使用,可以用於本地測試。
2.多Master模式
一個集羣無Slave,全是Master,例如2個Master或者3個Master,這種模式的優缺點如下:
優點:配置簡單,單個Master宕機或重啓維護對應用無影響,在磁盤配置爲RAID10時,即使機器宕機不可恢復情況下,由於RAID10磁盤非常可靠,消息也不會丟(異步刷盤丟失少量消息,同步刷盤一條不丟),性能最高;
缺點:單臺機器宕機期間,這臺機器上未被消費的消息在機器恢復之前不可訂閱,消息實時性會受到影響。
3.多Master多Slave模式-異步複製
每個Master配置一個Slave,有多對Master-Slave,HA採用異步複製方式,主備有短暫消息延遲(毫秒級),這種模式的優缺點如下:
優點:即使磁盤損壞,消息丟失的非常少,且消息實時性不會受影響,同時Master宕機後,消費者仍然可以從Slave消費,而且此過程對應用透明,不需要人工干預,性能同多Master模式幾乎一樣;
缺點:Master宕機,磁盤損壞情況下會丟失少量消息。
4.多Master多Slave模式-同步雙寫
每個Master配置一個Slave,有多對Master-Slave,HA採用同步雙寫方式,即只有主備都寫成功,才嚮應用返回成功,這種模式的優缺點如下:
優點:數據與服務都無單點故障,Master宕機情況下,消息無延遲,服務可用性與數據可用性都非常高;
缺點:性能比異步複製模式略低(大約低10%左右),發送單個消息的RT會略高,且目前版本在主節點宕機後,備機不能自動切換爲主機。
本篇文章介紹如何用兩臺服務器搭建雙Nameserver、雙主Broker、雙從Broker、無單點故障的高可用RocketMQ集羣,兩臺服務器IP分別爲:192.168.31.186和192.168.31.231。
二、安裝環境
- Linux版本:CenterOS 7
- RocketMQ版本:4.7.0
- Java版本:jdk1.8
三、安裝步驟
1.安裝java環境
在兩臺機器上分別安裝Java環境,以其中一臺爲例。
1.1.下載jdk安裝包
可以上Oracle官網下載Linux版的jdk安裝包,jdk-8u161-linux-x64.tar.gz
1.2.上傳jdk包到服務器
在/usr/local目錄下創建一個java目錄,並將jdk包上傳至java目錄下
cd /usr/local/
mkdir java
解壓jdk安裝包
tar zxvf jdk-8u161-linux-x64.tar.gz
1.3.配置環境變量
vi /etc/profile
在文件的最下方添加
JAVA_HOME=/usr/local/java/jdk1.8.0_161
CLASSPATH=.:$JAVA_HOME/lib/
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME CLASSPATH
使環境變量修改生效
source /etc/profile
1.4.驗證安裝是否成功
java -version
2.安裝NameServer
在兩臺機器上分別安裝啓動NameServer,以其中一臺爲例。
2.1.創建data目錄並進入到data目錄
mkdir /data
2.2.下載RocketMQ安裝包
[root@rocketmq-2 data]# wget https://mirror.bit.edu.cn/apache/rocketmq/4.7.0/rocketmq-all-4.7.0-bin-release.zip
2.3.解壓RocketMQ安裝包
[root@rocketmq-2 data]# unzip rocketmq-all-4.7.0-bin-release.zip
2.4.啓動NameServer
[root@rocketmq-2 data]# nohup sh /data/rocketmq-all-4.7.0-bin-release/bin/mqnamesrv &
2.5.驗證NameServer是否啓動成功
[root@rocketmq-2 data]# jps
1842 NamesrvStartup
通過jps命令查看有NameServer進程表示啓動成功,NameServer的日誌文件在/root/logs/rocketmqlogs目錄中,通過/data/rocketmq-all-4.7.0-bin-release/conf/logback_namesrv.xml可以配置日誌文件目錄。
2.6 停止NameServer
[root@rocketmq-2 rocketmq-all-4.7.0-bin-release]# sh /data/rocketmq-all-4.7.0-bin-release/bin/mqshutdown namesrv
3.安裝Broker
每臺機器上都要啓動一個Master角色和Slave角色的Broker,並互爲主備,即在A機器上啓動broker-a的master節點、broker-b-s的slave節點;在B機器上啓動broker-b的master節點、broker-a-s的slave節點。
在conf目錄下提供了幾種集羣方式配置文件的示例,2m-noslave=雙master模式;2m-2s-sync=雙master雙slave同步雙寫模式;2m-2s-async=雙master雙slave異步複製模式。
本次安裝採用2m-2s-async模式
3.1 編輯broker配置文件
在192.168.31.186機器上的Master Broker的配置文件broker-a.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#所屬集羣名字
brokerClusterName=rocketmq-cluster
#broker名字,注意此處不同的配置文件填寫的不一樣 例如:在a.properties 文件中寫 broker-a 在b.properties 文件中寫 broker-b
brokerName=broker-a
#0 表示 Master,>0 表示 Slave
brokerId=0
#刪除文件時間點,默認凌晨 4點
deleteWhen=04
#文件保留時間,默認 48 小時
fileReservedTime=120
#Broker 的角色,ASYNC_MASTER=異步複製Master,SYNC_MASTER=同步雙寫Master,SLAVE=slave節點
brokerRole=ASYNC_MASTER
#刷盤方式,ASYNC_FLUSH=異步刷盤,SYNC_FLUSH=同步刷盤
flushDiskType=SYNC_FLUSH
#Broker 對外服務的監聽端口
listenPort=10911
#nameServer地址,這裏nameserver是單臺,如果nameserver是多臺集羣的話,就用分號分割(即namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.31.186:9876;192.168.31.231:9876
#每個topic對應隊列的數量,默認爲4,實際應參考consumer實例的數量,值過小不利於consumer負載均衡
defaultTopicQueueNums=8
#是否允許 Broker 自動創建Topic,生產建議關閉
autoCreateTopicEnable=true
#是否允許 Broker 自動創建訂閱組,生產建議關閉
autoCreateSubscriptionGroup=true
#設置BrokerIP
brokerIP1=192.168.31.186
#存儲路徑
storePathRootDir=/data/rocketmq-all-4.7.0-bin-release/data/store-a
#commitLog 存儲路徑
storePathCommitLog=/data/rocketmq-all-4.7.0-bin-release/data/store-a/commitlog
#消費隊列存儲路徑存儲路徑
storePathConsumerQueue=/data/rocketmq-all-4.7.0-bin-release/data/store-a/consumequeue
#消息索引存儲路徑
storePathIndex=/data/rocketmq-all-4.7.0-bin-release/data/store-a/index
#checkpoint 文件存儲路徑
storeCheckpoint=/data/rocketmq-all-4.7.0-bin-release/data/store-a/checkpoint
#abort 文件存儲路徑
abortFile=/data/rocketmq-all-4.7.0-bin-release/data/store-a/abort
#commitLog每個文件的大小默認1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每個文件默認存30W條,根據業務情況調整
mapedFileSizeConsumeQueue=300000
在192.168.31.186機器上的Slave Broker的配置文件broker-b-s.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#所屬集羣名字
brokerClusterName=rocketmq-cluster
#broker名字,注意此處不同的配置文件填寫的不一樣 例如:在a.properties 文件中寫 broker-a 在b.properties 文件中寫 broker-b
brokerName=broker-b
#0 表示 Master,>0 表示 Slave
brokerId=1
#刪除文件時間點,默認凌晨 4點
deleteWhen=04
#文件保留時間,默認 48 小時
fileReservedTime=120
#Broker 的角色,ASYNC_MASTER=異步複製Master,SYNC_MASTER=同步雙寫Master,SLAVE=slave節點
brokerRole=SLAVE
#刷盤方式,ASYNC_FLUSH=異步刷盤,SYNC_FLUSH=同步刷盤
flushDiskType=SYNC_FLUSH
#Broker 對外服務的監聽端口
listenPort=11011
#nameServer地址,這裏nameserver是單臺,如果nameserver是多臺集羣的話,就用分號分割(即namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.31.186:9876;192.168.31.231:9876
#每個topic對應隊列的數量,默認爲4,實際應參考consumer實例的數量,值過小不利於consumer負載均衡
defaultTopicQueueNums=8
#是否允許 Broker 自動創建Topic,生產建議關閉
autoCreateTopicEnable=true
#是否允許 Broker 自動創建訂閱組,生產建議關閉
autoCreateSubscriptionGroup=true
#設置BrokerIP
brokerIP1=192.168.31.186
#存儲路徑
storePathRootDir=/data/rocketmq-all-4.7.0-bin-release/data/store-b
#commitLog 存儲路徑
storePathCommitLog=/data/rocketmq-all-4.7.0-bin-release/data/store-b/commitlog
#消費隊列存儲路徑存儲路徑
storePathConsumerQueue=/data/rocketmq-all-4.7.0-bin-release/data/store-b/consumequeue
#消息索引存儲路徑
storePathIndex=/data/rocketmq-all-4.7.0-bin-release/data/store-b/index
#checkpoint 文件存儲路徑
storeCheckpoint=/data/rocketmq-all-4.7.0-bin-release/data/store-b/checkpoint
#abort 文件存儲路徑
abortFile=/data/rocketmq-all-4.7.0-bin-release/data/store-b/abort
#commitLog每個文件的大小默認1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每個文件默認存30W條,根據業務情況調整
mapedFileSizeConsumeQueue=300000
在192.168.31.231機器上的Master Broker的配置文件broker-b.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#所屬集羣名字
brokerClusterName=rocketmq-cluster
#broker名字,注意此處不同的配置文件填寫的不一樣 例如:在a.properties 文件中寫 broker-a 在b.properties 文件中寫 broker-b
brokerName=broker-b
#0 表示 Master,>0 表示 Slave
brokerId=0
#刪除文件時間點,默認凌晨 4點
deleteWhen=04
#文件保留時間,默認 48 小時
fileReservedTime=120
#Broker 的角色,ASYNC_MASTER=異步複製Master,SYNC_MASTER=同步雙寫Master,SLAVE=slave節點
brokerRole=ASYNC_MASTER
#刷盤方式,ASYNC_FLUSH=異步刷盤,SYNC_FLUSH=同步刷盤
flushDiskType=SYNC_FLUSH
#Broker 對外服務的監聽端口
listenPort=10911
#nameServer地址,這裏nameserver是單臺,如果nameserver是多臺集羣的話,就用分號分割(即namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.31.186:9876;192.168.31.231:9876
#每個topic對應隊列的數量,默認爲4,實際應參考consumer實例的數量,值過小不利於consumer負載均衡
defaultTopicQueueNums=8
#是否允許 Broker 自動創建Topic,生產建議關閉
autoCreateTopicEnable=true
#是否允許 Broker 自動創建訂閱組,生產建議關閉
autoCreateSubscriptionGroup=true
#設置BrokerIP
brokerIP1=192.168.31.231
#存儲路徑
storePathRootDir=/data/rocketmq-all-4.7.0-bin-release/data/store-b
#commitLog 存儲路徑
storePathCommitLog=/data/rocketmq-all-4.7.0-bin-release/data/store-b/commitlog
#消費隊列存儲路徑存儲路徑
storePathConsumerQueue=/data/rocketmq-all-4.7.0-bin-release/data/store-b/consumequeue
#消息索引存儲路徑
storePathIndex=/data/rocketmq-all-4.7.0-bin-release/data/store-b/index
#checkpoint 文件存儲路徑
storeCheckpoint=/data/rocketmq-all-4.7.0-bin-release/data/store-b/checkpoint
#abort 文件存儲路徑
abortFile=/data/rocketmq-all-4.7.0-bin-release/data/store-b/abort
#commitLog每個文件的大小默認1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每個文件默認存30W條,根據業務情況調整
mapedFileSizeConsumeQueue=300000
在192.168.31.231機器上的Slave Broker的配置文件broker-a-s.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#所屬集羣名字
brokerClusterName=rocketmq-cluster
#broker名字,注意此處不同的配置文件填寫的不一樣 例如:在a.properties 文件中寫 broker-a 在b.properties 文件中寫 broker-b
brokerName=broker-a
#0 表示 Master,>0 表示 Slave
brokerId=1
#刪除文件時間點,默認凌晨 4點
deleteWhen=04
#文件保留時間,默認 48 小時
fileReservedTime=120
#Broker 的角色,ASYNC_MASTER=異步複製Master,SYNC_MASTER=同步雙寫Master,SLAVE=slave節點
brokerRole=SLAVE
#刷盤方式,ASYNC_FLUSH=異步刷盤,SYNC_FLUSH=同步刷盤
flushDiskType=SYNC_FLUSH
#Broker 對外服務的監聽端口
listenPort=11011
#nameServer地址,這裏nameserver是單臺,如果nameserver是多臺集羣的話,就用分號分割(即namesrvAddr=ip1:port1;ip2:port2;ip3:port3)
namesrvAddr=192.168.31.186:9876;192.168.31.231:9876
#每個topic對應隊列的數量,默認爲4,實際應參考consumer實例的數量,值過小不利於consumer負載均衡
defaultTopicQueueNums=8
#是否允許 Broker 自動創建Topic,生產建議關閉
autoCreateTopicEnable=true
#是否允許 Broker 自動創建訂閱組,生產建議關閉
autoCreateSubscriptionGroup=true
#設置BrokerIP
brokerIP1=192.168.31.231
#存儲路徑
storePathRootDir=/data/rocketmq-all-4.7.0-bin-release/data/store-a
#commitLog 存儲路徑
storePathCommitLog=/data/rocketmq-all-4.7.0-bin-release/data/store-a/commitlog
#消費隊列存儲路徑存儲路徑
storePathConsumerQueue=/data/rocketmq-all-4.7.0-bin-release/data/store-a/consumequeue
#消息索引存儲路徑
storePathIndex=/data/rocketmq-all-4.7.0-bin-release/data/store-a/index
#checkpoint 文件存儲路徑
storeCheckpoint=/data/rocketmq-all-4.7.0-bin-release/data/store-a/checkpoint
#abort 文件存儲路徑
abortFile=/data/rocketmq-all-4.7.0-bin-release/data/store-a/abort
#commitLog每個文件的大小默認1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue每個文件默認存30W條,根據業務情況調整
mapedFileSizeConsumeQueue=300000
3.2 啓動broker
啓動時,先啓動兩臺機器上的Master節點,再啓動兩臺機器上的Slave節點。
192.168.31.186上啓動broker-a
[root@rocketmq-1 rocketmq-all-4.7.0-bin-release]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a.properties &
192.168.31.231上啓動broker-b
[root@rocketmq-2 rocketmq-all-4.7.0-bin-release]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b.properties &
192.168.31.231上啓動broker-a-s
[root@rocketmq-2 rocketmq-all-4.7.0-bin-release]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a-s.properties &
192.168.31.186上啓動broker-b-s
[root@rocketmq-1 rocketmq-all-4.7.0-bin-release]# nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b-s.properties &
3.3 注意事項
3.3.1 內存不足
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000005c0000000, 8589934592, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /data/rocketmq-all-4.7.0-bin-release/hs_err_pid1841.log
- 原因:broker的java啓動內存參數配置的是8g,請根據服務器實際資源做調整。
- 解決方案:修改內存參數,在bin/runbroker.sh文件中
JAVA_OPT="${JAVA_OPT} -server -Xms512m -Xmx512m -Xmn512m"
3.3.2 開放RocketMQ相關端口
NameServer的9876端口、Broker的10911、11011、10909、11009端口
可以選擇關閉服務器防火牆或者開放相應的端口
3.4.驗證
查看broker.log日誌文件輸出正常,日誌文件的目錄在/data/rocketmq-all-4.7.0-bin-release/conf/logback_broker.xml可以查看/修改
3.5. mqadmin工具
- 查看集羣情況
[root@rocketmq-1 rocketmq-all-4.7.0-bin-release]# sh bin/mqadmin clusterList -n "192.168.31.186:9876;192.168.31.231:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
rocketmq-cluster broker-a 0 192.168.31.186:10911 V4_7_0 0.00(0,0ms) 0.00(0,0ms) 0 441917.21 -1.0000
rocketmq-cluster broker-a 1 192.168.31.231:11011 V4_7_0 0.00(0,0ms) 0.00(0,0ms) 0 441917.21 0.1308
rocketmq-cluster broker-b 0 192.168.31.231:10911 V4_7_0 0.00(0,0ms) 0.00(0,0ms) 0 441917.21 -1.0000
rocketmq-cluster broker-b 1 192.168.31.186:11011 V4_7_0 0.00(0,0ms) 0.00(0,0ms) 0 441917.21 -1.0000
mqadmin還有其他的用法,具體的可以參考文檔。
3.6.關閉服務
先關閉Broker、再關閉NameServer,服務啓動的時候正好相反。
[root@rocketmq-1 rocketmq-all-4.7.0-bin-release]# sh bin/mqshutdown broker
[root@rocketmq-1 rocketmq-all-4.7.0-bin-release]# sh bin/mqshutdown namesrv
4.安裝rocketmq-console(可選)
rocketmq-console是RocketMQ項目的擴展插件,是一個圖形化管理控制檯,提供Broker集羣狀態查看,Topic管理,Producer、Consumer狀態展示,消息查詢等常用功能,這個功能在安裝好RocketMQ後需要額外單獨安裝、運行。
在本地有git+maven的開發環境可以自行在https://github.com/apache/rocketmq-externals克隆下載項目後,再通過maven打包rocketmq-console,或者可以在服務器上參考如下步驟進行。
4.1 安裝maven
- 下載安裝包
[root@rocketmq-1 data]# wget http://mirrors.cnnic.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
- 解壓
[root@rocketmq-1 data]# tar zxvf apache-maven-3.5.4-bin.tar.gz
- 配置環境變量
[root@rocketmq-1 data]# vi /etc/profile
在最後輸入:
export MAVEN_HOME=/data/apache-maven-3.5.4
export PATH=$MAVEN_HOME/bin:$PATH
保存後退出使之生效
[root@rocketmq-1 data]# source /etc/profile
- 驗證
[root@rocketmq-1 data]# mvn -v
4.2 下載打包rocketmq-console
在https://github.com/apache/rocketmq-externals頁面下載zip包上傳至任意一個服務器上。
- 解壓
[root@rocketmq-1 data]# unzip rocketmq-externals-master.zip
- 進入rocketmq-console目錄
[root@rocketmq-1 data]# cd rocketmq-externals-master/rocketmq-console/
- 打包
[root@rocketmq-1 rocketmq-console]# mvn clean package -Dmaven.test.skip=true
4.3 啓動rocketmq-console
[root@rocketmq-1 rocketmq-console]# java -jar target/rocketmq-console-ng-1.0.1.jar --rocketmq.config.namesrvAddr='192.168.31.186:9876;192.168.31.231:9876'
4.4 驗證
啓動無異常後,訪問http://192.168.31.186:8080即可看到如下界面: