Thingsboard下Cassandra集羣化Docker部署
徐景周
一、環境
- CentOS
- 安裝Nano
apt-get update
apt-get install nano
- Oracle JDK:1.8及以上
查找Java安裝路徑
which java
ls /usr/bin/java (上一步結果路徑)
ls -lrt /etc/alternatives/java (上一步結果路徑)
ls /usr/lib/jvm/java-8-openjdk-amd64 (最終路徑)
- 關閉防火牆
systemctl stop firewalld
systemctl disable firewalld
二、單機部署(Docker版)
1. 停止宿主機Cassandra(如有必要)
sudo service cassandra stop
sudo chkconfig cassandra off
sudo service cassandra status
2. 安裝Docker
- 安裝
請瀏覽【參考文獻 1】。
- 修改Docker默認目錄
sudo service docker stop
mv /var/lib/docker /home/docker
ln -s /home/docker /var/lib/docker
sudo yum install -y nano
sudo nano /usr/lib/systemd/system/docker.service
文件修改項如下:
ExecStart=/usr/bin/dockerd --graph /home/docker -H fd:// --containerd=/run/containerd/containerd.soc
sudo systemctl daemon-reload
sudo service docker restart
sudo service docker status
docker info | grep Dir
- Docker默認磁盤空間
df -h
docker info
docker exec -it tb-cassandra df -h
- 自啓動
sudo systemctl enable docker
3. Docker中安裝Cassandra
1. 操作指令
-
安裝
docker stop tb-cassandra
docker rm tb-cassandra
docker run --name tb-cassandra -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 -d cassandra:3.0
- 啓動
docker start tb-cassandra
- 檢查
docker ps -a
- 自啓動
docker update --restart=always 容器ID
2. 實現細節
請瀏覽【參考文獻 2】和【參考文獻 3】。
4. 修改Cassandra配置
1. 操作指令
- 進入容器bash
docker exec -it tb-cassandra bash
- 啓動cql環境
cqlsh
- cql環境下,修改族名
update system.local set cluster_name = 'Thingsboard Cluster' where key='local';
- cql環境下,查看族名
select cluster_name from system.local;
- cql環境下,退出
exit
- 刷新配置
nodetool flush system
- 修改cassandra.yaml配置文件
nano /etc/cassandra/cassandra.yaml
或者採用文件替換方式(先退出docker bash)
docker exec -it tb-cassandra bash
ls etc/cassandra/cassandra.yaml
exit;
docker ps
docker cp cassandra.yaml 容器id:/etc/cassandra/cassandra.yaml
docker exec -it tb-cassandra bash
cat /etc/cassandra/cassandra.yaml
Cassandra.yaml配置文件修改
創建目錄及修改屬性
ls /var/lib/cassandra/
mkdir /var/lib/cassandra/data
mkdir /var/lib/cassandra/commitlog
mkdir /var/lib/cassandra/saved_caches
mkdir /var/lib/cassandra/hints
chown 777 /var/lib/cassandra/data
chown 777 /var/lib/cassandra/commitlog
chown 777 /var/lib/cassandra/saved_caches
chown 777 /var/lib/cassandra/hints
修改文件下面的選項
cluster_name: 'Thingsboard Cluster’
hints_directory: /var/lib/cassandra/hints
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
saved_caches_directory: /var/lib/cassandra/saved_caches
seeds: "127.0.0.1”
listen_address: localhost
# broadcast_address: 1.2.3.4
start_rpc: true
rpc_address: 0.0.0.0
broadcast_rpc_address: 1.2.3.4
- 退出docker bash
exit;
- 安裝Thingsboard對應數據庫及數據表(順序:schema-ts.cql -> schema-entities.cql)
docker ps
docker cp schema-ts.cql 容器id:/root/schema-ts.cql
docker cp schema-entities.cql 容器id:/root/schema-entities.cql
docker exec -it tb-cassandra bash
ls /root
cqlsh -f /root/schema-entities.cql
cqlsh -f /root/schema-ts.cql
exit;
- 重啓Cassandra
docker restart tb-cassandra
- 查看Cassandra是否啓動正常
docker logs tb-cassandra
實現細節
請瀏覽【參考文獻4】和【參考文獻 5】。
5. 修改Thingsboard配置
- 修改文件thingsboard.conf(/usr/share/thingsboard/conf)
# DB Configuration
export DATABASE_ENTITIES_TYPE=sql
export DATABASE_TS_TYPE=cassandra
export SPRING_JPA_DATABASE_PLATFORM=org.hibernate.dialect.PostgreSQLDialect
export SPRING_DRIVER_CLASS_NAME=org.postgresql.Driver
export SPRING_DATASOURCE_URL=jdbc:postgresql://192.168.1.131:5432/thingsboard
export SPRING_DATASOURCE_USERNAME=postgres
export SPRING_DATASOURCE_PASSWORD=123456
export CASSANDRA_CLUSTER_NAME=Thingsboard Cluster
export CASSANDRA_KEYSPACE_NAME=thingsboard
export CASSANDRA_URL=192.168.1.131:9042
export CASSANDRA_USE_CREDENTIALS=false
export CASSANDRA_USERNAME=
export CASSANDRA_PASSWORD=
- 重啓Thingsboard
sudo service thingsboard restart
nano /var/log/thingsboard/thingsboard.log
- 重啓Mosquitto(可選)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d
- 重啓TB-Gateway
sudo service tb-gateway restart
nano /var/log/tb-gateway/tb-gateway.log
6. Cassandra調優
實現細節
請瀏覽【參考文獻7】和【參考文獻8】。
7. 測試
- 打開Thingsboard後,運行測試腳本。請瀏覽【參考文獻20】。
node /opt/mqtt/mqtt_gateway_mqtt_sz_new.js
- 查看tb-cassandra容器中Cassandra數據
docker exec -it tb-cassandra bash
cqlsh
select count(1) from thingsboard.ts_kv_latest_cf;
exit
exit;
8. Cassandra查詢工具
DataStax Studio
Cassandra Manager
Razorsql
9. 配置示例
192.168.1.131(Cassandra + Postgres)
192.168.1.59(Thingsboard + TB-Gateway)
三、單機部署(非Docker版)
如果不希望基於Docker,直接在機器上安裝Cassandra的情況。
1. 停止Docker版Cassandra(如有必要)
sudo service docker stop
sudo systemctl disable docker
docker info
# 殺掉Cassandra端口占用程序
netstat -tunlp
kill PID
2. 實現細節
請瀏覽【參考文獻6】和“單機部署(Docker版)-修改Cassandra配置”章節。
四、集羣部署(Docker版)
1. 停止宿主機Cassandra(如有必要)
sudo service cassandra stop
sudo chkconfig cassandra off
sudo service cassandra status
2. 安裝Docker
請參考單機部署中對應章節。
3. 安裝Ping和IFConfig
apt-get update
apt-get install inetutils-ping
apt-get install net-tools
4. 修改主機名
# 查看主機名
Hostnamectl
# 133機器
hostnamectl set-hostname 133
# 134機器
hostnamectl set-hostname 134
# 135機器
hostnamectl set-hostname 135
# 136機器
hostnamectl set-hostname 136
5. 安裝IPTables
sudo yum install -y iptables-services
systemctl enable iptables.service
/usr/libexec/iptables/iptables.init save
systemctl daemon-reload
sudo service docker restart
6. Docker之間跨主機通訊(SWARM方式)
實現細節請瀏覽【參考文獻9】。
例如:將133做爲Manager機,其它做爲Work機。
① Manager上創建指令(133機器)
docker swarm init --advertise-addr=192.168.1.133
② Work上加入指令(136機器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.136 192.168.1.133:2377
③ Work上加入指令(135機器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.135 192.168.1.133:2377
④ Work上加入指令(135機器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.134 192.168.1.133:2377
⑤ Manager上查看集羣狀態(133機器)
docker node ls
⑥ Manager上創建overlay網絡(133機器)
docker network create -d overlay --attachable cassandra1-net
docker service update --network-add cassandra1-net --network-rm cassandranet my-cassandra
docker network create -d overlay --attachable cassandranet
docker network create -d overlay --subnet=172.172.0.0/24 --attachable cassandranet
# Manager機器和其它Work機器上都可查看
docker network ls
⑦ Manager上創建swarm服務(133機器)
docker service rm my-cassandra
docker network rm cassandranet
docker service create --name my-cassandra --network cassandra1-net --replicas 1 --publish published=8080,target=80 cassandra
# 黃色標記爲機器數
docker service create --name my-cassandra --network cassandranet --replicas 5 --publish published=8080,target=80 cassandra
docker service ls
⑧ Manager上查看新建服務是否連接到新建overlay網絡(133機器)
docker network inspect cassandranet
docker network inspect cassandra1-net
⑨ 查看服務
docker ps -a
⑩ 查看節點情況
docker node ls
docker service ps my-cassandra
7. Docker中安裝Cassandra
- 操作指令
# 安裝
docker stop tb-cassandra
docker rm tb-cassandra
# 133機器
sudo docker run -d --name tb-cassandra --network=cassandranet --ip 172.172.0.133 --restart always -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 cassandra:3.0
# 134機器
sudo docker run -d --network=cassandranet --ip 172.172.0.134 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0
# 135機器
sudo docker run -d --network=cassandranet --ip 172.172.0.135 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0
# 136機器
sudo docker run -d --network=cassandranet --ip 172.172.0.136 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0
# 啓動
docker start tb-cassandra
# 檢查
docker ps -a
# 自啓動
docker update --restart=always 容器ID
8. 修改Cassandra配置
① 按單機部署相關章節,執行前6步。
② 在修改後cassandra.yaml 基礎上,再修改下面幾項內容
1) # 選取一半數量的節點物理IP,做爲Seeds
例如:
- seeds: "192.168.1.133,192.168.1.134"
2) # 替換成各節點自己的物理IP地址
例如;
# 容器IP(docker inspect tb-cassandra)
listen_address: 172.172.0.133
# hostIP
broadcast_address: 192.168.1.133
3) # 同步策略
endpoint_snitch: GossipingPropertyFileSnitch
③ 修改文件cassandra-rackdc.properties
1) dc=datacenter1
2) rack=rack1
④ 修改文件cassandra-env.sh
1) #JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=192.168.1.134"
2) LOCAL_JMX=no
3) JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
4) JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"
⑤ 分別替換各個容器節點cassandra中對應的文件:cassandra.yaml、cassandra-rackdc.properties、cassandra-env.sh文件
例如:
docker exec -it tb-cassandra bash
ls etc/cassandra/cassandra.yaml
exit;
docker ps
# 133機器
# cassandra.yaml
docker cp /root/cassandra_133.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-133.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
ls /etc/cassandra/
# 修改權限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 134機器
# cassandra.yaml
docker cp /root/cassandra_134.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-134.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改權限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 135機器
# cassandra.yaml
docker cp /root/cassandra_135.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-135.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改權限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 136機器
# cassandra.yaml
docker cp /root/cassandra_136.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-136.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改權限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 退出
exit;
⑥ 安裝Thingsboard數據庫及數據表(順序:schema-entities.cql -> schema-ts.cql)
docker ps
docker cp /root/schema-entities.cql 容器id:/root/schema-entities.cql
docker cp /root/schema-ts.cql 容器id:/root/schema-ts.cql
docker exec -it tb-cassandra bash
ls /root
cqlsh -f /root/schema-entities.cql
cqlsh -f /root/schema-ts.cql
exit;
⑦ 重啓Cassandra
docker stop tb-cassandra
docker start tb-cassandra
⑧ 查看Cassandra是否啓動正常
docker logs tb-cassandra
9. 查看集羣情況
各節點的cassandra.yaml文件依次替換後,分別啓動各個Cassandra節點,查看集羣情況。
例如:
# 任一機器
docker exec -it tb-cassandra bash
nodetool -h 192.168.1.133 -u cassandra -pw cassandra status
nodetool -h 192.168.1.134 -u cassandra -pw cassandra status
nodetool -h 192.168.1.135 -u cassandra -pw cassandra status
nodetool -h 192.168.1.136 -u cassandra -pw cassandra status
nodetool -h 192.168.1.133 -u cassandra -pw cassandra ring
nodetool -h 192.168.1.134 -u cassandra -pw cassandra ring
nodetool -h 192.168.1.135 -u cassandra -pw cassandra ring
nodetool -h 192.168.1.136 -u cassandra -pw cassandra ring
cqlsh 192.168.1.133 -u cassandra -p cassandra
ping 172.172.0.134
ping 172.172.0.135
ping 172.172.0.136
exit;
# Manager(133機器
docker node ls
docker network inspect cassandranet
10. 修改Thingsboard配置
在單機部署修改後thingsboard.conf的基礎上,修改下面內容(/usr/share/thingsboard/conf):
1) Export CASSANDRA_URL=192.168.1.133:9042,192.168.1.134:9042,192.168.1.135:9042,192.168.1.136:9042
2) export CASSANDRA_USE_CREDENTIALS=true
3) export CASSANDRA_USERNAME=cassandra
4) export CASSANDRA_PASSWORD=cassandra
重啓Thingsboard
sudo service thingsboard restart
nano /var/log/thingsboard/thingsboard.log
重啓Mosquitto(可選,MQTT)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d
重啓TB-Gateway
sudo service tb-gateway restart
nano /var/log/tb-gateway/tb-gateway.log
11. 集羣工具
Nodetool
JConsole
例如:
nodetool status
nodetool info
nodetool proxyhistograms
12. 修復無法啓動的容器
Docker容器中,由於個別文檔(例如:cassandra-env.sh)導致cassandra無法啓動。
例如:
cd /home/docker
find ./ -name 'cassandra-env.sh'
對搜索到的文檔進行比對修改,再重啓容器即可!
13. 配置示例
192.168.1.133(Cassandra)
192.168.1.134(Cassandra)
192.168.1.135(Cassandra)
192.168.1.136(Cassandra)
192.168.1.131(Postgres)
192.168.1.59(Thingsboard + TB-Gateway)
五、集羣部署(非Docker版)
如果不希望基於Docker,直接在機器上安裝Cassandra集羣的情況。
1. 停止Docker版Cassandra(如有必要)
sudo service docker stop
sudo systemctl disable docker
docker info
2. 關閉運行中YUM
rm -f /var/run/yum.pid
3. 安裝Cassandra
請瀏覽【參考文獻6】
4. 修改配置文件
① 修改Cassandra默認族名
例如:
cqlsh
update system.local set cluster_name = 'Thingsboard Cluster' where key='local';
select cluster_name from system.local;
exit
nodetool flush system
② 修改配置文件
請瀏覽【參考文獻18】、【參考文獻19】,以及“單機部署(Docker版)-修改Cassandra配置”章節內容,進行修改。
例如:
sudo nano /etc/thingsboard/conf/thingsboard.conf
sudo nano /etc/cassandra/default.conf/cassandra.yaml
備註:其中修改的文件(cassandra.yaml和thingsboard.conf)
③ 安裝Thingsboard數據庫及數據表
備註:數據庫及表只需任選一臺安裝,其它節點會定時自動同步。執行順序:schema-entities.cql -> schema-ts.cql
例如:
# 安裝在133機器
ls /root
cqlsh 192.168.1.133 -u cassandra -p cassandra -f /root/schema-entities.cql
cqlsh 192.168.1.133 -u cassandra -p cassandra -f /root/schema-ts.cql
# 查看是否創建
cqlsh 192.168.1.133 -u cassandra -p cassandra
select * from thingsboard.user;
select count(1) from thingsboard.ts_kv_latest_cf;
exit
5. 查看集羣情況
重啓各cassandra節點,查看集羣情況。
例如:
# 如果出現前後datacenter不一致問題,執行下面語句
rm -rf /var/lib/cassandra/data/*
ls /var/lib/cassandra/data
# 殺掉Cassandra端口占用程序
netstat -tunlp
kill PID
sudo systemctl stop cassandra
sudo systemctl start cassandra
nano /var/log/cassandra/cassandra.log
sudo systemctl status cassandra
# 查看集羣
nodetool status
cqlsh 192.168.1.133 -u cassandra -p cassandra
nodetool -h 192.168.1.133 -u cassandra -pw cassandra status
nodetool -h 192.168.1.133 -u cassandra -pw cassandra ring
# 查看庫、表同步狀況
# 136機器
cqlsh 192.168.1.136 -u cassandra -p cassandra
select * from thingsboard.user;
select count(1) from thingsboard.ts_kv_latest_cf;
# 135機器
cqlsh 192.168.1.135 -u cassandra -p cassandra
select * from thingsboard.user;
# 查看集羣節點
select peer from system.peers;
# Cassandra各節點成功啓動後,查看日誌(cassandra.log)
6. 修改system_auth表的配置
此步驟可選,可視情況而定。
例如:
# 133機器上,”dc1”爲數據中心名(可通過nodetool status獲取)
cqlsh 192.168.1.133 -u cassandra -p cassandra
ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy','dc1': '5'};
exit
nodetool repair
# 其它機器上進行同步
nodetool repair
7. 測試
替換配置文件(thingsboard.conf)後,重啓Thingsboard和TB-Gateway,運行測試腳本進行測試!
例如:
# 重啓Mosquitto(可選)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d
# 重啓Thingsboard
sudo systemctl restart thingsboard
nano /var/log/thingsboard/thingsboard.log
sudo systemctl status thingsboard
# 重啓TB-Gateway
sudo systemctl restart tb-gateway
nano /var/log/tb-gateway/tb-gateway.log
sudo systemctl status tb-gateway
# 運行測試腳本
node /opt/mqtt/mqtt_gateway_mqtt_sz_new.js
# Thingsboard成功運行後,查看日誌(thingsboard.log)
3. 查看數據庫負載均衡及同步情況
測試成功運行後,可在任一臺Cassandra節點查看庫狀況。
例如:
# 133機器上
cqlsh 192.168.1.133 -u cassandra -p cassandra
select count(1) from thingsboard.ts_kv_latest_cf;
exit
9. 配置示例
192.168.1.133(Cassandra)
192.168.1.134(Cassandra)
192.168.1.135(Cassandra)
192.168.1.136(Cassandra)
192.168.1.131(Postgres)
192.168.1.59(Thingsboard + TB-Gateway)
參考文獻
- https://docs.docker.com/install/linux/docker-ce/centos/
- https://hub.docker.com/_/cassandra/
- https://thingsboard.io/docs/user-guide/install/docker/
- https://stackoverflow.com/questions/22006887/cassandra-saved-cluster-name-test-cluster-configured-name
- https://stackoverflow.com/questions/36133127/how-to-configure-cassandra-for-remote-connection
- https://thingsboard.io/docs/user-guide/install/rhel/?rhelThingsboardDatabase=hybrid
- https://docs.datastax.com/en/ddac/doc/datastax_enterprise/config/configRecommendedSettings.html#configRecommendedSettings
- https://www.jianshu.com/p/5bacb06e334b
- https://docs.docker.com/v17.12/network/network-tutorial-overlay/#walkthrough
- https://cloud.tencent.com/developer/article/1110563
- https://www.ibm.com/developerworks/cn/opensource/os-cn-cassandraxu1/index.html
- https://www.jianshu.com/p/b970876ee3cf
- https://blog.csdn.net/chenxingzhen001/article/details/8729623
- https://stackoverflow.com/questions/38405983/apache-cassandra-3-7-snitch-issue-cannot-start-data-center/39150744#39150744
- https://blog.csdn.net/walkerjong/article/details/52221126
- https://www.cnblogs.com/51kata/p/5276407.html
- https://www.cnblogs.com/capcom923/p/deploy_dockerize_cassandra_HA_cluster.html
- https://thingsboard.io/docs/user-guide/install/cluster-setup/
- https://thingsboard.io/docs/reference/performance-tests/