Spark搭建 之 高可用集羣

本文主線 準備 => ZooKeeper => Spark => 測試

本文基於Spark搭建 之 Standalone集羣

準備

# master
/opt/services/spark/sbin/stop-all.sh

ZooKeeper

# master
cd /opt/services

wget https://mirror.tuna.tsinghua.edu.cn/apache/zookeeper/stable/apache-zookeeper-3.6.2-bin.tar.gz

tar xf apache-zookeeper-3.6.2-bin.tar.gz

mv apache-zookeeper-3.6.2-bin zookeeper

cd zookeeper

mkdir data

vim conf/zoo.cfg
# tickTime=2000
# initLimit=10
# syncLimit=5
# dataDir=/opt/services/zookeeper/data
# clientPort=2181
# server.1=master:2888:3888
# server.2=slave01:2888:3888
# server.3=slave02:2888:3888

echo 1 > /opt/services/zookeeper/data/myid

scp -r /opt/services/zookeeper op@slave01:/opt/services

# ssh op@slave01
echo 2 > /opt/services/zookeeper/data/myid

scp -r /opt/services/zookeeper op@slave02:/opt/services

# ssh op@slave02
echo 3 > /opt/services/zookeeper/data/myid

# matser
/opt/services/zookeeper/bin/zkServer.sh start

# slave01 slave02
/opt/services/zookeeper/bin/zkServer.sh start

Spark

vim /opt/services/spark/conf/spark-env.sh
# export JAVA_HOME=/opt/services/jdk
# SPARK_MASTER_WEBUI_PORT=8090 # 和ZooKeeper衝突
# export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=master:2181,slave01:2181,slave02:2181 -Dspark.deploy.zookeeper.dir=/spark"

scp -r /opt/services/spark op@slave01:/opt/services

scp -r /opt/services/spark op@slave02:/opt/services

# master
/opt/services/spark/sbin/start-all.sh

• http://192.168.1.29:8090

# slave01
/opt/services/spark/sbin/start-master.sh

• http://192.168.1.30:8090

測試

# master
/opt/services/spark/bin/spark-submit --class "SimpleApp" --master spark://master:7077,slave01:7077 ~/HelloSpark.jar

jps
# 4166 Master

sudo kill -9 4166

Lines with a: 61, Lines with b: 30

• http://192.168.1.30:8090

參考

• Spark Standalone Mode