Spark搭建 之 Standalone集羣

本文的主線 系統 => 集羣 => 測試

本文基於Spark搭建 之 單機模式以及Spark第一個應用

系統

保證集羣ssh互通

hostname

sudo hostname master
# sudo hostname slave01
# sudo hostname slave02

sudo vim /etc/hostname
# master or slave01 or slave02

hosts

sudo vim /etc/hosts
# 192.168.1.29 master
# 192.168.1.30 slave01
# 192.168.1.31 slave02

集羣

# master
cd /opt/services

wget https://mirror.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz

tar xf spark-2.4.7-bin-hadoop2.7.tgz

mv spark-2.4.7-bin-hadoop2.7 spark

cd spark

cp conf/slaves.template conf/slaves

vim conf/slaves
# slave01
# slave02

cp conf/spark-env.sh.template conf/spark-env.sh

vim conf/spark-env.sh
# export JAVA_HOME=/opt/services/jdk
# SPARK_MASTER_PORT=master
# SPARK_MASTER_PORT=7077

scp -r /opt/services/spark [email protected]:/opt/services

scp -r /opt/services/spark [email protected]:/opt/services

方法1

# master
/opt/services/spark/sbin/start-all.sh

方法2

# master
/opt/services/spark/sbin/start-master.sh

# slave01 & slave02
/opt/services/spark/sbin/start-slave.sh spark://master:7077

• http://192.168.1.29:8080

測試

# slave01
echo apple >> /opt/services/spark/README.md
# slave02
echo blue >> /opt/services/spark/README.md

• 基於Spark第一個應用

/opt/services/spark/bin/spark-submit --class "SimpleApp" --master spark://master:7077 ~/HelloSpark.jar
# Lines with a: 61, Lines with b: 31

參考

• Spark Standalone Mode