spark集羣1.6.2升級爲2.0.2

參考文檔

http://dblab.xmu.edu.cn/blog/1187-2/ Spark 2.0分佈式集羣環境搭建
http://blog.csdn.net/andy572633/article/details/7211546 linux下殺死進程（kill）的N種方法

更改原有spark目錄名

可提前stop-all
將原有的spark改爲spark1.6.2

sudo mv /usr/local/spark  /usr/local/saprk1.6.2

Master安裝spark2.0.2

下載安裝包 spark-2.0.2-bin-without-hadoop.tgz

sudo tar -zxf ~/下載/spark-2.0.2-bin-without-hadoop.tgz -C /usr/local/
cd /usr/local
sudo mv ./spark-2.0.2-bin-without-hadoop/ ./spark
sudo chown -R hadoop ./spark

vim ~/.bashrc

在.bashrc添加如下配置：

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Master spark配置

slaves

將 slaves.template 拷貝到 slaves

cd /usr/local/spark/
cp ./conf/slaves.template ./conf/slaves

添加slaves

spark-env.sh

將 spark-env.sh.template 拷貝到 spark-env.sh

cp ./conf/spark-env.sh.template ./conf/spark-env.sh

添加內容如下：

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_IP=xxx (具體ip)

分發到slaves

將spark文件夾發送到各個node上：

cd /usr/local/
tar -zcf ~/spark.master.tar.gz ./spark
cd ~
scp ./spark.master.tar.gz n01:/home/hadoop
scp ./spark.master.tar.gz n02:/home/hadoop
scp ./spark.master.tar.gz n03:/home/hadoop
scp ./spark.master.tar.gz n04:/home/hadoop
scp ./spark.master.tar.gz n05:/home/hadoop
scp ./spark.master.tar.gz n06:/home/hadoop
scp ./spark.master.tar.gz n07:/home/hadoop
scp ./spark.master.tar.gz n08:/home/hadoop
scp ./spark.master.tar.gz n09:/home/hadoop
scp ./spark.master.tar.gz n10:/home/hadoop

在n01…n10上執行：

sudo rm -rf /usr/local/spark/
sudo tar -zxf ~/spark.master.tar.gz -C /usr/local
sudo chown -R hadoop /usr/local/spark

啓動spark集羣

cd /usr/local/spark/
sbin/start-master.sh

打開http://m01:8080,如下

左上角可看到已升級爲spark2.0.2

可能出現的問題

如果jps出現多個Master進程(可通過web ui 8080 8081 等端口反饋)
1.先停止當前spark-all
2.jps找到多餘的Master PID
3.然後kill終結此Master進程

ps -ef | grep master
kill -s 9 master_pid

其中-s 9 制定了傳遞給進程的信號是９，即強制、儘快終止進程。
世界清靜.