安裝高可用Hadoop生態 （四） 安裝Spark

4.    安裝Spark

4.1. 準備目錄

tar -xf /home/hadoop/install/spark-2.1.0-bin-without-hadoop.tgz -C /opt/cloud/packages/
ln -s /opt/cloud/packages/spark-2.1.0-bin-without-hadoop  /opt/cloud/bin/spark
ln -s /opt/cloud/packages/spark-2.1.0-bin-without-hadoop/conf  /opt/cloud/etc/spark
mkdir -p /opt/cloud/logs/spark
mkdir -p /opt/cloud/data/spark

4.2. 設置環境變量

vi ~/.bashrc
export SPARK_HOME=/opt/cloud/bin/spark
export PATH=$SPARK_HOME/bin:$PATH

即刻生效

source ~/.bashrc

4.3. 修改配置文件

4.3.1.    spark-env.sh

cd /opt/cloud/etc/spark
mv spark-env.sh.template spark-env.sh
vi spark-env.sh

export JAVA_HOME=/usr/lib/jvm/java
export HADOOP_HOME=/opt/cloud/bin/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_DIST_CLASSPATH=$(hadoop classpath)[1]
export SPARK_LOCAL_DIRS=/opt/cloud/data/spark
export SPARK_LOG_DIR=/opt/cloud/logs/spark
export SPARK_PID_DIR=/opt/cloud/hdfs/tmp

4.3.2.    spark-defaults.conf

mv spark-defaults.conf.template spark-defaults.conf
vi spark-defaults.conf

spark.driver.memory              512m

4.4. 測試

通過修改MASTER臨時變量，測試yarn模式

export MASTER=yarn
./bin/run-example SparkPi 1000

4.5. 卸載

復原環境變量，刪除spark相關行

vi ~/.bashrc           

刪除臨時數據和目錄

rm /opt/cloud/bin/spark
rm /opt/cloud/etc/spark -rf
rm /opt/cloud/packages/spark-2.1.0-bin-without-hadoop/ -rf
rm /opt/cloud/data/spark/ -rf
rm /opt/cloud/logs/spark/ -rf

 

 

 

---

[1] without hadoop 時需要