Spark 安裝

1. 軟件列表

 hadoop 2.6.0、 zookeeper 3.4.6和habase 1.2.0已安裝好，參見《HBase 安裝配置》。在此基礎上安裝scala-2.10.6和spark-1.6.0-bin-hadoop2.6。

2. 安裝

在d-hdp-01上執行操作

解壓縮 scala-2.10.6.tgz

tar -xvf scala-2.10.6.tgz
mv scala-2.10.6 ~/cloud/scala

解壓縮 spark-1.6.0-bin-hadoop2.6.tgz

tar -xvf spark-1.6.0-bin-hadoop2.6.tgz
mv spark-1.6.0-bin-hadoop2.6 ~/cloud/spark

配置~/.bashrc 增加

export SCALA_HOME=/home/hadoop/cloud/scala
export PATH=$PATH:$SCALA_HOME/bin

export SPARK_HOME=/home/hadoop/cloud/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

配置~/cloud/spark/conf/slaves

cp slaves.template slaves
d-hdp-02
d-hdp-03

配置~/cloud/spark/conf/spark-env.sh

export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64/jre"  
export HADOOP_HOME=/home/hadoop/cloud/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SCALA_HOME=/home/hadoop/cloud/scala
SPARK_MASTER_IP=d-hdp-01
SPARK_LOCAL_DIRS=/home/hadoop/cloud/workspace/spark
SPARK_DRIVER_MEMORY=2G

建工作目錄

mkdir -p /home/hadoop/cloud/workspace/spark

在d-hdp-02、d-hdp-03和d-hdp-client上

在d-hdp-01上建兩個鏈接

cd ~/cloud/spakr/sbin
ln -s start-all.sh start-all-spark.sh
ln -s stop-all.sh  stop-all-spark.sh

3. 運行測試

在d-hdp-01上執行

start-all-spark.sh

在d-hdp-client上用spark-submit提交作業，已SparkPI爲例

export SPARK_LIB=/home/hadoop/cloud/spark/lib
spark-submit --class org.apache.spark.examples.SparkPi --master spark://d-hdp-01:7077 $SPARK_LIB/spark-examples-1.6.0-hadoop2.6.0.jar 10

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster $SPARK_LIB/spark-examples-1.6.0-hadoop2.6.0.jar 10