（二）spark HA安裝部署

一、下載Spark安裝包

二、安裝基礎

1、Java8安裝成功

2、zookeeper安裝成功

3、hadoop2.7.5 HA安裝成功

4、Scala安裝成功（不安裝進程也可以啓動）

回到頂部

三、Spark安裝過程

1、上傳並解壓縮

[hadoop@hadoop1 ~]$ ls
apps     data      exam        inithive.conf  movie     spark-2.3.0-bin-hadoop2.7.tgz  udf.jar
cookies  data.txt  executions  json.txt       projects  student                        zookeeper.out
course   emp       hive.sql    log            sougou    temp
[hadoop@hadoop1 ~]$ tar -zxvf spark-2.3.0-bin-hadoop2.7.tgz -C apps/

2、爲安裝包創建一個軟連接

[hadoop@hadoop1 ~]$ cd apps/
[hadoop@hadoop1 apps]$ ls
hadoop-2.7.5  hbase-1.2.6  spark-2.3.0-bin-hadoop2.7  zookeeper-3.4.10  zookeeper.out
[hadoop@hadoop1 apps]$ ln -s spark-2.3.0-bin-hadoop2.7/ spark
[hadoop@hadoop1 apps]$ ll
總用量 36
drwxr-xr-x. 10 hadoop hadoop  4096 3月  23 20:29 hadoop-2.7.5
drwxrwxr-x.  7 hadoop hadoop  4096 3月  29 13:15 hbase-1.2.6
lrwxrwxrwx.  1 hadoop hadoop    26 4月  20 13:48 spark -> spark-2.3.0-bin-hadoop2.7/
drwxr-xr-x. 13 hadoop hadoop  4096 2月  23 03:42 spark-2.3.0-bin-hadoop2.7
drwxr-xr-x. 10 hadoop hadoop  4096 3月  23 2017 zookeeper-3.4.10
-rw-rw-r--.  1 hadoop hadoop 17559 3月  29 13:37 zookeeper.out
[hadoop@hadoop1 apps]$

3、進入spark/conf修改配置文件

（1）進入配置文件所在目錄

[hadoop@hadoop1 ~]$ cd apps/spark/conf/
[hadoop@hadoop1 conf]$ ll
總用量 36
-rw-r--r--. 1 hadoop hadoop  996 2月  23 03:42 docker.properties.template
-rw-r--r--. 1 hadoop hadoop 1105 2月  23 03:42 fairscheduler.xml.template
-rw-r--r--. 1 hadoop hadoop 2025 2月  23 03:42 log4j.properties.template
-rw-r--r--. 1 hadoop hadoop 7801 2月  23 03:42 metrics.properties.template
-rw-r--r--. 1 hadoop hadoop  865 2月  23 03:42 slaves.template
-rw-r--r--. 1 hadoop hadoop 1292 2月  23 03:42 spark-defaults.conf.template
-rwxr-xr-x. 1 hadoop hadoop 4221 2月  23 03:42 spark-env.sh.template
[hadoop@hadoop1 conf]$

（2）複製spark-env.sh.template並重命名爲spark-env.sh，並在文件最後添加配置內容

[hadoop@hadoop1 conf]$ cp spark-env.sh.template spark-env.sh
[hadoop@hadoop1 conf]$ vi spark-env.sh

export JAVA_HOME=/usr/local/jdk1.8.0_73
#export SCALA_HOME=/usr/share/scala
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.7.5
export HADOOP_CONF_DIR=/home/hadoop/apps/hadoop-2.7.5/etc/hadoop
export SPARK_WORKER_MEMORY=500m
export SPARK_WORKER_CORES=1
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop1:2181,hadoop2:2181,hadoop3:2181,hadoop4:2181 -Dspark.deploy.zookeeper.dir=/spark"

注：

#export SPARK_MASTER_IP=hadoop1 這個配置要註釋掉。
集羣搭建時配置的spark參數可能和現在的不一樣，主要是考慮個人電腦配置問題，如果memory配置太大，機器運行很慢。
說明：
-Dspark.deploy.recoveryMode=ZOOKEEPER #說明整個集羣狀態是通過zookeeper來維護的，整個集羣狀態的恢復也是通過zookeeper來維護的。就是說用zookeeper做了spark的HA配置，Master(Active)掛掉的話，Master(standby)要想變成Master（Active）的話，Master(Standby)就要像zookeeper讀取整個集羣狀態信息，然後進行恢復所有Worker和Driver的狀態信息，和所有的Application狀態信息；
-Dspark.deploy.zookeeper.url=hadoop1:2181,hadoop2:2181,hadoop3:2181,hadoop4:2181#將所有配置了zookeeper，並且在這臺機器上有可能做master(Active)的機器都配置進來；（我用了4臺，就配置了4臺）

-Dspark.deploy.zookeeper.dir=/spark
這裏的dir和zookeeper配置文件zoo.cfg中的dataDir的區別？？？
-Dspark.deploy.zookeeper.dir是保存spark的元數據，保存了spark的作業運行狀態；
zookeeper會保存spark集羣的所有的狀態信息，包括所有的Workers信息，所有的Applactions信息，所有的Driver信息,如果集羣

（3）複製slaves.template成slaves

[hadoop@hadoop1 conf]$ cp slaves.template slaves
[hadoop@hadoop1 conf]$ vi slaves

添加如下內容

hadoop1
hadoop2
hadoop3
hadoop4

（4）將安裝包分發給其他節點

[hadoop@hadoop1 ~]$ cd apps/
[hadoop@hadoop1 apps]$ scp -r spark-2.3.0-bin-hadoop2.7/ hadoop2:$PWD
[hadoop@hadoop1 apps]$ scp -r spark-2.3.0-bin-hadoop2.7/ hadoop3:$PWD
[hadoop@hadoop1 apps]$ scp -r spark-2.3.0-bin-hadoop2.7/ hadoop4:$PWD

創建軟連接

[hadoop@hadoop2 ~]$ cd apps/
[hadoop@hadoop2 apps]$ ls
hadoop-2.7.5  hbase-1.2.6  spark-2.3.0-bin-hadoop2.7  zookeeper-3.4.10
[hadoop@hadoop2 apps]$ ln -s spark-2.3.0-bin-hadoop2.7/ spark
[hadoop@hadoop2 apps]$ ll
總用量 16
drwxr-xr-x 10 hadoop hadoop 4096 3月  23 20:29 hadoop-2.7.5
drwxrwxr-x  7 hadoop hadoop 4096 3月  29 13:15 hbase-1.2.6
lrwxrwxrwx  1 hadoop hadoop   26 4月  20 19:26 spark -> spark-2.3.0-bin-hadoop2.7/
drwxr-xr-x 13 hadoop hadoop 4096 4月  20 19:24 spark-2.3.0-bin-hadoop2.7
drwxr-xr-x 10 hadoop hadoop 4096 3月  21 19:31 zookeeper-3.4.10
[hadoop@hadoop2 apps]$

4、配置環境變量

所有節點均要配置

[hadoop@hadoop1 spark]$ vi ~/.bashrc

#Spark
export SPARK_HOME=/home/hadoop/apps/spark
export PATH=$PATH:$SPARK_HOME/bin

保存並使其立即生效

[hadoop@hadoop1 spark]$ source ~/.bashrc

回到頂部

四、啓動

1、先啓動zookeeper集羣

所有節點均要執行

[hadoop@hadoop1 ~]$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[hadoop@hadoop1 ~]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: follower
[hadoop@hadoop1 ~]$

2、在啓動HDFS集羣

任意一個節點執行即可

[hadoop@hadoop1 ~]$ start-dfs.sh

3、在啓動Spark集羣

在一個節點上執行

[hadoop@hadoop1 ~]$ cd apps/spark/sbin/
[hadoop@hadoop1 sbin]$ start-all.sh

4、查看進程

5、問題

查看進程發現spark集羣只有hadoop1成功啓動了Master進程，其他3個節點均沒有啓動成功，需要手動啓動，進入到/home/hadoop/apps/spark/sbin目錄下執行以下命令，3個節點都要執行

[hadoop@hadoop2 ~]$ cd ~/apps/spark/sbin/
[hadoop@hadoop2 sbin]$ start-master.sh

6、執行之後再次查看進程

Master進程和Worker進程都以啓動成功

回到頂部

五、驗證

1、查看Web界面Master狀態

hadoop1是ALIVE狀態，hadoop2、hadoop3和hadoop4均是STANDBY狀態

hadoop1節點

hadoop2節點

hadoop3

hadoop4

2、驗證HA的高可用

手動幹掉hadoop1上面的Master進程，觀察是否會自動進行切換

幹掉hadoop1上的Master進程之後，再次查看web界面

hadoo1節點，由於Master進程被幹掉，所以界面無法訪問

hadoop2節點，Master被幹掉之後，hadoop2節點上的Master成功篡位成功，成爲ALIVE狀態

hadoop3節點

hadoop4節點

回到頂部

六、執行Spark程序on standalone

1、執行第一個Spark程序

[hadoop@hadoop3 ~]$ /home/hadoop/apps/spark/bin/spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master spark://hadoop1:7077 \
> --executor-memory 500m \
> --total-executor-cores 1 \
> /home/hadoop/apps/spark/examples/jars/spark-examples_2.11-2.3.0.jar \
> 100

其中的spark://hadoop1:7077是下圖中的地址

運行結果

2、啓動spark shell

[hadoop@hadoop1 ~]$ /home/hadoop/apps/spark/bin/spark-shell \
> --master spark://hadoop1:7077 \
> --executor-memory 500m \
> --total-executor-cores 1

參數說明：

--master spark://hadoop1:7077 指定Master的地址

--executor-memory 500m:指定每個worker可用內存爲500m

--total-executor-cores 1: 指定整個集羣使用的cup核數爲1個

注意：

如果啓動spark shell時沒有指定master地址，但是也可以正常啓動spark shell和執行spark shell中的程序，其實是啓動了spark的local模式，該模式僅在本機啓動一個進程，沒有與集羣建立聯繫。

Spark Shell中已經默認將SparkContext類初始化爲對象sc。用戶代碼如果需要用到，則直接應用sc即可

Spark Shell中已經默認將SparkSQl類初始化爲對象spark。用戶代碼如果需要用到，則直接應用spark即可

3、在spark shell中編寫WordCount程序

（1）編寫一個hello.txt文件並上傳到HDFS上的spark目錄下

[hadoop@hadoop1 ~]$ vi hello.txt
[hadoop@hadoop1 ~]$ hadoop fs -mkdir -p /spark
[hadoop@hadoop1 ~]$ hadoop fs -put hello.txt /spark

hello.txt的內容如下

you,jump
i,jump
you,jump
i,jump
jump

（2）在spark shell中用scala語言編寫spark程序

scala> sc.textFile("/spark/hello.txt").flatMap(_.split(",")).map((_,1)).reduceByKey(_+_).saveAsTextFile("/spark/out")

說明：

sc是SparkContext對象，該對象是提交spark程序的入口

textFile("/spark/hello.txt")是hdfs中讀取數據

flatMap(_.split(" "))先map再壓平

map((_,1))將單詞和1構成元組

reduceByKey(_+_)按照key進行reduce，並將value累加

saveAsTextFile("/spark/out")將結果寫入到hdfs中

（3）使用hdfs命令查看結果

[hadoop@hadoop2 ~]$ hadoop fs -cat /spark/out/p*
(jump,5)
(you,2)
(i,2)
[hadoop@hadoop2 ~]$

回到頂部

七、執行Spark程序on YARN

1、前提

成功啓動zookeeper集羣、HDFS集羣、YARN集羣

2、啓動Spark on YARN

[hadoop@hadoop1 bin]$ spark-shell --master yarn --deploy-mode client

報錯如下：

報錯原因：內存資源給的過小，yarn直接kill掉進程，則報rpc連接失敗、ClosedChannelException等錯誤。

解決方法：

先停止YARN服務，然後修改yarn-site.xml，增加如下內容

        <property>
                <name>yarn.nodemanager.vmem-check-enabled</name>
                <value>false</value>
                <description>Whether virtual memory limits will be enforced for containers</description>
        </property>
        <property>
                <name>yarn.nodemanager.vmem-pmem-ratio</name>
                <value>4</value>
                <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
        </property>

將新的yarn-site.xml文件分發到其他Hadoop節點對應的目錄下，最後在重新啓動YARN。

重新執行以下命令啓動spark on yarn

[hadoop@hadoop1 hadoop]$ spark-shell --master yarn --deploy-mode client

啓動成功

3、打開YARN的web界面

打開YARN WEB頁面：http://hadoop4:8088
可以看到Spark shell應用程序正在運行

單擊ID號鏈接，可以看到該應用程序的詳細信息

單擊“ApplicationMaster”鏈接

4、運行程序

scala> val array = Array(1,2,3,4,5)
array: Array[Int] = Array(1, 2, 3, 4, 5)

scala> val rdd = sc.makeRDD(array)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at makeRDD at <console>:26

scala> rdd.count
res0: Long = 5                                                                  

scala>

再次查看YARN的web界面

查看executors

5、執行Spark自帶的示例程序PI

[hadoop@hadoop1 ~]$ spark-submit --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode cluster \
> --driver-memory 500m \
> --executor-memory 500m \
> --executor-cores 1 \
> /home/hadoop/apps/spark/examples/jars/spark-examples_2.11-2.3.0.jar \
> 10

執行過程

[hadoop@hadoop1 ~]$ spark-submit --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode cluster \
> --driver-memory 500m \
> --executor-memory 500m \
> --executor-cores 1 \
> /home/hadoop/apps/spark/examples/jars/spark-examples_2.11-2.3.0.jar \
> 10
2018-04-21 17:57:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-04-21 17:57:34 INFO  ConfiguredRMFailoverProxyProvider:100 - Failing over to rm2
2018-04-21 17:57:34 INFO  Client:54 - Requesting a new application from cluster with 4 NodeManagers
2018-04-21 17:57:34 INFO  Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-04-21 17:57:34 INFO  Client:54 - Will allocate AM container, with 884 MB memory including 384 MB overhead
2018-04-21 17:57:34 INFO  Client:54 - Setting up container launch context for our AM
2018-04-21 17:57:34 INFO  Client:54 - Setting up the launch environment for our AM container
2018-04-21 17:57:34 INFO  Client:54 - Preparing resources for our AM container
2018-04-21 17:57:36 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-04-21 17:57:39 INFO  Client:54 - Uploading resource file:/tmp/spark-93bd68c9-85de-482e-bbd7-cd2cee60e720/__spark_libs__8262081479435245591.zip -> hdfs://myha01/user/hadoop/.sparkStaging/application_1524303370510_0005/__spark_libs__8262081479435245591.zip
2018-04-21 17:57:44 INFO  Client:54 - Uploading resource file:/home/hadoop/apps/spark/examples/jars/spark-examples_2.11-2.3.0.jar -> hdfs://myha01/user/hadoop/.sparkStaging/application_1524303370510_0005/spark-examples_2.11-2.3.0.jar
2018-04-21 17:57:44 INFO  Client:54 - Uploading resource file:/tmp/spark-93bd68c9-85de-482e-bbd7-cd2cee60e720/__spark_conf__2498510663663992254.zip -> hdfs://myha01/user/hadoop/.sparkStaging/application_1524303370510_0005/__spark_conf__.zip
2018-04-21 17:57:44 INFO  SecurityManager:54 - Changing view acls to: hadoop
2018-04-21 17:57:44 INFO  SecurityManager:54 - Changing modify acls to: hadoop
2018-04-21 17:57:44 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-04-21 17:57:44 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-04-21 17:57:44 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
2018-04-21 17:57:44 INFO  Client:54 - Submitting application application_1524303370510_0005 to ResourceManager
2018-04-21 17:57:44 INFO  YarnClientImpl:273 - Submitted application application_1524303370510_0005
2018-04-21 17:57:45 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:45 INFO  Client:54 - 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1524304664749
     final status: UNDEFINED
     tracking URL: http://hadoop4:8088/proxy/application_1524303370510_0005/
     user: hadoop
2018-04-21 17:57:46 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:47 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:48 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:49 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:50 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:51 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:52 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:53 INFO  Client:54 - Application report for application_1524303370510_0005 (state: ACCEPTED)
2018-04-21 17:57:54 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:57:54 INFO  Client:54 - 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.123.104
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1524304664749
     final status: UNDEFINED
     tracking URL: http://hadoop4:8088/proxy/application_1524303370510_0005/
     user: hadoop
2018-04-21 17:57:55 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:57:56 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:57:57 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:57:58 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:57:59 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:00 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:01 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:02 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:03 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:04 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:05 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:06 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:07 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:08 INFO  Client:54 - Application report for application_1524303370510_0005 (state: RUNNING)
2018-04-21 17:58:09 INFO  Client:54 - Application report for application_1524303370510_0005 (state: FINISHED)
2018-04-21 17:58:09 INFO  Client:54 - 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.123.104
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1524304664749
     final status: SUCCEEDED
     tracking URL: http://hadoop4:8088/proxy/application_1524303370510_0005/
     user: hadoop
2018-04-21 17:58:09 INFO  Client:54 - Deleted staging directory hdfs://myha01/user/hadoop/.sparkStaging/application_1524303370510_0005
2018-04-21 17:58:09 INFO  ShutdownHookManager:54 - Shutdown hook called
2018-04-21 17:58:09 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-93bd68c9-85de-482e-bbd7-cd2cee60e720
2018-04-21 17:58:09 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-06de6905-8067-4f1e-a0a0-bc8a51daf535
[hadoop@hadoop1 ~]$

（二）spark HA安裝部署

一、下載Spark安裝包

1、從官網下載

2、從微軟的鏡像站下載

3、從清華的鏡像站下載

二、安裝基礎

三、Spark安裝過程

1、上傳並解壓縮

2、爲安裝包創建一個軟連接

3、進入spark/conf修改配置文件

4、配置環境變量

四、啓動

1、先啓動zookeeper集羣

2、在啓動HDFS集羣

3、在啓動Spark集羣

4、查看進程

5、問題

6、執行之後再次查看進程

五、驗證

1、查看Web界面Master狀態

2、驗證HA的高可用

六、執行Spark程序on standalone

1、執行第一個Spark程序

2、啓動spark shell

3、在spark shell中編寫WordCount程序

七、執行Spark程序on YARN

1、前提

2、啓動Spark on YARN

3、打開YARN的web界面

4、運行程序

5、執行Spark自帶的示例程序PI

（六） Spark的Transformation和action介紹

（二）spark HA安裝部署

（九）spark on yarn

（七）Spark實戰之 wordCount

（八）spark學習之數據傾斜

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結

（二）spark HA安裝部署

一、下載Spark安裝包

1、從官網下載

2、從微軟的鏡像站下載

3、從清華的鏡像站下載

二、安裝基礎

三、Spark安裝過程

1、上傳並解壓縮

2、爲安裝包創建一個軟連接

3、進入spark/conf修改配置文件

4、配置環境變量

四、啓動

1、先啓動zookeeper集羣

2、在啓動HDFS集羣

3、在啓動Spark集羣

4、查看進程

5、問題

6、執行之後再次查看進程

五、驗證

1、查看Web界面Master狀態

2、驗證HA的高可用

六、執行Spark程序on standalone

1、執行第一個Spark程序

2、啓動spark shell

3、 在spark shell中編寫WordCount程序

七、 執行Spark程序on YARN

1、前提

2、啓動Spark on YARN

3、打開YARN的web界面

4、運行程序

5、執行Spark自帶的示例程序PI

3、在spark shell中編寫WordCount程序

七、執行Spark程序on YARN