Spark2.4.0源碼編譯支持hadoop-2.6.0-cdh5.7.0

軟件包

• JDK1.8.0_45
• apache-maven-3.6.0
• Scala2.11.8
• hadoop-2.6.0-cdh5.7.0

---

注意設置內存*****

export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

---

安裝git,源碼編譯過程中會需要用到git

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ scala-2.11.8]$ sudo yum install -y git

---

下載spark2.4.0源碼

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ software]$ wget https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0.tgz
--2019-01-03 07:05:31--  https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0.tgz
Resolving archive.apache.org... 163.172.17.199
Connecting to archive.apache.org|163.172.17.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16118412 (15M) [application/x-gzip]
Saving to: “spark-2.4.0.tgz”

100%[===============================================================================================================>] 16,118,412  3.17M/s   in 4.9s    

2019-01-03 07:05:37 (3.17 MB/s) - “spark-2.4.0.tgz” saved [16118412/16118412]

解壓spark源碼

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ software]$ tar -zxvf spark-2.4.0.tgz -C ~/app/

• 因爲使用的是CDH版本的Hadoop，所以需要在pom.xml添加repository

<repository>
     <id>cloudera</id>
     <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>

編譯方法

用make-distributed 腳本來編譯

• 修改dev目錄下make-distribution.sh腳本

VERSION=2.4.0        #Spark版本
SCALA_VERSION=2.11   #Scala版本（大版本）
SPARK_HADOOP_VERSION=2.6.0-cdh5.7.0 #Hadoop版本
SPARK_HIVE=1         #是否將Hive打包（非1表示不打包）

• 圖示
  

• 執行編譯命令

[hadoop001@iZuf6dz84cmy6o2y05pb8bZ spark-2.4.0]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0

耐心等待編譯完成即可，編成功後生成的部署包位於根目錄下。