軟件版本:
jdk:1.8
maven:3.61 http://maven.apache.org/download.cgi
spark:2.42 https://archive.apache.org/dist/spark/spark-2.4.2/
hadoop版本:hadoop-2.6.0-cdh5.7.0(spark編譯支持的hadoop版本,不需要安裝)
配置maven:
#配置環境變量 [root@hadoop004 soft]# cat /etc/profile.d/maven.sh MAVEN_HOME=/usr/local/maven export PATH=$MAVEN_HOME/bin:$PATH #確認maven版本 [root@hadoop004 maven]# mvn --version Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00) Maven home: /usr/local/maven Java version: 1.8.0_111, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_111/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-862.3.2.el7.x86_64", arch: "amd64", family: "unix" #配置mvn的本地存放地址:settings.xml文件 <localRepository>/usr/local/maven/repo</localRepository> #配置mvn下載源爲阿里雲的maven倉庫,加整下載 <mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> <mirrorOf>central</mirrorOf> </mirror>
配置Spark:
tar xf spark-2.4.2.tgz cd spark-2.4.2.tgz #修改pom.xml文件,添加clouder倉庫 <repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository>
執行編譯命令:
#在spark目錄下執行 ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0 注:本次編譯時長大約爲35分鐘,中間無任何報錯; 注:默認使用的scala版本爲最新的,如果要指定scala版本,通過以下方式修改 比如把scala版本改爲 2.10 ./dev/change-scala-version.sh 2.10
參數說明:
--name:生成壓縮包的後綴名字;前綴默認爲spark版本的名字,本例爲:spark-2.4.2-bin
--tgz:採用壓縮格式爲tar,壓縮的後綴名爲.tgz
-Pyarn:表示spark需要運行在yarn上面
-Phadoop-2.6:表示spark使用hadoop的profile的id
-Dhadoop.version=2.6.0-cdh5.7.0:表示spark使用hadoop的版本;如果不指定,默認使用的是2.2.0的hadoop
-Phive -Phive-thriftserver:表示支持hive
生成的文件:
在spark目錄下:spark-2.4.2-bin-2.6.0-cdh5.7.0.tgz
使用編譯後的spark部署:
tar xf spark-2.4.2-bin-2.6.0-cdh5.7.0.tgz ln -s spark-2.4.2-bin-2.6.0-cdh5.7.0 spark #配置spark的環境變量 [hadoop@hadoop001 ~]$ vim .bash_profile export SPARK_HOME=/home/hadoop/app/spark-2.4.2-bin-2.6.0-cdh5.7.0 export PATH=${SPARK_HOME}/bin:$PATH [hadoop@hadoop001 ~]$ source .bash_profile #運行spark測試 [hadoop@hadoop001 ~]$ spark-shell 19/04/29 10:51:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://hadoop001:4040 Spark context available as 'sc' (master = local[*], app id = local-1556506274719). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.2 /_/ Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111) Type in expressions to have them evaluated. Type :help for more information. scala>