软件环境:
CDH:5.7.3;Oozie:4.1.0-CDH5.7.3 ; Spark:1.6.0-cdh5.7.3-hadoop2.6.0-cdh5.7.3 ; Hadoop:hadoop2.6.0-cdh5.7.3(HDFS 采用HA方式);
问题描述:
解决思路:
1. 首先就是各种网上找
2. 查看源码
#oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib/lib_20161222004831/spark
采用第二行的方式,而非第一行的方式(第二行中的lib_2016...是时间戳,每个集群应该不一样);val cachedSecondaryJarLinks = ListBuffer.empty[String]
List(
(args.addJars, LocalResourceType.FILE, true),
(args.files, LocalResourceType.FILE, false),
(args.archives, LocalResourceType.ARCHIVE, false)
).foreach { case (flist, resType, addToClasspath) =>
if (flist != null && !flist.isEmpty()) {
flist.split(',').foreach { file => // add distinct operation to avoid multiple same jars
val (_, localizedPath) = distribute(file, resType = resType)
println("fansy: ---->file:"+file)
require(localizedPath != null)
if (addToClasspath) {
cachedSecondaryJarLinks += localizedPath
}
}
}
}
然后再次查看日志:解决方案:
1. 修改源码:
val cachedSecondaryJarLinks = ListBuffer.empty[String]
List(
(args.addJars, LocalResourceType.FILE, true),
(args.files, LocalResourceType.FILE, false),
(args.archives, LocalResourceType.ARCHIVE, false)
).foreach { case (flist, resType, addToClasspath) =>
if (flist != null && !flist.isEmpty()) {
flist.split(',').distinct.foreach { file => // add distinct operation to avoid multiple same jars
val (_, localizedPath) = distribute(file, resType = resType)
println("fansy: ---->file:"+file)
require(localizedPath != null)
if (addToClasspath) {
cachedSecondaryJarLinks += localizedPath
}
}
}
}
编译该源码(如果自己编译记得去掉那行打印),得到其class,如下:2. 替换Jar包
/user/oozie/share/lib/lib_20161222004831/spark/spark-yarn_2.10-1.6.0-cdh5.7.3.jar
把这个jar包先下载到linux,然后下载到windows;接着删掉HDFS上的该jar包:hdfs dfs -rm -r /user/oozie/share/lib/lib_20161222004831/spark/spark-yarn_2.10-1.6.0-cdh5.7.3.jar
在windows里面使用winRAR打开下载的spark-yarn_2.10-1.6.0-cdh5.7.3.jar包,并使用编译后的Client的所有class替换对应的class;替换完成后得到该spark jar(可以在这里下载 http://download.csdn.net/detail/fansy1990/9720059 )hdfs dfs -put spark-yarn_2.10-1.6.0-cdh5.7.3.jar /user/oozie/share/lib/lib_20161222004831/spark/
再次运行,发现Oozie任务成功运行: