文章目錄
提交命令
假定Yarn-Cluster方式提交：
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--num-executors 2 \
--master yarn \
--deploy-mode cluster \
./examples/jars/spark-examples_2.11-2.4.3.jar
任務提交流程
此代碼都是運行在Client端
任務提交初流程
上圖來之網上博客，有助於直觀感受，下面詳細解釋。
首先看 spark-submit 文件最終會執行
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"
會在 spark-class 文件中解析參數，其中重點說明如下代碼：
build_command() {
"$RUNNER" -Xmx128m $SPARK_LAUNCHER_OPTS -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
printf "%d\0" $?
}
會調用build_command()方法創建命令（其新啓動一個jvm，在 org.apache.spark.launcher.Main 類中根據不同的提交環境初步解析|添加|修改參數，這裏不再詳述），並把創建後的命令循環加到數組CMD中，最後執行exec執行CMD命令。
最終轉化後的CMD命令爲：
/software/servers/jdk1.8.0_121/bin/java -cp /software/conf/10k/mart_scr/bdp_jmart_fsh_union.bdp_jmart_fsh_formal/spark_conf/:/software/servers/10k/mart_scr/spark/jars/*:/software/conf/10k/mart_scr/bdp_jmart_fsh_union.bdp_jmart_fsh_formal/hadoop_conf/ org.apache.spark.deploy.SparkSubmit \
--master yarn \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--num-executors 2 \
./examples/jars/spark-examples_2.11-2.4.3.jar
上述命令啓動一個JVM進程，最終會執行 org.apache.spark.deploy.SparkSubmit 類
下面分析入口類 org.apache.spark.deploy.SparkSubmit 類，主要是對命令行參數進行了封裝，準備提交環境。在object SparkSubmit中，main(args)函數執行流程爲：
override def main(args: Array[String]): Unit = {
val submit = new SparkSubmit() {
self =>
override protected def parseArguments(args: Array[String]): SparkSubmitArguments = {
new SparkSubmitArguments(args) {
override protected def logInfo(msg: => String): Unit = self.logInfo(msg)
override protected def logWarning(msg: => String): Unit = self.logWarning(msg)
override protected def logError(msg: => String): Unit = self.logError(msg)
}
}
override protected def logInfo(msg: => String): Unit = printMessage(msg)
override protected def logWarning(msg: => String): Unit = printMessage(s"Warning: $msg")
override protected def logError(msg: => String): Unit = printMessage(s"Error: $msg")
override def doSubmit(args: Array[String]): Unit = {
try {
super.doSubmit(args)
} catch {
case e: SparkUserAppException =>
exitFn(e.exitCode)
}
}
}
submit.doSubmit(args)
}
創建一個匿名的SparkSubmit的匿名子類：在子類中重寫了 parseArguments() 和 doSubmit() 等，submit.doSubmit(args)進行提交，首先調用匿名類中的 doSubmit()，其實現爲：
def doSubmit(args: Array[String]): Unit = {
// Initialize logging if it hasn't been done yet. Keep track of whether logging needs to
// be reset before the application starts.
val uninitLog = initializeLogIfNecessary(true, silent = true)
val appArgs = parseArguments(args)
if (appArgs.verbose) {
logInfo(appArgs.toString)
}
appArgs.action match {
case SparkSubmitAction.SUBMIT => submit(appArgs, uninitLog)
case SparkSubmitAction.KILL => kill(appArgs)
case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
case SparkSubmitAction.PRINT_VERSION => printVersion()
}
}
可以看出首先解析參數 val appArgs = parseArguments(args) ，解析參數在 SparkSubmitArguments 類（注：如果要打印參數，請在啓動命令添加配置 --verbose true ），其中 SparkSubmitArguments 類重點說明如下代碼：
// Set parameters from command line arguments 即：解析提交命令行參數，調用父類的parse方法，最終會調用重載父類SparkSubmitOptionParser的該子類方法，例如： handle(opt, value) 對變量賦值、handleUnknown(opt)設置是否python|R。
parse(args.asJava)
// Populate `sparkProperties` map from properties file 即：獲取默認配置文件參數(./conf/spark-defaults.conf) 放在sparkProperties中
mergeDefaultSparkProperties()
// Remove keys that don't start with "spark." from `sparkProperties`. 即：剔除上面默認配置文件中非spark.開頭的配置參數
ignoreNonSparkProperties()
// Use `sparkProperties` map along with env vars to fill in any missing parameters 即加載環境變量中設置的參數，此代碼會合並參數設置，其中優先級爲：提交命令參數 > spark-defaults.conf參數 > 環境變量參數
loadEnvironmentArguments()
useRest = sparkProperties.getOrElse("spark.master.rest.enabled", "false").toBoolean
// 校驗參數，根據不同的運行環境校驗某些值不能爲null
validateArguments()
回到 doSubmit 函數 appArgs.action 進行模式匹配（SUBMIT、KILL、REQUEST_STATUS、PRINT_VERSION）。對於模式 SUBMIT ，如果匹配成功，調用 submit(appArgs)方法提交，它的執行邏輯是
/**
* Submit the application using the provided parameters, ensuring to first wrap
* in a doAs when --proxy-user is specified.
*/
@tailrec
private def submit(args: SparkSubmitArguments, uninitLog: Boolean): Unit = {
def doRunMain(): Unit = {
if (args.proxyUser != null) {
val proxyUser = UserGroupInformation.createProxyUser(args.proxyUser,
UserGroupInformation.getCurrentUser())
try {
proxyUser.doAs(new PrivilegedExceptionAction[Unit]() {
override def run(): Unit = {
runMain(args, uninitLog)
}
})
} catch {
case e: Exception =>
// Hadoop's AuthorizationException suppresses the exception's stack trace, which
// makes the message printed to the output by the JVM not very helpful. Instead,
// detect exceptions with empty stack traces here, and treat them differently.
if (e.getStackTrace().length == 0) {
error(s"ERROR: ${e.getClass().getName()}: ${e.getMessage()}")
} else {
throw e
}
}
} else {
runMain(args, uninitLog)
}
}
// In standalone cluster mode, there are two submission gateways:
// (1) The traditional RPC gateway using o.a.s.deploy.Client as a wrapper
// (2) The new REST-based gateway introduced in Spark 1.3
// The latter is the default behavior as of Spark 1.3, but Spark submit will fail over
// to use the legacy gateway if the master endpoint turns out to be not a REST server.
if (args.isStandaloneCluster && args.useRest) {
try {
logInfo("Running Spark using the REST application submission protocol.")
doRunMain()
} catch {
// Fail over to use the legacy submission gateway
case e: SubmitRestConnectionException =>
logWarning(s"Master endpoint ${args.master} was not a REST server. " +
"Falling back to legacy submission gateway instead.")
args.useRest = false
submit(args, false)
}
// In all other modes, just run the main class as prepared
} else {
doRunMain()
}
}
會進入 doRunMain() 函數，如果設置代理用戶，則設置對應參數，最終繼續調用 runMain(args, uninitLog) 函數，它的執行邏輯是：
/**
* Run the main method of the child class using the submit arguments.
*
* This runs in two steps. First, we prepare the launch environment by setting up
* the appropriate classpath, system properties, and application arguments for
* running the child main class based on the cluster manager and the deploy mode.
* Second, we use this launch environment to invoke the main method of the child
* main class.
*
* Note that this main class will not be the one provided by the user if we're
* running cluster deploy mode or python applications.
*/
private def runMain(args: SparkSubmitArguments, uninitLog: Boolean): Unit = {
val (childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitEnvironment(args)
// Let the main class re-initialize the logging system once it starts.
if (uninitLog) {
Logging.uninitialize()
}
if (args.verbose) {
logInfo(s"Main class:\n$childMainClass")
logInfo(s"Arguments:\n${childArgs.mkString("\n")}")
// sysProps may contain sensitive information, so redact before printing
logInfo(s"Spark config:\n${Utils.redact(sparkConf.getAll.toMap).mkString("\n")}")
logInfo(s"Classpath elements:\n${childClasspath.mkString("\n")}")
logInfo("\n")
}
val loader = getSubmitClassLoader(sparkConf)
for (jar <- childClasspath) {
addJarToClasspath(jar, loader)
}
var mainClass: Class[_] = null
try {
mainClass = Utils.classForName(childMainClass)
} catch {
case e: ClassNotFoundException =>
logError(s"Failed to load class $childMainClass.")
if (childMainClass.contains("thriftserver")) {
logInfo(s"Failed to load main class $childMainClass.")
logInfo("You need to build Spark with -Phive and -Phive-thriftserver.")
}
throw new SparkUserAppException(CLASS_NOT_FOUND_EXIT_STATUS)
case e: NoClassDefFoundError =>
logError(s"Failed to load $childMainClass: ${e.getMessage()}")
if (e.getMessage.contains("org/apache/hadoop/hive")) {
logInfo(s"Failed to load hive class.")
logInfo("You need to build Spark with -Phive and -Phive-thriftserver.")
}
throw new SparkUserAppException(CLASS_NOT_FOUND_EXIT_STATUS)
}
val app: SparkApplication = if (classOf[SparkApplication].isAssignableFrom(mainClass)) {
mainClass.getConstructor().newInstance().asInstanceOf[SparkApplication]
} else {
new JavaMainApplication(mainClass)
}
@tailrec
def findCause(t: Throwable): Throwable = t match {
case e: UndeclaredThrowableException =>
if (e.getCause() != null) findCause(e.getCause()) else e
case e: InvocationTargetException =>
if (e.getCause() != null) findCause(e.getCause()) else e
case e: Throwable =>
e
}
try {
app.start(childArgs.toArray, sparkConf)
} catch {
case t: Throwable =>
throw findCause(t)
}
}
首先看 val (childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitEnvironment(args) ，返回參數是四元組（childArgs=子進程參數，childClasspath=子進程classpath，sparkConf=Spark配置，childMainClass=子進程入口類），在該函數中工作包含：
1、在該方法中校驗提交的命令參數，如果設置不合理，提前報異常；
2、根據應用配置 -- class，這個參數代表我們的Driver啓動類，其中：
- Java|Scala應用：如果未設置
-- class，則取配置的Jar中的Main-Class參數（如果打包方式不對，則沒有該Main-Class，此時會報錯）- Python應用：此時如果是pyspark_shell則
-- class爲org.apache.spark.api.python.PythonGatewayServer，否則爲org.apache.spark.deploy.PythonRunner，最終相當於driver端執行PythonRunner 用戶主文件.py py-files參數 [參數列表]- R應用：此時如果是sparkr_shell則
-- class爲org.apache.spark.api.r.RBackend，否則爲org.apache.spark.deploy.RRunner，最終相當於driver執行RRunner 用戶主文件.r [參數列表]
3、根據部署模型設置childMainClass參數，childMainClass這個參數來決定下一步首先啓動哪個類，childMainClass根據部署模型有不同的值:
- 1.如果是部署模式爲Client模式那麼直接在客戶端運行啓動Driver，即上面所說的
-- class參數。- 2.如果是StandaloneCluster，如果啓用rest則childMainClass值爲RestSubmissionClientApp全類名(
org.apache.spark.deploy.rest.StandaloneRestClient)，否則childMainClass值爲ClientApp全類名(org.apache.spark.deploy.Client)。- 3.如果是Yarn集羣上運行，則childMainClass爲
org.apache.spark.deploy.yarn.YarnClusterApplication。- 4.如果是kubernetes集羣上運行，則爲
org.apache.spark.deploy.k8s.submit.KubernetesClientApplication。
此時客戶端準備工作已完成，當拿到childMainClass後，就反射實例化類並開始調用 app.start(childArgs.toArray, sparkConf) ，進入子進程，如果是client提交直接執行啓動driver，如果是cluster提交則提交集羣。
YarnClusterApplication提交集羣流程
由於我們以yarn-cluster舉例，所以直接進入org.apache.spark.deploy.yarn.YarnClusterApplication 類的 start(args, conf) ，此時就開始向集羣申請資源並提交任務了。本節介紹的環節基本都位於 org.apache.spark.deploy.yarn.Client中，其中YarnClusterApplication類也位於Client.scala文件內。
繼續分析，調用函數爲 YarnClusterApplication.start -> Client.run
private[spark] class YarnClusterApplication extends SparkApplication {
override def start(args: Array[String], conf: SparkConf): Unit = {
// SparkSubmit would use yarn cache to distribute files & jars in yarn mode,
// so remove them from sparkConf here for yarn mode.
conf.remove(JARS)
conf.remove(FILES)
new Client(new ClientArguments(args), conf, null).run()
}
}
private[spark] class Client(
val args: ClientArguments,
val sparkConf: SparkConf,
val rpcEnv: RpcEnv)
extends Logging {
...
/**
* Submit an application to the ResourceManager.
* If set spark.yarn.submit.waitAppCompletion to true, it will stay alive
* reporting the application's status until the application has exited for any reason.
* Otherwise, the client process will exit after submission.
* If the application finishes with a failed, killed, or undefined status,
* throw an appropriate SparkException.
*/
def run(): Unit = {
// 提交任務
this.appId = submitApplication()
if (!launcherBackend.isConnected() && fireAndForget) {
val report = getApplicationReport(appId)
val state = report.getYarnApplicationState
logInfo(s"Application report for $appId (state: $state)")
logInfo(formatReportDetails(report))
if (state == YarnApplicationState.FAILED || state == YarnApplicationState.KILLED) {
throw new SparkException(s"Application $appId finished with status: $state")
}
} else {
// 監控提交後任務並循環打印application狀態
val YarnAppReport(appState, finalState, diags) = monitorApplication(appId)
if (appState == YarnApplicationState.FAILED || finalState == FinalApplicationStatus.FAILED) {
diags.foreach { err =>
logError(s"Application diagnostics message: $err")
}
throw new SparkException(s"Application $appId finished with failed status")
}
if (appState == YarnApplicationState.KILLED || finalState == FinalApplicationStatus.KILLED) {
throw new SparkException(s"Application $appId is killed")
}
if (finalState == FinalApplicationStatus.UNDEFINED) {
throw new SparkException(s"The final status of application $appId is undefined")
}
}
}
由上面代碼可以知道，重點是 this.appId = submitApplication() ，下面主要看該函數
/**
* Submit an application running our ApplicationMaster to the ResourceManager.
*
* The stable Yarn API provides a convenience method (YarnClient#createApplication) for
* creating applications and setting up the application submission context. This was not
* available in the alpha API.
*/
def submitApplication(): ApplicationId = {
ResourceRequestHelper.validateResources(sparkConf)
var appId: ApplicationId = null
try {
// 初始化啓動yarnClient
launcherBackend.connect()
yarnClient.init(hadoopConf)
yarnClient.start()
logInfo("Requesting a new application from cluster with %d NodeManagers"
.format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
// Get a new application from our RM，從集羣RM獲取一個NodeManager用於啓動application
val newApp = yarnClient.createApplication()
val newAppResponse = newApp.getNewApplicationResponse()
// 獲取ApplicationId
appId = newAppResponse.getApplicationId()
// The app staging dir based on the STAGING_DIR configuration if configured
// otherwise based on the users home directory.
val appStagingBaseDir = sparkConf.get(STAGING_DIR)
.map { new Path(_, UserGroupInformation.getCurrentUser.getShortUserName) }
.getOrElse(FileSystem.get(hadoopConf).getHomeDirectory())
stagingDirPath = new Path(appStagingBaseDir, getAppStagingDir(appId))
new CallerContext("CLIENT", sparkConf.get(APP_CALLER_CONTEXT),
Option(appId.toString)).setCurrentContext()
// Verify whether the cluster has enough resources for our AM. 驗證集羣是否有足夠的資源
verifyClusterResources(newAppResponse)
// Set up the appropriate contexts to launch our AM. 設置用於提交的上下文設置
val containerContext = createContainerLaunchContext(newAppResponse)
val appContext = createApplicationSubmissionContext(newApp, containerContext)
// Finally, submit and monitor the application. 提交併監控application
logInfo(s"Submitting application $appId to ResourceManager")
yarnClient.submitApplication(appContext)
launcherBackend.setAppId(appId.toString)
reportLauncherState(SparkAppHandle.State.SUBMITTED)
appId
} catch {
case e: Throwable =>
if (stagingDirPath != null) {
cleanupStagingDir()
}
throw e
}
}
其中重點是 val containerContext = createContainerLaunchContext(newAppResponse) ，其主要設置ApplicationMaster的容器啓動的上下文，包含上傳hdfs和設置啓動參數等，代碼如下
/**
* Set up a ContainerLaunchContext to launch our ApplicationMaster container.
* This sets up the launch environment, java options, and the command for launching the AM.
*/
private def createContainerLaunchContext(newAppResponse: GetNewApplicationResponse)
: ContainerLaunchContext = {
logInfo("Setting up container launch context for our AM")
val appId = newAppResponse.getApplicationId
val pySparkArchives =
if (sparkConf.get(IS_PYTHON_APP)) {
findPySparkArchives()
} else {
Nil
}
// 讀取需提交給集羣啓動ApplicationMaster的環境變量
val launchEnv = setupLaunchEnv(stagingDirPath, pySparkArchives)
// 預處理資源文件，如果是集羣部署則上傳資源到hdfs，如果執行則爲下一個進程設置配置
val localResources = prepareLocalResources(stagingDirPath, pySparkArchives)
// 初始化ContainerLaunchContext，設置容器啓動參數：env和resources
val amContainer = Records.newRecord(classOf[ContainerLaunchContext])
amContainer.setLocalResources(localResources.asJava)
amContainer.setEnvironment(launchEnv.asJava)
...
// Command for the ApplicationMaster. 根據上面拼接設置的ApplicationMaster啓動命令commond
val commands = prefixEnv ++
Seq(Environment.JAVA_HOME.$$() + "/bin/java", "-server") ++
javaOpts ++ amArgs ++
Seq(
"1>", ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout",
"2>", ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr")
// TODO: it would be nicer to just make sure there are no null commands here
val printableCommands = commands.map(s => if (s == null) "null" else s).toList
// 繼續設置ContainerLaunchContext，並設置容器啓動參數：command
amContainer.setCommands(printableCommands.asJava)
// 打印加載ApplicationMaster啓動上下文參數
logDebug("===============================================================================")
logDebug("YARN AM launch context:")
logDebug(s" user class: ${Option(args.userClass).getOrElse("N/A")}")
logDebug(" env:")
if (log.isDebugEnabled) {
Utils.redact(sparkConf, launchEnv.toSeq).foreach { case (k, v) =>
logDebug(s" $k -> $v")
}
}
logDebug(" resources:")
localResources.foreach { case (k, v) => logDebug(s" $k -> $v")}
logDebug(" command:")
logDebug(s" ${printableCommands.mkString(" ")}")
logDebug("===============================================================================")
// send the acl settings into YARN to control who has access via YARN interfaces
// 繼續設置ContainerLaunchContext，並設置容器啓動參數：acl
val securityManager = new SecurityManager(sparkConf)
amContainer.setApplicationACLs(
YarnSparkHadoopUtil.getApplicationAclsForYarn(securityManager).asJava)
setupSecurityToken(amContainer)
amContainer
}
如上代碼，重點說明下 val localResources = prepareLocalResources(stagingDirPath, pySparkArchives) ，該方法預處理資源文件，如果是集羣部署則上傳資源到hdfs，如果執行則爲下一個進程設置配置等；
緊接着集羣設置提交給集羣的容器上下文，主要包含四個參數，分別爲：env、resources、command和acl，容器上下文參數（主要封裝ContainerLaunchContext類）是通過rpc的參數方式傳遞給服務端的，其中 resources 參數用於Yarn集羣調度的Node節點在加載Container容器前資源本地化使用，acl爲控制集羣安全機制，env和command是用於集羣啓動ApplicationMaster容器使用。
最終的生成的參數也能在日誌中打印（注：估計公司考慮集羣信息安全問題，取消了resources參數打印，因此網上又找了一個截圖可以參照），其中 user class 是Driver啓動類（不同的運行環境有的值，可以參照前面介紹的-- class參數設置）， 如下兩個圖
其中摘抄一下上圖中comman命令：
{{JAVA_HOME}}/bin/java -server -Xmx8192m -Djava.io.tmpdir={{PWD}}/tmp -Dspark.yarn.app.container.log.dir=<LOG_DIR> org.apache.spark.deploy.yarn.ApplicationMaster \
--class 'org.apache.spark.examples.SparkPi' \
--jar file:/work/code/sco_bigdata/spark/./examples/jars/spark-examples_2.11-2.4.3.jar \
--properties-file {{PWD}}/__spark_conf__/__spark_conf__.properties 1> <LOG_DIR>/spark_stdout 2> <LOG_DIR>/spark_stderr
最後，在 def submitApplication() 函數體的 yarnClient.submitApplication(appContext) 代碼正式提交給Yarn集羣。可以看出後面就是在ApplicationMaster上啓動Driver流程了，對應的入庫類是 org.apache.spark.deploy.yarn.ApplicationMaster，至此Spark任務的提交流程就分析完了 。
提交過程環節彙總
本節彙總記錄一些中間重要環節。
用戶Yarn-Cluster提交shell命令
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--num-executors 2 \
--master yarn \
--deploy-mode cluster \
./examples/jars/spark-examples_2.11-2.4.3.jar
提交給SparkSubmit類的cmd命令
/software/servers/jdk1.8.0_121/bin/java -cp /software/conf/10k/mart_scr/bdp_jmart_fsh_union.bdp_jmart_fsh_formal/spark_conf/:/software/servers/10k/mart_scr/spark/jars/*:/software/conf/10k/mart_scr/bdp_jmart_fsh_union.bdp_jmart_fsh_formal/hadoop_conf/ org.apache.spark.deploy.SparkSubmit \
--master yarn \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--num-executors 2 \
./examples/jars/spark-examples_2.11-2.4.3.jar
提交給集羣啓動driver的命令
{{JAVA_HOME}}/bin/java -server -Xmx8192m -Djava.io.tmpdir={{PWD}}/tmp -Dspark.yarn.app.container.log.dir=<LOG_DIR> org.apache.spark.deploy.yarn.ApplicationMaster \
--class 'org.apache.spark.examples.SparkPi' \
--jar file:/work/code/sco_bigdata/spark/./examples/jars/spark-examples_2.11-2.4.3.jar \
--properties-file {{PWD}}/__spark_conf__/__spark_conf__.properties 1> <LOG_DIR>/spark_stdout 2> <LOG_DIR>/spark_stderr
任務運行結果
上傳到hdfs的文件
其中，重點關注：application_1584006073801_1715800/__spark_conf__/__spark_conf__.properties，存在完整提交的參數信息。
整個任務運行日誌
執行日誌（直接在集羣上執行即可復現日誌），由於涉及公司集羣信息，如有需要參照閱讀請單獨留言。