從Application提交角度審視Executor

從Application提交的角度審視Executor，解密Executor到底是什麼時候啓動的以及Executor如何把結果交給Application。

Executor何時啓動

SparkContext啓動後，直接實例化createTaskScheduler方法，createTaskScheduler完成後，調用TaskScheduler的start方法，實際調用的是TaskSchedulerImpl的start方法，在TaskSchedulerImpl的start方法中實現SparkDeploySchedulerBackend（spark2.0後更名爲StandaleSchedulerBackend）的start方法。SparkDeploySchedulerBackend的start方法中將commend封裝註冊給Master，Master轉過來要Worker啓動具體的Executor，command已經封裝好指令，Executor具體要啓動進程入口類CoarseGrainedExecutorBackend。然後調用new()函數創建一個AppClient，AppClient中有個名爲ClientEndpoint的內部類，在創建ClientEndpoint時會傳入Command來指定具體爲當前應用程序啓動的Executor進行的入口類的名稱爲CoarseGrainedExecutorBackend。ClientEndpoint繼承自ThreadSafeRpcEndpoint，其通過RPC機制完成和Master的通信。在ClientEndpoint的start方法中，會通過registerWithMaster方法向Master發送RegisterApplication請求，Master收到該請求消息後，首先通過registerApplication方法完成信息登記，之後調用schedule方法，在Worker上啓動Executor（詳細查看Master-Driver和Worker-Executor剖析章節）。

override def receive: PartialFunction[Any, Unit] = {   ......
  case RegisterApplication(description, driver) => {
    if (state == RecoveryState.STANDBY) {
    } else {val app = createApplication(description, driver)
      registerApplication(app)
      persistenceEngine.addApplication(app)
      driver.send(RegisteredApplication(app.id, self))
      schedule()}
  }
......}

Master匹配收到的RegisterApplication請求，先判斷Master是否爲STANDBY（備用），若不是則爲ALIVE狀態，調用createApplicaton方法創建applicationInfo，把applicationInfo信息通過registerApplication方法進行註冊，通過persistenceEngine.addApplication方法進行持久化，完成後通過driver.send方法向AppClient返回註冊成功的信息。

對象ApplicationDescription和RpcEndpointRef傳入方法createApplication中，createApplication返回的對象是ApplicationInfo，Master類的createApplication方法的源碼：

private def createApplication(desc: ApplicationDescription, driver: RpcEndpointRef):
    ApplicationInfo = {
  val now = System.currentTimeMillis()
  val date = new Date(now)
  val appId = newApplicationId(date)
  new ApplicationInfo(now, appId, desc, date, driver, defaultCores)}

之後把ApplicationInfo對象傳入registerApplication方法，調用registerApplication方法完成application的註冊，registerApplication方法的源碼：

private def registerApplication(app: ApplicationInfo): Unit = {
  val appAddress = app.driver.address //Driver地址，用於Master和Driver通信
  if (addressToApp.contains(appAddress)) {
     return //若已有Driver地址，則Driver已經註冊過了，直接return
  }
  applicationMetricsSystem.registerSource(app.appSource)  //向度量系統註冊
  apps += app  //apps是一個HashSet，保證數據不重複
  idToApp(app.id) = app  //idToApp是一個HashMap，保存id和app的對應關係
  endpointToApp(app.driver) = app //endpoint是HashMap，保存Driver和app的對應關係
  addressToApp(appAddress) = app
  waitingApps += app
}

註冊完成後，調用shedule方法，此方法有兩個作用：1、完成Driver調度將waitingDrivers數組中的Driver發送到滿足Worker上運行；2、Worker節點上爲application啓動Executor。每一次新的Driver的註冊、application的註冊或資源變動都將調用schedule方法。Schedule方法用於當前等待調度的application調度可用資源，在滿足條件的Worker節點上啓動Executor，調用startExecutorsOnWorkers方法完成操作（詳細見Master-Driver和Worker-Executor剖析章節）。

啓動Executor的方法startExecutorsOnWorkers是調用scheduleExecutorsOnWorkers方法，此方法有兩種啓動Executor策略：輪流均攤策略和依次全佔策略，Master的scheduleExecutorsOnWorkers方法的源碼：

private def scheduleExecutorsOnWorkers(
    app: ApplicationInfo,
    usableWorkers: Array[WorkerInfo],
    spreadOutApps: Boolean): Array[Int] = {
  val coresPerExecutor = app.desc.coresPerExecutor
  val minCoresPerExecutor = coresPerExecutor.getOrElse(1)
  val oneExecutorPerWorker = coresPerExecutor.isEmpty
  val memoryPerExecutor = app.desc.memoryPerExecutorMB
  val numUsable = usableWorkers.length
  val assignedCores = new Array[Int](numUsable) // Number of cores to give to each worker
  val assignedExecutors = new Array[Int](numUsable) // Number of new executors on each worker
  var coresToAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
  /** Return whether the specified worker can launch an executor for this app. */
  def canLaunchExecutor(pos: Int): Boolean = {
    val keepScheduling = coresToAssign >= minCoresPerExecutor
    val enoughCores = usableWorkers(pos).coresFree - assignedCores(pos) >= minCoresPerExecutor
    val launchingNewExecutor = !oneExecutorPerWorker || assignedExecutors(pos) == 0
    if (launchingNewExecutor) {
      val assignedMemory = assignedExecutors(pos) * memoryPerExecutor
      val enoughMemory = usableWorkers(pos).memoryFree - assignedMemory >= memoryPerExecutor
      val underLimit = assignedExecutors.sum + app.executors.size < app.executorLimit
      keepScheduling && enoughCores && enoughMemory && underLimit
    } else {......}

scheduleExecutorOnWorkers方法爲application分配好邏輯意義上的資源後，還不能真正在Worker節點爲application分配出資源，當調用allocateWorkerResourceToExecutors方法時將會在Worker節點上真正分配資源。

private def allocateWorkerResourceToExecutors( app: ApplicationInfo,
......
    val exec = app.addExecutor(worker, coresToAssign)
    launchExecutor(worker, exec)
    app.state = ApplicationState.RUNNING}

launchExecutor方法的源碼：

private def launchExecutor(worker: WorkerInfo, exec: ExecutorDesc): Unit = {
  logInfo("Launching executor " + exec.fullId + " on worker " + worker.id)
  worker.addExecutor(exec)
  worker.endpoint.send(LaunchExecutor(masterUrl,
    exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory))
  exec.application.driver.send(
    ExecutorAdded(exec.id, worker.id, worker.hostPort, exec.cores, exec.memory))}

Worker收到LaunchExecutor（worker.endpoint.send()）消息會相應的處理。匹配LaunchDriver成功構建DriverRunner對象，調用DriverRunner的start方法。在DriverRunner的start方法中調用fetchAndRunExecutor方法，此方法中的CommandUtils.buildProcessBuilder(appDesc.command...)傳入的入口類是“org.apache.spark.executor.CoarseGrainedExecutorBackend”，當Worker節點中啓動ExecutorRunner時，ExecutorRunner中會啓動CoarseGrainedExecutorBackend進程，在CoarseGrainedExecutorBackend的onStart方法中，向Driver發出RegisterExecutor註冊請求。Driver端收到註冊請求，將會註冊Executor的請求。CoarseGrainedSchedulerBackend的receiveAndReply方法的源碼：

override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
  case RegisterExecutor(executorId, executorRef, hostPort, cores, logUrls) =>
    if (executorDataMap.contains(executorId)) { ......}

Driver向CoarseGrainedExecutorBackend發送RegisteredExecutor消息，CoarseGrainedExecutorBackend收到RrgisteredExecutor消息後將會新建一個Executor執行器，併爲此Executor充當信使與Driver通信。CoarseGrainedExecutorBackend收到RegisteredExecutor消息的實現方法receive源碼：

override def receive: PartialFunction[Any, Unit] = {
  case RegisteredExecutor(hostname) =>
    logInfo("Successfully registered with driver")
    executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
    ......}

從CoarseGrainedExecutorBackend的receive方法中可知，CoarseGrainedExecutorBackend收到RegisteredExecutor消息後，創建一個org.apache.spark.executor.Executor對象，至此Executor創建完畢。