spark2.3源碼分析之ResultTask讀取並處理shuffle file的流程

ResultTask

概述

ResultTask 執行當前分區的計算，首先從ShuffleMapTask拿到當前partition的數據，會從所有的ShuffleMapTask都拿一遍當前的partition數據。最後合併所有的ResultTask輸出結果，返回給driver application。

成員變量

private[spark] class ResultTask[T, U](
    stageId: Int,
    stageAttemptId: Int,
    taskBinary: Broadcast[Array[Byte]],
    partition: Partition,
    locs: Seq[TaskLocation],
    val outputId: Int,
    localProperties: Properties,
    serializedTaskMetrics: Array[Byte],
    jobId: Option[Int] = None,
    appId: Option[String] = None,
    appAttemptId: Option[String] = None,
    isBarrier: Boolean = false)
  extends Task[U](stageId, stageAttemptId, partition.index, localProperties, serializedTaskMetrics,
    jobId, appId, appAttemptId, isBarrier)

runTask方法

  override def runTask(context: TaskContext): U = {
    // Deserialize the RDD and the func using the broadcast variables.
    val threadMXBean = ManagementFactory.getThreadMXBean
    val deserializeStartTime = System.currentTimeMillis()
    val deserializeStartCpuTime = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
      threadMXBean.getCurrentThreadCpuTime
    } else 0L
    val ser = SparkEnv.get.closureSerializer.newInstance()
//反序列化ResultTask，結果爲rdd,和func函數
//taskBinary的值是在DAGScheduler.submitMissingTasks()方法中進行序列化的
    val (rdd, func) = ser.deserialize[(RDD[T], (TaskContext, Iterator[T]) => U)](
      ByteBuffer.wrap(taskBinary.value), Thread.currentThread.getContextClassLoader)
    _executorDeserializeTime = System.currentTimeMillis() - deserializeStartTime
    _executorDeserializeCpuTime = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
      threadMXBean.getCurrentThreadCpuTime - deserializeStartCpuTime
    } else 0L
//整個ResultTask計算在 rdd.iterator(partition, context) 中完成
//此時的RDD爲:ShuffleRDD,所以rdd.iterator()方法調用的是ShuffleRDD.iterator()方法，會調用ShuffleRDD.compute()方法
//func 函數將Iterator轉換爲數組: RDD.collect()方法中的 (iter: Iterator[T]) => iter.toArray
    func(context, rdd.iterator(partition, context))
  }

 ShuffleRDD.compute()方法

調用 BlockStoreShuffleReader.read()方法去各個節點拉取ShuffleMapTask輸出的當前partition的磁盤文件。

override def compute(split: Partition, context: TaskContext): Iterator[(K, C)] = {
    val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, C]]
    SparkEnv.get.shuffleManager.getReader(dep.shuffleHandle, split.index, split.index + 1, context)
      .read()
      .asInstanceOf[Iterator[(K, C)]]
  }

spark shuffle過程中典型的fetch shuffle file堆棧如下：

org.apache.spark.shuffle.IndexShuffleBlockResolver.getBlockData(IndexShuffleBlockResolver.scala:191)
	at org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:298)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks(ShuffleBlockFetcherIterator.scala:238)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:269)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:112)
	at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:43)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:98)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)

參考：Spark源碼分析之ResultTask處理