下面是StorageLevel类的代码解释
/**
* :: DeveloperApi ::
* Flags for controlling the storage of an RDD. Each StorageLevel records whether to use memory,
* or ExternalBlockStore, whether to drop the RDD to disk if it falls out of memory or
* ExternalBlockStore, whether to keep the data in memory in a serialized format, and whether
* to replicate the RDD partitions on multiple nodes.
*
* 用于控制RDD存储的标志,每个StorageLevel记录是否使用内存或ExternalBlockStore,如果它脱离内存或ExternalBlockStore,
* 是否将RDD丢弃到磁盘,是否将数据保存在内存中的序列化格式,以及是否复制多个节点上的RDD分区
*
* The [org.apache.spark.storage.StorageLevel$]singleton object contains some static constants
* for commonly useful storage levels. To create your own storage level object, use the
* factory method of the singleton object (`StorageLevel(...)`).
* [org.apache.spark.storage.StorageLevel $] singleton对象包含一些常用的存储级别的静态常量,
* 要创建自己的存储级别对象,请使用单例对象（`StorageLevel（...）`）的工厂方法
*/StorageLevel的成员如下：
private var _useDisk: Boolean,
private var _useMemory: Boolean,
private var _useOffHeap: Boolean,//使用扩展存储
private var _deserialized: Boolean,
private var _replication: Int = 1) //默认的复本数是1StorageLevel中的Object方法解释如下：
object StorageLevel {
//不会保存任务数据
val NONE = new StorageLevel(false, false, false, false)
//直接将RDD的partition保存在该节点的Disk上
val DISK_ONLY = new StorageLevel(true, false, false, false)
//直接将RDD的partition保存在该节点的Disk上,在其他节点上保存一个相同的备份
val DISK_ONLY_2 = new StorageLevel(true, false, false, false, 2)
//将RDD的partition对应的原生的Java Object保存在JVM中,如果RDD太大导致它的部分partition不能存储在内存中
//那么这些partition将不会缓存,并且需要的时候被重新计算,默认缓存的级别
val MEMORY_ONLY = new StorageLevel(false, true, false, true)
//将RDD的partition对应的原生的Java Object保存在JVM中,在其他节点上保存一个相同的备份
val MEMORY_ONLY_2 = new StorageLevel(false, true, false, true, 2)
val MEMORY_ONLY_SER = new StorageLevel(false, true, false, false)
val MEMORY_ONLY_SER_2 = new StorageLevel(false, true, false, false, 2)
//将RDD的partition反序列化后的对象存储在JVM中,如果RDD太大导致它的部分partition不能存储在内存中
//超出的partition将被保存在Disk上,并且在需要时读取
val MEMORY_AND_DISK = new StorageLevel(true, true, false, true)
//在其他节点上保存一个相同的备份
val MEMORY_AND_DISK_2 = new StorageLevel(true, true, false, true, 2)
val MEMORY_AND_DISK_SER = new StorageLevel(true, true, false, false)
val MEMORY_AND_DISK_SER_2 = new StorageLevel(true, true, false, false, 2)
//将RDD的partition序列化后存储在Tachyon中
val OFF_HEAP = new StorageLevel(false, false, true, false)下面是我们的RDD存储的调用
RDD和Block关系如下：
一个RDD有多个Partition,每个Partition对应一个Block块，也就是说一个RDD有多个Block,同时每个BlockId又有拥有唯一的编号BlockId，对应的数据块的编号规则为：“rdd_”+”rddid_”+”splitIndex”,其中这个splitIndex对应的是数据块对应的Partition的序列号
RDD的存储过程分析如下：
/**
* Mark this RDD for persisting using the specified level.
* 标记此RDD以使用指定的级别进行持久化
* this.type表示当前对象（this)的类型。this指代当前的对象。
* this.type被用于变量,函数参数和函数返回值的类型声明
* @param newLevel the target storage level
* @param allowOverride whether to override any existing level with the new one
*/
private def persist(newLevel: StorageLevel, allowOverride: Boolean): this.type = {
// TODO: Handle changes of StorageLevel
if (storageLevel != StorageLevel.NONE && newLevel != storageLevel && !allowOverride) {
throw new UnsupportedOperationException(
"Cannot change storage level of an RDD after it was already assigned a level")
}
// If this is the first time this RDD is marked for persisting, register it
// with the SparkContext for cleanups and accounting. Do this only once.
//如果这是RDD第一次标记为持久性,请注册与SparkContext进行清理和计费,只做一次
if (storageLevel == StorageLevel.NONE) {
sc.cleaner.foreach(_.registerRDDForCleanup(this))
sc.persistRDD(this)
}
storageLevel = newLevel
this
}
/**
* 设置RDD存储级别在操作之后完成,这里只能分配RDD尚未确认的新存储级别,检查点是一个例别
* Set this RDD's storage level to persist its values across operations after the first time
* it is computed. This can only be used to assign a new storage level if the RDD does not
* have a storage level set yet. Local checkpointing is an exception.
* 设置此RDD的存储级别，以便在第一次操作之后保持其值计算,如果RDD没有,这只能用于分配新的存储级别
*还有一个存储级别,本地检查点是一个例外。
*/
def persist(newLevel: StorageLevel): this.type = {
if (isLocallyCheckpointed) {
//之前已经调用过localCheckpoint(),这里应该标记RDD待久化,在这里我们应该重写旧的存储级别,一个是由用户显式请求
// This means the user previously called localCheckpoint(), which should have already
// marked this RDD for persisting. Here we should override the old storage level with
// one that is explicitly requested by the user (after adapting it to use disk).
persist(LocalRDDCheckpointData.transformStorageLevel(newLevel), allowOverride = true)
} else {
persist(newLevel, allowOverride = false)
}
}
/**
* Persist this RDD with the default storage level (`MEMORY_ONLY`).
* 持久化RDD,默认存储级别MEMORY_ONLY,内存存储
* */
def persist(): this.type = persist(StorageLevel.MEMORY_ONLY)
/**
* Persist this RDD with the default storage level (`MEMORY_ONLY`).
* 持久化RDD使用默认存储级别(内存存储)
* */
def cache(): this.type = persist()
/**
* Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
* 删除持久化RDD,同时删除内存和硬盘
* @param blocking Whether to block until all blocks are deleted.
* @return This RDD.
*/
def unpersist(blocking: Boolean = true): this.type = {
logInfo("Removing RDD " + id + " from persistence list")
sc.unpersistRDD(id, blocking)
storageLevel = StorageLevel.NONE //将存储级别设置为None
this
}
/**
* Get the RDD's current storage level, or StorageLevel.NONE if none is set.
* 获得RDD当前的存储级别
* */
def getStorageLevel: StorageLevel = storageLevel下面是我们的RDD的Iterator方法的解析如下：
/**
* Task的执行起点,计算由此开始
* Internal method to this RDD; will read from cache if applicable(可用), or otherwise(否则) compute it.
* This should ''not'' be called by users directly, but is available for implementors of custom
* subclasses of RDD.
* RDD内部方法,从缓存中读取可用,如果没有则计算它,
*/
final def iterator(split: Partition, context: TaskContext): Iterator[T] = {
//如果存储级别不是NONE 那么先检查是否有缓存,没有缓存则要进行计算
if (storageLevel != StorageLevel.NONE) {
getOrCompute(split, context)
//SparkEnv包含运行时节点所需要的环境信息
//cacheManager负责调用BlockManager来管理RDD的缓存,如果当前RDD原来计算过并且把结果缓存起来.
//接下来的运行都可以通过BlockManager来直接读取缓存后返回
} else {
//如果没有缓存,存在检查点时直接获取中间结果
computeOrReadCheckpoint(split, context)
}
}getOrComputey方法如下：
/** Gets or computes an RDD partition. Used by RDD.iterator() when an RDD is cached.
* 获取或计算一个RDD的分区,当RDD被缓存时由RDD.iterator()使用
* */
private[spark] def getOrCompute(partition: Partition, context: TaskContext): Iterator[T] = {
//通过RDD的编号和Partition的序号获取到数据块Block的编号
val blockId = RDDBlockId(id, partition.index)
var readCachedBlock = true
// This method is called on executors, so we need call SparkEnv.get instead of sc.env
//因为这个方法是由executor来调用这个方法，所以可以使用SparkEnv代替sc.env
//先根据数据块BlockId来读取数据，然后更新数据，这个方法是读写数据的入口
SparkEnv.get.blockManager.getOrElseUpdate(blockId, storageLevel, elementClassTag,
//如果是数据块不存在，则尝试读取检查点的结果进行迭代计算
() => {
readCachedBlock = false
computeOrReadCheckpoint(partition, context)
}) match {
case Left(blockResult) =>
if (readCachedBlock) {
val existingMetrics = context.taskMetrics().inputMetrics
existingMetrics.incBytesRead(blockResult.bytes)
new InterruptibleIterator[T](context, blockResult.data.asInstanceOf[Iterator[T]]) {
override def next(): T = {
existingMetrics.incRecordsRead(1)
delegate.next()
}
}
} else {
new InterruptibleIterator(context, blockResult.data.asInstanceOf[Iterator[T]])
}
case Right(iter) =>
new InterruptibleIterator(context, iter.asInstanceOf[Iterator[T]])
}
}下面是我们的getOrElseUpdate方法的解析如下：
/**
*这个方法是我们的Spark存写数据的入口点，同时这个方法由我们的Executor调用
* Retrieve the given block if it exists, otherwise call the provided `makeIterator` method
* to compute the block, persist it, and return its values.
*
* @return either a BlockResult if the block was successfully cached, or an iterator if the block
* could not be cached.
*/
def getOrElseUpdate[T](
blockId: BlockId,
level: StorageLevel,
classTag: ClassTag[T],
makeIterator: () => Iterator[T]): Either[BlockResult, Iterator[T]] = {
// Attempt to read the block from local or remote storage. If it's present, then we don't need
// to go through the local-get-or-put path.
//读取数据的入口点，尝试从本地或者远程读取数据
get[T](blockId)(classTag) match {
case Some(block) =>
return Left(block)
case _ =>
// Need to compute the block.
}
// Initially we hold no locks on this block.
//写数据入口
doPutIterator(blockId, makeIterator, level, classTag, keepReadLock = true) match {
case None =>
// doPut() didn't hand work back to us, so the block already existed or was successfully
// stored. Therefore, we now hold a read lock on the block.
val blockResult = getLocalValues(blockId).getOrElse {
// Since we held a read lock between the doPut() and get() calls, the block should not
// have been evicted, so get() not returning the block indicates some internal error.
releaseLock(blockId)
throw new SparkException(s"get() failed for block $blockId even though we held a lock")
}
// We already hold a read lock on the block from the doPut() call and getLocalValues()
// acquires the lock again, so we need to call releaseLock() here so that the net number
// of lock acquisitions is 1 (since the caller will only call release() once).
releaseLock(blockId)
Left(blockResult)
case Some(iter) =>
// The put failed, likely because the data was too large to fit in memory and could not be
// dropped to disk. Therefore, we need to pass the input iterator back to the caller so
// that they can decide what to do with the values (e.g. process them without caching).
Right(iter)
}
}