Spark架构
1、Standalone架构
- 整个集群分为 Master 节点和 Worker 节点，相当于 Hadoop 的 Master 和 Slave 节点。
- Master 节点上常驻 Master 守护进程，负责管理全部的 Worker 节点。
- Worker 节点上常驻 Worker 守护进程，负责与 Master 节点通信并管理 executors。
- Driver 官方解释是 “The process running the main() function of the application and creating the SparkContext”。Application 就是用户自己写的 Spark 程序（driver program），比如 WordCount.scala。如果 driver program 在 Master 上运行，比如在 Master 上运行，那么 SparkPi 就是 Master 上的 Driver。如果是 YARN 集群，那么 Driver 可能被调度到 Worker 节点上运行（比如上图中的 Worker Node 2）。
- 每个 Worker 上存在一个或者多个 ExecutorBackend 进程（一个application在一个Worker上只有一个ExecutorBackend ）。每个进程包含一个 Executor 对象，该对象持有一个线程池，每个线程可以执行一个 task。
- 每个
application 包含一个 driver 和多个 executors，每个 executor 里面运行的 tasks 都属于同一个 application
- Worker
通过持有 ExecutorRunner 线程来控制 CoarseGrainedExecutorBackend 的启停
2、Spark on yarn架构
上图是yarn-client模式，下图是yarn-cluster模式
从深层次的含义讲，yarn-cluster和yarn-client模式的区别其实就是Spark Driver运行在哪的区别，yarn-cluster模式下，driver运行在AM(Application Master)中，它负责向YARN申请资源，并监督作业的运行状况。当用户提交了作业之后，就可以关掉Client，作业会继续在YARN上运行。然而yarn-cluster模式不适合运行交互类型的作业。而yarn-client模式下，Client中运行Driver，在Client中能看到Application的运行信息，也就是说Client不能离开。
注意：本文以及后面的文章全是在standalone架构下分析的
作业提交
export HADOOP_CONF_DIR=/home/fw/hadoop-2.2.0/etc/hadoop
$SPARK_HOME/bin/spark-submit --class cn.edu.hdu.threshold.Threshold \
--master yarn-client \
--num-executors 2 \
--driver-memory 4g \
--executor-memory 4g \
--executor-cores 4 \
/home/fw/Cluster2.jar打开脚本spark-submit.sh，最后会发现调用的是org.apache.spark.deploy.SparkSubmit这个类
SparkSubmit.main
def main(args: Array[String]) {
val appArgs = new SparkSubmitArguments(args)
if (appArgs.verbose) {
printStream.println(appArgs)
}
val (childArgs, classpath, sysProps, mainClass) = createLaunchEnv(appArgs)
launch(childArgs, classpath, sysProps, mainClass, appArgs.verbose)
}先看createLaunchEnv方法
private[spark] def createLaunchEnv(args: SparkSubmitArguments)
: (ArrayBuffer[String], ArrayBuffer[String], Map[String, String], String) = {
if (args.master.startsWith("local")) {
clusterManager = LOCAL
} else if (args.master.startsWith("yarn")) {
clusterManager = YARN
} else if (args.master.startsWith("spark")) {
clusterManager = STANDALONE
} else if (args.master.startsWith("mesos")) {
clusterManager = MESOS
} else {
printErrorAndExit("Master must start with yarn, mesos, spark, or local")
}
......
(childArgs, childClasspath, sysProps, childMainClass)
}
接下来是launch方法
private def launch(
childArgs: ArrayBuffer[String],
childClasspath: ArrayBuffer[String],
sysProps: Map[String, String],
childMainClass: String,
verbose: Boolean = false) {
......
val mainClass = Class.forName(childMainClass, true, loader)
val mainMethod = mainClass.getMethod("main", new Array[String](0).getClass)
try {
mainMethod.invoke(null, childArgs.toArray)
} catch {
case e: InvocationTargetException => e.getCause match {
case cause: Throwable => throw cause
case null => throw e
}
}
}利用反射机制运行childMainClass
启动Client
Client.main
def main(args: Array[String]) {
......
// TODO: See if we can initialize akka so return messages are sent back using the same TCP
// flow. Else, this (sadly) requires the DriverClient be routable from the Master.
val (actorSystem, _) = AkkaUtils.createActorSystem(
"driverClient", Utils.localHostName(), 0, conf, new SecurityManager(conf))
actorSystem.actorOf(Props(classOf[ClientActor], driverArgs, conf))
actorSystem.awaitTermination()
}
} 咦？这三行代码什么？一开始看到这也懵了，后来一查才知道这是Spark使用了一个很经典的消息通信机制：Akka
下面是官方的例子：
//定义一个case class用来传递参数
case class Greeting(who: String)
//定义Actor，比较重要的一个方法是receive方法，用来接收信息的
class GreetingActor extends Actor with ActorLogging {
def receive = {
case Greeting(who) ⇒ log.info("Hello " + who)
}
}
//创建一个ActorSystem
val system = ActorSystem("MySystem")
//给ActorSystem设置Actor
val greeter = system.actorOf(Props[GreetingActor], name = "greeter")
//向greeter发送信息，用Greeting来传递
greeter ! Greeting("Charlie Parker")ClientActor.preStart
override def preStart() = {
masterActor = context.actorSelection(Master.toAkkaUrl(driverArgs.master))
......
driverArgs.cmd match {
case "launch" =>
......
masterActor ! RequestSubmitDriver(driverDescription)
case "kill" =>
val driverId = driverArgs.driverId
val killFuture = masterActor ! RequestKillDriver(driverId)
}
}2、参数配置及代码的封装
3、最后向master提交程序
启动Driver
Master.receive
override def receive = {
......
case RequestSubmitDriver(description) => {
if (state != RecoveryState.ALIVE) {
val msg = s"Can only accept driver submissions in ALIVE state. Current state: $state."
sender ! SubmitDriverResponse(false, None, msg)
} else {
logInfo("Driver submitted " + description.command.mainClass)
val driver = createDriver(description)
persistenceEngine.addDriver(driver)
waitingDrivers += driver
drivers.add(driver)
schedule()
// TODO: It might be good to instead have the submission client poll the master to determine
// the current status of the driver. For now it's simply "fire and forget".
sender ! SubmitDriverResponse(true, Some(driver.id),
s"Driver successfully submitted as ${driver.id}")
}
}1、将Driver封装成DriverInfor对象
2、再将Driver加入到相应的集合中
3、Master执行自己的调度机制
4、向Client返回Resonse
Master的调度机制我们也放到后面的章节
Master.launchDriver
def launchDriver(worker: WorkerInfo, driver: DriverInfo) {
logInfo("Launching driver " + driver.id + " on worker " + worker.id)
worker.addDriver(driver)
driver.worker = Some(worker)
worker.actor ! LaunchDriver(driver.id, driver.desc)
driver.state = DriverState.RUNNING
}Worker.receive
override def receive = {
......
case LaunchDriver(driverId, driverDesc) => {
logInfo(s"Asked to launch driver $driverId")
val driver = new DriverRunner(driverId, workDir, sparkHome, driverDesc, self, akkaUrl)
drivers(driverId) = driver
driver.start()
coresUsed += driverDesc.cores
memoryUsed += driverDesc.mem
}
......
}
DriverRunner.start会启动一个线程执行命令，通过ProcessBuilder类启动Driver进程,这是万里长征的第一步，之后还有初始化SparkContext、DAGScheduler、TaskScheduler等，将Application中的RDD划分成Stage，调度Tasks提交执行，这些会在之后的章节陆续展开