深入浅出Spark（三）：Spark调度系统之“权力的游戏”

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"专题介绍"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2009 年，Spark 诞生于加州大学伯克利分校的 AMP 实验室（the Algorithms, Machines and People lab），并于 2010 年开源。2013 年，Spark 捐献给阿帕奇软件基金会（Apache Software Foundation），并于 2014 年成为 Apache 顶级项目。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"如今，十年光景已过，Spark 成为了大大小小企业与研究机构的常用工具之一，依旧深受不少开发人员的喜爱。如果你是初入江湖且希望了解、学习 Spark 的“小虾米”，那么 InfoQ 与 FreeWheel 技术专家吴磊合作的专题系列文章——"},{"type":"link","attrs":{"href":"https:\/\/www.infoq.cn\/theme\/84","title":"xxx","type":null},"content":[{"type":"text","text":"《深入浅出 Spark：原理详解与开发实践》"}]},{"type":"text","text":"一定适合你！"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文系专题系列第三篇。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"令狐冲：“我要退出江湖，从此不问江湖之事。”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"italic"}],"text":"任我行：“这个世界有人的地方就有恩怨，有恩怨就有江湖，人就是江湖，你怎么退出？”"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"感谢各位看官在百忙之中来听我说书，真是太给面子啦！前文书《"},{"type":"link","attrs":{"href":"https:\/\/www.infoq.cn\/article\/LBzKJPoaFAre5c0cI4ur","title":"xxx","type":null},"content":[{"type":"text","text":"内存计算的由来 ——DAG"}]},{"type":"text","text":"》咱们说到DAGScheduler以首尾倒置的方式从后向前回溯DAG计算图，沿途以Shuffle为边界划分Stages。那么，这些Stages划分出来之后有什么用呢？DAGScheduler如何将DAG划分出的Stages转化为可执行的分布式任务？本期“权力的游戏”将带您走进Spark调度系统，笔者将竭尽全力与您一起揭开Spark调度系统的神秘面纱。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"在本段书正式开始之前，咱们先得铺垫铺垫，毕竟保不齐有刚入座的看官头一次来咱们书棚，咱们都得照顾到不是。在讲Spark调度系统之前，咱们先来简单回顾一下Spark分布式系统架构和重要概念。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}