How to build a cluster scheduler 项目简介

Aim / Learning objectives

The aim of this project is to explore the resource allocation problem in data-intensive systems. Schedulers need to distribute the workload across a set of resources to achieve a balance between various quality of service measures and the cost of operating a system.

Background

The background to the resource allocation problem is explained in the paper on schedulers that is already in the list of core papers for the module.

Diptanu Gon Choudhury and Timothy Perrett. Designing cluster schedulers for internet-scale services. Communications of the ACM, 61(6):34–40, May 2018

The problem to tackle in this project is a simplified version of the general problem of scheduling workloads onto a set of infrastructure components (servers).

• 本篇博客是关于"如何编写一个集群调度器"系列博客的背景介绍。这一系列博客将围绕该项目进行阐述，并以实例代码的方式逐步地编写一个集群调度器。在前面的博客中，我为大家转发了关于Apache开源的两种资源调度框架的文章，其中就提到了"Scheduler"这个重要的组件。在这里就不介绍了，有兴趣的小伙伴可以去查阅了解一下。本篇博客中，我为大家推荐一篇论文 - “Designing cluster schedulers for internet-scale services”, 该论文主要讲解了什么是"cluster scheduler" 和如何设计scheduler；论文关于设计的部分是以实际应用中所遇到的问题为前提和角度去阐述的，通读该论文应该能对"cluster scheduler"能有个大致的了解。

University of St Andrews 的"Data-intensive System"课程的一个项目