概述
ResourceManager是一系列组件的集合。它会加载core-site.xml和yarn-site.xml,同时创建yarn scheduler、RMAppManager等一系列组件。
创建yarn scheduler
获取yarn.resourcemanager.scheduler.class配置项,默认是CapacityScheduler
public static final String RM_SCHEDULER = "yarn.resourcemanager.scheduler.class";
public static final String DEFAULT_RM_SCHEDULER = "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler";
protected ResourceScheduler createScheduler() {
//获取yarn.resourcemanager.scheduler.class配置项,默认是CapacityScheduler
String schedulerClassName = conf.get(YarnConfiguration.RM_SCHEDULER,
YarnConfiguration.DEFAULT_RM_SCHEDULER);
LOG.info("Using Scheduler: " + schedulerClassName);
try {
Class<?> schedulerClazz = Class.forName(schedulerClassName);
if (ResourceScheduler.class.isAssignableFrom(schedulerClazz)) {
return (ResourceScheduler) ReflectionUtils.newInstance(schedulerClazz,
this.conf);
} else {
throw new YarnRuntimeException("Class: " + schedulerClassName
+ " not instance of " + ResourceScheduler.class.getCanonicalName());
}
} catch (ClassNotFoundException e) {
throw new YarnRuntimeException("Could not instantiate Scheduler: "
+ schedulerClassName, e);
}
}
Application管理模块
protected RMAppManager createRMAppManager() {
return new RMAppManager(this.rmContext, this.scheduler, this.masterService,
this.applicationACLsManager, this.conf);
}
ApplicationMaster管理模块
创建ApplicationMasterService
protected ApplicationMasterService createApplicationMasterService() {
Configuration config = this.rmContext.getYarnConfiguration();
if (isOpportunisticSchedulingEnabled(conf)) {
if (YarnConfiguration.isDistSchedulingEnabled(config) &&
!YarnConfiguration
.isOpportunisticContainerAllocationEnabled(config)) {
throw new YarnRuntimeException(
"Invalid parameters: opportunistic container allocation has to " +
"be enabled when distributed scheduling is enabled.");
}
OpportunisticContainerAllocatorAMService
oppContainerAllocatingAMService =
new OpportunisticContainerAllocatorAMService(this.rmContext,
scheduler);
this.rmContext.setContainerQueueLimitCalculator(
oppContainerAllocatingAMService.getNodeManagerQueueLimitCalculator());
return oppContainerAllocatingAMService;
}
return new ApplicationMasterService(this.rmContext, scheduler);
}
ApplicationMasterLauncher
protected ApplicationMasterLauncher createAMLauncher() {
return new ApplicationMasterLauncher(this.rmContext);
}
NodeManager管理模块
protected ResourceTrackerService createResourceTrackerService() {
return new ResourceTrackerService(this.rmContext, this.nodesListManager,
this.nmLivelinessMonitor,
this.rmContext.getContainerTokenSecretManager(),
this.rmContext.getNMTokenSecretManager());
}
用户管理模块
创建AdminService
protected AdminService createAdminService() {
return new AdminService(this);
}
创建ClientRMService
protected ClientRMService createClientRMService() {
return new ClientRMService(this.rmContext, scheduler, this.rmAppManager,
this.applicationACLsManager, this.queueACLsManager,
this.rmContext.getRMDelegationTokenSecretManager());
}
ResourceManager的交互协议
-
ResourceTracker
NodeManager通过该协议向ResourceManager中注册、汇报节点健康情况以及Container的运行状态,并且领取ResourceManager下达的重新初始化、清理Container等命令。NodeManager和ResourceManager这种RPC通信采用了和MRv1类似的“pull模型”(ResourceManager充当RPC server角色,NodeManager充当RPC client角色),NodeManager周期性主动地向ResourceManager发起请求,并且领取下达给自己的命令。
-
ApplicationMasterProtocol
应用程序的ApplicationMaster同过该协议向ResourceManager注册、申请和释放资源。该协议和上面协议同样也是采用了“pull模型”,其中在RPC机制中,ApplicationMaster充当RPC client角色,ResourceManager充当RPC server角色。
-
ApplicationClientProtocol
-
客户端通过该协议向ResourceManager提交应用程序、控制应用程序(如杀死job)以及查询应用程序的运行状态等。在该RPC 协议中应用程序客户端充当RPC client角色,ResourceManager充当RPC server角色。
整理一下ResourceManager与NodeManager、ApplicationMaster和客户端RPC协议交互的信息:
上图中的ResourceTrackeServer、ApplicationMasterService 、ClientRMServer是ResourceManager中处理上述功能的组件。