客戶端MapReduce提交到YARN過程（二）

客戶端通過RPC協議ClientRMProtocol提交Application，其實是提交了一個SubmitApplicationRequest，在Hadoop 1.0時代，是使用Writable作爲序列化和反序列化框架的，而在2.0中hadoop已經廢棄掉了，改用了ProtocolBuffer，它除了支持多種語言外最大的好處是向後兼容性，這樣不同版本的AM，Client，RM和NM之間能相互通信。

比如對於SubmitApplicationRequest, 源代碼只有PB的實現SubmitApplicationRequestPBImpl，當然用戶也可以實現其他的序列化反序列化框架的實現，包括Writable實現

SubmitApplicationRequest中會封裝提交上下文信息ApplicationSubmissionContext. ClientRMProtocol在RM的實現類是ClientRMService, RM在啓動的時候會初始化這個Service，該Service負責端口監聽RPC請求(由yarn.resourcemanager.address設置)

服務端根據提交上下文信息構造RMAppManagerSubmitEvent對象，交由rmAppManager來處理，RMAppManager在RM中的作用是管理所有的Application，比如提交/完成application，恢復一組applications等等。

ClientRMService的submitApplication方法：

public SubmitApplicationResponse submitApplication(
      SubmitApplicationRequest request) throws YarnRemoteException {
		 ApplicationSubmissionContext submissionContext = request
		 .getApplicationSubmissionContext();
		 ApplicationId applicationId = submissionContext.getApplicationId();
		 String user = submissionContext.getUser();
		 user = UserGroupInformation.getCurrentUser().getShortUserName();
		 if (rmContext.getRMApps().get(applicationId) != null) {
		 throw new IOException("Application with id " + applicationId
		 + " is already present! Cannot add a duplicate!");
		 }
		 submissionContext.setUser(user);
		 // 將Application信息提交給rmAppManager來啓動ApplicationMaster
		 rmAppManager.handle(new RMAppManagerSubmitEvent(submissionContext, System
		 .currentTimeMillis()));
		 
		 // If recovery is enabled then store the application information in a 
		 // blocking call so make sure that RM has stored the information needed 
		 // to restart the AM after RM restart without further client communication
		 RMStateStore stateStore = rmContext.getStateStore();
		 LOG.info("Storing Application with id " + applicationId);
		 try {
		 stateStore.storeApplication(rmContext.getRMApps().get(applicationId));
		 } catch (Exception e) {
		 LOG.error("Failed to store application:" + applicationId, e);
		 ExitUtil.terminate(1, e);
		 }
		 LOG.info("Application with id " + applicationId.getId() + 
		 " submitted by user " + user);
		 RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST,
		 "ClientRMService", applicationId);


		 SubmitApplicationResponse response = recordFactory
		 .newRecordInstance(SubmitApplicationResponse.class);
		 return response;
}

YARN採用了基於事件驅動的編程模型，一個狀態的改變可以觸發一個或多個事件，同時可以觸發其他狀態發生變化。對於RMAppManagerSubmitEvent他有兩種狀態APP_SUBMIT和APP_COMPLETED，在RMAppManager Handle這個Event的時候，如果是APP_COMPLETED則執行finishApplication將ApplicationId加入completedApps隊列中，若狀態爲APP_SUBMIT則執行submitApplication方法生成一個RMApp，放入RMContext的Map<ApplicationId, RMApp> applications總，然後生成一個狀態爲START的RMAppEvent交由AsyncDispatcher中央調度器來處理。

調用棧

RMAppManager的handle函數

  public void handle(RMAppManagerEvent event) {
    ApplicationId applicationId = event.getApplicationId();
    LOG.debug("RMAppManager processing event for " 
        + applicationId + " of type " + event.getType());
    switch(event.getType()) {
      case APP_COMPLETED: 
      {
        finishApplication(applicationId);
        ApplicationSummary.logAppSummary(
            rmContext.getRMApps().get(applicationId));
        checkAppNumCompletedLimit(); 
      } 
      break;
      case APP_SUBMIT:
      {
        ApplicationSubmissionContext submissionContext = 
            ((RMAppManagerSubmitEvent)event).getSubmissionContext();
        long submitTime = ((RMAppManagerSubmitEvent)event).getSubmitTime();
        submitApplication(submissionContext, submitTime);
      }
      break;
      default:
        LOG.error("Invalid eventtype " + event.getType() + ". Ignoring!");
      }
  }

submitApplication邏輯

 protected void submitApplication(
      ApplicationSubmissionContext submissionContext, long submitTime) {
    ApplicationId applicationId = submissionContext.getApplicationId();
    RMApp application = null;
    try {
      // Create RMApp
      application =
          new RMAppImpl(applicationId, rmContext, this.conf,
            submissionContext.getApplicationName(),
            submissionContext.getUser(), submissionContext.getQueue(),
            submissionContext, this.scheduler, this.masterService,
            submitTime);
   rmContext.getRMApps().putIfAbsent(applicationId, application) != 
          null);
      // All done, start the RMApp
      this.rmContext.getDispatcher().getEventHandler().handle(
          new RMAppEvent(applicationId, RMAppEventType.START));
    } catch (IOException ie) {
        LOG.info("RMAppManager submit application exception", ie);
        if (application != null) {
          // Sending APP_REJECTED is fine, since we assume that the 
          // RMApp is in NEW state and thus we havne't yet informed the 
          // Scheduler about the existence of the application
          this.rmContext.getDispatcher().getEventHandler().handle(
              new RMAppRejectedEvent(applicationId, ie.getMessage()));
        }
    }
  }

本文鏈接http://blog.csdn.net/lalaguozhe/article/details/10142707，轉載請註明