Hadoop源碼之JobTracker

JobTracker是Map/Reducer中任務調度的服務器。

1、有如下線程爲其服務：

1）提供兩組RPC服務（InterTrackerProtocol、JobSubmissionProtocol）的1個Listener線程與默認10個Handler線程；

 2）提供任務執行情況查詢的一組web服務線程，包括Socker Listener等；

 3）ExpireTrackers：用來停止已經無效的TaskTracker服務；

synchronized (taskTrackers) {
    synchronized (trackerExpiryQueue) {
	long now = System.currentTimeMillis();
	TaskTrackerStatus leastRecent = null;
	while ((trackerExpiryQueue.size() > 0) &&
	       ((leastRecent = (TaskTrackerStatus) trackerExpiryQueue.first()) != null) &&
	       (now - leastRecent.getLastSeen() > TASKTRACKER_EXPIRY_INTERVAL)) {

	    // Remove profile from head of queue
	    trackerExpiryQueue.remove(leastRecent);
	    String trackerName = leastRecent.getTrackerName();

	    // Figure out if last-seen time should be updated, or if tracker is dead
	    TaskTrackerStatus newProfile = (TaskTrackerStatus) taskTrackers.get(leastRecent.getTrackerName());
	    // Items might leave the taskTracker set through other means; the
	    // status stored in 'taskTrackers' might be null, which means the
	    // tracker has already been destroyed.
	    if (newProfile != null) {
		if (now - newProfile.getLastSeen() > TASKTRACKER_EXPIRY_INTERVAL) {
		    // Remove completely
		    updateTaskTrackerStatus(trackerName, null);
		    lostTaskTracker(leastRecent.getTrackerName());
		} else {
		    // Update time by inserting latest profile
		    trackerExpiryQueue.add(newProfile);
		}
	    }
	}
    }
}

 

4）RetireJobs：用來除去已經完成任務的TaskTracker；

synchronized (jobs) {
    synchronized (jobInitQueue) {
	synchronized (jobsByArrival) {
	    for (Iterator it = jobs.keySet().iterator(); it.hasNext(); ) {
		String jobid = (String) it.next();
		JobInProgress job = (JobInProgress) jobs.get(jobid);

		if (job.getStatus().getRunState() != JobStatus.RUNNING &&
		    job.getStatus().getRunState() != JobStatus.PREP &&
		    (job.getFinishTime() + RETIRE_JOB_INTERVAL < System.currentTimeMillis())) {
		    it.remove();
	    
		    jobInitQueue.remove(job);
		    jobsByArrival.remove(job);
		}
	    }
	}
    }
}

5）JobInitThread：用來對job做一些初始化的工作；

synchronized (jobInitQueue) {
    if (jobInitQueue.size() > 0) {
	job = (JobInProgress) jobInitQueue.elementAt(0);
	jobInitQueue.remove(job);
    } else {
	try {
	    jobInitQueue.wait(JOBINIT_SLEEP_INTERVAL);
	} catch (InterruptedException iex) {
	}
    }
}
try {
    if (job != null) {
	job.initTasks();
    }
} catch (Exception e) {
    LOG.log(Level.WARNING, "job init failed", e);
    job.kill();
}

2、實現了兩組rpc服務（協議），其中InterTrackerProtocol如下：

1）TaskTracker間隔幾秒鐘發送的心跳服務；

int emitHeartbeat(TaskTrackerStatus status, boolean initialContact);

2）向JobTracker獲取新的任務；

Task pollForNewTask(String trackerName);

3）詢問JobTracker，任務是否可以終結；

String pollForTaskWithClosedJob(String trackerName);

4）Reduce Task詢問JobTracker，哪些Map Task已經結束；

  MapOutputLocation[] locateMapOutputs(String taskId, String[][] mapTasksNeeded);

5）獲取文件系統名；

public String getFilesystemName() throws IOException;

 

JobSubmissionProtocol如下：

1）提交一個待執行的job；

public JobStatus submitJob(String jobFile) throws IOException;

2）殺死一個job；

public void killJob(String jobid);

3）獲取job的名字、id等信息；

public JobProfile getJobProfile(String jobid);

4）獲取job的狀態；

public JobStatus getJobStatus(String jobid);

5）獲取Map任務的報告；

public TaskReport[] getMapTaskReports(String jobid);

6）獲取Reduce任務的報告；

public TaskReport[] getReduceTaskReports(String jobid);