Yarn Scheduler Load Simulator YARN調度負載模擬器

    項目起源是因爲有人希望有個模擬環境來模擬公平調度器和容量調度器，以便合理配置調度器，降低生產環境出問題的風險，詳見https://issues.apache.org/jira/browse/YARN-1021。之後在hadoop2.3.0就增加了這個工具。

    首先設定環境變量：

     export HADOOP_HOME=/usr/hadoop-2.3.0

     export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop #此目錄放置sls-runner.xml文件

     sls-runner.xml文件內容如下：

<configuration>

  <!-- SLSRunner configuration -->
  <property>
    <name>yarn.sls.runner.pool.size</name>
    <value>100</value>
  </property>
  
  <!-- Nodes configuration -->
  <property>
    <name>yarn.sls.nm.memory.mb</name>
    <value>10240</value>
  </property>
  <property>
    <name>yarn.sls.nm.vcores</name>
    <value>10</value>
  </property>
  <property>
    <name>yarn.sls.nm.heartbeat.interval.ms</name>
    <value>1000</value>
  </property>
  
  <!-- Apps configuration -->
  <property>
    <name>yarn.sls.am.heartbeat.interval.ms</name>
    <value>1000</value>
  </property>
  <property>
    <name>yarn.sls.am.type.mapreduce</name>
    <value>org.apache.hadoop.yarn.sls.appmaster.MRAMSimulator</value>
  </property>

  <!-- Containers configuration -->
  <property>
    <name>yarn.sls.container.memory.mb</name>
    <value>1024</value>
  </property>
  <property>
    <name>yarn.sls.container.vcores</name>
    <value>1</value>
  </property>

  <!--  metrics  -->
  <property>
    <name>yarn.sls.metrics.switch</name>
    <value>ON</value>
  </property>
  <property>
    <name>yarn.sls.metrics.web.address.port</name>
    <value>10001</value>
  </property>
  <property>
    <name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler</name>
    <value>org.apache.hadoop.yarn.sls.scheduler.FifoSchedulerMetrics</value>
  </property>
  <property>
    <name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</name>
    <value>org.apache.hadoop.yarn.sls.scheduler.FairSchedulerMetrics</value>
  </property>
  <property>
    <name>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</name>
    <value>org.apache.hadoop.yarn.sls.scheduler.CapacitySchedulerMetrics</value>
  </property>
  
</configuration>

• yarn.sls.runner.pool.size

  The simulator uses a thread pool to simulate the NM and AM running , and this parameter specifies the number of threads in the pool.

• yarn.sls.nm.memory.mb

  The total memory for each NMSimulator.

• yarn.sls.nm.vcores

  The total vCores for each NMSimulator.

• yarn.sls.nm.heartbeat.interval.ms

  The heartbeat interval for each NMSimulator.

• yarn.sls.am.heartbeat.interval.ms

  The heartbeat interval for each AMSimulator.

• yarn.sls.am.type.mapreduce

  The AMSimulator implementation for MapReduce-like applications. Users can specify implementations for other type of applications.

• yarn.sls.container.memory.mb

  The memory required for each container simulator.

• yarn.sls.container.vcores

  The vCores required for each container simulator.

• yarn.sls.runner.metrics.switch

  The simulator introduces Metrics to measure the behaviors of critical components and operations. This field specifies whether we open (ON) or close (OFF) the Metrics running.

• yarn.sls.metrics.web.address.port

  The port used by simulator to provide real-time tracking. The default value is 10001.

• org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler

  The implementation of scheduler metrics of Fifo Scheduler.

• org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

  The implementation of scheduler metrics of Fair Scheduler.

• org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

  The implementation of scheduler metrics of Capacity Scheduler.

然後使用apache rumen對jobhistory文件進行解析，生成json格式文件以便sls解析：

hadoop jar hadoop-rumen-2.3.0.jar org.apache.hadoop.tools.rumen.TraceBuilder -write-job-trace file:///home/user/job-trace.json file:///home/user/topology.output file:///home/user/logs/history/done

file:///home/user/logs/history/done 用戶集羣存放運行完成jobhistory的目錄，一般在hdfs裏，可以通過hadoop fs -get取到本地目錄

file:///home/user/job-trace.json file:///home/user/topology.output 生成的sls要讀取的文件

運行模擬器

slsrun.sh --input-rumen=/home/user/  --output-dir=/usr/sls/sample-result

--input-rumen 本例就是file:///home/user/job-trace.json file:///home/user/topology.output 對應的路徑

如果運行報錯：

java.lang.NullPointerException
at org.apache.hadoop.yarn.sls.web.SLSWebApp.<init>(SLSWebApp.java:82)
at 

在https://issues.apache.org/jira/browse/YARN-1021中查看comment，解決方法如下：

bin/slsrun.sh --input-sls=sls-file/sls-jobs.json --output-dir=output_sls --nodes=sls-file/sls-nodes.json