【Yarn】配置公平調度器

目錄

添加如下屬性到yarn-site.xml

添加配置文件fair-scheduler.xml

將新的配置同步到集羣所有節點,重啓Yarn

創建相應的用戶進行測試

提交任務

查看任務

---

• 添加如下屬性到yarn-site.xml

<!--  指定使用fairScheduler的調度方式  -->
<property>
       <name>yarn.resourcemanager.scheduler.class</name>
       <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
 
<!--  指定配置文件路徑  -->
<property>
       <name>yarn.scheduler.fair.allocation.file</name>
       <value>/export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop/fair-scheduler.xml</value>
</property>
 
<!-- 是否啓用資源搶佔，如果啓用，那麼當該隊列資源使用
yarn.scheduler.fair.preemption.cluster-utilization-threshold 這麼多比例的時候，就從其他空閒隊列搶佔資源
  -->
<property>
       <name>yarn.scheduler.fair.preemption</name>
       <value>true</value>
</property>
 
<property>
       <name>yarn.scheduler.fair.preemption.cluster-utilization-threshold</name>
       <value>0.8f</value>
</property>
 
<!-- 默認提交到default隊列  -->
<property>
       <name>yarn.scheduler.fair.user-as-default-queue</name>
       <value>true</value>
</property>
 
<!-- 如果提交一個任務沒有到任何的隊列，是否允許創建一個新的隊列，設置false不允許  -->
<property>
       <name>yarn.scheduler.fair.allow-undeclared-pools</name>
       <value>false</value>
</property>

 

• 添加配置文件fair-scheduler.xml

<?xml version="1.0"?>
<allocations>
<!-- users max running apps  -->
<userMaxAppsDefault>30</userMaxAppsDefault>
<!-- 定義隊列  -->
<queue name="root">
         <minResources>512mb,4vcores</minResources>
         <maxResources>102400mb,100vcores</maxResources>
         <maxRunningApps>100</maxRunningApps>
         <weight>1.0</weight>
         <schedulingMode>fair</schedulingMode>
         <aclSubmitApps> </aclSubmitApps>
         <aclAdministerApps> </aclAdministerApps>
 
         <queue name="default">
                   <minResources>512mb,4vcores</minResources>
                   <maxResources>30720mb,30vcores</maxResources>
                   <maxRunningApps>100</maxRunningApps>
                   <schedulingMode>fair</schedulingMode>
                   <weight>1.0</weight>
                   <!--  所有的任務如果不指定任務隊列，都提交到default隊列裏面來 -->
                   <aclSubmitApps>*</aclSubmitApps>
         </queue>
 
<!--
 
weight
資源池權重
 
aclSubmitApps
允許提交任務的用戶名和組；
格式爲： 用戶名 用戶組
 
當有多個用戶時候，格式爲：用戶名1,用戶名2 用戶名1所屬組,用戶名2所屬組
 
aclAdministerApps
允許管理任務的用戶名和組；
 
格式同上。
 -->
         <queue name="hadoop">
                   <minResources>512mb,4vcores</minResources>
                   <maxResources>20480mb,20vcores</maxResources>
                   <maxRunningApps>100</maxRunningApps>
                   <schedulingMode>fair</schedulingMode>
                   <weight>2.0</weight>
                   <aclSubmitApps>hadoop hadoop</aclSubmitApps>
                   <aclAdministerApps>hadoop hadoop</aclAdministerApps>
         </queue>
 
         <queue name="develop">
                   <minResources>512mb,4vcores</minResources>
                   <maxResources>20480mb,20vcores</maxResources>
                   <maxRunningApps>100</maxRunningApps>
                   <schedulingMode>fair</schedulingMode>
                   <weight>1</weight>
                   <aclSubmitApps>develop develop</aclSubmitApps>
                   <aclAdministerApps>develop develop</aclAdministerApps>
         </queue>
 
         <queue name="test1">
                   <minResources>512mb,4vcores</minResources>
                   <maxResources>20480mb,20vcores</maxResources>
                   <maxRunningApps>100</maxRunningApps>
                   <schedulingMode>fair</schedulingMode>
                   <weight>1.5</weight>
                   <aclSubmitApps>test1,hadoop,develop test1</aclSubmitApps>
                   <aclAdministerApps>test1 group_businessC,supergroup</aclAdministerApps>
         </queue>
</queue>
</allocations>

 

• 將新的配置同步到集羣所有節點,重啓Yarn

同步：

scp yarn-site.xml  fair-scheduler.xml node02:$PWD
scp yarn-site.xml  fair-scheduler.xml node03:$PWD

 

重啓Yarn：

stop-yarn.sh
start-yarn.sh

 

• 創建相應的用戶進行測試

useradd hadoop
passwd hadoop
groupadd supergroup
usermod -a -G supergroup hadoop    修改用戶所屬的附加羣主
su - root -s /bin/bash -c "hdfs dfsadmin -refreshUserToGroupsMappings"
刷新用戶組信息
su hadoop  切換用戶

 

• 提交任務

hadoop jar /export/servers/hadoop-2.6.0-cdh5.14.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.14.0.jar pi 10 20

 

• 查看任務

http://192.168.100.201:8088/cluster/scheduler