案例分享：如何使用yarn中的容量调度器

偏执的太偏执、 2022-04-11 10:09 251阅读 0赞

### 容量调度器capacity scheduler的特性 ###

1，FIFO Scheduler的局限性，多个用户需要共享集群资源，集群资源以队列为单位划分；  
2，可以控制每一个队列资源最低保障和最高限制，最高使用限制是防止某个队列占用过多资源,导致其他队列资源紧张；  
3，可以针对用户设置每个用户最高资源使用限制，以防止该用户滥用或者频繁使用资源；  
4，每一个队列内部也是按照先进先出的原则调度资源；  
5，如果A队列资源使用紧张，B队列资源有剩余，那么可暂时把B剩余的资源共享给资源比较紧张的队列A，一旦B队列有应用程序提交，那么刚才被分配给队列A的资源会归还给该B队列。这种特性是弹性队列。

### 弹性队列与非弹性队列的区别 ###

弹性队列是当某个队列占用量超出了分配给本队列配额的情况下，(如果集群中其他队列有空余的资源)该队列仍然可以抢占额外的资源。  
非弹性队列是只能使用分给自己队列的配额，不能超额，即使集群中有空余资源也不能从其他的队列抢占资源。

### 如何决定使用弹性功能或者非弹性功能 ###

取决于你的集群资源情况和用户使用情况。当有多个部门的人共同使用一个集群的时候，建议使用capacity scheduler中的非弹性功能，因为资源是有限的，不能像fair scheduler那样自由抢占资源。

### 如何实施capacity scheduler ###

1，规划资源队列  
分为3个队列feature、dw、report\_config\_dev  
指定每个队列占用整个集群资源的下限分别是：35.0%、10.0%、55.0%

2，按照刚才的规划，修改每个nodeManager节点上的capacity-scheduler.xml

<configuration>
    <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
    Maximum number of applications that can be pending and running.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description>
    Maximum percent of resources in the cluster which can be used to run 
    application masters i.e. controls number of concurrent running
    applications.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
    The ResourceCalculator implementation to be used to compare 
    Resources in the scheduler.
    The default i.e. DefaultResourceCalculator only uses Memory while
    DominantResourceCalculator uses dominant-resource to compare 
    multi-dimensional resources such as Memory, CPU etc.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>dw,report_config_dev,feature</value>
    <description>
    The queues at the this level (root is the root queue).
    </description>
    </property>
    
    <property>
    <name>yarn.scheduler.capacity.root.feature.capacity</name>
    <value>29.0</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.report_config_dev.capacity</name>
    <value>61.0</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.dw.capacity</name>
    <value>10.0</value>
    </property>
    
    <property>
    <name>yarn.scheduler.capacity.root.feature.user-limit-factor</name>
    <value>1.0</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.report_config_dev.user-limit-factor</name>
    <value>1.0</value>
    </property>
    <property>
    
    <name>yarn.scheduler.capacity.root.dw.user-limit-factor</name>
    <value>1.0</value>
    </property>
    <property>
    
    <name>yarn.scheduler.capacity.root.feature.maximum-capacity</name>
    <value>100.0</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.report_config_dev.maximum-capacity</name>
    <value>100.0</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.dw.maximum-capacity</name>
    <value>100.0</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.dw.state</name>
    <value>RUNNING</value>
    <description>
    The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.feature.state</name>
    <value>RUNNING</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.report_config_dev.state</name>
    <value>RUNNING</value>
    </property>
    
    <property>
    <name>yarn.scheduler.capacity.root.feature.acl_submit_applications</name>
    <value>feature</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.report_config_dev.acl_submit_applications</name>
    <value>report_config_dev</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.dw.acl_submit_applications</name>
    <value>hive</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.feature.acl_administer_queue</name>
    <value>feature</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.root.report_config_dev.acl_administer_queue</name>
    <value>report_config_dev</value>
    </property>
    
    <property>
    <name>yarn.scheduler.capacity.root.dw.acl_administer_queue</name>
    <value>hive</value>
    </property>
    <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>31</value>
    <description>
    Number of missed scheduling opportunities after which the CapacityScheduler 
    attempts to schedule rack-local containers. 
    Typically this should be set to number of nodes in the cluster, By default is setting 
    approximately number of nodes in one rack which is 40.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value>u:root:dw,u:hive:dw,u:report_config_dev:report_config_dev,u:feature:feature</value>
    <description>
    A list of mappings that will be used to assign jobs to queues
    The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
    Typically this list will be used to map users to queues,
    for example, u:%user:%user maps all users to queues with the same name
    as the user.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
    If a queue mapping is present, will it override the value specified
    by the user? This can be used by administrators to place jobs in queues
    that are different than the one specified by the user.
    The default is false.
    </description>
    </property>
    <property>
    <name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name>
    <value>1</value>
    <description>
    Controls the number of OFF_SWITCH assignments allowed
    during a node's heartbeat. Increasing this value can improve
    scheduling rate for OFF_SWITCH containers. Lower values reduce
    "clumping" of applications on particular nodes. The default is 1.
    Legal values are 1-MAX_INT. This config is refreshable.
    </description>
    </property>
    </configuration>

【附图说明非弹性队列设置的效果】  
![非弹性队列使用效果][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNTk4MTEz_size_16_color_FFFFFF_t_70]  
其他的参数意义请参考官网  
[https://hadoop.apache.org/docs/r2.7.6/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html][https_hadoop.apache.org_docs_r2.7.6_hadoop-yarn_hadoop-yarn-site_CapacityScheduler.html]  
3，将配置文件yarn-site.xml的调度类修改为CapacityScheduler

<property>
        <name>yarn.resourcemanager.scheduler.class</name>
        
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>    
    </property>

4，将配置文件capacity-scheduler.xml、yarn-site.xml同步到其他NodeManager节点  
\[hadoop@tony-hdp-01 work\]$ cd /home/hadoop/hadoop/etc/hadoop/  
\[hadoop@tony-hdp-01 hadoop\]$ sync-all capacity-scheduler.xml  
\[hadoop@tony-hdp-01 hadoop\]$ sync-all yarn-site…xml

5，到active 状态的ResourceManager上刷新队列配置即可生效  
\[hadoop@tony-hdp-01 hadoop\]$ yarn rmadmin -getServiceState rm1  
active  
\[hadoop@tony-hdp-01 hadoop\]$ yarn rmadmin -getServiceState rm2  
standby  
\[hadoop@tony-hdp-01 hadoop\]$ yarn rmadmin -refreshQueues

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxNTk4MTEz_size_16_color_FFFFFF_t_70]: /images/20220411/9bed7a70f3d845d59895b7079574bbaf.png
[https_hadoop.apache.org_docs_r2.7.6_hadoop-yarn_hadoop-yarn-site_CapacityScheduler.html]: https://hadoop.apache.org/docs/r2.7.6/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html