Phoenix Job priorities

水深无声 2022-05-09 05:46 194阅读 0赞

## Contents ##

\[[hide][]\]

*  [1Introduction][]
 *  [2How the FairShare algorithm work – a test example][2How the FairShare algorithm work _ a test example]
 *  [3Diagnostic tool – rcquota, rcshare and rcstat][3Diagnostic tool _ rcquota_ rcshare and rcstat]
 *  [4My job seems to take forever to run ? What is happening?][4My job seems to take forever to run _ What is happening]
 *  [5I used to wait a few hours before my job started running, but now it takes a few days !][5I used to wait a few hours before my job started running_ but now it takes a few days]

## Introduction\[[edit][] | [edit source][]\] ##

The SLURM job scheduler dynamically organises the queue in order to maximise the computational workload and thus the efficiency of the supercomputer. When you submit a job, it will be placed onto the queue and its priority will mainly depend the FairShare factor. The FairShare algorithm ensures that all project groups/schools can use their quarterly quotas. The following section will showcase an example to help understand how the FairShare algorithm works.

## How the FairShare algorithm work – a test example\[[edit][edit 1] | [edit source][edit source 1]\] ##

In this example, the supercomputer resources are shared among four groups : ACDC , Metallica, KISS and Credence Clearwater Revival (CCR). Each group is allocated a given percentage of the total resources (see Table 1). The fair share algorithm must ensure that the two pie charts shown in Figure 1 are very similar, that is the fraction of allocation spent by each group equals their allocated share.

<table>  Table 1: Example of the resources shared among four schools and the current resources spent by each school  
 <tbody> 
  <tr> 
   <th>School/Group</th> 
   <th colspan="2">Resources allocation</th> 
   <th colspan="3">Current usage</th> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>Service units [SU]</td> 
   <td>[%]</td> 
   <td>Quota [%]</td> 
   <td>Service unit spent [SU]</td> 
   <td>[%]</td> 
  </tr> 
  <tr> 
   <td>ACDC</td> 
   <td>100,000</td> 
   <td>10%</td> 
   <td>50%</td> 
   <td>50,000</td> 
   <td>7.9%</td> 
  </tr> 
  <tr> 
   <td>Metallica</td> 
   <td>300,000</td> 
   <td>30%</td> 
   <td>75%</td> 
   <td>225,000</td> 
   <td>35.7%</td> 
  </tr> 
  <tr> 
   <td>KISS</td> 
   <td>250,000</td> 
   <td>25%</td> 
   <td>100%</td> 
   <td>250,000</td> 
   <td>39.6%</td> 
  </tr> 
  <tr> 
   <td>CCR</td> 
   <td>350,000</td> 
   <td>35%</td> 
   <td>30%</td> 
   <td>105,000</td> 
   <td>16.6&nbsp;%</td> 
  </tr> 
  <tr> 
   <td>Total</td> 
   <td>1,000,000</td> 
   <td>100&nbsp;%</td> 
   <td>&nbsp;</td> 
   <td>630,000</td> 
   <td>100&nbsp;%</td> 
  </tr> 
 </tbody> 
</table>

As shown in Figure 1, it is easily noticeable that the KISS group have ran most of the jobs (~40% of the allocated resources so far compared to the expected 25 %). As a result, their job priorities will be considerably reduced. We also note from Table 1, that they have reached their quarterly quota. Although jobs can still be submitted, they will receive low priorities.

[![HPC_FairShare_Example.png][]][HPC_FairShare_Example.png 1]

Figure 1: Pie charts illustrating the expected share of the allocated resources (left panel) vs the actual current share allocated to the four schools (right panel, see Table 1)

ACDC and Metallica have received ~8 %<10 % and ~36%>30 % of the allocated resources so far. The FairShare algorithm will consequently slightly enhance the priority of ACDC and lowering the priority of Metallica.

Finally, the CCR group has not submitted enough jobs. The FairShare algorithm will therefore boost the priority of these jobs. However, they are required to submit these jobs before the end of the quarter or they will waste the remaining service units (SU).

To summarize, the FairShare algorithm will increase/decrease the job priority if the fraction of research allocation used is lower/greater than the fraction of the system share.

## Diagnostic tool – rcquota, rcshare and rcstat\[[edit][edit 2] | [edit source][edit source 2]\] ##

rcquota and rcshare are powerful tools to diagnose whether your jobs will be on the high priority list or not.

rcshare : shows a summary of allocation and usage statistics by project for the current accounting period. As shown in Figure 2, the last three columns can help understand how the FairShare algorithm will assess your job. In the case where the grant use is greater than 100 %, then the school/project group you belong to will have their job priority lowered.

On one hand, the system share represents the fractional amount of the quarterly allocation granted to a school/research group.

SYSTEM SHARE = GRANT SU / TOTAL GRANT SU

On the other hand, the system use represents the current fractional amount of service units allocated to a given school/research group :

SYSTEM USE = USAGE SU / TOTAL USAGE SU

If SYSTEM SHARE < SYSTEM USE, then the FairShare algorithm will decrease the priority of the jobs from your school/research project. If SYSTEM SHARE > SYSTEM USE, then the FairShare algorithm will increase the priority of the jobs from your school/research project.

[![HPC_rcshare.png][]][HPC_rcshare.png 1]

Figure 2: Output from rcshare. The columns indicate, in order, the project groups/schools, the quarterly service unit (SU) grant, the quarterly service unit used, the fraction of service unit used, the system share and the system use.

rcquota : This command outputs the allocation usage from your account as well as the school/project group you belong to.

[![HPC_rcquota.png][]][HPC_rcquota.png 1]

Figure 3: Output from rcquota. The first column displays the project group/school you belong to. The second column indicates the number of quarterly service units (SU) granted to your research project/school. The third and four columns show the number of service units used and the fraction over the service unit granted.

Sprio | -rn -k2 | less : ordering the submitted jobs according to their priority number (mostly determined by the FairShare factor). You can access the job id by using squeue -u <a1234567>

rcstat <job\_id> (see wiki-page): display the statistics of the job. This is particularly useful to optimize the resource allocation of a given SLURM job.

[![HPC_rcstat.png][]][HPC_rcstat.png 1]

Figure 4:Output of rcstat.

The first few rows displays the allocation requested when the job has been submitted. It then shows the time when the job was submitted, when it started and ended.

Finally, it displays the fraction of your requested wall-time used, the fraction CPU used, and finally the amount of memory written. If you intend to submit similar jobs, these information help you optimize your allocation request when submitting a job. A general consensus is that a SLURM job is optimised when the wall-time, the CPU used and the memory used is around 80 % of the reserved allocation.

In Figure 4, we notice that the job used 96,7 % of the CPU allocated. This indicates that the code makes good use of all the cores available (no cores are idle). However, we can considerably reduce the requested wall-time and the requested memory per node. By optimizing your job request, your submitted job is expected to start sooner.

## My job seems to take forever to run ? What is happening?\[[edit][edit 3] | [edit source][edit source 3]\] ##

It is expected that the larger is the allocation requested, the longer is the waiting time. The job scheduler ensures that the supercomputer is efficiently making use of the nodes at all time. Whenever you submit a large job, the scheduler has to find a time when all the resources requested is available. To reduce the waiting time, you must ensure you optimize your SLURM job using rcstat (see above section).

## I used to wait a few hours before my job started running, but now it takes a few days !\[[edit][edit 4] | [edit source][edit source 4]\] ##

The queue is constantly being updated every minutes. High priority jobs that have recently been submitted will be pushed ahead of the queue. By using rcshare, rcquota, and sprio (see above section), you can diagnose whether your job is given lower or higher priority. If your project group/school has exceeded the quota, then the FairShare algorithm will give you low priority. If your school system share is smaller than your system use (see rcshare), then FairShare will also lower your priority. This will result in extending your waiting time.

[hide]: https://wiki.adelaide.edu.au/hpc/Job_priorities#
[1Introduction]: https://wiki.adelaide.edu.au/hpc/Job_priorities#Introduction
[2How the FairShare algorithm work _ a test example]: https://wiki.adelaide.edu.au/hpc/Job_priorities#How_the_FairShare_algorithm_work_.E2.80.93_a_test_example
[3Diagnostic tool _ rcquota_ rcshare and rcstat]: https://wiki.adelaide.edu.au/hpc/Job_priorities#Diagnostic_tool_.E2.80.93_rcquota.2C_rcshare_and_rcstat
[4My job seems to take forever to run _ What is happening]: https://wiki.adelaide.edu.au/hpc/Job_priorities#My_job_seems_to_take_forever_to_run.C2.A0.3F_What_is_happening.3F
[5I used to wait a few hours before my job started running_ but now it takes a few days]: https://wiki.adelaide.edu.au/hpc/Job_priorities#I_used_to_wait_a_few_hours_before_my_job_started_running.2C_but_now_it_takes_a_few_days.C2.A0.21
[edit]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&veaction=edit&vesection=1
[edit source]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&action=edit&section=1
[edit 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&veaction=edit&vesection=2
[edit source 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&action=edit&section=2
[HPC_FairShare_Example.png]: /images/20220509/c9b5dcd5d07743bdb9cead043ecb8607.png
[HPC_FairShare_Example.png 1]: https://wiki.adelaide.edu.au/hpc/File:HPC_FairShare_Example.png
[edit 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&veaction=edit&vesection=3
[edit source 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&action=edit&section=3
[HPC_rcshare.png]: /images/20220509/f8a9ad7a77c54747b40c85ba2a550d56.png
[HPC_rcshare.png 1]: https://wiki.adelaide.edu.au/hpc/File:HPC_rcshare.png
[HPC_rcquota.png]: /images/20220509/2e4f65bcc98d4bb687a239953508754d.png
[HPC_rcquota.png 1]: https://wiki.adelaide.edu.au/hpc/File:HPC_rcquota.png
[HPC_rcstat.png]: /images/20220509/a052b3a635f54d41826cccd456f6fe88.png
[HPC_rcstat.png 1]: https://wiki.adelaide.edu.au/hpc/File:HPC_rcstat.png
[edit 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&veaction=edit&vesection=4
[edit source 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&action=edit&section=4
[edit 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&veaction=edit&vesection=5
[edit source 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=Job_priorities&action=edit&section=5