Scheduling policy

Priority and backfill

Jobs are primarily started in priority order but the backfilling algorithm may give precedence to low priority jobs under certain conditions.

Job priorities

We make use of the priority/multifactor plugin. For a full documentation please refer to: https://slurm.schedmd.com/priority_multifactor.html

Each jobs' priority at any given time is determined by this formula:

PRIORITY =  AGE + FAIRSHARE + JOBSIZE + QoS + TRES - NICE

Depending on the QoS of a job there may be restrictions on the maximum number of allowed resources, like concurrently running jobs per user. This command can be used to inspect the settings for a particular QoS:

sacctmgr show QoS <QoS>

Can priority be increased for urgent jobs?

Please send an email with your request to our support team.

Can one request a walltime longer than the default for special jobs?

If no time limit was specified in a job script, the default time limit for the partition applies. The maximum time limit that can be set, is defined with the QoS for the job. Use these commands to find out the default and maximum allowed runtimes:

scontrol -o show partition <Partition> | tr ' ' '\n'|grep -i DefaultTime

sacctmgr show QoS zen3_0512 format=name%30,maxwall
                          Name     MaxWall
------------------------------ -----------
                     zen3_0512  3-00:00:00

sacctmgr show QoS skylake_0096 format=name%30,maxwall
                          Name     MaxWall
------------------------------ -----------
                  skylake_0096  3-00:00:00

The default time limit is set to 24h. If you need a longer walltime please specify an appropriate duration in your job script, e.g.:

#SBATCH -t 2-00:00:00

for a runtime of 48h.

Can walltime be extended once the job has been started?

Please send an email with your request to our support team.

Backfill

Lower priority jobs are sometimes started using "backfill" scheduling if this can be done without affecting higher-priority jobs. Backfilling is the process of scheduling jobs on compute nodes that would otherwise be idle while waiting for enough nodes to become available to start a large job. If there are idle nodes available and a lower priority job can be started without affecting the estimated start time of the highest priority job, the lower priority job is started. If more than one low-priority job could be started using backfill, the highest priority one is selected. In general, jobs shorter than a few hours have a good chance to be started using backfill. Once started, a job is never terminated to allow a higher-priority job to start.

It is highly recommended to specify a proper time limit for your jobs. Only then backfilling is activated and your job can possibly start sooner.