National Center for Computational Sciences

General Information

In a simple batch queue system, jobs run in a first-in, first-out (FIFO) order. This often does not make effective use of the system. A large job may be next in line to run. If the system is using a strict FIFO queue, then many processors may sit idle while the large job waits to run. Backfilling would allow smaller, shorter jobs to use those otherwise idle resources, and with the proper algorithm, the start time of the large job would not be delayed. While this does make more effective use of the system, it encourages smaller jobs to ensure quick turnaround time.

Basic Queue Policies

The NCCS has implemented the following queue policies to enable large jobs to run in a timely fashion on LCF systems.

The basic priority factor for jobs remains the time a given job has been waiting relative to other jobs in the queue. However, several factors are applied by the batch system to modify the apparent time a job has been waiting. These factors include the size of the job and the queue to which it was submitted (both of which are described below) as well as the percent usage of the project under which the job is run (see the next section). If your jobs require resources outside these limits, please complete the relevant request form on the special requests page. If you have any questions or comments on the queue policies below, please direct them to the User Assistance Center.

Jobs are aged according to the processor count. Each processor count drops a job into a specific bin (see the table below) used for scheduling. Each bin has a different priority mechanism for aging. All jobs in each bin have the same aging boost. All jobs in each bin are scheduled based on an 8-week fair-share mechanism.

Current Configuration

Production Queue Limits

• The system may be further constrained such that jobs in bin 1 consume a minimum aggregate time on the system in a given period. For example, jobs in those bins must consume at least 80% of all system resources in a month.
• The batch queue system places a limit of two production jobs in the queued (i.e., eligible-to-run) state per user. If a user submits more than two jobs, the additional jobs will enter a held state. Once one of the user’s queued jobs begins execution, one of the held jobs will be moved into the queued state. Note that this is not a limit on the number of jobs that a user may have running simultaneously. Instead, it is a limit on the number eligible to enter a running state.
• A user may have only two production jobs in bin 5 running at any time.

Debug Queue Policies

• Approximately 10% of the system (3,100 cores) is reserved for debugging and development. These cores may be access by jobs submitted to the debug queue. At the center’s discretion, this debug partition may be altered in size to meet the debugging and development needs of the user community.
• The debug reservation is in effect from 10 a.m. until 7 p.m. (Eastern time) Monday-Thursday.
• Debug jobs requesting more than 3,100 cores will not run on the reserved debug cores. A priority boost is not given to debug jobs. For these reasons the debug queue is not beneficial to jobs requesting more than 3,100 cores.
• The maximum wall time for debug jobs is 1 hour.
• Only one debug job (in any state) is allowed per user.
• Production jobs are not allowed in the debug queue.

Allocation Overuse Policies

Projects that overrun their allocation are still allowed to run on LCF systems, although at a reduced priority. Like the adjustment for the number of processors requested, this is an adjustment to the apparent submit time of the job. However, this adjustment has the effect of making jobs appear much younger than jobs submitted under projects that have not exceeded their allocation. In addition to the priority change, these jobs are also limited in the amount of wall time that can be used.

For example, if job1 is submitted at the same time as job2, and the project for job1 is over its allocation (while the project for job2 is not over its allocation), the batch system will consider job2 to have been waiting for a longer time than job1. The length of time applied as an adjustment to the project that is over its allocation depends on the system being used and the percentage that the project is over its allocation.

• For projects that have used between 100% and 125% of their allocations, the following rules apply:
  • Jobs have their priority reduced by 30 days.
  • Jobs have a maximum wall time limit of 12 hours.
• For projects that have used greater than 125% of their allocation, the following rules apply:
  • Jobs have their priority reduced by 365 days.
  • Jobs have a maximum wall time limit of 4 hours.

System Reservations

Projects may request to reserve a set of processors for a period of time through the reservation request form. If the reservation is granted, the reserved processors will be blocked from general use for a given period of time. Only users that have been authorized to use the reservation can utilize those resources. Since no other users can access the reserved resources, it is crucial that groups given reservations take care to ensure the utilization on those resources remains high. To prevent reserved resources from remaining idle for an extended period of time, reservations are monitored for inactivity. If activity falls below 50% of the reserved resources for more than 30 minutes, the reservation will be canceled and the system will be returned to normal scheduling. A new reservation must be requested if this occurs.

Since a reservation makes resources unavailable to the general user population, projects that are granted reservations will be charged (regardless of their actual utilization) a cputime equivalent to
(# of cores reserved) * (length of reservation in hours).