Queues and Scheduling Policies
Users submit jobs to a submit queue and wait in line until nodes become available to run a job. NERSC's queue structures are intended to be fair and to allow jobs of various sizes to run efficiently. Balancing the job size and throughput requirements of a large number of users is always a challenge. We encourage users to send questions, feedback, or concerns about the queue structures, to the consultants.
Queue Classes
| Submit Queue | Execution Queue1 | Nodes | Processors | Max Wallclock | Relative Priority | Run Limit2 | Queued Limit3 | Queue Charge Factor |
|---|---|---|---|---|---|---|---|---|
| interactive | interactive | 1-256 | 1-6,144 | 30 mins | 2 | 1 | 1 | 1 |
| debug | debug | 1-512 | 1-12,288 | 30 mins | 3 | 1 | 1 | 1 |
| regular | reg_1hour | 1-256 | 1-6,144 | 1 hr | 5 | 8 | 8 | 1 |
| reg_short | 1-682 | 1-16,368 | 6 hrs | 5 | 16 | 16 | 1 | |
| reg_small | 1-682 | 1-16,368 | 48 hrs | 5 | 16 | 16 | 1 | |
| reg_med | 683-2,048 | 16,369-49,152 | 36 hrs | 4 | 4 | 4 | 0.6 | |
| reg_big | 2,049-4,096 | 49,153-98,304 | 36 hrs | 4 | 2 | 2 | 0.6 | |
| reg_xbig4 | 4,097-6,100 | 98,305-146,400 | 12 hrs | 1 | 2 | 2 | 0.6 | |
| -- | bigmem5 | 1-384 | 1-9,216 | 24 hrs | 5 | 1 | 1 | 1 |
| low | low | 1-683 | 1-16,392 | 24 hrs | 7 | 6 | 6 | 0.5 |
| thruput6 | thruput | 1-2 | 1-48 | 168 hrs | 5 | 250 | 500 | 1.0 |
| scavenger7 | scavenger | 1-683 | 1-16,392 | 6 hrs | 8 | 2 | 2 | 0 |
| premium | premium | 1-2,048 | 1-49,152 | 12 hrs | 3 | 1 | 1 | 2 |
| ccm_queue8 | ccm_queue | 1-682 | 1-16,368 | 96 hrs | 4 | 16 | 16 | 1 |
| ccm_int9 | ccm_int | 1-512 | 1-12,288 | 30 mins | 3 | 1 | 1 | 1 |
| iotask9 | iotask | 1-682 | 1-16,368 | 12 hrs | 6 | 1 | - | 1 |
| xfer10 | xfer | -- | -- | 12 hrs | -- | 4 | 3 | 0 |
1 - Jobs are submitted to the submit queues and automatically routed to the appropriate execution queue.
2 - The maximum number of jobs a single user is permitted to have running concurrently in the specified queue.
3 - The maximum number of jobs a single user is permitted to have eligible for scheduling in the specified queue
4 - The reg_xbig queue is only turned on at 21:00 Pacific Time on Fridays (jobs need to be submitted at least 48 hours in advance) and after system reboots.
5 - bigmem is an execution queue. Users may not submit jobs to it directly, but must include "-l mpplabels=bigmem" in the batch script. The relative priority, run limit, idle limit, and charging factor are those for the execution queue that this job is routed to.
6 - thruput is a queue designed for running high throuput jobs (usually many small jobs that need long wall clock time). The maximum run limit for this queue is set to 250 (not per user limit, but the global limit for this queue).
7 - scavenger is a queue available only to users with a zero or negative balance in one of their repositories. This applies to both total repository balances as well as per-user balances. The queue is not available for jobs submitted against a repository with a positive balance.
8 - ccm_queue and ccm_int are the queues designed for jobs to run under the Cluster Compatibility Mode (CCM).
9 - iotask is a queue designed specifically for running I/O intensive benchmarking codes in a more controlled way. The maximum run limit for this queue is set to 1 (not per user limit, but the global limit for this queue).
10 - xfer is a queue designed specifically for backing up your files to HPSS. The maximum run limit for this queue is 12 (this is not a per user limit, but the gloabl limit for all jobs in this queue). xfer jobs are submitted to a local batch server on hopper06.
Notes about queue policies:
- Do NOT submit scripts directly to an execution queue. Always use the submit queue name.
- If you have reached the run limit in an execution queue, then the queued limit becomes zero.
- There is a limit of 500 submitted jobs per execution queue per user.
- The debug and interactive queues are to be used for code development, testing, and debugging. Production runs are not permitted in the debug queue. User accounts are subject to suspension if they are determined to be using the debug queue for production computing. In particular, job "chaining" in the debug queue is not allowed. Chaining is defined as using a batch script to submit another batch script.
- User jobs can only be submitted to the scavenger queue when charged against a repository with a negative balance. If a user has multiple repositories, the user should add the line "#PBS -A <repo>" to the jobscript in order to specify which repo a job is to be charged against.
- 512 nodes (12,288 cores) are reserved for debugging and interactive jobs from 5am - 6pm Pacific Time.
- User jobs should not be in "user held" status for 14 or more days. Jobs over the user held limit will be removed from the system.
Tips for getting your job through the queue faster
- Submit shorter jobs. If your application has the capability to checkpoint and restart, consider submitting your job for shorter time periods. On a system as large as Hopper there are many opportunities for backfilling jobs. Backfill is a technique the scheduler uses to keep the system busy. If there is a large job at the top of the queue the system will need to drain resources in order to schedule that job. During that time, short jobs can run. Jobs that qualify for reg_short are good candidates for backfill.
- Make sure the wall clock time you request is accurate. As noted above, shorter jobs are easier to schedule. Many users unnecessarily enter the largest wall clock time possible as a default.
- Run jobs before a system maintenance. A system must drain all jobs before a maintenance so there is an opportunity for good turn around for shorter jobs.
