HPC Help Desk Feedback

HPC Help Desk

When to Contact the HPC Help Desk Users should contact the HPC Help Desk when assistance is needed for unclassified problems, issues, or questions.
Hours of Operation 8:00 a.m. - 11:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).
HPC Centers Home Page https://centers.hpc.mil/
Phone Number 1-877-222-2039 or (937) 255-0679
Fax Number (937) 656-9538
Help E-mail help@helpdesk.hpc.mil
Accounts E-mail accounts@helpdesk.hpc.mil
HPC Help Desk Manager E-mail manager@helpdesk.hpc.mil
After Hours Calls, e-mails and tickets received after normal operating hours will be addressed the following business day.
Mailing Address DoD HPCMP HPC Help Desk
AFRL/RCM
2435 Fifth Street
Wright-Patterson Air Force Base, OH - 45433-7802
Ticket Requests Active users may submit tickets from the Active User Help Ticket.
Inactive users may submit tickets using the Inactive User Help Ticket.

Getting Started

New to our HPC systems, or need a quick refresher? This page will help you get started.

Obtaining An Account

Click here for instructions on obtaining an HPC systems account.

Application Process

Prior to requesting access to any of the systems at one or more of the DoD Supercomputing Resource Centers (DSRCs), a user must obtain an account on the Portal to Information Environment (commonly referred to as a pIE account).

  • Your Service Agency Approval Authority (S/AAA) will provide you assistance through the account process. If you don't currently have a S/AAA or are unsure as to who is your S/AAA, contact require@hpc.mil.
  • NOTE: All users must have a National Agency Check with Inquiries (NACI) or a Security Clearance to run on all of the HPCMP resources except those who plan to run only on the Open Research Systems.
  • Go to the pIE web site and request a login account:
    1. You will need to know the following information to complete your request:
      • Citizenship Status, including registration numbers and expiration dates, if known
      • Preferred Kerberos Realm (HPCMP.HPC.MIL for US Citizens, Green Card Holders, and non-US citizens with a NACI; ORS.HPC.MIL for anyone without a clearance or NACI)
      • Organization ID (Your S/AAA will let you know which Org ID to choose from the drop down menu, for example, WPASC, AFVAW, NRLDC, etc.)
      • Name, Title, Position
      • Mailing address (No PO Box), Company Name, Phone, Fax, E-mail address
      • Preferred Username
      • Government Employee status. If not a Government Employee, you will need to provide a Government Point of Contact
      • Contract Number and Contract Expiration date, if known (optional - except for Air Force Projects)
      • Preferred Shell
    2. If you have a Common Access Card (CAC), click "Register My Common Access Card (CAC)" to register it to your pIE account.
    3. If you do not have a CAC, you should click the "I do NOT have a CAC" button.
    4. After you hit the submit button, you'll be asked to accept the HPCMP User Agreement. If you don't accept, your request will be terminated.
  • Complete the Information Assurance Training (also known as the Cyber Awareness Challenge) and send a copy of your signed Certificate of Completion to your S/AAA.

Back to Top

Approval

At this point, your pIE user account will be rejected or approved by your S/AAA. Once your S/AAA completes this action, you will receive a pIE notification informing you of the status of your application. You can contact the HPC Help Desk by e-mail help@helpdesk.hpc.mil or by phone 1-877-222-2039 at any time throughout this process to determine your account status.

If your pIE user account is rejected, talk to your S/AAA or alternate S/AAA for additional information and discuss re-submitting your request.

If your pIE user account is approved and your Preferred Kerberos Realm is HPCMP.HPC.MIL, your information will be sent to the HPC Help Desk to complete your HPCMP account.

If your pIE user account is approved and you only plan to run on the Open Research Systems, your information will be sent to Engineer Research and Development Center (ERDC) DoD Supercomputing Resource Center (DSRC) to complete your HPCMP account.

Back to Top

Activating your Account

To complete and activate your pIE user account with a Preferred Kerberos Realm of HPCMP.HPC.MIL:

  • Have your Facility Security Officer (FSO) send a Visit Request to the ERDC Security Office:
    • Security Specialist (Attn: HPC FSO)
    • Fax: 601-634-3134 (Call 601-634-3177 before faxing)
    • JPAS SMO Code: W03GAA (preferred)

It is recommended that you have your Security Office send your Visit Request to ERDC Security as soon as you apply for an account in pIE. This may help to expedite activation of your account. This single visit request will suffice for your pIE account and access to HPCMP resources.

NOTE: A Visit Request is a vehicle to transmit personal (Privacy Act) information from one security office to another, and is used for the purpose of HPC Accounts only.

  • Agree to the terms of the HPCMP Information System User Agreement.
    • You should have received an email with a "Subject" line of "[pIE NOTIFICATION] HPCMP User Agreement needed for <your name>." In the body of this email will be a unique URL that you must visit in order to agree to the terms of the HPCMP Information System User Agreement.

If you require a YubiKey (i.e., if you don't have a Common Access Card / CAC):

  • The HPC Help Desk will FedEx the YubiKey to the address provided in pIE when the account is approved and a visit request has been received.

To complete and activate your pIE user account to run on the Open Research Systems only, ERDC DSRC must receive:

  • Agree to the terms of the HPCMP Information System User Agreement.
    • You should have received an email with a "Subject" line of "[pIE NOTIFICATION] HPCMP User Agreement needed for <your name>." In the body of this email will be a unique URL that you must visit in order to agree to the terms of the HPCMP Information System User Agreement.

After all of the above is complete, your pIE user account will be activated within pIE. Additional steps must be taken in order for you to access HPC resources at the DoD Supercomputing Resource Centers (DSRCs). Please work directly with your S/AAA or Principal Investigator to get access to these resources.

When you no longer need access to HPC resources, you must return your YubiKey to the HPC Help Desk.

  • Put your YubiKey into a bubble wrap envelope and send it by regular US mail to:

    HPC Help Desk
    2435 Fifth St
    ATTN: HPC Accounts
    WPAFB OH 45433-7802

Back to Top

Kerberos & Authentication

Overview

Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography. The HPCMP employs Kerberos to authenticate user access to many of its resources, including all of its HPC systems and Utility Server systems, and many of its web sites. In order to login to a Kerberos-protected system, you must obtain an electronic Kerberos "ticket." The process for obtaining a ticket varies depending upon the operating system of your local computer (the "client" of the Kerberos transaction), and the type of cryptographic credential you have in your possession. Windows and Mac Kerberos clients are GUI-based; Linux clients are command line-based.

The most common cryptographic credential among HPCMP users is the DoD Common Access Card, or "CAC." The CAC is fitted with an integrated circuit chip containing information about the owner, including a PIN and one or more Public Key Infrastructure (PKI) certificates. One of these certificates is used to obtain a Kerberos ticket. This is accomplished through the use of PKINIT software that is included with the HPCMP-provided Kerberos Client Kit.

For users who do not possess a DoD CAC, the HPCMP provides an alternate method for Kerberos authentication: YubiKey. The YubiKey is a small hardware authentication token that looks like a USB memory stick (thumb drive), but is actually a small read-only device that presents itself to your computer as a keyboard. When plugged into a USB port, it generates a one-time alphanumeric string that enables you to obtain a Kerberos ticket.

Kerberos Documentation

Kerberos For Windows

Download and Install

Note: Export of this software from the United States of America may be subject to the Export Administration Regulations of the United States Department of Commerce. You are responsible for complying with all applicable export regulations, including obtaining an export license if required. You may not download this software if you are located in, or are a citizen or national of, any country for which the US government prohibits the export of encryption source code.

To provide flexibility, the HPCMP offers the Kerberos for Windows software for download in standard Windows Installer and a self-extracting archive form. Please note that the Windows Installer method of installing Kerberos for Windows must be installed by a system administrator. The Self-Extracting Archive form does not require administrator privileges.

Important! Please review the following "Installation Notes" before continuing.

Installation Notes

  • Be aware of the clock skew problem described on our HPCMP Kerberos FAQ page.
  • HPCMP Kerberos for Windows Client Kit Contains:
    • PuTTY 0.60
    • FileZilla 2.2.32
  • Prior to removing your current KfW, you may wish to save C:\Program Files\HPCMP Kerberos\FileZilla.xml to another location, as it will be deleted when the kit is removed
  • You MUST remove the old Kerberos for Windows (KfW) kit before installing the new kit.
  1. Click on either of the following, depending on which version you wish to install:
  2. Depending upon your web browser, you may be prompted to select a location to save the file, or the location may default to the Downloads folder of your Windows profile.
  3. Move the downloaded file to your Desktop.

Installing using the Windows Installer

The Windows Installer file guides the user through installing the Kerberos for Windows client software using a series of click-through windows and will create program directories, services, and application shortcuts for the user.

Users familiar with installing software on Windows will already be familiar with this process.

  1. Double click the installer executable (from Step 3 above)to begin the installation.
  2. You may be prompted by the Windows User Account Control (UAC) subsystem to provide your administrator credentials. If you do not have administrator privileges, click the Cancel button to exit the installer and contact your local administrator or security team for more support. If you do not have administrator credentials, you may want to download the self extracting executable and follow the installation instructions in the "Installing Using the Self-Extracting Archive" section below.
  3. Proceed through the Installer by selecting the Next button until the Installer completes
  4. Click the OK button to complete the Installer.

Installing using the Self-Extracting Archive

For users without administrator privileges or who prefer an alternative method for software installation, we also provide a self-extracting archive that can be extracted to a user-writeable directory.

Please note that while the HPCMP provides the Kerberos for Windows client software in self-extracting archive form, users are urged to contact their local computer support or security staff to ensure that they are permitted to use this software on their local computer system.

The self-extracting archive version of the Kerberos for Windows software doesn't require elevated or administrator privileges, provided that the user installs it to a location they have the ability to write to (i.e., the Desktop or My Documents locations).

  1. Double click the self-extracting executable to begin the installation.
  2. Click Extract to have the archive extracted to a folder on your Desktop.
  3. The software will be extracted to a folder on your Desktop.

Back to Top

Obtain a Ticket using a CAC

For guidance on choosing and configuring a CAC reader, please go to http://iase.disa.mil/pki-pke/getting_started/Pages/windows.aspx.

  1. Ensure your CAC is inserted into your smart card reader.
  2. Start the Kerberos client. If you installed using the Windows Installer, press Start>All Programs>HPCMP Kerberos>krb5. Windows start menu showing the krb5 icon highlighted
  3. If you installed using the Self-Extracting Archive, open the folder containing the extracted files and double click krb5.exe. Folder contents extracted from the self-extracting archive, with the krb5.exe icon highlighted
  4. Once the client starts, you will be presented the following application window. The krb5.exe dialog box showing No Tickets
  5. Enter your HPCMP username into the Name field.
  6. Leave the Password field blank.
  7. Enter your Kerberos Realm into the Realm field. For most users, this is HPCMP.HPC.MIL (capitalization required).
  8. Click Login.
  9. Enter your CAC PIN when prompted, then press OK. The PIN Prompt dialog for the ActivClient tool.
  10. When the ticket has been granted, you will see the following indication. The krb5.exe dialog box showing a ticket.

Back to Top

Obtain a Ticket using a YubiKey

  1. Plug your YubiKey into an open USB port on your local system. You will know it is inserted correctly and working when the green light on the side lights.
  2. Start the Kerberos client. If you installed using the Windows Installer, press Start>All Programs>HPCMP Kerberos>krb5. Windows start menu showing the krb5 icon highlighted
  3. If you installed using the Self-Extracting Archive, open the folder containing the extracted files and double click krb5.exe. Folder contents extracted from the self-extracting archive, with the krb5.exe icon highlighted
  4. Once the client starts, you will be presented the following application window. The krb5.exe dialog box showing No Tickets
  5. Enter your HPCMP username into the Name field.
  6. Enter your Kerberos password into the Password field.
  7. Enter your Kerberos Realm into the Realm field. For most users, this is HPCMP.HPC.MIL (capitalization required).
  8. Click Login.
  9. When prompted for your Passcode, press the button on the side of the YubiKey. This will enter your YubiKey passcode and press the OK button for you.
  10. When the ticket has been granted, you will see the following indication. The krb5.exe dialog box showing a ticket.

Back to Top

Login to an HPCMP System

The HPCMP provides Windows users the PuTTY client in the Kerberos client kit. PuTTY is an open source terminal emulator, serial console, and network file transfer application. It supports the Secure Shell (ssh) protocol, which is used in the HPCMP to provide a command line shell on HPCMP systems.

  1. Obtain a Kerberos ticket using one of the previously described methods.
  2. Start the PuTTY client. If you installed using the Windows Installer, press Start>All Programs>HPCMP Kerberos>putty. Windows start menu showing the putty icon highlighted
  3. If you installed using the Self-Extracting Archive, open the folder containing the extracted files and double click putty.exe. Folder contents extracted from the self-extracting archive, with the PuTTy.ext icon highlighted
  4. You will be presented with the following application window. There are a plethora of options available for you to customize the PuTTY window, but the default settings are sufficient. Default settings of the PuTTY interface
  5. Enter the system name that you wish to login to in the Host Name field, then press the Open button. PuTTy interface with kilrain.navo.hpc.mil in Host Name field
  6. The following shell window will appear and you will be logged into the target system. Shell window showing a successful login

Back to Top

Kerberos For Linux

Download and Install

Note: Export of this software from the United States of America may be subject to the Export Administration Regulations of the United States Department of Commerce. You are responsible for complying with all applicable export regulations, including obtaining an export license if required. You may not download this software if you are located in, or are a citizen or national of, any country for which the US government prohibits the export of encryption source code.

The Kerberos client kit for Linux is provided in a gzipped tar file and can be downloaded below. Client kits are available for 32-bit and 64-bit Linux systems. Even though the clients were built on a particular version and flavor of Linux distribution, they will work on most popular Linux distributions, including SuSE, Red Hat, Fedora, Ubuntu, Debian, and others.

In addition to the Kerberos client kit itself, you must also download and install the HPCMP Open Secure Shell (SSH) kit, also covered in this instructions.

As a general overview of these instructions, you will download two gzipped tar files and extract them into /usr/local. You will need to be root to do this.

  1. Click on either the 32-bit or 64-bit link below, depending on which version you wish to install.
  2. You will be prompted to save the file. Click Save File, then click OK.
  3. Depending upon your web browser, you may be prompted to select a location to save the file, or the location may default to the Downloads folder of your home directory.
  4. After the download is complete, you will next download the OpenSSH client. Click on the appropriate link below, depending on which version you wish to install. The i686 version is for 32-bit Intel and AMD based PC architecture. The x86_64 version is for 64-bit Intel and AMD based PC architecture. The Sun versions are for Sparc and x86 architectures, respectively.
  5. You will be prompted to save the file. Click Save File, then click OK.
  6. Depending upon your web browser, you may be prompted to select a location to save the file, or the location may default to the Downloads folder of your home directory.

  7. Open a terminal shell and, as root, copy the downloaded files to /usr/local. The Kerberos client kit will extract into ./krb5. The OpenSSH kit will extract into ./ossh. The directories will be created if they do not exist.

    # cd /usr/local
# cp ~username/Downloads/HPCMP_RELEASE_20111225b_client-2.6.9-103.EL-Linux-64.tar.gz .
# cp ~username/Downloads/ user-openssh-6.2p1a-x86_64-unknown-linux-gnu.tgz .
# tar xzvf HPCMP_RELEASE_20111225b_client-2.6.9-103.EL-Linux-64.tar.gz
# tar xzvf user-openssh-6.2p1a-x86_64-unknown-linux-gnu.tgz
# rm HPCMP_RELEASE_20111225b_client-2.6.9-103.EL-Linux-64.tar.gz
# rm user-openssh-6.2p1a-x86_64-unknown-linux-gnu.tgz
  8. The Kerberos and OpenSSH kits are now installed in /usr/local/krb5 and /usr/local/ossh, respectively.

Back to Top

Obtain a Ticket using a CAC

For guidance on choosing and configuring a CAC reader, please go to http://iase.disa.mil/pki-pke/getting_started/Pages/linux.aspx.

  1. Insert your CAC into the smart card reader.
  2. Open a restricted shell, using the kshell program provided in the HPCMP Kerberos client kit. Shell window executing the kshell command
  3. Initiate Kerberos by executing pkinit, also provided in the kit. Shell window executing the pkinit command
  4. Enter your CAC PIN when prompted. Shell window showing the PIN prompt
  5. You now have an active Kerberos ticket. You can verify this with the klist command. Shell window executing the klist command

Back to Top

Obtain a Ticket using a YubiKey

  1. Plug your YubiKey into an open USB port on your local system. You will know it is inserted correctly and working when the green light on the side lights.
  2. Open the terminal app and run the following commands: 
    /usr/local/krb5/bin/kshell

/usr/local/krb5/bin/kinit <kerberos_principal_name>@HPCMP.HPC.MIL
  3. Enter your kerberos password. When prompted for your Passcode, press the button on the side of the YubiKey. This will enter your YubiKey passcode and press the enter key for you.

Back to Top

Login to an HPCMP System

The HPCMP provides Windows users the PuTTY client in the Kerberos client kit. PuTTY is an open source terminal emulator, serial console, and network file transfer application. It supports the Secure Shell (ssh) protocol, which is used in the HPCMP to provide a command line shell on HPCMP systems.

  1. Obtain a Kerberos ticket using one of the previously described methods.

  2. Initiate the ssh command to the target host.

    $ /usr/local/ossh/bin/ssh kilrain.navo.hpc.mil
    Shell window showing a successful ssh login to a host
  3. You are now logged into the target system.

Back to Top

Kerberos For Mac

Download and Install

Note: Export of this software from the United States of America may be subject to the Export Administration Regulations of the United States Department of Commerce. You are responsible for complying with all applicable export regulations, including obtaining an export license if required. You may not download this software if you are located in, or are a citizen or national of, any country for which the US government prohibits the export of encryption source code.

The Kerberos client kit for Mac is provided in a gzipped tar file. The Kerberos client kit works on OSX 10.9 - 10.11 systems. OSX 10.8 and below are no longer supported. You must also download and install the HPCMP Open Secure Shell (SSH) kit, also covered in these instructions.

As a general overview of these instructions, you will download two gzipped tar files and extract them into /usr/local. You will need to be root to do this.

  1. Click on link below.
  2. You will be prompted to save the file. Click Save File, then click OK.
  3. Depending upon your web browser, you may be prompted to select a location to save the file, or the location may default to the Downloads folder of your home directory.
  4. After the download is complete, you will next download the OpenSSH client. Click on link below to download and install.
  5. You will be prompted to save the file. Click Save File, then click OK. Save File dialog
  6. Depending upon your web browser, you may be prompted to select a location to save the file, or the location may default to the Downloads folder of your home directory.
  7. Open a terminal shell and, as root, copy the downloaded files to /usr/local. The Kerberos client kit will extract into ./krb5. The OpenSSH kit will extract into ./ossh. The directories will be created if they do not exist. On a Macintosh system, the terminal shell can be found inside of the "Utilities" folder, which is inside of the "Applications" folder. It is named "terminal.app". To become root on a Macintosh system, you must typically use the "sudo" command along with your administrator password.
    # sudo -i
<enter administrator password>
# cd /usr/local
# cp ~username/Downloads/HPCMP_RELEASE_20120704_client-10.6-OSX.tar.gz .
# cp ~username/Downloads/openssh-6.0p1a-OSX-10.6.tar.gz .
# tar xzvf HPCMP_RELEASE_20120704_client-10.6-OSX.tar.gz
# tar xzvf openssh-6.0p1a-OSX-10.6.tar.gz
# rm HPCMP_RELEASE_20120704_client-10.6-OSX.tar.gz
# rm openssh-6.0p1a-OSX-10.6.tar.gz
  8. The Kerberos and OpenSSH kits are now installed in /usr/local/krb5 and /usr/local/ossh, respectively.

Back to Top

Obtain a Ticket using a CAC

For guidance on choosing and configuring a CAC reader, please go to http://iase.disa.mil/pki-pke/getting_started/Pages/mac.aspx.

To utilize a CAC on a Macintosh system, a smart card reader must be installed and the proper libraries (pcsc and libcoolkey/libcackey or OSX keychain) must be present. Contact your system administrator to install these items.

To obtain a Kerberos ticket with a CAC, follow these instructions.

  1. Insert your CAC into the smart card reader.
  2. Initiate Kerberos by executing pkinit, also provided in the kit.
  3. Enter your CAC PIN when prompted.
  4. You now have an active Kerberos ticket. You can verify this with the klist command. Shell window executing the klist command

Back to Top

Obtain a Ticket using a YubiKey

  1. Plug your YubiKey into an open USB port on your local system. You will know it is inserted correctly and working when the green light on the side lights.
  2. Open the terminal app and run the following commands: 
    /usr/local/krb5/bin/kshell

/usr/local/krb5/bin/kinit <kerberos_principal_name>@HPCMP.HPC.MIL
  3. Enter your kerberos password. When prompted for your Passcode, press the button on the side of the YubiKey. This will enter your YubiKey passcode and press the enter key for you.

Back to Top

Login to an HPCMP System

The HPCMP provides Linux users a modified version of OpenSSH in the OpenSSH client kit. OpenSSH is used to login to HPC systems.

  1. Obtain a Kerberos ticket using the previously described method.

  2. Initiate the ssh command to the target host.

    $ /usr/local/ossh/bin/ssh kilrain.navo.hpc.mil

    Shell window shoting a successful ssh login to a host

  3. You are now logged into the target system.

  4. Note that in the default configuration, your ticket will not "follow" you to the target system. To change this behavior, run the following as root:

    # echo "GSSAPIDelegateCredentials yes" >> /usr/local/ossh/etc/ssh_config

Back to Top

Connecting to a System

Many systems can then be accessed via Kerberized ssh as follows: 

%ssh user@system

A Kerberos client kit must be installed on your desktop to enable you to get a Kerberos ticket. Information about installing Kerberos clients on your Windows desktop can be found in the Kerberos & Authentication Section of this page.

For some systems, however, you may have to specify a numbered login node.

Please review the table below, to get specific system login information.

System Login Information
SystemLoginCenter
Armstrongarmstrong.navydsrc.hpc.milNAVY
Conradconrad.navydsrc.hpc.milNAVY
Coppercopper0[1-2].ors.hpc.milORS
Excaliburexcalibur.arl.hpc.milARL
Garnetgarnet.erdc.hpc.milERDC
Gordongordon.navydsrc.hpc.milNAVY
Haisehaise.navo.hpc.milNAVY
Lightninglightning.afrl.hpc.milAFRL
Predatorpredator.afrl.hpc.milAFRL
Riptideriptide.mhpcc.hpc.milMHPCC
Shepardshepard.navydsrc.hpc.milNAVY
Spiritspirit.afrl.hpc.milAFRL
Thunderthunder.afrl.hpc.milAFRL
Topaztopaz.erdc.hpc.milERDC
Utility Server us.[CENTER].hpc.mil (CENTER=afrl,arl,erdc,mhpcc,navo,ors) All Centers

Computing Environment

The HPCMP Centers Team provides an assortment of classified and unclassified computational, storage, visualization, and support resources for DoD scientists and engineers. Please select the Systems tab in the main menu bar to find detailed information about the equipment we make available to users.

While the specific computing environment on our HPC systems may vary by vendor, architecture, and the DoD Supercomputing Resource Center (DSRC) at which the systems are located, we provide certain common elements to help create a similar user experience as you move from system to system within our Program. These elements include environment variables, modules, math libraries, performance and profiling tools, high productivity languages, and others.

Each HPC system consists broadly of a set of login nodes, compute nodes, home directory space, and working directory (scratch) space, along with a large suite of software tools and applications. Access to HPC systems is typically gained through the use of a command line within a secure shell (ssh) instance. Specific authentication and login steps are provided in the Kerberos section of this web site.

Each DSRC operates a similar petascale mass storage system for long-term data storage, and users have the option of storing vital data files at an off-site disaster recovery facility. We also provide short-term storage on HPC systems themselves and on a Center-wide File System (CWFS) located at each DSRC.

Compiling Code

Need to compile your own source code instead of using the COTS and GOTS applications available on our HPC systems? No problem. Each HPC system offers multiple compiler choices for users. Available compilers and instructions for using them are provided in section "5.2 Available Compilers" of each HPC system's User Guide. The User Guide for a particular system is located on the Systems page; just click the Systems link in the main menu bar above, then navigate to the system of interest and look for the User Guide in the Available Documentation box.

Queues

In order to manage the volume of work demanded of HPCMP supercomputers, the HPC Centers Team employs a batch queuing system, PBS Pro, for workload management. The batch system makes use of queues, which hold a job until sufficient resources are available to execute the job. The characteristics of the queues found on each HPC system may vary, depending upon the size of the system, the type of workload for which it is optimized, the size of the job, and the priority of the work. To see details of the queues on specific HPC Systems, select the system of interest from the Systems menu in the main menu bar. Look for the "Queue Descriptions and Limits" box.

In a typical workflow, a user submits a job to a queue, and then at a future time when resources are available, a scheduler dispatches the job for execution. Upon completion, the job ends and relevant files are collected and deposited in a location specified by the user. The user generally has no control over when the job starts. If such control is needed, the HPC Centers Team provides the Advance Reservation Service (ARS), which allows users to choose a future time at which the job is guaranteed to run. Note, however, that the number of CPUs dedicated to ARS is limited.

The priority assigned to each queue is dictated by the priority of the work the queue is allowed to run. DoD Service/Agency computational projects may have different types of accounts and may run at different priorities. All foreground and background usage will be tracked by project and subproject and reported to pIE by subproject.

Queue Name/Priority Type of Queue Available To
Standard Allows users to run in foreground at standard priority. All Users
Background Allows users to run in background at lowest priority without charging the user's allocation. Impact on foreground usage is minimal. Some accounts may have background-only allocation, if they have no other allocation on that system. All Users
Debug Allows user to run short jobs at very high priority for program development and testing. All Users
Frontier Reserved for users/projects who received Frontier priority allocation via a proposal review process. Frontier Users Only
High-priority Reserved for high-priority, time-critical jobs on a regular or recurring basis. User works with Service Agency Approval Authority (S/AAA) to request special permission.
Urgent Reserved for high priority, single time-sensitive events arising from an unexpected need requiring faster-than-normal turnaround and special handling. Jobs run at highest priority on system. User works with Service Agency Approval Authority (S/AAA) to request special permission.

Running Jobs

In order to manage the volume of work demanded of HPCMP supercomputers, the HPC Centers Team employs a batch queuing system, PBS Pro, for workload management. To run a job in a typical workflow, a user submits a job to a queue, and then at a future time when resources are available, a scheduler dispatches the job for execution. Upon completion, the job ends and relevant files are collected and deposited in a location specified by the user. The user generally has no control over when the job starts. If such control is needed, the HPC Centers Team provides the Advance Reservation Service (ARS), which allows users to choose a future time at which the job is guaranteed to run. Note, however, that the number of CPUs dedicated to ARS is limited.

Batch jobs are controlled by scripts written by the user and submitted to the batch queuing system that manages the compute resource and schedules the job to run based on a set of policies. Batch scripts consist of two parts:

1) a set of directives that describe your resource requirements (time, number of processors, etc.) and
2) UNIX commands that perform your computations.

These UNIX command may create directories, transfer files, etc.; anything you can type at a UNIX shell prompt.

Please refer to each HPC system's PBS Guide for details regarding the format and execution of batch scripts. The PBS Guide for a particular system is located on the Systems page; just click the Systems link in the main menu bar above, then navigate to the system of interest and look for the PBS Guide in the Available Documentation box.