Oak Ridge Leadership Computing Facility

Contents

• 1. Computing Policy
• 2. Data Management Policy
• 3. Cyber Security Policy
• 4. Titan Scheduling Policy
• 5. INCITE Allocation Under-utilization Policy
• 6. Project Reporting Policy

This guide presents all official policies of the OLCF.

1. Computing Policy

(Back to Top)

Computers, software, and communications systems provided by the OLCF are to be used for work associated with and within the scope of the approved project. The use of OLCF resources for personal or non-work-related activities is prohibited. All computers, networks, E-mail, and storage systems are property of the United States Government. Any misuse or unauthorized access is prohibited, and is subject to criminal and civil penalties.

OLCF systems are provided to our users without any warranty. OLCF will not be held liable in the event of any system failure or data loss or corruption for any reason including, but not limited to: negligence, malicious action, accidental loss, software errors, hardware failures, network losses, or inadequate configuration of any computing resource or ancillary system.

The OLCF computer systems are operated as research systems and only contain data related to scientific research and do not contain personally identifiable information (data that falls under the Privacy Act of 1974 5U.S.C. 552a). Use of OLCF resources to store, manipulate, or remotely access any national security information is strictly prohibited. This includes, but is not limited to: classified information, unclassified controlled nuclear information (UCNI), naval nuclear propulsion information (NNPI), the design or development of nuclear, biological, or chemical weapons or any weapons of mass destruction.

Authors/generators/owners of information are responsible for its correct categorization as sensitive or non-sensitive. Owners of sensitive information are responsible for its secure handling, transmission, processing, storage, and disposal on OLCF systems.

Principal investigators, users, or project delegates that use OLCF resources, or are responsible for overseeing projects that use OLCF resources, are strictly responsible for knowing whether their project generates any of these prohibited data types or information that falls under Export Control. For questions, contact help@nccs.gov.

The OLCF systems provide protections to maintain the confidentiality, integrity, and availability of user data. Measures include the availability of file permissions, archival systems with access control lists, and parity and CRC checks on data paths and files. It is the user’s responsibility to set access controls appropriately for the data. In the event of system failure or malicious actions, the OLCF makes no guarantee against loss of data or that a user’s data can be accessed, changed, or deleted by another individual. It is the user’s responsibility to insure the appropriate level of backup and integrity checks on critical data and programs.

Users are prohibited from taking unauthorized actions to intentionally modify or delete information or programs.

The OLCF reserves the right to remove any data at any time and/or transfer data to other users working on the same or similar project once a user account is deleted or a person no longer has a business association with the OLCF. After a sensitive project has ended or has been terminated, all data related to the project must be purged from all OLCF computing resources within 30 days.

All software used on OLCF computers must be appropriately acquired and used according to the appropriate software license agreement. Possession, use, or transmission of illegally obtained software is prohibited. Likewise, users shall not copy, store, or transfer copyrighted software, except as permitted by the owner of the copyright. Only export-controlled codes approved by the Export Control Office may be run by parties with sensitive data agreements.

Users must not intentionally introduce or use malicious software such as computer viruses, Trojan horses, or worms.

Users are not allowed to reconstruct information or software for which they are not authorized. This includes but is not limited to any reverse engineering of copyrighted software or firmware present on OLCF computing resources.

Users are accountable for their actions and may be held accountable to applicable administrative or legal sanctions.

Users are advised that there is no expectation of privacy of your activities on any system that is owned by, leased or operated by UT-Battelle on behalf of the U.S. Department of Energy (DOE). The Company retains the right to monitor all activities on these systems, to access any computer files or electronic mail messages, and to disclose all or part of information gained to authorized individuals or investigative agencies, all without prior notice to, or consent from, any user, sender, or addressee. This access to information or a system by an authorized individual or investigative agency is in effect during the period of your access to information on a DOE computer and for a period of three years thereafter.

OLCF personnel and users are required to address, safeguard against, and report misuse, abuse and criminal activities. Misuse of OLCF resources can lead to temporary or permanent disabling of accounts, loss of DOE allocations, and administrative or legal actions. Users who have not accessed a OLCF computing resource in at least 6 months will be disabled. They will need to reapply to regain access to their account. All users must reapply annually.

All users are required to use a one-time password for authentication. Tokens will be distributed to OLCF users. Users will be required to create a Personal Identification Number (PIN). This is used in conjunction with a generated token code as part of a two-factor authentication implementation.

Accounts on the OLCF machines are for the exclusive use of the individual user named in the account application. Users should not share accounts or tokens with anyone. If evidence is found that more than one person is using an account, that account will be disabled immediately. Users are not to attempt to receive unintended messages or access information by some unauthorized means, such as imitating another system, impersonating another user or other person,
misuse of legal user credentials (usernames, tokens, etc.), or by causing some system component to function incorrectly.

Users are prohibited from changing or circumventing access controls to allow themselves or others to perform actions outside their authorized privileges. Users must notify the OLCF immediately when they become aware that any of the accounts used to access OLCF have been compromised.

Users should inform the OLCF promptly of any changes in their contact information (E-mail, phone, affiliation, etc.) Updates should be sent to accounts@ccs.ornl.gov.

Applicants who appear on a restricted foreign country listing in section 15 CFR 740.7 License Exceptions for Computers are denied access based on US Foreign Policy. The countries cited are Cuba, Iran, North Korea, Sudan, and Syria. Additionally, no work may be performed on OLCF computers on behalf of foreign nationals from these countries.

Users may not deliberately interfere with other users accessing system resources.

2. Data Management Policy

(Back to Top)

Users are provided with several storage areas, each of which serve different purposes. These areas are intended for storage of data for a particular user and not for storage of project data.

There are (3) subcategories of User storage areas: User Home, User Work, and User Archive.

Home directories for each user are NFS-mounted on OLCF systems. This is online disk space intended for long-term, frequently used storage, and is backed up on a daily basis. This file system does not generally provide the input/output (I/O) performance required by most jobs. A User Home directory has a 5GB quota.

Individual User Work directories reside in the center-wide high-capacity Lustre file system on large, fast disk areas intended for global (parallel) access to temporary/scratch storage. User Work directories are provided across each system. Because of the scratch nature of the file system, it is not backed up and files are automatically purged on a regular basis. Files should not be retained in this file system and should be migrated to archival space as soon as the files are not actively being used.

If a file system associated with User Work storage runs out of free space or gets low on free space, then data on that file system will be subject to involuntary deletion without prior notice.

The High Performance Storage System (HPSS) is the archival storage system at the OLCF. Space on HPSS is intended for files that are not immediately needed.

Each user archive space belongs to a single user. An individual user has a 2TB space quota and a 2,000 file quota. User archive areas in HPSS are created in /home/[username].
User archive data stored on HPSS will only be retained up to three months past user account termination. There is no lifetime retention. Users are expected to migrate their data off HPSS before their account is deactivated.

Projects are provided with several storage areas for the data they need. Project directories provide members of a project with a common place to store code, data files, documentation, and other files related to their project. While this information could be stored in one or more user directories, storing in a project directory provides a common location to gather all files.

There are (3) subcategories of Project storage areas: Project Home, Project Work, and Project Archive.

Project home directories are NFS-mounted on OLCF systems. This is online disk space intended for long-term, frequently used storage and is backed up on a daily basis. This file system does not generally provide the input/output (I/O) performance required by most jobs. The standard project directory has a 50GB quota.

Project work directories reside in the center-wide high-capacity Lustre file system on large, fast disk areas intended for global (parallel) access. This is online disk space intended for long-term, frequently used storage, but this area is not backed up. An individual project directory has a 2TB quota. Project spaces on the center-wide high-capacity Lustre directories may be denied or revoked at any time.

The High Performance Storage System (HPSS) is the archival storage system at the OLCF. Space on HPSS is intended for files that are not immediately needed.

Project archive areas are shared between all users of the project. Project archive areas quotas have 100TB space quotas and 100,000 file number quotas. Project areas in HPSS are created in /proj/[projectid].

Larger quotas may be granted for a project on a case-by-case basis; the projects must include storage needs in their proposals along with strategies for data migration after the project has ended. Even though HPSS is a very large storage system, space is not unlimited. Users must not store files unrelated to their OLCF projects on HPSS. They must also periodically review their files and remove unneeded ones. Quotas will be enforced.

Projects should not duplicate executables or data sets in each user’s area, but instead should set privileges to share a master copy. Contact the User Assistance Center for help in setting privileges to facilitate sharing.

Project data stored on HPSS will only be retained up to three months past the end of a project allocation. There is no lifetime retention. Projects are expected to migrate their data off HPSS before project deactivation.

A large, fast disk area intended for parallel access to temporary storage in the form of scratch directories may be provided on a limited number of systems. This area is local to the specific system. This directory is, for example, intended to hold output generated by a user’s job. Because of the scratch nature of the file system, it is not backed up and files are automatically purged on a regular basis. Files should not be retained in this file system and should be migrated to archival storage as soon as the files are not actively being used.

If a file system runs out of or gets low on space, data on that file system will be subject to involuntary deletion without prior notice. Quotas may be instituted on a machine-by- machine basis if deemed necessary.

There are three purge/retention policy types: the Scratch Purge Policy, Project Purge Policy, and the User Purge Policy. Please note there is no lifetime retention for any data on OLCF machines. Projects and users are expected to migrate data off of OLCF systems when projects and users are finished with OLCF systems. When a project ends, active users in that project will have limited timeframes to migrate all data off of OLCF file systems.

Scratch purge policies are intended to maintain file systems so there is always a large amount of scratch space available for executing jobs. Files are automatically purged on a regular basis. Any file older than 14 days is subject to deletion. If a file system runs out of free space or gets low on free space, data on that file system will be subject to involuntary deletion without prior notice.

User purge policies are intended to reclaim space after a user account is disabled for any reason. When a user account is deactivated prior to the user migrating needed data off of OLCF storage areas, the user must request a temporary data access account extension, and then will have no more than 1 month to remove all user data from OLCF global and local storage systems, and no more than 3 months to remove all user data from OLCF archival storage systems.

In cases where project data is stored in a User Home area, a project representative must submit a request for data from the user home area within 1 month of the user account being disabled.

In cases where a deadline for an extension is missed, OLCF reserves the right to delete all files and directories designated as user data across all applicable storage spaces.

Project purge policies are intended to reclaim space after a project ends. When a project ends, project members will have no more than 1 month to remove all data from OLCF global and local storage systems, and no more than 3 months to remove all data from OLCF archival storage systems. If a project needs additional time to remove project data from OLCF systems, a project member must request an extension.

In cases where a project misses the deadline for an extension, OLCF reserves the right to delete all files and directories designated as project data across all applicable storage spaces.

Storage usage will be monitored continually. When time permits, offenders will be warned to clean up their space. Ignoring these warnings will result in loss of access privileges.

3. Cyber Security Policy

(Back to Top)

The Oak Ridge Leadership Computing Facility (OLCF) computing resources are provided to users for research purposes. All users must agree to abide by all security measures described in this document. Failure to comply with security procedures will result in termination of access to OLCF computing resources and possible legal actions.

The requirements outlined in this document apply to all individuals who have an OLCF account. It is your responsibility to ensure that all individuals have the proper need-to-know before allowing them access to the information on OLCF computing resources. This document will outline the main security concerns. Specific use policies are covered in the OLCF Computing Policy.

OLCF computing resources are for business use only. Installation or use of software for personal use is not allowed. Incidents of abuse will result in account termination.
Inappropriate uses include, but are not limited to:

• Sexually oriented information
• Downloading, copying, or distributing copyrighted materials without prior permission from the owner
• Downloading or storing large files or utilizing streaming media for personal use (e.g., music files, graphic files, internet radio, video streams, etc.)
• Advertising, soliciting, or selling

Access to systems is provided via Secure Shell version 2 (sshv2). You will need to ensure that your ssh client supports keyboard-interactive authentication. The method of setting up this authentication varies from client to client, so you may need to contact your local administrator for assistance. Most new implementations support this authentication type, and many ssh clients are available on the web. Login sessions will be automatically terminated after a period of inactivity.

When you apply for an account, you will be mailed an RSA SecurID token. You will also be sent a request to complete identity verification. When your account is approved, your RSA SecurID token will also be enabled. Please refer to Authenticating to OLCF Systems for more information setting your PIN and logging in; refer to OLCF System Hostnames for more information on host access specifics.

DO NOT share your PIN or RSA SecurID token with anyone. Sharing of accounts will result in termination. If your SecurID token is stolen or misplaced, contact the OLCF immediately and report the missing token. Upon termination of your account access, return the token to the OLCF in person or via mail.

The OLCF uses a standard file system structure to assist users with data organization on OLCF systems. Complete details about all file systems available to OLCF users can be found in the OLCF Data Management Policy.

Additional file systems and file protections may be employed for sensitive data. If you are a user on a project producing sensitive data, further instructions will be given by the OLCF. The following guidelines apply to sensitive data:

• Only store sensitive data in designated locations. Do not store sensitive data in your User Home directory.
• Never allow access to your sensitive data to anyone outside of your group.
• Transfer of sensitive data must be through the use encrypted methods (scp, sftp, etc).
• All sensitive data must be removed from all OLCF resources when your project has concluded.

The OLCF offers two dedicated data transfer nodes to users. The nodes have been tuned specifically for wide area data transfers, and also perform well on the local area. There are also several utilities that the OLCF recommends for data transfer. Please refer to the Data Transfer Nodes page for information about the DTNs and available utilities.

4. Titan Scheduling Policy

(Back to Top)

In a simple batch queue system, jobs run in a first-in, first-out (FIFO) order. This often does not make effective use of the system. A large job may be next in line to run. If the system is using a strict FIFO queue, many processors sit idle while the large job waits to run.

Backfilling would allow smaller, shorter jobs to use those otherwise idle resources, and with the proper algorithm, the start time of the large job would not be delayed. While this does make more effective use of the system, it indirectly encourages the submission of smaller jobs.

As a DOE Leadership Computing Facility, the OLCF has a mandate that a large portion of Titan’s usage come from large, leadership-class (aka capability) jobs. To ensure the OLCF complies with DOE directives, we strongly encourage users to run jobs on Titan that are as large as their code will warrant. To that end, the OLCF implements queue policies that enable large jobs to run in a timely fashion.

The basic priority-setting mechanism for jobs waiting in the queue is the time a job has been waiting relative to other jobs in the queue. However, several factors are applied by the batch system to modify the apparent time a job has been waiting. These factors include:

• The number of nodes requested by the job.
• The queue to which the job is submitted.
• The 8-week history of usage for the project associated with the job.
• The 8-week history of usage for the user associated with the job.

If your jobs require resources outside these queue policies, please complete the relevant request form on the Special Requests page. If you have any questions or comments on the queue policies below, please direct them to the User Assistance Center.

Jobs are aged according to the job’s requested processor count (older age equals higher queue priority). Each job’s requested processor count places it into a specific bin. Each bin has a different aging parameter, which all jobs in the bin receive.

FairShare, as its name suggests, tries to push each user and project towards their fair share of the system’s utilization: in this case, 5% of the system’s utilization per user and 10% of the system’s utilization per project.

To do this, the job scheduler adds (30) minutes priority aging per user and (1) hour of priority aging per project for every (1) percent the user or project is under its fair share value for the prior (8) weeks. Similarly, the job scheduler subtracts priority in the same way for users or projects that are over their fair share.

For instance, a user who has personally used 0.0% of the system’s utilization over the past (8) weeks who is on a project that has also used 0.0% of the system’s utilization will get a (12.5) hour bonus (5 * 30 min for the user + 10 * 1 hour for the project).

In contrast, a user who has personally used 0.0% of the system’s utilization on a project that has used 12.5% of the system’s utilization would get no bonus (5 * 30 min for the user – 2.5 * 1 hour for the project).

The batch queue system places a limit of (2) production jobs in the queued (i.e., eligible-to-run) state per user. If a user submits more than (2) jobs, the additional jobs will enter a held state. If (1) one of the user’s queued jobs were to begin execution, (1) of the held jobs would be moved into the queued state.

Additionally, a user may have only (2) jobs in bin 5 running at any time.

Jobs submitted to the debug queue will receive a (2)-day priority aging boost.

Production jobs are not allowed in the debug queue. Proper use of the debug queue would be, for example, submitting (1) job at a time as part of a software development, testing, or debugging cycle. Interactive parallel work is an ideal use for the debug queue.

Projects that overrun their allocation are still allowed to run on OLCF systems, although at a reduced priority. Like the adjustment for the number of processors requested above, this is an adjustment to the apparent submit time of the job. However, this adjustment has the effect of making jobs appear much younger than jobs submitted under projects that have not exceeded their allocation. In addition to the priority change, these jobs are also limited in the amount of wall time that can be used.

For example, consider that job1 is submitted at the same time as job2. The project associated with job1 is over its allocation, while the project for job2 is not. The batch system will consider job2 to have been waiting for a longer time than job1.

The adjustment to the apparent submit time depends upon the percentage that the project is over its allocation, as shown in the table below:

Effective July 19th 2012 and until further notice, leadership-class jobs (i.e. jobs requesting 60,000 or more cores) submitted against an INCITE project will not be subject to the usual over-allocation priority reduction penalties on Titan. The over-allocated project will be limited to (1) job running at any given time, but their submitted jobs will not be penalized with a priority reduction as in the table above.

Projects may request to reserve a set of processors for a period of time through the reservation request form, which can be found on the Special Requests page.

If the reservation is granted, the reserved processors will be blocked from general use for a given period of time. Only users that have been authorized to use the reservation can utilize those resources. Since no other users can access the reserved resources, it is crucial that groups given reservations take care to ensure the utilization on those resources remains high.

To prevent reserved resources from remaining idle for an extended period of time, reservations are monitored for inactivity. If activity falls below 50% of the reserved resources for more than (30) minutes, the reservation will be canceled and the system will be returned to normal scheduling. A new reservation must be requested if this occurs.

Since a reservation makes resources unavailable to the general user population, projects that are granted reservations will be charged (regardless of their actual utilization) a CPU-time equivalent to
(# of cores reserved) * (length of reservation in hours).

5. INCITE Allocation Under-utilization Policy

(Back to Top)

The OLCF has a pull-back policy for under-utilization of INCITE allocations. Under-utilized INCITE project allocations will have core-hours removed from their outstanding core-hour project balance at specific times during the INCITE calendar year. The following table summarizes the current under-utilization policy:

For example, a 1,000,000 core-hour INCITE project that has utilized only 50,000 core-hours (5% of the allocation) on May 1st would forfeit (0.30 * 950,000) = 285,000 core-hours from their remaining allocation.

6. Project Reporting Policy

(Back to Top)

Principal Investigators of current OLCF projects must submit a quarterly progress report. The quarterly reports are essential as the OLCF must diligently track the use of the center’s resources. In keeping with this, the OLCF (and DOE Leadership Computing Facilities in general) imposes the following penalties for late submission: