NERSC File Systems
Overview
NERSC file systems can be divided into two categories: local and global. Local file systems are only accessible on a single platform, providing best performance; global file systems are accessible on multiple platforms, simplifying data sharing between platforms.
File systems are configured for different purposes. On each machine you have access to at least three different file system
- Home: Permanent, relatively small storage for data like source code, shell scripts, etc. that you want to keep. This file system is not tuned for high performance from parallel jobs. Referenced by the environment variable $HOME.
- Scratch: Large, high-performance file system. Place your large data files in this file system for capacity and capability computing. Data is purged as described below, so you must save important files elsewhere (like HPSS). Referenced by the environmental variable $SCRATCH.
- Project: Large, permanent, medium-performance file system. "Project directories" are available upon request are intended for sharing data within a group of researchers.
The following table shows the availability of the various file systems on each of the primary NERSC platforms.
| File System | Hopper | Carver | Euclid | Genepool | Data Transfer Nodes | PDSF |
|---|---|---|---|---|---|---|
Global homes |
Y |
Y |
Y |
Y |
Y |
|
Global scratch |
Y |
Y |
Y |
Y |
Y |
|
Global project |
Y |
Y |
Y |
Y |
Y |
Y |
Global projectb |
Y |
Y |
Y |
Y |
||
Local scratch |
Y |
Y |
Summary of File System Policies
| File System | Path | Type | Default Quota | Backups | Purge Policy |
|---|---|---|---|---|---|
| Global homes | $HOME | GPFS | 40GB 1,000,000 Inodes |
Yes | Not purged |
| Global scratch | $GSCRATCH | GPFS | 20TB 2,000,000 Inodes |
No | Files not accessed for 12 weeks are deleted |
| Global project | /project/projectdirs/projectname | GPFS | 4TB 4,000,000 Inodes |
Yes | Not purged |
| Hopper local scratch | $SCRATCH and $SCRATCH2 | Lustre | 5TB 5,000,000 Inodes (Combined) |
No | Files not accessed for 12 weeks are deleted. |
Summary of File System Capacities
| File system | Capacity |
| Global homes | 246 TB |
| Global scratch | 1092 TB |
| Global project | 3965 TB |
| Global projectb | 2620 TB |
| Hopper local scratch1 | 1117 TB |
| Hopper local scratch2 | 1106 TB |
Finding your disk usage
NERSC provides the myquota command which allows you to find your current disk space and inode usage for all file systems.
bash-3.2$ myquota
==== Space (GB) ====
FileSystem Used InDoubt Quota
---------- ---- ------- -----
HOME 10 0 40
GSCRATCH 6 3 20480
==== Inodes ====
FileSystem Used InDoubt Quota
---------- ---- ------- -----
HOME 89468 39 1000000
GSCRATCH 20732 757 2000000
The prjquota command gives you the usage for a project directory.
bash-3.2$ prjquota mpccc
------ Space (GB) ------- ----------- Inode -----------
Project Usage Quota InDoubt Usage Quota InDoubt
-------- ------- ------- ------- ------- ------- -------
mpccc 5818 10240 11 3755653 4000000 1462
Purge Policy
In order to provide equitable access to scarce file system resources, NERSC "purges" certain scratch file systems on a regular basis, as indicated in the above table. Scratch file systems are intended to provide temporary storage on high-performace hardware. Data that is not in active use should be placed in HPSS.
Users are not allowed to run commands (e.g., "touch" scripts) whose sole purpose is to update the time-of-last-access on purgeable files.
Local Scratch on Hopper
Hopper uses Lustre for its local scratch file systems. These file systems are not available on any other computer systems. For more information, please see the File Storage Page for Hopper.
NERSC monitors I/O rates and I/O performance on the local Lustre file systems. Users can see live and historical I/O rates.
Global Home Filesystem
Global home directories (or "global homes") provide a convenient means for a user to have access to source files, input files, configuration files, etc., regardless of the platform the user is logged in to. Wherever possible, you should refer to your home directory using the environment variable $HOME. Read More »
Global Scratch File System
The global scratch file system (or "global scratch") can be accessed by multiple NERSC systems. Its default quota is much larger than the Global Home quota, so it can be used to temporarily store large amounts of data. This file system is periodically purged. Read More »
Project Directory
The Project Directory is a global file system available to all NERSC computational systems. It allows groups of NERSC users to store and share data. The Project Directory is available by request. Read More »
Disk Quota Increase Request Form
Use this form to request quota increases. You need to log in using your NIM password. Read More »
Project Directory Request Form
Use this form to request project directory space in the NERSC Global File system (NGF). Here are some guidelines for completing this form: The name, login, e-mail address, and phone number should be those of the requestor filling out this form. The sponsoring repository will typically be that of the requestor, although this is not a requirement. Participating repositories are those that are collaborating with the sponsoring repository. This field may be left blank for a single-repository… Read More »
