NERSC Global Filesystem

Overview

The NERSC Global Filesystem (NGF) is a large, shared filesystem that can be accessed from any of the major compute platforms. This facilitates file sharing between platforms, as well as file sharing among NERSC users working on a common project. NGF is based on IBM's General Parallel File System (GPFS). It currently contains approximately 150 TB of user-accessible storage, and 50 million inodes.

NGF provides project directory space for NERSC computational systems. Access control and quota enforcement for project directories are based on Unix file groups. In most situations, the name of the project directory is the same as the associated file group.

Quotas and Performance

Default project directory quotas are 1 TB and 250,000 inodes. If your directory needs more than that, fill out the Disk Quota Change Request Form.

To check your current usage and quota in a project directory, use the myquota command, specifying the file group name with the -g option:

% myquota -G -g bigsci
Displaying quota usage for group bigsci:
            ---------- Block (GB) ----------  ------------ Inode -------------
FileSystem   Usage    Quota   InDoubt  Grace   Usage    Quota   InDoubt  Grace
----------  -------  -------  -------  -----  -------  -------  -------  -----
scratch           0      250        0      -        1    50000       41      -
project        1187     2560        3      -   181071   250000     2209      -

In the above example, the project directory (file group) "bigsci" has used about 1 TB of its 2 TB block quota, and about 181000 inodes out of its 250000 inode quota. The -G option requests the output be displayed in gigabytes; other options include -M for megabytes, and -K for kilobytes.

The system has a sustainable bandwidth of 1 GB/sec bandwidth for streaming I/O, although actual performance for user applications will depend on a variety of factors. Because NGF is a distributed network filesystem, performance will be slightly less than that of filesystems that are local to NERSC compute platforms. This should only be an issue for applications whose performance is I/O bound.

Policies

There must be a project directory administrator associated with each project directory. This user must have the NIM role of PI, PI Proxy, or Project Manager.

Unlike the $SCRATCH filesystems on the compute platforms, files in NGF are not subject to purging. However, similar to the $HOME filesystems, they are also not backed up (except for disaster recovery of the entire filesystem).

It is the responsibility of the users to back up their files to HPSS or some other archival resource.

Requesting Space

NERSC allocates project space on NGF to groups of users that need to share files among themselves and/or between machines. These users can all be members of a single repository, or collaborating members of different repositories. To request a project directory on NGF, please use the Project Directory Request Form.

Usage

Project directories are created in /project/projectdirs. The name of the project directory will usually be used as the associated Unix file group. This name will sometimes be the same as a NERSC repository, and all active users of that repository (which is already a Unix file group) will thereby have access to the project directory.

However, there are cases where a repository name is not suitable for a project directory. For example, some large projects might want a project directory to be accessible by members of multiple repositories. Also, some long-term projects outlive the specific repositories that constitute them. In these cases, a project directory administrator may request the creation of a new project name. This will result in the creation of a new Unix file group consisting soley of the project directory administrator, followed by the creation of the project directory itself. The project directory administrator must then use NIM to add users to the newly-created file group (this is a very simple operation). Only these users will be able to access the project directory.