GUPFS - Global Unified Parallel File System

The Global Unified Parallel File System (GUPFS) project at NERSC aims to deploy a center-wide file system and consolidated disk storage to be used by all NERSC production and support systems. The deployed file system will be used to provide a unified name space for user data. It is expected that several instances of the file system, tuned for different workload characteristics and possibly utilizing different storage devices, will be used to create the unified name space. The deployed file system will be integrated with the NERSC HPSS archival system and will provide HSM services using HPSS. A major use of the file system will be in support of parallel scientific applications performing high volume concurrent and simultaneous I/O.

The deployed GUPFS file system is expected to have a long life, of 10 to 15 years or more. It is expected that during this time the file system will change and evolve, as will the systems in the center that are utilizing it. It is also expected that the user data will have long term persistence in the file system, ranging from months and years up to the deployed life of the file system, at the discretion of the users.

NERSC traditionally procures a major, new computational system every three years, and operates it for DOE research usage for five years. These systems are procured through open competitions. Consequently, NERSC operates in, and must continue to operate in, a heterogeneous environment with systems from multiple vendors, multiple platforms, different system architectures, and multiple operating systems. The deployed file system must operate in the same heterogeneous client environment throughout its life time.

Over its life-time the deployed file system is expected to grow in performance, capacity, and capability. During the first five years, the file system is expected to supply 10s to 100s of GB/s sustained I/O bandwidth, capacities of 100s of TB to PB, support 100s to 10,000s direct client systems, and millions to trillions of files.