Data & Networking
We recognize the importance of being able to manage your files and data. Scientific datasets are growing very rapidly and there is an increasing need for large scale data storage and high performance data transfers. As science becomes more distributed organizing and sharing data becomes very important.
This section covers several topics on how to successfully manage your data at NERSC.
Connecting to NERSC
All NERSC computers (except HPSS) are reached by use of programs that implement the Secure Shell (SSH) communication and encryption protocol, version 2, or by Grid tools that use trusted certificates. Read More »
NERSC File Systems
This page compares the various file systems at NERSC in terms of availability per machine, purging, quota limits, and other key characteristics. Read More »
Transferring Data
Learn about the best way and most efficient tools to move data in, out and around the center. Read More »
HPSS Data Archive
HPSS, the High Performance Storage System, is the NERSC system you should use to back up your files to prevent data loss from accidental deletion and file purging. Read More »
Optimizing I/O performance on the Lustre file system
The Lustre file system is mounted locally on Hopper. This page describes how get the best performance out of the Lustre file system. Read More »
I/O Formats
I/O continues to be one of the main bottlenecks for scientific applications. This page describes the HDF5 and NetCDF software. Read More »
Sharing Data
Information on how to share data across NERSC systems, with other users within NERSC, or between NERSC and systems elsewhere. Read More »
Network Performance
In this ESnet Network Performance Knowledge Base you will find information on how to make your wide area data transfers go faster. The goal of this site is to help you maximize your wide-area network bulk data transfer performance by tuning the TCP settings for your end hosts and by using file transfer tools that are designed to maximize network throughput. Read More »
