Transferring Data

Advice and Overview

However, when file transfer is necessary, we provide several ways for transferring data both inside and outside NERSC. To transfer files from/to NERSC, we suggest you to use the dedicated Data Transfer Nodes, which is optimized for bandwidth and have access for most of the NERSC file systems.

File transfer bandwidth is also optimized for transferring file between ORNL and NERSC.

NERSC File Systems

Please refer to the table showing the NERSC file systems  on each NERSC machine.  Note that the "Local Non-shared Scratch" are different for Hopper and Franklin, while the Global File Systems (Global Home, Global Scratch and Project Directory) are shared file systems. I.e. the Global Home you see on Hopper is identical to the Global Home you see on a Data Transfer Node.

Having the file systems available on multiple platforms allows the user to minimize data movement. E.g. by putting data on Global Scratch, a user can generate the data with Hopper and visualize it on Euclid without moving the data.

Data Transfer Nodes

The Data Transfer Nodes (DTN) are servers dedicated for data transfer. DTNs have access to most of the NERSC File Systems, and are tuned to transfer data efficiently. The Data Transfer Nodes are also tuned for transferring large data files between NERSC and ORNL or ANL.

External Data Transfers

There are a number of ways to transfer data to and from NERSC.

• SCP/SFTP: for smaller files (<1GB).
• Globus Online: for large files, with extra features for auto-tuning and auto-fault recovery without a client install
• BaBar Copy (bbcp): for large files
• GridFTP: for large files
• HSI: can be an efficient way to transfer files already in the HPSS system

Transferring Data Within NERSC

• Do you need to transfer at all? If your data is on NERSC Global File Systems it's available at high performance center-wide.  No data transfer is necessary if files are in /project, /global  homes or /global scratch because these file systems are mounted on almost all NERSC systems.
• Use the the unix command "cp" to copy files within the same computation system.
• To transfer files between computational systems (e.g. Franklin local scratch to Euclid $GSCRATCH), we suggest you to use SCP/SFTP to transfer smaller files (<10GB), and BaBar Copy (bbcp) or GridFTP for bigger files.
• HPSS can also be used to transfer files within NERSC.  For example users can backup data to HPSS from Franklin $SCRATCH and retrieve necessary files from HPSS on Euclid's $GSCRATCH. For details about HPSS data transfer, see Storing and Retrieving HPSS Data.