Data Storage

Disk Storage: The supercomputer’s data systems consist of 640 I/O nodes that connect to 16 storage area networks (SANs) that control 7,680 disk drives with a total capacity of 7.6 petabytes of raw storage and a maximum aggregate transfer speed of 88 gigabytes per second. The ALCF uses two parallel file systems—PVFS and GPFS—to access the storage.

Tape Storage: The tape drives have built-in hardware compression with compression ratios typically between 1.25:1 and 2:1, depending on the data, giving an effective capacity of 16-24 petabytes. An HPSS automated tape storage system provides archival storage with 16,000 tapes in two 10,000-slot libraries.

Networking

The Blue Gene/P uses five different networks for different communication operations. The 3D torus network is used for general-purpose, point-to-point message passing as well as for collective operations using irregular communication or large message sizes. Each node has six nearest neighbors. Each link provides a bandwidth of 425 MB/s per direction, for a total bidirectional bandwidth of 5.1 GB/s. Though each node has six bidirectional links on each node, there is only one shared DMA engine. The 3D torus network is also usable as a 3D mesh. The supercomputer connects to other research institutions using a total of 30 Gb/s of public network connectivity. Scientists can transfer datasets to and from other institutions over fast research networks such as the Energy Science Network (ESNet) and the Metropolitan Research and Education Network (MREN).