I/O Formats
Software
I/O continues to be one of the main bottlenecks for scientific applications. Here are two software packages that many application developers use to manage input/output of heterogeneous types of binary application data used on many different platforms. HDF5 and NETCDF are both implemented on top of MPI‐IO and have gained popularity as alternatives to basic POSIX API. HDF5 is a machine-independent and self-documenting file format. Each HDF5 file "looks" like a directory tree, with subdirectories, and leaf nodes that contain the actual data. This means that data can be found in a file by referring to its name, rather than its location in the file. NetCDF is a file format and support library developed at the National Center for Atmospheric Research (NCAR). Like HDF5, NetCDF is self-describing and portable across platforms.
Advice and Overview
Introduction to Scientific I/O
Optimizing I/O Performance on the Lustre File System
