IO optimization for Lustre

Home	About	News & Publications	HPC Users	Projects
Systems	Services	Analytics	Status & Statistics	Help

Status & Stats

NERSC MOTD
Announcements
Known Problems
Current Queue Look
Completed Jobs List
Job Stats

Franklin IO Performance Tips

Lustre File Striping for Parallel IO

Franklin uses the Lustre file system. For many applications a technique called file striping will increase IO performance. File striping will primarily improve performance for codes doing parallel IO on either a single node or multiple nodes using multiple nodes simultaneously to access one or more large files such as with MPI-IO, parallel HDF5 or parallel NetCDF.

The default striping on Franklin is set to 4 on the /scratch Lustre file system. This is optimized for the greatest number of codes running at NERSC. Users who feel that an application would benefit from a different striping pattern should contact consult@nersc.gov.

What is file striping?

The Lustre file system is made up of an underlying set of file systems called Object Storage Targets (OST's), which are essentially a set of parallel IO servers. A file is said to be striped when read and write operations access multiple OST's concurrently. File striping is a way to increase IO performance since writing or reading from multiple OST's simultaneously increases the available IO bandwidth.

Striping will likely have little impact for the following codes:

Serial IO where a single processor or node performs all of the IO for an application.
Multiple nodes perform IO, access files at different times.
Multiple nodes perform IO simultaneously to different files that are small (each < 100 MB).

Lustre allows users to set file striping at the file or directory level. As mentioned above, striping will not improve IO performance for all files. For example, in a parallel application, if each processor writes its own file then file striping will not provide any benefit. Each file will already be placed in its own OST and the application will be using OST's concurrently. File striping, in this case, could lead to a performance decrease due to contention between the processors as they try to write (or read) pieces of their files spread across multiple OST's.

For MPI applications with parallel IO, multiple processors accessing multiple OST's can provide large IO bandwidths. Using all the available OST's on Franklin will provide maximum performance.

There are a few disadvantages to striping. Interactive commands such as ls -l will be slower for striped files. Additionally, striped files are more likely to suffer from data loss from a hardware failure since the the file is spread across multiple OST's.

Striping commands

To set striping for a file or directory use the command


% lfs setstripe

Each file and directory can have a separate striping pattern and a directory's striping setting can be overwritten for a particular file by issuing the lfs setstripe command for individual files within that directory. However, striping settings for a file must be set before it is created. If the settings for an existing file are changed, it will only get the new striping setting if the file is recreated. If the settings for an existing directory are changed, the files need to be copied elsewhere and then copied back to the directory in order to inherit the new settings. The lfs setstripe syntax is:


% lfs setstripe filename  [stripe-size]  [OST-offset]  [stripe-count]

For parallel IO applications or for applications where 1 node is writing out a very large file start with a stripe-size of 0 (the default) OST-offset to -1 (which means choose the starting OST in a round robin fashion ) and a stripe-count of -1 (which means the file should be spread across all available OST's). For example, the command below sets the stripe count for myfile to be all available OST's.


% lfs setstripe myfile  0 -1 -1

For applications where each processor writes its own file, file striping is not recommended. In this case set the stripe-count to 1 and each file will be placed on its own OST. Example:


% lfs setstripe myfile  0 -1 1

An applications best performance may likely fall between these two extreme examples.
The lfs find command will give the striping pattern for a file or directory.


% lfs find -v directory/file

If the command returns no stripe, then that means the directory/file is not set to stripe, or in other words the stripe width is 1.


% lfs find -v testdirectory

OBDS:
0: ost1_UUID ACTIVE
1: ost2_UUID ACTIVE
2: ost3_UUID ACTIVE
3: ost4_UUID ACTIVE
4: ost5_UUID ACTIVE
5: ost6_UUID ACTIVE
6: ost7_UUID ACTIVE
7: ost8_UUID ACTIVE
8: ost9_UUID ACTIVE
9: ost10_UUID ACTIVE
10: ost11_UUID ACTIVE
11: ost12_UUID ACTIVE
12: ost13_UUID ACTIVE
13: ost14_UUID ACTIVE
14: ost15_UUID ACTIVE
15: ost16_UUID ACTIVE
testdirectory/
default stripe_count: 1 stripe_size: 0 stripe_offset: -1

This shows a stripe count of 1 (no striping), the stripe size is set to 0 (meaning use the default), and the stripe offset is set to -1 which means to round robin the files across the OST's. Always use -1 for stripe_offset. See the man page for lfs for more info.

	National Energy Research Scientific Computing Center
	A DOE Office of Science User Facility at Lawrence Berkeley National Laboratory	Site Map \| Help \| Search Login

HPC Users

Systems

Franklin

Status & Stats

Franklin IO Performance Tips

Lustre File Striping for Parallel IO