Stagesub
Category: Tools-Workflow
Description
The stagesub command uses simple directives such as "#Stagein" and "#Stageout" within a standard PBS script to coordinate data staging and stageout jobs alongside compute jobs. The data jobs are submitted to the data transfer queue and compute jobs to the regular batch queue. This enables timely staging and offload-ing of job input and output data.
A PBS script that is intrumented with #Stagein and #Stageout directives is parsed into three jobs namely, stagein, stageout and the original compute job. The stagein and stageout jobs are submitted to a data transfer queue and scheduled as data jobs. The compute job is executed as usual in the batch queue. The three jobs, submitted to different queues, are setup in a way that the compute job starts only after the staging in of the input data completes and the stageout job commences only after the termination of the compute job. The PBS script can be instrumented as follows:
-
#Stagein [-d HH:MM:SS] <source_uri> <dest_uri>
-
#Stageout [-d HH:MM:SS] <source_uri> <dest_uri>
The "-d" option is optional and is used to specify the walltime for the stagin and stageout jobs. If duration is not specified, a default walltime of thirty minutes is used. Multiple #Stagein and #Stageout directives can each have their own durations. The stagein data job is a sequence of multiple stagein operations, whose wall- time is the sum of the durations of all the #Stagein directives.
Several #Stagin and #Stageout directives can be issued to code in multiple data transfer commands. Supported protocols for source and destination URIs include: NFS cp (file://), scp (scp://), hsi-HPSS (hpss:// for individual files and hpssdir:// for an entire directory recursively). The destination protocol for a stagein needs to be file:// and the source protocol for a stageout needs to be file://, indicating that the file is moved in and out of the scratch file system.
More information may be found through man stagesub
.
Use
The stagesub utility and man page can be added to a user's environment through the stagesub module.
Support
This package has the following support level : Supported
Available Versions
Version |
Available Builds |
pgi |
pathscale |
gnu |
Other |
1.0 |
|
|
|
|