Transferring Data from Batch Jobs

Examples 

Once you are set up for automatic authentication (see HPSS Passwords) you can access HPSS within batch scripts.  You can add the following lines at the end of your batch script.

HSI will accept one-line commands on the HSI command line, e.g.,

hsi put filename

HSI, ftp and pftp read from the standard input (stdin) and a list of commands can be placed in a text file (script) and redirected into the given utility, e.g.,

htp < file_with_ftp_commands

"Here" Documents

Another method uses what are called "Here Documents", in which the commands are embedded in the batch script rather than in a separate file external to the main script.  The start of a "here-doc" block in a script is signaled by the presence of double angle brackets:  << followed by an identifying tag.  Lines up to the line containing the tag are treated as if they had been typed at the command prompt.

Here is a simple script which performs an ftp file transfer:

pftp -v -i archive <<_EOS
cd my_HPSS_directory
mget data*
quit
_EOS

This example will execute the FTP command between the "_EOS" strings.

Transfer (xfer) Queues

Many of the computational systems have an xfer queue set up so users can submit data transfer requests to the batch system.  Users always have the option of adding an htar or hsi command to the end of a batch script to transfer data.  However, if a batch job requests thousands of nodes and transfer to HPSS is expected to take a significant amount of time, a user will be charged for the time on all of the compute nodes requested in a batch script.  The alternative is to submit a batch job to the xfer queue on a single core.