Job Arrays

Job Arrays

Job arrays have many advantages, including reduced load on UGE, faster job submission, and easier job management. If you find yourself submitting thousands of jobs at a time that are largely identical, you should use job arrays. For example, if you have many different data sets, but want to run the same program on all of the data sets, you can either use Perl to generate one script for each data set, or you can use a job array with a single script.  Using a UGE job array is cleaner, easier and more efficient!

Here is a description of how UGE job arrays work:

Job arrays can be submitted from the command line with the -t option to qsub, e.g.,

genepool01% qsub -t 1-20:1 myjob.csh

This would submit 20 identical jobs, each running instances of myjob.csh, with job indices from 1-20.

To use different input file for each job you need to reference $SGE_TASK_ID in myjob.csh.  For this example, if it is the case that the inputs are named "inputs.0", ..., "inputs.20" and you want to have corresponding outputfiles, then you can do the following in myjob.csh:

./myprogram inputs.$SGE_TASK_ID results.$SGE_TASK_ID

The $SGE_TASK_ID variable corresponds to the task currently being run on the compute node and for this example will number between 1 and 20, and the program will execute on each input file.

This is computationally equivalent to 20 individual queue submissions, however,

• only ONE qsub command is issued (and only ONE qdel command is required to delete all these jobs)
• only ONE entry appears in the qstat output
• the load on the UGE submit node (cluster login node) is significantly lighter than if it has to process 1000 individial jobs

Other options for job arrays

If the ":1" was replaced with ":n" it would do every "n"th job.  The job index does not need to start at one, you could instead do something like

genepool01% qsub -t 25-78:5 myjob.csh

which would start the tasks at 25, increment by 5 and end at 75.  Why does it end at 75 and not 78 as is entered?  UGE automatically changes the upperbound so that it is consistent with the lower bound plus an integer-multiple of the increment that is less than 78, i.e. 25+5*10 = 75.

For files with more complicated naming conventions, you can do the following in a perl script:

open(LIST,"jobFiles.list") or die "Error: $!\n";
my @labels;
while(<LIST>){
my $name = $_;
push @labels,$name;
}
my $whichFile = $ENV{SGE_TASK_ID} - 1;
my $filename = $labels[$whichFile];

Here "jobFiles.list" is a file containing a list of input files and $filename is the input file for the job. This snippet of code is run during execution (not at submission). The qsub -t option in your job submission should reflect the actual number of files in your list.