HSI Tape Ordering Script
Below is an example of a perl script that shows how one might create a list of hsi retrievals from HPSS that are ordered by tape and tape position for the most efficient way to get the files. Note that the source directory in HPSS and the destination directory on disk are hardwired along with a selection on what files to list. The script is based on using "hsi ls" with the "-P" option. Also shown is what the script prints out as it is run and a snippet of the output file it produces containing the transfer commands.
#! /usr/bin/perl
use strict;
### This script lists an hpss directory, sorts the files according to
### tape and position on the tape and creates a file containing
### the transfer commands. Written by ELH, 10/29/10.
### Set up transfer source and destination directories and select files for transfer list
my $hpssDir = "/nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048";
my $files = "st_physics*.StrangeMuDst.root";
my $diskDir = $hpssDir;
$diskDir =~ s/\/nersc\/l\/luyan/\/eliza3\/starprod\/reco/;
###
my $output = substr $hpssDir.".txt", 1;
$output =~ s/\//_/g;
(-e $output) && `rm -f $output`;
open(XFER, ">$output");
print "Source HPSS directory = $hpssDir\n";
print "Destination disk directory = $diskDir\n";
### Get a list of desired files with tape info from hsi:
print "Doing hsi listing...\n";
my $hsistart = time;
my @hsi = `hsi -q ls -P $hpssDir/$files 2>&1`;
my $hsidone = time;
my $hsitime = $hsidone - $hsistart;
### Keep track of what tapes the files are on
### and create 2d array of hsi info
print "Done with hsi in $hsitime seconds, now processing output...\n";
my %seenTapes = ();
my $fileCount = 0;
my $procstart = time;
my @hsiarray = ();
foreach my $line (@hsi){
my @info = split /\s+/,$line;
# $info[1] = filename, $info[4] = position on tape, $info[5] = tape name
$seenTapes{$info[5]} = 1;
push @hsiarray, [split /\s+/, $line];
$fileCount++;
}
print "Found $fileCount files, creating transfer list\n";
### @inorder is a 2d array sorted by tape position
my @inorder = sort{$a->[4] <=> $b->[4]}(@hsiarray);
### Create transfer list
my @tapes = keys %seenTapes;
foreach my $tape (@tapes){
print "Working on tape $tape...";
my $posCount = 0;
my $i = 0;
for($i=0; $i<$fileCount; $i++){
if ($tape eq $inorder[$i][5]){
$posCount++;
$inorder[$i][1] =~ /$hpssDir\//;
my $filename = $';
print XFER "hsi cget $diskDir/$filename : $inorder[$i][1]\n";
}
}
print " $posCount files\n";
}
close XFER;
my $procdone = time;
my $proctime = $procdone - $procstart;
print "Done with processing in $proctime seconds\n";
print "File transfer commands listed in $output\n";
An example of the printout from the script when it is run:
pdsf3 85% tapeorder.pl
Source HPSS directory = /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048
Destination disk directory = /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048
Doing hsi listing...
Done with hsi in 67 seconds, now processing output...
Found 971 files, creating transfer list
Working on tape EA111900... 9 files
Working on tape EA114400... 2 files
Working on tape EA476700... 45 files
Working on tape EA118500... 19 files
Working on tape EB289900... 27 files
Working on tape EA113500... 6 files
Working on tape EA809700... 221 files
Working on tape EA737100... 46 files
Working on tape EA010800... 126 files
Working on tape EB287300... 14 files
Working on tape EB029200... 64 files
Working on tape EA785000... 31 files
Working on tape EB039700... 78 files
Working on tape EB021900... 143 files
Working on tape EB283500... 19 files
Working on tape EA677900... 27 files
Working on tape EA811300... 64 files
Working on tape EA114300... 13 files
Working on tape EB278600... 6 files
Working on tape EA777700... 11 files
Done with processing in 0 seconds
File transfer commands listed in nersc_l_luyan_StrangeMuDst_auau200_minibias_proLowMidHigh_P05ic_FullField_P05ic_2004_048.txt
pdsf3 86%
A few lines from the files that lists the commands to get the files out of HPSS:
pdsf3 95% head nersc_l_luyan_StrangeMuDst_auau200_minibias_proLowMidHigh_P05ic_FullField_P05ic_2004_048.txt
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_2080002.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_2080002.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3050001.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3050001.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3050002.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3050002.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3060001.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3060001.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3070001.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048030_raw_3070001.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1040002.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1040002.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1040003.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1040003.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1050001.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1050001.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1050002.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_adc_5048033_raw_1050002.StrangeMuDst.root
hsi cget /eliza3/starprod/reco/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_5048024_raw_2010006.StrangeMuDst.root : /nersc/l/luyan/StrangeMuDst/auau200/minibias/proLowMidHigh_P05ic/FullField/P05ic/2004/048/st_physics_5048024_raw_2010006.StrangeMuDst.root
pdsf3 96%
