pftp
On the NERSC computational systems both the ftp and pftp commands are aliases to a special, high-performance FTP client that will transfer data to and from the NERSC HPSS systems in parallel for high bandwidth transfers. This pftp client also supports an extended command set not typically available on standard ftp clients. The Pftp client software is not available for download.
Note that the pftp client supports the standard internet File Transfer Protocol and can be used to transfer files to and from remote non-HPSS systems. In such a situation, the transfers will not be parallel and only the command supported by that remote FTP server will work (the HPSS specific command will not work).
Access Example
To access the archive storage system using the parallel client from a NERSC computational platform you must specify the system name on the command line:
% pftp archive.nersc.gov
or
% ftp archive.nersc.gov
Likewise, to access the hpss backup system (not all users have accounts on this system) type:
% pftp hpss.nersc.gov
or
% ftp hpss.nersc.gov
As with the other HPSS clients (hsi and htar) pftp will look for your HPSS token in the .netrc file of your local home directory for authentication to the HPSS server. If you have not generated an HPSS token pftp will prompt you for your NIM password, generate a token and store it in your .netrc file for you. See the HPSS Passwords page for details.
Extended Command Set
The pftp client has a rich extended command set for use when connected to the NERSC HPSS system. To see a list of all the commands, type "help" and to get a short description of a specific command, type "help <command>":
%pftp archive
...
ftp> help
Commands may be abbreviated. Commands are:
! macdef quit
$ mchgrp quote
account mchmod recv
append mdelete reget
ascii mdir rstatus
autoparallel mget rhelp
bell mkdir rename
binary mlfget reset
bye mlfput restart
case mls rmdir
cd mode runique
cdup modtime safe
chgrp mput send
chmod multinode setpwidth
clear newer setpblocksize
close nmap getremtuningparms
cr nlist gettuningparms
delete ntrans setsockbufsize
debug open setxferbufsize
dir p64bit site
disconnect passive size
form pdata status
get pdatapush struct
getprot pmover system
glob private sunique
hash prompt tenex
help protect term
features proxy time_read
hfeatures sendport time_write
idle mpget tput
image mpput trace
lcd pappend type
lfappend pget user
lfget pput umask
lfput put verbose
ls pwd ?
ftp> help chmod
chmod change file permissions of remote file
A subset of commonly used extended commands are listed in the fillowing table:
| Desired Action: | Command to Execute: |
|---|---|
| change group id of a file |
chgrp groupname hpssfile |
| change permissions of a file |
chmod mode hpssfile |
| store a local file into HPSS in parallel |
pput localfile [hpssfile] |
| get an HPSS file in parallel |
pget hpssfile [localfile] |
| retrieve multiple files from HPSS in parallel |
mpget hpssfile1 hpssfile2 ... |
| store multiple local files into HPSS in parallel |
mpput localfile1 localfile2 ... |
| get file from HPSS if it is newer thatn local file | newer hpssfile [localfile] |
See the pftp man page for details.
Sleepers
The pftp clients on the NERSC computational systems will detect if the HPSS system you are trying to connect to is temporairly unavailable. If so, the session will be suspended until the system is available again. When running in a batch script, you may not want the command to suspend. You can modify your script to detect this condition by following the instructions on the sleepers page.
Usage Examples
The following example shows how you can use pftp in a batch script by placing the commands in a file and having pftp read the command from standard input:
% cat send_files
mkdir testdir
cd testdir
pput file1.tar
pput file2 myfile2
ls -l
quit
% pftp archive < send_files
Using ...
ftp> mkdir testdir
ftp> cd testdir
ftp> pput file1.tar
All Mover transfer information received, beginning data transmission.
ftp> pput file2 myfile2
All Mover transfer information received, beginning data transmission.
ftp> ls -l
-rw-r----- 1 joeuser ccc 93104 Apr 6 12:17 file1.tar
-rw-r----- 1 joeuser ccc 59752 Apr 6 12:17 myfile2
ftp> quit
%
Another way to do this is by using HEREIS documents:
% cat putfiles
#!/bin/bash
pftp archive <<EOF
mkdir testdir
cd testdir
pput file1.tar
pput file2 myfile2
ls -l
quit
EOF
% ./putfiles
Using ...
ftp> mkdir testdir
ftp> cd testdir
ftp> pput file1.tar
All Mover transfer information received, beginning data transmission.
ftp> pput file2 myfile2
All Mover transfer information received, beginning data transmission.
ftp> ls -l
-rw-r----- 1 joeuser ccc 93104 Apr 6 12:22 file1.tar
-rw-r----- 1 joeuser ccc 59752 Apr 6 12:22 myfile2
ftp> quit
%
