File Transfers using bbcp

There is a faster alternative to scp available on Jacquard, Davinci, and Franklin. bbcp. The developer provides information for it at bbcp. You need to download and install the command on your local machine to use it to transfer files to and from NERSC.

The basic syntax is quite similar to that of scp:

% bbcp -option(s) sourcespec targetspec 

There are three machines involved in a bbcp file transfer: the agent - the machine on which the bbcp command is run, the source - the machine on which the file(s) to be transferred currently reside, and the target - the machine to which the file(s) will be transferred. The agent is typically the same machine as either the source or the target, but it does not have to be.

NOTE: It is currently necessary to add this option to the bbcp command if either Jacquard or Davinci is the source system:

-S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp"

-T "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp"

Sometimes "-z" option (use reverse connection protocol, i.e., sink to source) could be used to resolve transfer problem caused by firewall issues.

Peformance of bbcp

We have done several tests of the performance of bbcp between Jacquard and Davinci, involving one 11.5 GByte file. scp always gives pretty much the same level of performance for both puts and gets on any given agent, regardless of which system is the target and which the source, about 20 MBytes/sec on Jacquard and 23 Mbytes/sec on Davinci.

The performance of bbcp with no optimization arguments depends on the source and target systems rather than the agent. Transfers with the same source and target perform the same regardless of the agent on which bbcp is run.

If Jacquard is the source system and Davinci the target, the default bbcp attains 115-120 MBytes/sec. If Davinci is the source system and Jacquard the target, the default bbcp attains 43-46 MBytes/sec.

We investigated two performance options to increase the default performance, -s for number of parallel network streams (the default is four), and -w for size of disk I/O buffers (the default size is 64k).

When the number of streams on the Davinci source Jacquard target transfer was reduced to 1 and the buffer size doubled to 128k (-s 1 -w 128k, the performance of the file transfer more than doubled to almost that of the Jacquard source Davinci target transfer, 115 MBytes/sec.

Neither of these options was able to improve on the default performance of the the Jacquard source Davinci target transfer.