QCDOC Computing
User Environment
A beginner’s guide to QCDOC at BNL
This guide is an introduction to using the QCDOC computing resources at BNL.
Its purpose is to help new users get their programs running on QCDOC machine
partitions. Please send your comments, corrections, suggestions, etc. to
stratos@bnl.gov. This guide is updated
frequently.
Existing QCDOC user guides have been used (and sometimes closely followed or
even copied) when writing this guide. They are listed in the References
section.
- Required BNL Accounts
- Accessing QCDOC Resources
- Transferring Files
- Setting Up the User Environment
- Compiling a Simple Example
- Allocating a QCDOC Machine Partition
- Running a Job
- Leaving QCDOC
- Reporting Problems
- References
- Acknowledgements
Required Accounts
In order to access and use QCDOC computing resources at BNL users need three
separate accounts: A QCDOC computer account, an ssh gateway account and a
Web Allocator account. For a detailed description of each account type and
how to apply for them see the accounts page.
Accessing QCDOC Resources
QCDOC computational resources can be accessed interactively from the
front-end nodes. Members of the US DOE QCD collaboration can simply ssh to
qcdochostb.qcdoc.bnl.gov, the front-end node of the US DOE QCD
collaboration. RBRC users should ssh to qcdochosta.qcdoc.bnl.gov.
Remote (outside BNL) users will need to go through one of the general BNL
ssh gateways ([ssh1, ssh2, ssh4].bnl.gov) and then ssh to the front-end node
(soon we’ll have dedicated ssh gateways for the QCDOC users). Remote users
can tunnel via the ssh gateways to access the front-end nodes:
remote host$ ssh –t user@ssh1.bnl.gov
ssh user@qcdochostb.qcdoc.bnl.gov
You can avoid typing your password by setting up your
ssh keys. Instructions can be found at
http://www.bnl.gov/cybersecurity/ssh_unix_protocol2.asp
Transferring Files
You can transfer files in and out of QCDOC servers using secure copy (scp).
Remote users can initiate the transfer from the front-end nodes to avoid
file staging on the ssh gateways:
qcdochostb> scp
user@qcdoc.ed.ac.uk:/home/user/*.C
/home/user/
Alternatively, incoming files can be staged on the ssh gateways (two-hop
file transfers) or remote users can set up ssh port forwarding to tunnel
files via the gateways. Detailed instructions are provided in File
Transfers.
Setting up the User Environment
To set up you environment correctly you need to source the setup script
of the QOS version. There are several version of the QOS installed under
/qcdoc/sfw/qos, the most recent one being v2.5.9.
If, for example, qos version v2.5.9 is the version of your choice,
you should source the script:
(bash)
$ source /qcdoc/sfw/qos/v2.5.9/aix5.2/scripts/setup.sh
(tcsh) $ . /qcdoc/sfw/qos/v2.5.9/aix5.2/scripts/setup.csh
It is recommended that users source the QOS setup
scripts in their shell startup scripts (.bash_profile and .tcshrc). The
setup scripts set QOS related variables, including the user PATH, in order
to use the appropriate cross-compilers and QOS utilities. Examples of shell
startup scripts can be copied from /home/stratos/examples/(.bash_profile/.tcshrc).
Bear in mind that the example startup scripts source the setup scripts for
QOS v2.5.9.
Compiling a Simple Example
The next step would be to compile your code. I describe here how to
compile a very simple “HelloWorld” type example. Under your home directory
create the test directory, copy the hello.C source file and the
Makefile from /home/stratos/examples/ and run make:
$ cd ~/test
$ cp /home/stratos/examples/hello.C .
$ cp /home/stratos/examples/Makefile .
$ make
powerpc-gnu-elf-g++ -c hello.C -o hello.o
powerpc-gnu-elf-g++ hello.o -o prog.out
Allocating a QCDOC Machine Partition
Users need to allocate a machine partition before running their jobs. A
machine partition is a set of nodes grouped together to run a single job.
The Web Allocator page (https://rbc.bnl.gov)
lists all the available partitions and allows authorized users to
allocate them. Currently a set of four single Mother Board (64 node)
partitions on ACC7 is available to Alpha Testers of the US DOE QCD
collaboration, while another set of four MBs on ACC5 is available to the
RCRC testers. They can be allocated under “DOE (or RBRC) General use QCDOC
machines”.
It is important that you remember the partition name that you
reserved. You will need it when you run your job. The partition names
for the four single MB partitions are: accX/slot0, accX/slot1, accX/slot2
and accX/slot3, where X is 5 for the RBRC MBs and 7 for the DOE MBs.
Running a job
Now that the code has been compiled and a machine partition has been
allocated you can run your job. First you need to communicate with the
machine partition you have allocated by running the qdaemon. On the
front-end node you start the qdeamon with the allocated machine
partition name as an argument:
$ qdaemon –m
acc7/slot1
On another shell on the front-end node you must then start the qcsh
shell to communicate with the qdaemon.
$ qcsh
The file .qcshoc in the user’s home directory is the qcsh start up
file. It contains qcsh commands that are executed at qcsh start up. An
initial .qcshoc file can be copied from /home/stratos/examples/.qcshoc.
A simple shell script, qsession, starts both the qdaemon
and the qcsh and sets the QMACHINE variable to the qsession argument.
The QMACHINE variable can be used in your .qcshoc file:
$ qsession
acc7/slot1
qsession starts an xterm for the qdaemon. Make sure you have the
$DISPLAY variable set correctly.
Connecting to the qdaemon
The qinit command starts up the communication with the qdeamon
for the allocated machine partition:
QCSH> qinit acc7/slot1
If you started the qdaemon and the qcsh using
the qsession script, you can write:
QCSH> qinit $QMACHINE
You should get the following output:
qhelper on socket
5
..created system socket for commands.. ..authorization complete.
Child
Exec /qcdoc/sfw/qos/v2.5.9/aix5.2/qhost/bin/qhelper 5
The default.qrb file
qinit loads the default.qrb file that contains
configuration parameters, such as the boot and the run kernel that will be
loaded, the clock frequency, etc. In QOS v2.5.9 the default.qrb file
is loaded from the appropriate machine directory (in the example this is /qcdoc/machines/v2.5/status/acc7/slot1/default.qrb).
The qcsh command qset_reset_boot command (without any arguments)
prints a list of the parameters that have been set. My recommendation is
that you should not trust the provided default.qrb file. Instead, you
should copy it into your working directory, examine and possibly modify the
parameter settings and then read it in running qset_reset_boot with
the –R <file> argument:
QCSH> qset_reset_boot –R ./default.2.5.9.qrb
executing...
qset_reset_boot -R default.2.5.9.qrb
CWD is /home/stratos/test/
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr
QCSH>qset_reset_boot
executing... qset_reset_boot
CWD is /home/stratos/test/
RBC:ResetBootCtl::Print.> bker /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/bker/bker.x
RBC:ResetBootCtl::Print.> rker /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/rker/rker.x
RBC:ResetBootCtl::Print.> ClkFreq 420
RBC:ResetBootCtl::Print.> BootNonCacheable NO
RBC:ResetBootCtl::Print.> AppExecPriveledged NO
RBC:ResetBootCtl::Print.> ReproduceRate 10
RBC:ResetBootCtl::Print.> InterruptOnExit YES
RBC:ResetBootCtl::Print.> InterruptOnSoftMemoryError YES
RBC:ResetBootCtl::Print.> InterruptOnSoftLinkError YES
RBC:ResetBootCtl::Print.> MaxDmaPolls 1000000
RBC:ResetBootCtl::Print.> MaxPintPolls 100000000
RBC:ResetBootCtl::Print.> MaxGintPolls 100000000
RBC:ResetBootCtl::Print.> MaxTrainingPolls 1000000
RBC:ResetBootCtl::Print.> MaxPllPolls 1000000
RBC:ResetBootCtl::Print.> HsslPllTune 0
RBC:ResetBootCtl::Print.> DoEdramInit YES
RBC:ResetBootCtl::Print.> DoEdramTest YES
RBC:ResetBootCtl::Print.> DoEdramZero YES
RBC:ResetBootCtl::Print.> DoDDRInit YES
RBC:ResetBootCtl::Print.> DoDDRTest YES
RBC:ResetBootCtl::Print.> DoDDRZero YES
RBC:ResetBootCtl::Print.> DoGPIOLight YES
RBC:ResetBootCtl::Print.> DoGPIOInit YES
RBC:ResetBootCtl::Print.> DoNetworkInit YES
RBC:ResetBootCtl::Print.> DoMdioInit NO
RBC:ResetBootCtl::Print.> DoGintInit YES
RBC:ResetBootCtl::Print.> DoGintTest YES
RBC:ResetBootCtl::Print.> DoPintTest NO
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr
The qset_reset_boot command could be put in the qcsh startup script (.qcshoc).
One of the parameters that you should pay special attention to is the
Clock Frequency. The QOS does not have a way to discover the clock frequency
set in the partition, thus it is set manually in the default.qrb
file. The default.qrb file in the appropriate machine directory has a
Clock Frequency of 360 (Edinburgh setting). When you copy the file to your
working directory, you should change this to 420, which is the BNL setting.
In the above output of the qset_reset_boot command example,
the clock frequency is in bold.
Clock frequencies do change and may be different from one machine
partition to another. When you are allocating a machine partition, the Web
Allocator page should list the clock frequency for each partition. Remember
that the clock frequency displayed in the web allocator is also set
manually.
Connecting to a machine partition
You can get a list of the available machine partition by running
qpartition_list. To connect to a machine partition you issue the
qpartition_connect command:
QCSH> qpartition_connect –p 0
executing... qpartition_connect -p 0
CWD is /home/stratos/test/
QD:TextParser::doqpartition_connect.M> partition connect
QD:TextParser::doqpartition_op.M> doqpartition_op
QD:TextParser::doqpartition_op.M> got client
QD:TextParser::doqpartition_op.M> trying to parse
QD:TextParser::parse_partitionID.M> Operating on partition 0
QD:TextParser::doqpartition_op.M> Calling PartitionMgr
QD:PartitionManager::process().M> PartitionManager thread:....
QD:PartitionManager::process().M> Connect request
QD:PartitionManager::qpartition_connect.M> Connecting for execute
QD:PartitionManager::qpartition_connect.M> Reserved partition 0 for
client--1
QD:PartitionManager::qpartition_connect.M> Connection Established
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr
Booting the machine
To boot you partition you must first run qreset that will
load the boot kernel (via Ethernet JTAG) and will do some basic setup
testing. qboot loads the run kernel. The two commands (qboot
and qreset) have been merged into one: qreset_boot.
QCSH> qreset_boot
executing... qreset_boot
CWD is /home/stratos/test/
QD:Partition::PrintState.E> No idea what is running
QD:Partition::PrintState.M> Serial communications are NOT up
QD:Partition::PrintState.M> Application axes are NOT mapped
RBC:Boot0::EstablishComms.M> Successfully contacted all nodes
RBC:Boot0::EstablishComms.M> Checking PVR register is 0x40a204c0
RBC:Boot0::BuildPackets.M> Processing Elf boot0 kernel
RBC:Boot0::BuildPackets.M> /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/bker/bker.x
RBC:Boot0::elf2jtag.M> 6 P-sections
RBC:Boot0::elf2jtag.M> 143 write packets
RBC:Boot0::elf2jtag.M> 3 stuff packets
RBC:Boot0::BootNodes.M> 1 Spooling threads
RBC:Boot0::BootNodes.M> Spooler 0 has 64 nodes
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG UpdateIp
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG ResetProc
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG HaltProc
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Instruction Pipe Stuff
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Kernel Write
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Set IP/MCaddr
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Exec Kernel
RBC:Boot0::doBoot.M> Waiting for DRAM tests
RBC:Boot0::AnalyzeBoot.M> Contacted all nodes
RBC:Boot0::doBoot.M> Boot Phase Zero is SUCCEEDED
QD:Partition::qreset.M> Reset was successful
QD:Partition::qboot.M> Booting nodes
RBC:Boot1::BuildPackets.M> Processing elf kernel
RBC:Boot1::BuildPackets.M> /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/rker/rker.x
RBC:Boot1::elf2bkerp.M> 9 P-sections
RBC:Boot1::process_psect.M> Discarding section 00010034
RBC:Boot1::process_psect.M> Discarding section 00000000
RBC:Boot1::process_psect.M> Discarding section 00100000
RBC:Boot1::process_psect.M> Loading : 200000 -> 200100 : filesize = 100
RBC:Boot1::process_psect.M> Loading : 200100 -> 22253c : filesize = 2243c
RBC:Boot1::process_psect.M> Loading : 300000 -> 3065b4 : filesize = 65b4
RBC:Boot1::process_psect.M> Loading : 306600 -> 882b00 : filesize = 0
RBC:Boot1::process_psect.M> Discarding section 00b00000
RBC:Boot1::process_psect.M> Discarding section 00000000
RBC:Boot1::elf2bkerp.M> Total of 220 packets
RBC:Boot1::BootNodes.M> Spooler 0 has 64 nodes
RBC:Boot1::NICBootNodes.M> NICBootNodes: BKERP SYNC
RBC:Boot1::NICBootNodes.M> NICBootNodes: BKERP Write
RBC:Boot1::NICBootNodes.M> NICBootNodes: BKERP Execute
QD:Partition::qboot.M> Boot was successful
QD:Partition::qreset_boot.M> Success
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr
The discover process
Finally, the qdiscover command discovers the machine partition
topology by figuring out how many nodes exist in each of the six physical
(machine) dimensions.
QCSH> qdiscover
executing... qdiscover
CWD is /home/stratos/test/
QD:Partition::PrintState.M> Run kernel is running
QD:Partition::PrintState.M> Serial communications are NOT up
QD:Partition::PrintState.M> Application axes are NOT mapped
QD:Partition::qscu_train.> Setting up SCU hardware
QD:Partition::qscu_train.> Starting training process
QD:Partition::qscu_train.> Completing training process
QD:Partition::qscu_train.> Switching to idle bytes
QD:Partition::qscu_train.> Releasing Dircom units
QD:Partition::qscu_test.M> ScuDMATestStart
QD:Partition::qscu_test.M> ScuDMATestComplete
QD:Partition::qscu_test.M> SCU Test Successful
QCDOC:Partition::CheckLinks.M> 0 link bit errors on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 checksum mismatches on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCceErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCceErrs on ENTIRE machine
QD:Partition::DiscoverTopology.M> Preparing neighbour swap
QD:Partition::DiscoverTopology.M> Starting neighbour swap
QD:Partition::DiscoverTopology.M> Completing neighbour swap
QCDOC:Partition::CheckLinks.M> 0 link bit errors on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 checksum mismatches on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCceErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCceErrs on ENTIRE machine
QD:Partition::CheckHdwIdentfiers.M> Opening machine file /qcdoc/machines/v2.5/status/acc7/slot2//database/hdw/hdw.vml.1
QD:Partition::DiscoverTopology.M> Dimension 0 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 1 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 2 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 3 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 4 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 5 has length 2 nodes
QD:Partition::DiscoverTopology.M> Checking for well formed 6d mesh
QD:Partition::DiscoverTopology.M> Setting up MBTopology map
QD:Partition::DiscoverTopology.M> Motherboard topology mapped
QD:Partition::DiscoverTopology.M> Motherboard grid is 1 x 1 x 1
QD:Partition::qgints.M> Acquired mutex lock for Gints
QD:Partition::qgints.M> Running Gint tests
QD:Partition::qgints.M> Sensitive to bad pins on
U15,U16,U17,U18,U19,U20,U21,U22,U23
QD:Partition::qgints.M> Running test with Tx2Rx off
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with Tx2Rx on all Tx off
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with Tx2Rx on all Tx on
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with motherboards in isolation
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with no boundary isolation
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Testing each nodes gints individually
QD:Partition::qgints.M> Gint Irq 0
................................................................
QD:Partition::qgints.M> Gint Irq 1
................................................................
QD:Partition::qgints.M> Gint Irq 2 |
................................................................
QD:Partition::qscu_train.> Setting up SCU hardware
QD:Partition::qscu_train.> Starting training process
QD:Partition::qscu_train.> Completing training process
QD:Partition::qscu_train.> Switching to idle bytes
QD:Partition::qscu_train.> Releasing Dircom units
QD:Partition::qscu_test.M> ScuDMATestStart
QD:Partition::qscu_test.M> ScuDMATestComplete
QD:Partition::qscu_test.M> SCU Test Successful
QCDOC:Partition::CheckLinks.M> 0 link bit errors on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 checksum mismatches on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCceErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCceErrs on ENTIRE machine
QD:Partition::qpints.M> Acquired mutex lock for Gints
QD:Partition::qpints.M> PintInitPrepare
QD:Partition::qpints.M> PintInitStart
QD:Partition::qpints.M> PintInitComplete
QD:Partition::qpints.M> PintInitComplete Done
QD:Partition::DiscoverTopology.M> QCDOC is ready
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr
(qcdochostb:/home/stratos/test:QCSH)%
Running your program
You are now ready to run your program (prog.out) using qrun. qrun
loads your program into each machine node and runs it. Output from the o
node is echoed to the qcsh session.
QCSH> qrun prog.x arg1 arg2
As mentioned earlier, some of the qcsh commands that are repeated when
you start qcsh could go in the .qcshoc startup file. A sleep
command is added before running qinit to give enough time to
the qdaemon to start before connecting to it.
Stopping a running job
The following might help: first try CONTROL-C to get back to the qcsh
prompt and then run qkill. You may also have to run qreset_sys.
Leaving QCDOC
Once your job has finished you can detach from the machine partition and
exit:
QCSH> qdetach
QCSH> exit
The exit command also kills the qdaemon xterm window. Don’t forget to
release the machine partition you have allocated from the web Allocator
page.
Reporting Problems
Currently we are in the process of setting up Call Tracking System (CTS)
services where users can report software and hardware issues. The software
CTS system can be accessed at
https://sfwcts.phys.columbia.edu (account required). In the mean time,
if you are having problems with you user environment you should send an
email to both Stratos Efstathiadis (stratos@bnl.gov)
and Zhihua Dong (dong@physics.columbia.edu).
References/Acknowledgments
This QCDOC user guide was put together by updating and modifying already
existing user guides, such as those from Balint Joo, Zbyszek Sroczynskiand
and Chulwoo Jung. Balint’s guide is available at
http://www.ph.ed.ac.uk/~bj/QCDOC_quick_start, while Chulwoo’s guide is
available as a ps file from the Columbia web site (account required). Also,
Eric Blum, Bob Mawhinney and the alpha testers (Dry Renner, James Osborn,
Steve Gottlieb) contributed with comments and corrections. Fruitful
discussions with Peter Boyle and Balint Joo during my Edinburgh visit have
also help me understand better some of the QOS internals.
Last Modified: September 3, 2008
|