LQCD Homepage

QCDOC Computing

User Environment
 

A beginner’s guide to QCDOC at BNL

This guide is an introduction to using the QCDOC computing resources at BNL. Its purpose is to help new users get their programs running on QCDOC machine partitions. Please send your comments, corrections, suggestions, etc. to stratos@bnl.gov. This guide is updated frequently.

Existing QCDOC user guides have been used (and sometimes closely followed or even copied) when writing this guide. They are listed in the References section.

  • Required BNL Accounts
  • Accessing QCDOC Resources
  • Transferring Files
  • Setting Up the User Environment
  • Compiling a Simple Example
  • Allocating a QCDOC Machine Partition
  • Running a Job
  • Leaving QCDOC
  • Reporting Problems
  • References
  • Acknowledgements

Required Accounts

In order to access and use QCDOC computing resources at BNL users need three separate accounts: A QCDOC computer account, an ssh gateway account and a Web Allocator account. For a detailed description of each account type and how to apply for them see the accounts page.

Accessing QCDOC Resources

QCDOC computational resources can be accessed interactively from the front-end nodes. Members of the US DOE QCD collaboration can simply ssh to qcdochostb.qcdoc.bnl.gov, the front-end node of the US DOE QCD collaboration. RBRC users should ssh to qcdochosta.qcdoc.bnl.gov.

Remote (outside BNL) users will need to go through one of the general BNL ssh gateways ([ssh1, ssh2, ssh4].bnl.gov) and then ssh to the front-end node (soon we’ll have dedicated ssh gateways for the QCDOC users). Remote users can tunnel via the ssh gateways to access the front-end nodes:

remote host$ ssh –t user@ssh1.bnl.gov ssh user@qcdochostb.qcdoc.bnl.gov

You can avoid typing your password by setting up your ssh keys. Instructions can be found at http://www.bnl.gov/cybersecurity/ssh_unix_protocol2.asp

Transferring Files

You can transfer files in and out of QCDOC servers using secure copy (scp). Remote users can initiate the transfer from the front-end nodes to avoid file staging on the ssh gateways:

 qcdochostb> scp user@qcdoc.ed.ac.uk:/home/user/*.C /home/user/

Alternatively, incoming files can be staged on the ssh gateways (two-hop file transfers) or remote users can set up ssh port forwarding to tunnel files via the gateways. Detailed instructions are provided in File Transfers.

Setting up the User Environment

To set up you environment correctly you need to source the setup script of the QOS version. There are several version of the QOS installed under /qcdoc/sfw/qos, the most recent one being v2.5.9.

If, for example, qos version v2.5.9 is the version of your choice, you should source the script:

(bash)    $ source /qcdoc/sfw/qos/v2.5.9/aix5.2/scripts/setup.sh

(tcsh) $ . /qcdoc/sfw/qos/v2.5.9/aix5.2/scripts/setup.csh

It is recommended that users source the QOS setup scripts in their shell startup scripts (.bash_profile and .tcshrc). The setup scripts set QOS related variables, including the user PATH, in order to use the appropriate cross-compilers and QOS utilities. Examples of shell startup scripts can be copied from /home/stratos/examples/(.bash_profile/.tcshrc). Bear in mind that the example startup scripts source the setup scripts for QOS v2.5.9.

Compiling a Simple Example

The next step would be to compile your code. I describe here how to compile a very simple “HelloWorld” type example. Under your home directory create the test directory, copy the hello.C source file and the Makefile from /home/stratos/examples/ and run make:

$ cd ~/test
$ cp /home/stratos/examples/hello.C .
$ cp /home/stratos/examples/Makefile .
$ make

powerpc-gnu-elf-g++ -c hello.C -o hello.o
powerpc-gnu-elf-g++ hello.o -o prog.out

Allocating a QCDOC Machine Partition

Users need to allocate a machine partition before running their jobs. A machine partition is a set of nodes grouped together to run a single job. The Web Allocator page (https://rbc.bnl.gov) lists all the available partitions and allows authorized users to allocate them. Currently a set of four single Mother Board (64 node) partitions on ACC7 is available to Alpha Testers of the US DOE QCD collaboration, while another set of four MBs on ACC5 is available to the RCRC testers. They can be allocated under “DOE (or RBRC) General use QCDOC machines”.

It is important that you remember the partition name that you reserved. You will need it when you run your job. The partition names for the four single MB partitions are: accX/slot0, accX/slot1, accX/slot2 and accX/slot3, where X is 5 for the RBRC MBs and 7 for the DOE MBs.

Running a job

Now that the code has been compiled and a machine partition has been allocated you can run your job. First you need to communicate with the machine partition you have allocated by running the qdaemon. On the front-end node you start the qdeamon with the allocated machine partition name as an argument:

$ qdaemon –m acc7/slot1

On another shell on the front-end node you must then start the qcsh shell to communicate with the qdaemon.

$ qcsh

The file .qcshoc in the user’s home directory is the qcsh start up file. It contains qcsh commands that are executed at qcsh start up. An initial .qcshoc file can be copied from /home/stratos/examples/.qcshoc.

A simple shell script, qsession, starts both the qdaemon and the qcsh and sets the QMACHINE variable to the qsession argument. The QMACHINE variable can be used in your .qcshoc file:

$ qsession acc7/slot1

qsession starts an xterm for the qdaemon. Make sure you have the $DISPLAY variable set correctly.

Connecting to the qdaemon

The qinit command starts up the communication with the qdeamon for the allocated machine partition:

QCSH> qinit acc7/slot1

If you started the qdaemon and the qcsh using the qsession script, you can write:

QCSH> qinit $QMACHINE

You should get the following output:

qhelper on socket 5
..created system socket for commands.. ..authorization complete.
Child
Exec /qcdoc/sfw/qos/v2.5.9/aix5.2/qhost/bin/qhelper 5

The default.qrb file

qinit loads the default.qrb file that contains configuration parameters, such as the boot and the run kernel that will be loaded, the clock frequency, etc. In QOS v2.5.9 the default.qrb file is loaded from the appropriate machine directory (in the example this is /qcdoc/machines/v2.5/status/acc7/slot1/default.qrb).

The qcsh command qset_reset_boot command (without any arguments) prints a list of the parameters that have been set. My recommendation is that you should not trust the provided default.qrb file. Instead, you should copy it into your working directory, examine and possibly modify the parameter settings and then read it in running qset_reset_boot with the –R <file> argument:

QCSH> qset_reset_boot –R ./default.2.5.9.qrb

executing... qset_reset_boot -R default.2.5.9.qrb
CWD is /home/stratos/test/
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr 

QCSH>qset_reset_boot

executing... qset_reset_boot
CWD is /home/stratos/test/
RBC:ResetBootCtl::Print.> bker /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/bker/bker.x
RBC:ResetBootCtl::Print.> rker /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/rker/rker.x
RBC:ResetBootCtl::Print.> ClkFreq 420
RBC:ResetBootCtl::Print.> BootNonCacheable NO
RBC:ResetBootCtl::Print.> AppExecPriveledged NO
RBC:ResetBootCtl::Print.> ReproduceRate 10
RBC:ResetBootCtl::Print.> InterruptOnExit YES
RBC:ResetBootCtl::Print.> InterruptOnSoftMemoryError YES
RBC:ResetBootCtl::Print.> InterruptOnSoftLinkError YES
RBC:ResetBootCtl::Print.> MaxDmaPolls 1000000
RBC:ResetBootCtl::Print.> MaxPintPolls 100000000
RBC:ResetBootCtl::Print.> MaxGintPolls 100000000
RBC:ResetBootCtl::Print.> MaxTrainingPolls 1000000
RBC:ResetBootCtl::Print.> MaxPllPolls 1000000
RBC:ResetBootCtl::Print.> HsslPllTune 0
RBC:ResetBootCtl::Print.> DoEdramInit YES
RBC:ResetBootCtl::Print.> DoEdramTest YES
RBC:ResetBootCtl::Print.> DoEdramZero YES
RBC:ResetBootCtl::Print.> DoDDRInit YES
RBC:ResetBootCtl::Print.> DoDDRTest YES
RBC:ResetBootCtl::Print.> DoDDRZero YES
RBC:ResetBootCtl::Print.> DoGPIOLight YES
RBC:ResetBootCtl::Print.> DoGPIOInit YES
RBC:ResetBootCtl::Print.> DoNetworkInit YES
RBC:ResetBootCtl::Print.> DoMdioInit NO
RBC:ResetBootCtl::Print.> DoGintInit YES
RBC:ResetBootCtl::Print.> DoGintTest YES
RBC:ResetBootCtl::Print.> DoPintTest NO
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr

The qset_reset_boot command could be put in the qcsh startup script (.qcshoc).

One of the parameters that you should pay special attention to is the Clock Frequency. The QOS does not have a way to discover the clock frequency set in the partition, thus it is set manually in the default.qrb file. The default.qrb file in the appropriate machine directory has a Clock Frequency of 360 (Edinburgh setting). When you copy the file to your working directory, you should change this to 420, which is the BNL setting. In the above output of the qset_reset_boot command example, the clock frequency is in bold.

Clock frequencies do change and may be different from one machine partition to another. When you are allocating a machine partition, the Web Allocator page should list the clock frequency for each partition. Remember that the clock frequency displayed in the web allocator is also set manually.

Connecting to a machine partition

You can get a list of the available machine partition by running qpartition_list. To connect to a machine partition you issue the qpartition_connect command:

QCSH> qpartition_connect –p 0

executing... qpartition_connect -p 0
CWD is /home/stratos/test/
QD:TextParser::doqpartition_connect.M> partition connect
QD:TextParser::doqpartition_op.M> doqpartition_op
QD:TextParser::doqpartition_op.M> got client
QD:TextParser::doqpartition_op.M> trying to parse
QD:TextParser::parse_partitionID.M> Operating on partition 0
QD:TextParser::doqpartition_op.M> Calling PartitionMgr
QD:PartitionManager::process().M> PartitionManager thread:....
QD:PartitionManager::process().M> Connect request
QD:PartitionManager::qpartition_connect.M> Connecting for execute
QD:PartitionManager::qpartition_connect.M> Reserved partition 0 for client--1
QD:PartitionManager::qpartition_connect.M> Connection Established
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr

Booting the machine

To boot you partition you must first run qreset that will load the boot kernel (via Ethernet JTAG) and will do some basic setup testing. qboot loads the run kernel. The two commands (qboot and qreset) have been merged into one: qreset_boot.

QCSH> qreset_boot

executing... qreset_boot
CWD is /home/stratos/test/
QD:Partition::PrintState.E> No idea what is running
QD:Partition::PrintState.M> Serial communications are NOT up
QD:Partition::PrintState.M> Application axes are NOT mapped
RBC:Boot0::EstablishComms.M> Successfully contacted all nodes
RBC:Boot0::EstablishComms.M> Checking PVR register is 0x40a204c0
RBC:Boot0::BuildPackets.M> Processing Elf boot0 kernel
RBC:Boot0::BuildPackets.M> /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/bker/bker.x
RBC:Boot0::elf2jtag.M> 6 P-sections
RBC:Boot0::elf2jtag.M> 143 write packets
RBC:Boot0::elf2jtag.M> 3 stuff packets
RBC:Boot0::BootNodes.M> 1 Spooling threads
RBC:Boot0::BootNodes.M> Spooler 0 has 64 nodes
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG UpdateIp
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG ResetProc
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG HaltProc
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Instruction Pipe Stuff
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Kernel Write
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Set IP/MCaddr
RBC:Boot0::NICBootNodes.M> Spooler 0: JTAG Exec Kernel
RBC:Boot0::doBoot.M> Waiting for DRAM tests
RBC:Boot0::AnalyzeBoot.M> Contacted all nodes
RBC:Boot0::doBoot.M> Boot Phase Zero is SUCCEEDED
QD:Partition::qreset.M> Reset was successful
QD:Partition::qboot.M> Booting nodes
RBC:Boot1::BuildPackets.M> Processing elf kernel
RBC:Boot1::BuildPackets.M> /qcdoc/sfw/qos/v2.5.9/aix5.2/qker/rker/rker.x
RBC:Boot1::elf2bkerp.M> 9 P-sections
RBC:Boot1::process_psect.M> Discarding section 00010034
RBC:Boot1::process_psect.M> Discarding section 00000000
RBC:Boot1::process_psect.M> Discarding section 00100000
RBC:Boot1::process_psect.M> Loading : 200000 -> 200100 : filesize = 100
RBC:Boot1::process_psect.M> Loading : 200100 -> 22253c : filesize = 2243c
RBC:Boot1::process_psect.M> Loading : 300000 -> 3065b4 : filesize = 65b4
RBC:Boot1::process_psect.M> Loading : 306600 -> 882b00 : filesize = 0
RBC:Boot1::process_psect.M> Discarding section 00b00000
RBC:Boot1::process_psect.M> Discarding section 00000000
RBC:Boot1::elf2bkerp.M> Total of 220 packets
RBC:Boot1::BootNodes.M> Spooler 0 has 64 nodes
RBC:Boot1::NICBootNodes.M> NICBootNodes: BKERP SYNC
RBC:Boot1::NICBootNodes.M> NICBootNodes: BKERP Write
RBC:Boot1::NICBootNodes.M> NICBootNodes: BKERP Execute
QD:Partition::qboot.M> Boot was successful
QD:Partition::qreset_boot.M> Success
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr

The discover process

Finally, the qdiscover command discovers the machine partition topology by figuring out how many nodes exist in each of the six physical (machine) dimensions.

QCSH> qdiscover

executing... qdiscover
CWD is /home/stratos/test/
QD:Partition::PrintState.M> Run kernel is running
QD:Partition::PrintState.M> Serial communications are NOT up
QD:Partition::PrintState.M> Application axes are NOT mapped
QD:Partition::qscu_train.> Setting up SCU hardware
QD:Partition::qscu_train.> Starting training process
QD:Partition::qscu_train.> Completing training process
QD:Partition::qscu_train.> Switching to idle bytes
QD:Partition::qscu_train.> Releasing Dircom units
QD:Partition::qscu_test.M> ScuDMATestStart
QD:Partition::qscu_test.M> ScuDMATestComplete
QD:Partition::qscu_test.M> SCU Test Successful
QCDOC:Partition::CheckLinks.M> 0 link bit errors on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 checksum mismatches on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCceErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCceErrs on ENTIRE machine
QD:Partition::DiscoverTopology.M> Preparing neighbour swap
QD:Partition::DiscoverTopology.M> Starting neighbour swap
QD:Partition::DiscoverTopology.M> Completing neighbour swap
QCDOC:Partition::CheckLinks.M> 0 link bit errors on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 checksum mismatches on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCceErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCceErrs on ENTIRE machine
QD:Partition::CheckHdwIdentfiers.M> Opening machine file /qcdoc/machines/v2.5/status/acc7/slot2//database/hdw/hdw.vml.1
QD:Partition::DiscoverTopology.M> Dimension 0 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 1 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 2 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 3 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 4 has length 2 nodes
QD:Partition::DiscoverTopology.M> Dimension 5 has length 2 nodes
QD:Partition::DiscoverTopology.M> Checking for well formed 6d mesh
QD:Partition::DiscoverTopology.M> Setting up MBTopology map
QD:Partition::DiscoverTopology.M> Motherboard topology mapped
QD:Partition::DiscoverTopology.M> Motherboard grid is 1 x 1 x 1
QD:Partition::qgints.M> Acquired mutex lock for Gints
QD:Partition::qgints.M> Running Gint tests
QD:Partition::qgints.M> Sensitive to bad pins on U15,U16,U17,U18,U19,U20,U21,U22,U23
QD:Partition::qgints.M> Running test with Tx2Rx off
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with Tx2Rx on all Tx off
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with Tx2Rx on all Tx on
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with motherboards in isolation
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Running test with no boundary isolation
QD:Partition::qgints.M> Passed
QD:Partition::qgints.M> Testing each nodes gints individually
QD:Partition::qgints.M> Gint Irq 0
................................................................
QD:Partition::qgints.M> Gint Irq 1
................................................................
QD:Partition::qgints.M> Gint Irq 2 |
................................................................
QD:Partition::qscu_train.> Setting up SCU hardware
QD:Partition::qscu_train.> Starting training process
QD:Partition::qscu_train.> Completing training process
QD:Partition::qscu_train.> Switching to idle bytes
QD:Partition::qscu_train.> Releasing Dircom units
QD:Partition::qscu_test.M> ScuDMATestStart
QD:Partition::qscu_test.M> ScuDMATestComplete
QD:Partition::qscu_test.M> SCU Test Successful
QCDOC:Partition::CheckLinks.M> 0 link bit errors on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 checksum mismatches on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 DDRECCceErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCueErrs on ENTIRE machine
QCDOC:Partition::CheckLinks.M> 0 EdramECCceErrs on ENTIRE machine
QD:Partition::qpints.M> Acquired mutex lock for Gints
QD:Partition::qpints.M> PintInitPrepare
QD:Partition::qpints.M> PintInitStart
QD:Partition::qpints.M> PintInitComplete
QD:Partition::qpints.M> PintInitComplete Done
QD:Partition::DiscoverTopology.M> QCDOC is ready
QDaemonReturn 0
QKerReturn 0
AppReturn 0
ExitStr
(qcdochostb:/home/stratos/test:QCSH)%

Running your program

You are now ready to run your program (prog.out) using qrun. qrun loads your program into each machine node and runs it. Output from the o node is echoed to the qcsh session.

QCSH> qrun prog.x arg1 arg2

As mentioned earlier, some of the qcsh commands that are repeated when you start qcsh could go in the .qcshoc startup file. A sleep command is added before running qinit to give enough time to the qdaemon to start before connecting to it.

Stopping a running job

The following might help: first try CONTROL-C to get back to the qcsh prompt and then run qkill. You may also have to run qreset_sys.

Leaving QCDOC

Once your job has finished you can detach from the machine partition and exit:

QCSH> qdetach
QCSH> exit

The exit command also kills the qdaemon xterm window. Don’t forget to release the machine partition you have allocated from the web Allocator page.

Reporting Problems

Currently we are in the process of setting up Call Tracking System (CTS) services where users can report software and hardware issues. The software CTS system can be accessed at https://sfwcts.phys.columbia.edu (account required). In the mean time, if you are having problems with you user environment you should send an email to both Stratos Efstathiadis (stratos@bnl.gov) and Zhihua Dong (dong@physics.columbia.edu).

References/Acknowledgments

This QCDOC user guide was put together by updating and modifying already existing user guides, such as those from Balint Joo, Zbyszek Sroczynskiand and Chulwoo Jung. Balint’s guide is available at http://www.ph.ed.ac.uk/~bj/QCDOC_quick_start, while Chulwoo’s guide is available as a ps file from the Columbia web site (account required). Also, Eric Blum, Bob Mawhinney and the alpha testers (Dry Renner, James Osborn, Steve Gottlieb) contributed with comments and corrections. Fruitful discussions with Peter Boyle and Balint Joo during my Edinburgh visit have also help me understand better some of the QOS internals.

Top of Page

Last Modified: September 3, 2008
 


One of ten national laboratories overseen and primarily funded by the Office of Science of the U.S. Department of Energy (DOE), Brookhaven National Laboratory conducts research in the physical, biomedical, and environmental sciences, as well as in energy technologies and national security. Brookhaven Lab also builds and operates major scientific facilities available to university, industry and government researchers. Brookhaven is operated and managed for DOE’s Office of Science by Brookhaven Science Associates, a limited-liability company founded by Stony Brook University, the largest academic user of Laboratory facilities, and Battelle, a nonprofit, applied science and technology organization.

Privacy and Security Notice  | Contact Web Services for help