The Experimental Computing Lab (ExCL) was established in 2004 with the goal of providing application users and computer scientists with access to leading-edge computing systems. ExCL is managed by the Future Technologies Group. ExCL researchers investigate architectures such as multicore processors, Field Programmable Gate Arrays (FPGAs), Graphics Processing Units (GPUs), Cell Broadband Engines (CBEs), and Multi-Threaded Array Processors (MTAPs). Most hardware is located in a large access-controlled server room at ORNL in the JICS/NICS building, near the Future Technology Group.
First, log into excl.ornl.gov via ssh. Then log into the desired target system via ssh. The person sponsoring your ExCL access should be able to tell you the name of the resource(s) you should attempt to use.
In general, each ExCL system provides a similar software environment. The x86_64-based systems run a Red Hat family OS distribution, usually CentOS or Fedora.
The easiest way to manage your environment on ExCL systems is to use the 'modules' command. In your shell login script, either 'dot' or 'source' the appropriate initialization script from /opt/shared/sw/$PLAT/modules/default/init, where $PLAT is a GNU-style architecture-vendor-os triple (e.g., as produced by a GNU config.guess script). Currently, most software is built for x86_64-unknown-linux-gnu since most ExCL systems have this configuration. Then, commands like 'module avail' can be used to see which software packages are available. 'module load <pkg>' and 'module unload <pkg>' can be used to adjust your environment variables (e.g., PATH, LD_LIBRARY_PATH) to use the specified pkg.
Home directories are shared across most ExCL systems.
Commonly-used software is installed in /opt/shared/sw/$PLAT, where $PLAT is a GNU-style architecture-vendor-os triple (e.g., as produced by a GNU config.guess script). Currently, most software is built for x86_64-unknown-linux-gnu since most ExCL systems have this configuration.
Some compiler/tool software packages are shared and mounted under /opt. Look there to see what is available on the system that you are using.
There is a shared project file system mounted at /proj on most ExCL systems. Your ExCL sponsor should be able to tell you whether you are to use this project space
Shared ExCL file systems are hosted by a storage system that uses
RAID, but this storage is
not backed up. If this concerns you, you are responsible for transferring your data to a more stable archiving system.
Currently, there are no file system quotas enforced on user accounts. Please do not abuse this freedom - if you do, we will be forced to implement storage quotas for some or all users.
A 31-node Linux Networx cluster (Yoda1.ornl.gov) consisting of 32-bit Intel Xeon 2.6GHz processors, networked with Gigabit Ethernet and 4x SDR IB, which serves as a testbed for system software research including operating systems and parallel filesystems.
One Dual Socket 2.6 Ghz AMD Istanbul system with 16GB memory (ankara.ftpn.ornl.gov)
Two Dual Socket 2.6 Ghz AMD Shanghai systems (shanghai|peking.ftpn.ornl.gov)
Dual socket 2.4
GHz quad-core Intel Clovertown (springfield.ftpn.ornl.gov)
Quad socket 2.2
GHz quad-core AMD Opteron Barcelona B3 (malaga.ftpn.ornl.gov)
Dual socket 2.1
GHz quad-core AMD Opteron Barcelona B3 (madrid.ornl.gov)
Dual socket 2.6
GHz dual-core Intel Woodcrest (wc0[012].ftpn.ornl.gov), connected with InfiniBand DDR HCAs and Myrinet 10GigE cards.
Dual socket 2.2
GHz dual-core AMD Opterons (dmz0[0123].ftpn.ornl.gov)
Two NVIDIA Ion based systems, each with a dual core Intel Atom processor and a TDP of only 12 watts (cation|anion.ftpn.ornl.gov)
A Sun SPARC Enterprise T5120 server with one 1.165GHz UltraSPARC T2 processor (Victoria Falls) (olafsun.ftpn.ornl.gov)
Dual socket 2.3
GHz quad-core Intel Harpertown (iot0[1-5].ftpn.ornl.gov), with InfiniBand DDR+QDR and Chelsio 10GigE network
Several variants of GPU accelerators that include
Three Digilent Virtex-II Pro FPGA Development System boards, with a variety of I/O ports, including USB and Ethernet.
A Nallatech XtremeDSP Development Kit with the Xilinx Virtex-II Pro FPGA and dual-channel high-performance ADCs and DACs.
A 4.5 TB Panasas ActiveStore storage system (one shelf, two Director Blades and nine Storage Blades) serving home directories and project areas for ExCL systems
A 500
GB RAID server with two dual-core Intel Xeons
An InfiniBand network consists of a 48-port DDR switch, 32 dual-port SDR HCAs and 8 dual-port DDR HCAs. The 31-node LNXI cluster is currently connected with the InifiniBand SDR HCAs.
Two InfiniBand connectX QDR (quad data rate) HCAs
Two Myrinet 10-GigaBit EtherNet cards
Two Chelsio 10-GigaBit EtherNet cards
An SRC-6C MAPstation Reconfigurable Computing Platform pairing dual 2.8GHz Xeon processors with the Xilinx Virtex-II FPGA connected via DIMM slots.
An ATI FireStream 1GB PCIe GPU-based stream processing card.
Two ClearSpeed CS-301 PCI boards, each with two Multi-Threaded Array Processors with 64 parallel execution units.
An
Iwill H8501 server with 8 1.8GHz dual core Opteron processors with 32
GB of memory on a NUMA HT interconnect, configured as a 16-way SMP.
A 144 processor Cray XD1 containing 2.2GHz Opteron processors with six of these nodes containing a Virtex-II Pro FPGA connected to a pair of Opterons via HyperTransport.
An AGEIA PhysX P1 PCI 128MB GDDR3 physics accelerator board.
Two ClearSpeed Avalon PCI boards, each capable of 100GF.
A CUDA development machine with NVIDIA 8600GT (athens.ftpn.ornl.gov)
Two Cell Broadband Engine (CBE) blade systems with dual 2.4GHz Cell processors, each with a 64-bit Power Architecture PPE core and eight SPE SIMD cores. (cell0[01].ornl.gov)
