Quad-Core Changes
The upgraded jaguar system contains 7,832 quad-core 2.1 GHz AMD Opteron processors and 62 TB of memory (2 GB per core). Aggregate system performance is approximately 250 TF. Approximately 600 TB are available in the scratch filesystems.
While use of the system will be similar to jaguarcnl and the 119 TF dual-core jaguar system, there are a number of notable differences listed below.
Table of Contents
Access
To access the quad-core system, ssh to jaguar.ccs.ornl.gov.
Compiling
We recommend that codes built on the dual-core system be recompiled prior to running on the quad-core system.
As when building for the dual-core nodes, the cc, CC, and ftn compiler wrappers should be used to compile an application for quad-core nodes.
More details can be found in the XT compiling page.
Batch Jobs
On the XT systems, access to compute resources is managed by the Torque system. Job scheduling is handled by Moab, which interacts with Torque and the XT system software.
Batch script options used on the dual-core system should also work on the quad-core system.
As is the case with the dual-core system, the -lsize option to qsub requests cores. Please note that since jaguar is a quad-core system and users cannot share sockets, size requests must be a multiple of 4.
An example script follows:
1: #PBS -A XXXYYY 2: #PBS -N test 3: #PBS -j oe 4: #PBS -l walltime=1:00:00,size=8192 5: 6: cd $PBS_O_WORKDIR 7: 8: aprun -n8192 ./a.out
Details
1: The job will be charged to the “XXXYYY” project.
2: The job will be named “test.”
3: The job’s standard output and error will be combined.
4: The job will request 8,192 compute cores for 1 hour.
6: This command will change the directory into the script’s submission directory.
8: This command will run the executable a.out on 8,192 cores (on four cores of each 2,048 compute nodes) using aprun.
More details can be found in the XT running jobs page.
Launching an Application
As on the dual-core systems, aprun is used to launch an application.
Basic Options
- -D - Debug option that can be used to view layout
- -N - Number of cores per socket
- -n - Number of cores
- -m - Memory required per task (maximum 2,000 MB per core; 2,100 MB will allocate two cores for the task)
- -d - Number of threads per task
Examples
Launch a.out on 1,024 cores, four cores per socket as follows:
aprun -n 1024 a.out
Launch a.out on 1,024 cores, one core per socket as follows:
aprun -n 1024 -N1 a.out
Message Passing Interface (MPI) Task Layout
As on the dual-core system, the default MPI task layout is sequential. For example,
aprun -n8 a.out
will run the MPI executable a.out on a total of eight cores, four cores on two compute nodes. The MPI tasks will be allocated in the following sequential fashion:
| Compute Node 0 | Compute Node 1 | |||||||
|---|---|---|---|---|---|---|---|---|
| core 0 | core 1 | core 2 | core 3 | core 0 | core 1 | core 2 | core 3 | |
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
The layout order can be changed using the environment variable MPICH_RANK_REORDER_METHOD. See man intro_mpi for more information.
Task layout can be seen by setting the environment variable PMI_DEBUG variable to 1.
More details can be found in the XT running jobs page.
Using more than 16,000 cores
When using more than ~16,000 cores you may need to set MPICH_MSGS_PER_PROC to a number larger than the job’s requested cores.
The MPICH_MSGS_PER_PROC variable “Specifies the maximum number of internal message headers that can be allocated by MPI.” The default is 16,384.
File System Access
Work space is not shared between the jaguar and jaguarcnl sytems.
The systems do not share local file systems, including /tmp/work/$USER. The systems both have access to the same NFS-mounted home, project, and software directories. The NCCS HPSS is also accessable from both systems.
Compute nodes can see only the Lustre work space.
The NFS-mounted home, project, and software directories are not accessible to the compute nodes.
- Executable must be executed from within the Lustre work space.
- Batch jobs can be submitted from the home or work space. If submitted from a user’s home area, the user should cd into the Lustre work space directory prior to running the executable through
aprun. An error similar to the following may be returned if this is not done:aprun: [NID 94]Exec /tmp/work/userid/a.out failed: chdir /autofs/na1_home/userid No such file or directory
- Input must reside in the Lustre work space.
- Output must also be sent to the Lustre file system.
More details can be found in the XT file systems page.
Third Party Software
Third party applications that were previously stored in the /apps filesystem have been moved to the /sw filesystem. All software in this filesystem is available via the modules utility. Users are strongly encouraged to use the modules utility instead of full path names to files whenever possible.