Evaluation of Early Systems

AORSA3D Benchmarks

The AORSA3D code uses an all orders spectral algorithm to solve for the wave electric field and heating in a 3D stellerator plasma heated by radio frequency waves. It is an important application in the Fusion Sciences SciDAC project "Numerical Calculation of Wave-Plasma Interactions in Multidimensional Systems".

AORSA3D is a Fortran 90 code that uses MPI and SCALAPACK to solve linear systems arising from the spectral discretization. AORSA3D has three major computational phases:

1. matrix generation
2. complex linear system solution
3. current calculation

The problem sizes of scientific interest currently require that AORSA3D be run on very large parallel systems. This is inconvenient for benchmarking. Instead, we fix the maximum amount of memory per process to a value similar to that used in production runs. We then run the problem in a "scale-up" mode, increasing the problem size as we increase the number of processors. This allows us to assign some relevance to the benchmark results when evaluating the appropriateness of a given system for production AORSA3D runs.

The problem size is characterized by the total number of Fourier modes retained by the model. The following experimental results describe the performance in terms of the ratio of the number of modes to the execution time. The scaling behavior as a function of the number of processors is not important. If N is the number of modes, then the total memory requirement is O(N**2), while the computational complexity contains both O(N**2) and O(N**3) terms. Because the number of processors is proportional to O(N**2), the parallel complexity (and the modes/runtime ratio) necessarily decreases for increasing N once the O(N**3) term becomes dominant.

Two different versions of AORSA3D have been used for benchmarking, both written by Fred Jaeger at Oak Ridge National Laboratory. The first version of the code was released in August, 2001, and data was collected using this version until May 2002. The data in the following graphs were collected by Patrick H. Worley on the following systems:

• Compaq AlphaServer SC at Pittsburgh Supercomputer Center
  • 750 4-way ES45 SMP nodes (1.0 GHz Alpha 21264c with 8MB L2 cache)
• Compaq AlphaServer SC at Oak Ridge National Laboratory
  • 64 4-way ES40 SMP nodes (667 MHz Alpha 21264a with 8MB L2 cache)
• IBM p690 at Oak Ridge National Laboratory
  • 27 32-way p690 "Turbo" SMP nodes (1.3 GHz POWER4)
• IBM SP (375MHz) at Oak Ridge National Laboratory
  • 184 4-way Winterhawk II SMP nodes (375 MHz POWER3-II with 8MB L2 cache)
• NEC SX6/8 at Arctic Region Supercomputing Center
  • 8-way SX-6/8 SMP node (500 MHz processor with 8 vector pipes)

From these results, AORSA3D performance varies between the different platforms roughly in proportion to the relative processor capabilities, with the exception of the SX-6 vector system. This code was not modified for vectorization before running these experiments. An attempt was made to choose appropriate compiler options, and the vendor-supplied math library subroutines were used for the linear system solution. The runtime of AORSA3D is dominated by the solution of the linear system on the nonvector platforms. On the SX-6/8, the linear system solution is very efficient and the opposite is true. For more details, see the SX-6 evaluation pages.

The second version of AORSA3D was released in January, 2002. It has slightly improved science, and unnecessary work in the matrix generation and current calculations phases was eliminated. These modifications are in the vector-unfriendly parts of the code, and this version performs much better on the SX-6.

Performance on the SX-6 is still less than that of the IBM p690 for the small test cases, but is 50% better for the largest benchmark problem. As can be seen in the following graph, the linear system solution is a factor of 3 faster on the SX-6 than on the p690 on a per processor basis. As larger problem sizes will be dominated by the linear system solution, the advantage of the vector system is likely to increase, assuming that the interconnection network in an SX-6/8 cluster does not become a performance bottleneck.