Selected 2003 Science of Scale Projects at NERSC

Spectrum Synthesis of Supernovae

Magnetic Materials: Bridging Basic and Applied Science

NIMROD Project: Extended Magnetohydrodynamic Modeling for Fusion Experiments

Spectrum Synthesis of Supernovae

Peter Nugent and Daniel Kasen, Lawrence Berkeley National Laboratory; Edward Baron, University of Oklahoma; Peter Hauschildt, Jason Aufdenberg, Christopher Shore, Andreas Schweitz, Travis Barman, and Eric Lentz, University of Georgia.

An imaginative view of a binary star system going supernova: the companion star excavates a "pit" in the expanding cloud of matter.

Using astrophysics techniques developed in the LBNL Physics Division's Supernova Cosmology Group, this team has begun to measure the fundamental parameters of cosmology that shape our current understanding of particle physics through the observation of very distant Type Ia supernovae. The Supernova Cosmology Project has discovered approximately 120 spectroscopically confirmed high-redshift supernovae over the past four years. Interpretation of such a large, highly redshifted data set of spectra is best performed through spectrum synthesis calculations on a supercomputer. Project objectives include:

1. Completion of spectrum synthesis calculations for both distant and nearby supernovae to look for any systematic differences which might bias the cosmological parameters we measure.
2. Development of an objective classification scheme for SNe~Ia to accurately describe their age and luminosity through their spectral features.
3. Creation of a grid of synthetic supernova spectra which will be used to understand the supernova discovered via LBNL's Supernova Factory and prepare for the SNAP satellite through a comprehensive set of simulations.
4. Detailed study of core-collapse supernovae and their use for determining the cosmological parameters and nucleosynthesis products. The project concentrates on calculating spectrum synthesis models from both of the Supernova SciDAC projects.
5. Improve 3D spectropolarimetric synthesis calculations, interpret observations, and confront the theoretical models with these 3D parameterizations of the supernova events.

One of the results of this work was a set of alternative models to interpret the first evidence of polarization in a "normal" Type Ia supernova.

Magnetic Materials: Bridging Basic and Applied Science

G. Malcolm Stocks, Oak Ridge National Laboratory; Bruce N. Harmon, Ames Laboratory, Iowa State University; Micheal Weinert, University of Wisconsin-Milwaukee; David P. Landau, University of Georgia.

This project aims to develop a comprehensive capability to model magnetism and magnetic materials, with a particular focus on low dimensional (2D, 1D, and 0D) nanosystems (interfaces, nanowires, inclusions). Magnetic nanosystems are currently the focus of intensive experimental research. From the theory point of view, they provide a unique opportunity for theory and simulation to contribute to the understanding and prediction of properties. Indeed, it is at the nano scale where theory and experiment converge with respect to the system sizes that are the object of study. For example, a 5 nm cube of Fe contains ~12,000 atoms (which is of the size of interest in the onset of the phenomenon of superparamagnetism, which places limits on the further miniaturization of magnetic storage elements). Researchers are currently able to perform first principles calculations on systems containing ~ 2000-3000 atoms. This is true even for inhomogeneous systems such as multi-layers, wires, and particles. Using spin dynamics, Monte Carlo methods, and extended Heisenberg models, they are able to treat much larger systems, including the effects of temperature and magnetic fields.

The central difficulty in magnetic modeling is that it is fundamentally a multiscale problem. At the atomic level, first principles electronic structure techniques are required to describe moment formation and the complex magnetic ground states that can occur in inhomogeneous systems. At intermediate length scales, Monte Carlo and spin dynamics techniques implemented for spin models such as the Heisenberg model are needed to describe magnetic phase transitions and the dynamics of local moments. At the longest length scale, micromagnetics modeling describes the behavior of small regions of material and is the method of choice in industry for device modeling. The complexity of the mechanisms of moment formation and the multiscale nature of the interactions pose problems of two types: (1) significantly extending the system size that can be modeled at each of the length scales, with particular emphasis on first principles approaches; (2) developing rigorous approaches to both refining and bridging the models that describe magnetic phenomena on different length scales, with particular emphasis in two areas: (a) using first principles methods to extract parameters (magnetic moments, exchange interactions, and magneto-crystalline anisotropy) that can be used as inputs to spin and micromagnetics models, and (b) developing rigorous coarse graining approaches initially to couple spin models and micromagnetics.

Visualization of the exchange-bias system initial state, viewed from the direction of the y-axis. The arrows represent the orientation of the magnetic moments. The gold balls represent the Fe atoms, the grey ones the Mn atoms, while the green and pink balls stand for the relaxed and unrelaxed Co atoms. On the left, the 3Q antiferromagnetic ordering is schematically displayed within the conventional fcc structure.

A consequence of achieving the goals of this project will be modeling tools capable of integrating atomic level understanding of magnetic properties and interactions with structure and microstructure. Such a capability would enable the science based understanding and prediction of technologically relevant magnetic properties and the design of improved permanent and magneto-electronic devices.

This research has yielded important clues toward solving the mystery of exchange bias, a shift in magnetization that occurs when two different kinds of magnet come into contact. Exchange bias is used in numerous electronic applications such as magnetic multilayer storage and read head devices, but the effect is still not properly understood. Canning et al. [1] have performed first principles spin dynamics simulations of the magnetic structure of iron-manganese/cobalt (FeMn/Co) interfaces. These quantum mechanical simulations, involving 2016-atom super-cell models, reveal details of the orientational configuration of the magnetic moments at the interface that are unobtainable by any other means.

[1] A. Canning, B. Ujfalussy, T. C. Schulthess, X.-G. Zhang, W. A. Shelton, D. M. C. Nicholson, G. M. Stocks, Y. Wang, and T. Dirks, "Parallel Multi-Teraflops Studies of the Magnetic Structure of FeMn Alloys," Proc. Int. Parallel and Distributed Processing Symposium (IPDPS'03, April 22-26, 2003, Nice, France), p. 256b.

NIMROD Project: Extended Magnetohydrodynamic Modeling for Fusion Experiments

Dalton Schnack and Scott Kruger, Science Applications International Corp.; Carl Sovinec and Charlson Kim, University of Wisconsin-Madison; Eric Held, Utah State University; Scott Parker, University of Colorado-Boulder; and Rick Nebel, Los Alamos National Laboratory.

Developers of the NIMROD code, which is used to simulate fusion reactor plasmas, collaborated with members of the SciDAC Terascale Optimal PDE Simulations Center to implement the SuperLU linear solver software within NIMROD. As a result, NIMROD runs four to five times faster for cutting-edge simulations of nonlinear macroscopic electromagnetic dynamics-with a corresponding increase in scientific productivity.

The NIMROD project, funded by the DOE Office of Fusion Energy Sciences and the SciDAC Center for Extended Magnetohydrodynamic Modeling (CEMM), is developing a modern computer code suitable for the study of long-wavelength, low-frequency, nonlinear phenomena in fusion reactor plasmas. These phenomena involve large-scale changes in the shape and motion of the plasma and severely constrain the operation of fusion experiments. CEMM also supports a complementary plasma simulation code, AMRMHD, which uses different mathematical formulations.

By applying modern computational techniques to the solution of extended magnetohydrodynamics (MHD) equations, the NIMROD team is providing the fusion community with a flexible, sophisticated tool which can lead to improved understanding of these phenomena, ultimately leading to a better approach to harnessing fusion energy. Since the beginning of the project, the NIMROD code has been developed for massively parallel computation, enabling it to take full advantage of the most powerful computers to solve some of the largest problems in fusion. The team's primary high-end computing resource is Seaborg at NERSC.

NIMROD's algorithm requires solution of several large sparse matrices in parallel at every time step in a simulation. The stiffness inherent in the physical system leads to matrices that are ill-conditioned, since rapid wave-like responses provide global communication within a single time-step. The preconditioned conjugate gradient (CG) solver that has been used was the most computationally demanding part of the algorithm, so Carl Sovinec of the NIMROD team consulted with the SciDAC Terascale Optimal PDE Simulations (TOPS) Center to find a replacement.

Conversations with David Keyes of Old Dominion University, Dinesh Kaushik of Argonne National Laboratory, and Sherry Li and Esmond Ng of Lawrence Berkeley National Laboratory pointed to SuperLU as a possibly more efficient matrix solver for NIMROD. SuperLU is a library of software for solving nonsymmetric sparse linear systems in parallel using direct methods.

Full 3D numerical simulation of plasma particle drift orbits in a tokamak. From Charlson C. Kim, Scott E. Parker, and the NIMROD team, "Kinetic particles in the NIMROD fluid code," 2003 Sherwood Fusion Theory Conference, Corpus Christi, Texas.

It took less than a month to implement, test, and release SuperLU into the full NIMROD production code, and the performance improvements were dramatic. For two-dimensional linear calculations of MHD instabilities, NIMROD runs 100 times faster with SuperLU than it does with the CG solver. While linear calculations do not require supercomputer resources, they are extremely useful for preliminary explorations that help determine which cases require in-depth nonlinear simulations. For linear problems, this performance improvement makes NIMROD competitive with special-purpose linear codes.

For cutting-edge three-dimensional, nonlinear tokamak simulations, which require supercomputing resources, NIMROD with SuperLU runs four to five times faster than before. This improved efficiency yields four to five times more physics results for the same amount of time on the computer-a major improvement in scientific productivity.

The nonlinear simulations accumulate relatively small changes in the matrix elements at each time step, but there are no changes to the sparsity pattern. The NIMROD implementation allows SuperLU to reuse its factors repeatedly until the matrix elements accumulate a significant change and refactoring becomes necessary. Refactoring makes the performance improvement less dramatic in nonlinear simulations than in linear calculations, but it is still very significant.

The net result is that computationally demanding simulations of macroscopic MHD instabilities in tokamak plasmas, which until now were considered too difficult, have become routine.