The Department of Energy is one of the world’s leading users of high-performance computers.  The world’s most powerful supercomputer, Roadrunner*, is located at DOE’s Los Alamos National Laboratory, and is being used to simulate the performance, safety, and reliability of nuclear weapons and to certify their functionality.  Second on the list is Jaguar, located at DOE’s Oak Ridge National Laboratory.  Less than 5% slower, it is the only other system to top 1 petaflops and is the fastest open science system in the world*.  Jaguar is used to support open science research in many areas including superconductivity, climate modeling, astrophysics, and fusion energy.

These two machines present an exciting contrast in approaches to petaflops computing.  The Roadrunner architecture uses four traditional general purpose cores (the AMD Opteron) with four accelerator chips (the IBM double-precision eDP/PowerXCell) on each node to achieve high performance at relatively low power and chip-count.  This approach requires new programming approaches to fully utilize the co-processors.  The Jaguar system takes a more traditional approach using general purpose cores (the quad-core AMD Opteron processors) to achieve petaflops performance.  This approach minimizes the changes to the programming environment but results in greater energy usage.

DOE has seven other high performance computing systems in the world’s top 10 in performance*.  They are located at DOE’s Oak Ridge National Laboratory, Lawrence Livermore National Laboratory, Argonne National Laboratory, Sandia National Laboratories, and Los Alamos National Laboratory.

The underlying architecture of the systems in the Top 500 has converged to MPP (Massively Parallel Processing) and clusters using commodity or RISC chips representing almost all systems.  The Earth Simulator which held the number 1 position for over 2 years (June 2002 through June 2004) is the only remaining traditional vector system and is now listed at position 74, but newer systems such as Roadrunner have a SIMD (Single Instruction, Multiple Data) aspect similar to vectors.  Another critical aspect of a supercomputer is the communications fabric that ties the nodes together.  This year the fastest machines use either an infiniband fabric or purpose-build connection from companies such as IBM, Cray, SGI, Sun, Quadrics, or Myrinet, while gigabit Ethernet, the most popular interconnect does not appear until position 80 on the list.

*based on the November 2008 Top 500 list.

The DOE Advanced Simulation and Computing (ASC) Program, part of the National Nuclear Security Administration (NNSA) Stockpile Stewardship Program, funds fundamental and applied research and development in high performance computing, to support the broad nuclear security mission through leading edge science-based, computational science. NNSA must assure confidence in the nuclear stockpile, now and in the future, and the computational science activities in ASC are a key part of that mission.

ASC has led the nation and the world in the development and application of tera-scale computing and advanced architectures, including Roadrunner, the world's first Petaflops computing platform, and Blue Gene/L (360 Teraflops), the first platform to achieve the 100 \Teraflops milestone. As a world leader in large scale cluster computing, ASC has helped to foster the emergence of a new commercial market in commodity and advanced cluster architectures for the science, engineering, and information sectors.

The ASC Program focus on the development and integration of predictive science capabilities for stockpile transformation requires a balanced portfolio of activities.  These include development and validation of fundamental physics and engineering models and codes, deployment of the next generation of peta-scale platforms, and sponsorship of the supporting computer science to meet the unique capability and capacity computing needs of stockpile stewardship.

The ASC Academic Alliances Program is a strategic component of the overall ASC Program that brings computational scientists from NNSA Defense Programs National Laboratories (Los Alamos, Lawrence Livermore, and Sandia) together with researchers from a spectrum of leading academic institutions. These collaborations focus on fundamental problems in integrated, multi-scale computational physics to accelerate the development, integration and validation of simulation-based predictive science capabilities for complex, multi-disciplinary problems of national and global interest.

The DOE Office of Science, through its Advanced Scientific Computing Research (ASCR) Program, supports fundamental research in applied mathematics, computer science, and networking – and provides world-class high performance computational networking tools that enable DOE to succeed in its science, energy, environmental remediation, and national security missions.  The primary mission of the ASCR program is to discover, develop, and deploy the computational and networking tools that enable researchers in the scientific disciplines to analyze, model, simulate, and predict complex phenomena important to the Department of Energy.

This program includes the Scientific Discovery through Advanced Computing (SciDAC) Program, which is a set of coordinated investments in all Office of Science mission areas with the goal of achieving breakthrough scientific advances through computer simulation that were impossible using theoretical or laboratory studies alone.  The program also includes the  Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, which provides computing resources and consulting support for a small number of computationally intensive large-scale research projects that make high-impact scientific advances through the use of a substantial allocation of computer time and data storage at DOE high performance computing centers.

DOE laboratories that provide high performance computing for these programs are located at DOE’s Lawrence Berkeley National Laboratory, Argonne National Laboratory, and Oak Ridge National Laboratory.  A major expansion of high performance computing capacity is underway within this program, with new and expanded systems being delivered over the next few years with performance in the Petaflops range.

Last Reviewed: 1/2/2009