U.S. Dept. of Commerce / NOAA / OAR / GFDL *Disclaimer

 

1. SOFTWARE INFRASTRUCTURE FOR SUPPORT
OF SCIENTIFIC MODELING

GOALS


  1.1 LOW-LEVEL SUPPORT FOR MODELING

ACTIVITIES FY00

     1.1.1 Memory Management, Communication, and I/O on Scalable Systems           All GFDL models planned for use on GFDL's next supercomputing platform have been implemented for scalable systems. The implementation was based on a memory management scheme providing domain decomposition on rectilinear grids spanning shared or distributed address spaces. This versatile model of memory management provides simple user interfaces for programming across a variety of scalable architectures, including clusters, distributed memory massively parallel processors (MPPs), symmetric multiprocessors (SMPs), "clusters-of-SMPs", and cache-coherent non-uniform memory access (ccNUMA) shared address space systems.

          The Flexible Modeling System (FMS), as well as the other lab models, use the MPP modules developed at GFDL for memory management, communication, and I/O on scalable systems. These modules provide the message-passing and parallel I/O infrastructure for FMS and other participating lab models, including the gridpoint and spectral atmospheric GCMs, MOM3 and MOM4, and ZETANC, a non-hydrostatic mesoscale atmospheric model.

          The MPP modules consist of three separate F90 programming interfaces: 1) mpp_mod, the low level interface provides basic routines for message passing; 2) mpp_domains, the higher level interface, provides routines for defining domain decompositions and performing halo updates and data transposition across processors; 3) mpp_io, the I/O interface, provides routines for writing output in different formats (including netCDF) from distributed arrays.

          The MPP modules have been stable and operational since 1999. In the last year, they have been validated and benchmarked on a variety of scalable systems, including: Parallel Vector (Cray T90); massively parallel distributed memory (Cray T3E); ccNUMA (Origin 2000 and 3000); SMP clusters (IBM-SP, Sun UE-10000); and Beowulf clusters (SGI-1200, DEC alpha cluster over Myrinet, Intel Pentium cluster on fast ethernet).

     1.1.2 Abstract Parallel Dynamical Kernels

          A recent development within the MPP fabric is the development of abstract parallel dynamical kernels, called the "distributed_grid module". These kernels, coded in Fortran 90, describe physical quantities in terms of vector and scalar field types that contain detailed information about the distribution of the component arrays across a processor set. Operators such as dot and cross products, gradient, divergence, and laplacian can be written in a simple abstract high-level form. The operators will invoke highly optimized kernels for the actual numerics, as well as built-in detection and invocation of halo updates as needed.

          The coding of model numerics in terms of abstract operators has many advantages. In particular, it permits many details of how differencing and averaging are performed on different grid types (e.g., Arakawa B- and C-grid) to be hidden from the user. Thus, it may be invoked for any grid type provided that particular method is available, or can be supplied by the user, for the grid in question.

          Second, since the detection and invocation of halo updates is automatic, halos of differing widths may be used as needed in different parts of the code. For instance, the barotropic solver in the ocean component of FMS is latency-bound. The distributed_grid module will permit the use of wide halos in this portion of the code, so that halo updates can be called less frequently.

          The distributed_grid module has been benchmarked for a shallow water code on the Cray T90 and T3E. While there is some penalty for abstraction (about 20% on 1 processor), it exhibits its strength in its high scalability (80% scaling on 25 PEs on a 125x125 grid on the T3E).

     1.1.3 Interpolation Between Model Grids on Scalable Architectures

          The exchange grid software which is used to do conservative interpolation between model grids in FMS was rewritten to increase performance on scalable architectures. Only minor modifications to external interfaces were required, but communications patterns were changed to reduce bottlenecks. Additional interfaces to support disjoint longitude latitude grids for support of spectral models with hemispheric windows were implemented. Exchange grid software has been ported to NCEP (National Centers for Environmental Prediction) and IRI (International Research Institute for Climate Prediction), where it is being used as part of coupled model developments at those institutions.

PLANS FY01

          Additional optimization of the MPP modules and the exchange grid will be performed as needed to support FMS models on GFDL's new supercomputer.

          The distributed_grid module will be implemented for testing in the B-grid atmospheric model and the MOM4 ocean model.

          Versions of the exchange grid overlap interfaces to support additional grid types required by GFDL modelers will be developed.

  1.2 PHYSICAL PARAMETERIZATIONS, COMPONENT,
AND COUPLED MODELS

ACTIVITIES FY00

     1.2.1 Convective Parameterizations and Diagnostics           A Fortran-90, modular version of the "Relaxed Arawaka Schubert 2" (RAS2) convective parameterization, which includes a downdraft parameterization and an ice phase, has been produced and is being tested in the FMS B-grid model. The same has been done with the full Arakawa-Schubert scheme which was used at GFDL for many years. Versions of the Emanuel convective parameterization that were originally obtained from GFDL FMS implementations are now being used operationally in the Navy's global prediction models.

          The programs used for the tropical storm analysis, as well as the program to compute the Tibaldi/Molteni blocking index, and a number of other analysis programs have been rewritten as streamlined Fortran 90 modules.

     1.2.2 Diagnostic Cloud Model

          A modular version of the Gordon/Slingo diagnostic cloud parameterization scheme has been tested in single column calculations. This parameterization diagnoses cloud fraction for various cloud types, cloud optical depth of non-anvil cirrus, and specifies fixed values of cloud optical depth for other cloud types. The modular version does not yet support the diagnosis of marine stratocumulus cloud fraction and has been modestly revised to enhance scalability to finer vertical resolution. It is currently being tested in FMS in both the B-grid and spectral dynamical cores.

          An empirically based relative humidity threshold cloud parameterization scheme has been designed to closely resemble the diagnostic cloud parameterization used in the Experimental Prediction spectral GCM v197. This scheme calculates cloud fractions diagnostically using relative humidity, vertical velocity, and stability. Seven cloud types and three cloud vertical layer specifications are possible, each with its associated optical properties. A concise, modular scheme written in Fortran-90 has been completed for both the calculation of fractional cloud amounts and the handling of the cloud optical properties. Column tests give very good agreement with the non-modular v197 version.

     1.2.3 Global Atmospheric Grid Point Model

          A number of improvements have been made to the B-grid dynamical core. First, the polar filter can be used with two-dimensional domain decomposition. Second, the vertical differencing scheme has been replaced with an energy and angular-momentum conserving scheme. This scheme is also used by the spectral dynamical core. Third, the horizontal mass flux computations have been modified, reducing the two-grid-interval noise associated with the B-grid configuration so that the grid separation correction term is no longer needed. Fourth, the correction to horizontal diffusion for sloping sigma surfaces has been improved, reducing the problem of spurious precipitation over steep mountains. Fifth, the horizontal borrowing scheme for filling regions of negative tracer concentration has been improved, and lastly, a local filter option was added at the top of the model to remove undesirable modes.

          Evaluation of the B-grid core is underway in several coupled model configurations. Extensive testing has been done using a 2.5 x 2.0 degrees x 18 level version of the core coupled to models providing prescribed SSTs and sea ice. Additional tests have also been made with horizontal resolutions up to 1.25 x 1.0 degrees and vertical resolutions up to 50 levels. Initial testing has begun using a version of the core coupled to MOM3 and using prescribed sea-ice.

     1.2.4 Dynamic/Thermodynamic Sea Ice Model

           Development has continued on GFDL's dynamic/thermodynamic sea ice model (renamed SIS for Sea Ice Simulator). The model was parallelized and the horizontal differencing was generalized to accommodate arbitrary horizontal coordinates. An n-category ice thickness distribution scheme was implemented and a reference document has been written. SIS has been tested with daily NCEP reanalysis forcing over the period 1948-1999, and has also been coupled into an FMS based climate model along with a spectral atmosphere, a MOM3 based ocean model, and the LaD (Land Dynamics) land model.

     1.2.5 High Level Language Support for Coupled Models

          A high level language for model coupling is being designed and implemented (a compiler is being written). This language will allow assignment of expressions on the grid of one model to variables on another without explicit calls to exchange grid routines. A similar capability will exist for the subroutine calls that implement the component models' highest level interfaces. The high level coupling language will greatly facilitate the coupling of models to each other and to datasets.

PLANS FY01

          Design work on a single column model testing facility will be completed and the single column model will become part of official FMS releases.

          Further improvements to the SIS sea ice model will continue along with tests of SIS in a variety of coupled model configurations.

          The high level coupling language compiler will be completed and implemented to facilitate better coupled model design at GFDL.

          A number of enhancements to the FMS diagnostic manager will be made to support more elaborate run-time diagnostics. Among the new features will be diagnostics for limited spatial domains, better spatial averaging facilities, and the ability to output diagnostics for particular times of day.

          Evaluation of the current FMS version of the diagnostic cloud parameterization scheme will continue. The standard parameterization and a new cloud anomaly parameterization for marine stratus cloud fraction will both be incorporated in FMS. Comparative coupled model integrations with the Klein prognostic and Gordon/Slingo diagnostic clouds will be performed.

  1.3 SOFTWARE MANAGEMENT

ACTIVITIES FY00

     1.3.1 FMS Management           The Flexible Modeling System represents the first time that GFDL has embarked upon a software project of a level of complexity warranting formal management. Upon recommendation from a team of scientists involved in FMS, the director of GFDL appointed an Oversight and Planning (OP) Committee for FMS initially under his own chairmanship. The initial act of the newly founded committee was to approve a charter for FMS. The charter calls for model development to be under the direct supervision of its principal working subcommittee, the Model Infrastructure (MI) team. The MI team in turn implements the recommendations of the OP committee. Recent activities of the OP committee include the adoption of a task list mechanism to guide GFDL in assigning activities to contracted software engineers.

     1.3.2 Software Version Control for the Flexible Modeling System

          A significant development within FMS over the past year has been the implementation of an automated software version control system. The need for such a system is evident given the scope of FMS and the number of contributors involved. Modern software version control systems feature detailed history tracking of individual files, the ability to create and track multiple versions (branches) from a single file, the ability to group together files in a logical manner with appropriate labels, and the ability to support a broad remote community through internet-based protocols.

          FMS uses GNU's CVS (Concurrent Versions System) version control system. The FMS source tree is stored in a single CVS repository that is accessible in its entirety to all users at GFDL and accessible in part to external users. Considerable effort has been made to create a repository structure and policy that will serve the needs of FMS well into the future. The repository structure is split functionally into "shared" code, which consists of utilities common to all FMS codes, "component-model" code, which consists of the core codes to run each of the dynamical cores, and "coupler" code, which consists of codes, such as the surface flux calculations and the main drivers, that couple component models together. The repository policy outlines naming conventions, requirements for introducing "branch" or "trunk" code into the repository, and the coordination of the quarterly release schedule of FMS (1.3.3) with the repository.

     1.3.3 FMS Software Releases

          The FMS development team has established a schedule of quarterly releases of the FMS code. The releases provide regular deadlines for FMS contributors to coordinate, review, communicate, and test their recent development efforts. The three-month spacing between releases is sufficiently long to complete a sizeable development project and sufficiently short that the project deadline can slip to the next release without any great loss or undue pressure on the developers involved. The release cycle begins with meetings of the MI team that establish short-term (8 to 10 week) goals and a schedule of development activities. There follows an announcement of the release and its main features, and the creation of a prerelease for testing, three to four weeks before the release deadline. Supporting documentation for the release is created in the days following the official release day, and bug fixes are appended to the release as necessary.

          The releases are named by an alphabetical sequence of city names: Antwerp (5/2000), Bombay (8/2000), Calgary (11/2000), etc. The Antwerp release consisted mainly of parallelized "benchmark" code for the GFDL computer procurement. The Bombay release featured coordinated atmospheric dynamical cores, and the introduction of the MOM3 ocean model, the LaD land surface model, the SIS sea-ice model, and enhanced web documentation.

          In conjunction with the Bombay release, an FMS workshop was held at GFDL to familiarize the laboratory with the capabilities of FMS. One and a half days were devoted to a series of presentations ranging from overviews for casual users to detailed studies of low-level support modules. Representatives from nearly a dozen outside modeling institutions were invited to attend the workshop as observers, and participated in an afternoon session discussing possible ways in which GFDL could interact with the outside world using FMS.

     1.3.4 Web Page and Documentation Support

          A comprehensive web-based documentation system to support FMS is under development. Automatically generated version-specific documentation is created with each checkout of FMS code from the CVS software repository. Comprehensive documentation of updates and modifications to code are available. In addition, static documentation of FMS code and other aspects of the system are under development. Examples include a web-based start-up facility to allow new users to begin running GCMs, and a web-based bulletin board facility.

     1.3.5 Optimization Team, Migration, and Evaluation

          Until recently, FMS efficiency was considered secondary to code correctness and clarity. Now that FMS has matured and a new computer system is to be delivered, optimization issues are moving to center stage. An initial optimization team has been formed and has begun to instrument existing FMS code and document performance on existing platforms.

PLANS FY01

          The primary focus of FMS development will be to optimize FMS models for use on GFDL's new supercomputing system. It is anticipated that a number of simple modifications will lead to significant performance enhancements during the first six months that the new system is available.

          Continued coordination of development of different dynamical cores and component models should increase the amount of shared code and reduce maintenance burdens.

          Improved coordination of run-time scripts and data archiving facilities will be explored to avoid redundant effort between scientific groups in these areas.

          Quarterly releases of FMS will continue. Several major new features are planned for inclusion, including a comprehensive biosphere model, a standardized coupled model, as well as the new radiation code of the atmospheric processes group.

*Portions of this document contain material that has not yet been formally published and may not be quoted or referenced without explicit permission of the author(s).