ESRL Integrating Research and Technology Themes

Building a Service-based Grid Computing Infrastructure

  1. Goal and Scope
    2. Thumbnail of Supercomputing 2-pager.

    "Supercomputing at ESRL. (pdf)

    Computer facilities are expensive to build and maintain, so being able to share cycles across the Earth System Research Laboratory (ESRL) represents a more effective way to utilize IT resources. In addition, laboratory requirements for computing and data storage are growing rapidly in response to the increased flow and availability of data through its facilities to support (1) increasingly complex models, (2) new areas of research such as ensembles, data assimilation, and coupled models, and (3) the creation and integration of new high resolution datasets coming from next-generation observational platforms. The widespread acceptance and use of the web, coupled with recent advancements in distributed computing offer tremendous benefit to NOAA and ESRL if fully explored and developed.

    Distributed computing encompasses both web services and grid computing. Web services can be simple (authorization, searching, naming, registry) or complex, combining multiple services into a complex service that encompasses the comprehensive requirements of an application. Creating services to accomplish common tasks allows an organization to reduce the effort required to develop, port, and maintain its hardware and software systems. Grid computing extends the web services concept by providing task and resource management functions across heterogeneous computing environments. Compute grids can be deployed at a local level or encompass computing facilities around the nation. Data grids can provide an "umbrella" technology to link ESRL data into a more coherent and integrated system. Grid and web services can be built using commercially available tools; no research or development of grid or web tools will be done.

  2. Rationale and Payoffs

    3. NOAA's IT infrastructure is currently a set of independent systems that were developed over time to support its diverse programs and requirements. NOAA must integrate its computational resources to reduce costs, to improve systems utilization, to support new scientific challenges and to run and verify increasingly complex models using next generation high-density data streams. NOAA needs integrated IT systems capable of handling the rapid increase in data volumes from expected launches of GOES-R, NPOESS, and the Earth Observation System.

    ESRL can develop a secure, flexible, robust, adaptable IT infrastructure, based on grid and web technologies that showcase what is possible for the larger NOAA community.

    Benefits include:

    • development of a coordinated data management structure which permits research groups within the laboratory to create, locate and share data
    • incorporation of technology such as grid meta-schedulers to permit scientists to share compute resources more effectively
    • cost savings enabled by sharing resources, better collaborations, improved efficiency, and adaptable to changing demands and requirements of the organization

  3. Major Collaborators and Their Research Foci
    1. Earth System Research Laboratory
      2. Development
      • Physical Sciences Division: is investigating web services and service registries necessary for building a web-based infrastructure at ESRL.
      • Global Systems Division: is exploring both web services and grid computing in areas including meta-scheduling and portal development with GFDL and PMEL.
        • Users
      • Chemical Sciences Division: uses the WRF and MM5 models for atmospheric chemistry research and prediction and will benefit by the availability of a web / grid infrastructure.
      • Global Systems Division: strong involvement in the development and testing of WRF and has a critical need for significant network, data and compute resources.
    2. Other NOAA
      • NCEP/EMC: coordinated modeling efforts on WRF development require the large -scale compute and data resources at the Boulder Laboratories. The creation of a grid infrastructure and web services can simplify and coordinate mutually beneficial activities.
      • NESDIS/NCDC: The NOMADS project archives model data that can be used to initialize WRF and other models. NOMADS and FSL have been exploring opportunities to make this data available via web protocols including OpenDAP and web services.
      • NESDIS/NGDC: will provide direct and local access to satellite and gridded data via the CLASS project, and is interested in collaborating with ESRL in a NOAA grid.
      • NOAA CIO Office: Unified NOAA design and application of information systems.
      • NOS and NMFS: Design and application of models to marine systems.
    3. Others
      • CIRA: works in collaboration with ESRL staff to build web and grid capabilities and to utilize these advancements in scientific research and development.
      • NCAR: The WRF development community has strong interactions with FSL on WRF portal development and the creation of a virtual grid computing environment to be used by the Developmental Testbed Center (DTC).
      • AFWA: participates in, and could make resources available for DTC activities.
      • NSF / LEAD project: Is collaborating with FSL on the development of a web-based grid- enabled portal for WRF modeling activities.
  4. Contributions to NOAA Goals
    • NOAA Strategic Plan FY 2005 - FY 2010. Cross Cutting Priority 1: (Integrated Global Environmental Observation and Data Management System) focuses on developing a global-to-local environmental observation and data management systems for the comprehensive, continuous monitoring of coupled ocean / atmospheric / land systems that enhance NOAA's ability to protect lives, property, expand economic opportunities, understand climate variability, and promote healthy ecosystems.
  5. Major Information Products, Customers, and Linkages

    6. The Grid: an IT Infrastructure for NOAA in the 21st Century, A white paper on grid computing and its relevance to NOAA, Govett, et al, 2004.