Data Management and Communications

Central to the success of IOOS is the presence of a Data Management and Communication (DMAC) system capable of delivering real-time, delayed-mode, and historical data for in-situ and remotely-sensed physical, chemical and biological observations.  The system should provide model-generated outputs, including forecasts, to IOOS users and be able to deliver all forms of information to and from secure archive facilities. The DMAC design must also be responsive to user needs.  

Regional IOOS partners are working with the U.S. IOOS Program Office to develop this system, to make data and products discoverable and accessible, and to provide essential metadata regarding information sources, methods and quality.

Technical Information

Data Management and Communication (DMAC)Implementation Plan

  • IOOS DIF Draft Architecture & Recommendations This set of slides includes a partial set of architecture diagrams and provides an overview of the data access services used by the DIF.
  • XML Encoding: IOOS DIF XML Schema beta v0.6.1 - This Geography Markup Language (GML) application schema defines the XML encoding used by the NDBC and CO-OPS SOS to serve in situ data. This encoding is designed to be complete and machine-readable, and may be transformed to simplified representations using code such as the Extensible Styleheet Language Transformation (XSLT) example below.
  • Text encoding: IOOS conventions for comma-separated value (CSV) (PDF, 84kB) and tab-separated value (TSV) (PDF, 95kB) encoding. CSV and TSV are offered by the NDBC SOS and eventually by other IOOS SOS. The CSV is directly readable by common spreadsheet programs. The TSV includes a few specialized column headers to make it more easily readable by the Ocean Data View (ODV) software package.
  • DIF Design Document:(pdf, 1,039kb) This document defines the high-level DIF architecture, standards, conventions, and services to be employed, intersections with NOAA Enterprise Architecture, and risks and mitigation strategies.
  • IOOS Biological Data Terminology and Schema

 
Software

    The following software is provided on an as-is basis with no support or warranty. Read the full disclaimer before downloading.

      Server Code

    • NDBC SOS software - version 1: This software was developed by the National Data Buoy Center (NDBC) to provide a SOS interface to their MySQL database back-end. Offers data in GML, CSV, TSV and KML encodings.
    • GCOOS SOS software beta version 0.6.1(zip, 88kb)This PHP script was modified from the NDBC version to support SQLite database engine. This beta software is still under active development and test.
    • THREDDS Data Server (TDS) is open-source software from Unidata that provides OpenDAP and WCS functionality. It is used by the NDBC and CoastWatch.


      Format Converters

    • IOOS SOS to CSV v0.6.1This XSLT file converts the IOOS SOS v0.6.1 observation values from GML to comma-separated value (CSV) format suitable for use by Excel and other simple applications.
    • netCDF to BUFR converter beta v0.1 This sample application was developed at AOML. It converts data from a netCDF format accepted by DIF to the Binary Universal Form for the Representation of meteorological data (BUFR) format maintained by the World Meteorological Organization (WMO). The archive contains C source code, descriptor file, and brief help on compilation and execution.


      Other Tools

    • Tcl/Tk Example These Tcl/Tk routines are used by the Coastal Inundation customer for retrieving water level observations from the CO-OPS SOS.

    Core DIF Standards

    These are some of the basic standards and specifications adopted by the NOAA IOOS Data Integration Framework.

  • OGC Sensor Observation Service (SOS) specification NOAA IOOS uses this service type to provide access to in-situ oceanographic data in an XML encoding defined by the GML application schema referenced above.
  • OGC Web Coverage Service (WCS) specification NOAA IOOS uses this service type to provide access to gridded remotely sensed data in binary formats such as NetCDF and GeoTIFF.
  • OPeNDAP information This service type is used by NOAA IOOS to provide access to gridded remotely sensed data in binary formats such as NetCDF and GeoTIFF.
  • OGC Web Map Service (WMS) specification This service type will be used to provide georeferenced images of data.

 
Data Management and Communications Documents

  • Guidance for Implementation of the IOOS Data Management and Communications Subsystem (pdf, 39kb)

  • DMAC Plan for Research and Operational Integrated Ocean Observing Systems,
    2005     (pdf, 4.7Mb)

  • IOOS High Level Functional Requirements:(pdf, 168kb)As a part of the IOOS implementation planning effort, the NOAA IOOS Program collected functional requirements from a variety of existing sources and compiled them into this single document. This IOOS High-Level Functional Requirements Document will be used to assist developing more detailed documents as the program focuses its efforts in developing and deploying a National DMAC to serve all IOOS stakeholders.

  • IOOS DMAC Concept of Operations:(pdf, 168kb)This document describes the initial high-level concept of operations (ConOps) for the DMAC subsystem; it is not intended to be an implementation or transition plan. The focus of the document is to define the functions and services that IOOS stakeholders desire the DMAC to perform. It does not address the technology or architecture of how it will perform those functions and services. NOAA is currently running a pilot project, the Data Integration Framework (DIF), to address some of the DMAC functionality and some of the technology issues.

 

Data Integration Framework


Project Documents

  • Data Integration Framework (DIF) Final Assessment Report (pdf, 4.85Mb)

  • IOOS REGIONAL DATA INTEGRATION FRAMEWORK IMPLEMENTATION WORKSHOP:(pdf, 13mb)The Integrated Ocean Observing System (IOOS) Regional Data Integration Framework (DIF) Implementation Workshop was organized with the goal of improving the overall production and execution of IOOS data management across the eleven IOOS Regions. To accomplish the goal of DIF implementation, a set of objectives were developed in the workshop planning process.

  • The NOAA IOOS Data Integration Framework: Initial Implementation Report:Slides (pdf, 315kb) and associated paper (pdf, 125kb) presented at MTS/IEEE Oceans 2008 conference in Quebec, Canada 2008-09-17.

  • Project Plan:(pdf, 168kb)The Project Plan records planning ideas and agreement details, approved scope, cost, and schedule baselines.

  • Functional Requirements:(pdf, 333kb)This document defines the functional requirements of the DIF and is a guide to further system documentation and design.

  • Concept of Operations:(pdf, 410kb)The Concept of Operations (ConOps) document is the translation of the DIF functional requirements into operational scenarios. The ConOps provides important input into the high-level system design.

  • DIF Customer Implementation Project Summary and Performance Assessment Plan:(pdf, 457kb) This document describes the 4 customer implementation projects used to assess the value of DIF's integrated data. Provides overview of benchmarks, metrics, and performance assessment methodology.

 

 

Workshops

IOOS Regional Data Management Workshop, Sept 11-13, 2012

IOOS Regional Associations DMAC Coordination Briefings

  • AOOS (pdf), Rob Bochenek
  • CariCOOS (pdf), Jorge Capella
  • CeNCOOS (pdf), Steven Le and Jennifer Patterson
  • GCOOS (pdf), Matt Howard
  • GLOS (pdf), Tad Slawecki
  • MARACOOS (pdf), Kyle Wilcox
  • NANOOS (pdf), Emilio Mayorga
  • NERACOOS (pdf), Eric Bridger
  • PacIOOS (pdf), Jim Potemra
  • SCCOOS (pdf), Lisa Hazard and Darren Wright
  • SECOORA (pdf), Vembu Subramanian and Jeremy Cothran

IOOS DMAC Workshop Session Presentations

(top)


Interagency Ocean Observation Committee (IOOC)
Data Management and Communications (DMAC) Steering Team Meeting, May 11-12, 2011

On May 11-12, the U.S. Data Management and Communications Steering Team of the Integrated Ocean Observing Committee met to discuss ocean observation data management standards for the nation. The team is comprised of federal members across seven core agencies and regional participants from non-federal organizations and industry. The U.S. Integrated Ocean Observing System (IOOS®), which resides within NOAA, will chair these important deliberations.  

Meeting Agenda, the Federal Register Announcement and notes are availabe at:
http://www.iooc.us/committee-news/dmac/dmac-st-meetings/

 

IOOS Regional Data Management Workshop, April 27 - 28, 2011

Workshop Details

Workshop Theme: Advancing IOOS RA DMAC Interoperability
Workshop Goal: Resolve challenges and issues to advancing interoperability among the eleven IOOS Regional Associations.
Workshop Objectives:

    • Review state of the IOOS Regional Association’s Web Services.

    • Identify and discuss challenges (technical and/or governance) to implementing interoperable Web Services.

    • Coordinate with other IOOS activities


IOOS Regional Data Management Workshop, March 23-24, 2010

Workshop Report (pdf, 11,380kb)


IOOS Regional Data Integration Framwork Implementation Workshop

IOOS Regional Data Integration Framwork Implementation Workshop:(pdf, 13mb)

 

The overall objectives are for IOOS partners to develop and maintain DMAC capabilities to: