Data Format StandardsThe following table lists data formats that have been approved for use and officially endorsed by the NASA Earth Science Division (ESD) for use in Earth science data systems. The full list of NASA ESD approved standards and technical notes, as well as information on the standards submission and approval process, is available on the Standards Process Group (SPG) page.
The following table lists data formats that are approved for use in Earth science data systems by the NASA Earth Science Data and Information Systems (ESDIS) Project but have not been officially endorsed by the NASA Earth Science Division (ESD).
Data FormatsHDFNASA ESD-approved Standards NASA ESD Technical Note The Hierarchical Data Format (HDF) is designed to facilitate managing and sharing scientific data. HDF includes two formats (HDF4 and HDF5), software for accessing data in HDF formats, and applications for working with HDF data. HDF is designed for efficient storage and access of high volume, complex data, and for mixing varieties of data types in a single container. HDF libraries are used to read and write data, to define data types and structures for applications, and to control how data is stored. HDF applications include commercial and free software for viewing, creating, comparing, searching, analyzing and visualizing HDF data, and for converting between HDF and other formats. There are specialized libraries for HDF in application domains. These libraries promote the standard use of HDF, enabling data consumers to more easily share their data and applications. Some libraries, such as HDF-EOS, are broad in scope, and support a very wide range of applications. For more information about HDF as a scientific data format, see http://hdfgroup.org. HDF-EOSNASA ESD-approved Standard NASA ESD Technical Notes Hierarchical Data Format for the Earth Observing System (HDF-EOS) is NASA’s primary format for standard data products derived from EOS instruments. Because many Earth science data structures need to be geolocated, NASA developed the HDF-EOS format with additional conventions and data types for HDF files. There are two versions of HDF-EOS: HDF-EOS2 and HDF-EOS5. HDF-EOS2 uses HDF4 and HDF-EOS5 uses HDF5. HDF-EOS2 and HDF-EOS5 support three geospatial data types (grid, point, swath) and HDF-EOS5 also supports a “Zonal Average” datatype. HDF-EOS provides uniform access to diverse data types in a geospatial context. The HDF-EOS software libraries allow a user to query or subset the contents of a file by Earth coordinates and time if there is a spatial dimension in the data. HDF-EOS also provides a container for EOS inventory, archive and product specific metadata. HDF-EOS2 is used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments. HDF-EOS5 is used by EOS Aura instruments. Tools that process standard HDF files will also read HDF-EOS files; however, standard HDF library calls cannot access geolocation data, time data, and product metadata as easily as with HDF-EOS library calls. For an overview of data tools, see Section 5. For more information on HDF-EOS, see http://www.hdfeos.org. netCDFNASA ESD-approved Standards The network Common Data Form (netCDF) is an interface for array-oriented data access and a freely distributed collection of software libraries for C, FORTRAN, C++, Java, and Perl that provide implementations of the interface. The netCDF software was developed at the Unidata Program Center in Boulder, Colorado, and augmented by contributions from other netCDF users. The netCDF libraries define a machine-independent format for representing scientific data. Together, the interface, libraries, and format support the creation, access, and sharing of scientific data. For more information or to obtain netCDF software, see http://www.unidata.ucar.edu/software/netcdf. (The above information on netCDF was taken from the Unidata Web site.) ICARTTNASA ESD-approved Standard The International Consortium for Atmospheric Research on Transport and Transformation (ICARTT) file format was initially developed as an airborne data file standard format to satisfy the data archival and exchange needs of a multi-agency, multi-national group of researchers conducting coordinated observations in pursuit of a common research goal. Thus, the design of the ICARTT format was specifically tailored to accommodate airborne observations and arose from a consensus established across the atmospheric chemistry community. The success of the ICARTT study propelled much wider use of the ICARTT data format for subsequent airborne campaigns. Most recently, the ICARTT file format was adopted for use by international atmospheric chemistry community to support observations during the International Polar Year for the Polar Study using Aircraft, Remote Sensing, Surface Measurements and Models, of Climate, Chemistry, Aerosols, and Transport (PolarCAT), including NASA’s component - Arctic Research of the Composition of the Tropospheric from Aircraft and Satellites (ARCTAS). However, the ICARTT file format is not suitable for large 3-dimensional data sets. Also, while the file format includes metadata specification, the metadata fields themselves do not follow any other standards that may limit its use in large automated systems. GeoTIFFNASA SPG Heritage Standard GeoTIFF implements the geographic metadata formally, using compliant TIFF tags and structures. GeoTIFF refers to TIFF files which have geographic (or cartographic) data embedded as tags within the TIFF file. The geographic data can then be used to position the image in the correct location and geometry on the screen of a geographic information display. GeoTIFF is a metadata format, which provides geographic information to associate with the image data. But the TIFF file structure allows both the metadata and the image data to be encoded into the same file. GeoTIFF makes use of a public tag structure which is platform interoperable between any and all GeoTIFF-savvy readers. GIS, CAD, image processing, desktop mapping and any other types of systems using geographic images can read any GeoTIFF files created on any system to the GeoTIFF specification. Data Format Conventions - Lessons LearnedNASA ESD Technical Notes This technical note describes the creation of a common file format developed and used by the individual teams working on the four instruments on NASA’s Aura satellite (HIRDLS, MLS, OMI and TES). Each of these teams was independent and there was no mandate that they all use a common file format. The decision to do so and the implementation of it was a grassroots effort that was accepted by all of the PIs and instrument teams’ leading scientists. This document describes the process used in developing the guidelines and the keys to its success. It covers how the team came together; the processes used for developing, refining, documenting and verifying the guidelines; summarizes key considerations for developing similar guidelines; and documents lessons learned. The teams agreed upon the names, data types and dimension order of fields. There was also agreement regarding the file-, group- and field-level attributes to include in each product file. A file-naming convention was also adopted. Future NASA missions can use this technical note to learn from the process that the Aura instrument teams went through to develop their own set of guidelines. The process can be adopted and modified as needed. |
