Satellite Data Formats and Compression

GeoTIFF

The GeoTIFF* format was chosen as one of our standards, primarily for its ability to bring image-oriented data into GIS systems. GeoTIFF is merely a variant of standard TIFF, with some additional tags to include georeferencing information. This information can be utilized by some programs, notably GIS software, to properly place the data with respect to other georeferenced objects or "layers". Other applications will simply ignore the GeoTIFF tags and display the images as if they were regular TIFF. Additionally, unlike some image formats, the original data remain intact within the TIFF image and can be loaded and analyzed in various data analysis environments like Matlab and IDL.

HDF4-Scientific Data Set

In addition to GeoTIFF, the Satellite Oceanography Group employs the Hierarchical Data Format Version 4 (HDF4) Scientific Data Set (SDS) model. This HDF4-SDS format is extremely useful for large datasets, as it permits internal metadata tags, inclusion of multiple data layers, and perhaps most importantly an internal compression scheme known as tiling, which breaks the large data set into individually compressed pieces. These pieces can be individually decompressed, resulting in far greater access speeds when selecting subsetted regions from the global fields. Many software tools exist for working with HDF data. Please see a very nice list at the HDF Group web site: http://www.hdfgroup.org/tools.html*. Currently, our AVHRR Pathfinder Version SST data are provided in HDF4-SDS.

netCDF-3

Similar to HDF, the network Common Data Form (netCDF) file format allows for multiple data layers with internal metadata tags. Maintenance of the format and a full description of netCDF are provided by Unidata*. All of our GHRSST data are currently provided in netCDF-3.

Gzip and Bzip2 Compression

Because of the large size of many high resolution and global datasets, they are often compressed using gzip or bzip2. Compressing with gzip adds the ".gz" extension to each of the file names while bzip2 adds ".bz2". These are free programs to efficiently compress files.

Gzip is developed by GNU*. Executables to compress and decompress "gzipped" files are available for a wide variety of operating systems, including UNIX and Linux platforms, Windows machines, and Macs. Please see the gzip home page* for more information, including access to the source code, executables, user manuals, etc. Other programs are also capable of compressing and decompressing gzipped files. PowerArchiver 6.1* (freeware), PKZIP*, and WinZip* are three such programs. Decompressing a gzipped file is very straightforward. On UNIX/Linux systems, simply "gunzip filename.gz". On other platforms, once you have installed the software, simply double-clicking the filename should launch the proper application.