AIRS

Frequently Asked Questions


Back to FAQ list

Q.  Is there a tool that creates NetCDF files from HDF or is there a way to
get AIRS data in NetCDF format?  

A.  Look at the URL

http://hdfeos.net/software.php

Under HDF-EOS Tools there is an entry
# hdfeos-netcdf converts HDF-EOS4/5 files to netCDF
#Detailed description: hdfeos2netcdf converts HDF-EOS4/5 files to
netCDF. hdfeos2netcdf takes two file name arguments: the name of the
HDF-EOS5 input file and the name of the netCDF output file. If the
converter encounters an error it can't recover from, it prints an
error message and exits; otherwise, it creates the output file and
exits quietly.

The download link points to

http://hdfeos.net/software/hdfeos-netcdf/v1_1/

The date of this tool is Oct 4, 2006. We have had no experience with
it. The contents of the tar file is a set of C modules with a make
file.

The README is:

hdfeos2netcdf - Converts HDF-EOS4/5 files to netCDF

Installing

Prerequisites

This program uses both the HDF-EOS4 and HDF-EOS5 libraries
(which require the HDF4 and HDF5 libraries, respectively)
and the netCDF libraries, so they must all be installed before
trying to build this program. The HDF-EOS and HDF libraries can be
obtained from:


http://hdfeos.gsfc.nasa.gov

netCDF can be obtained from:

http://www.unidata.ucar.edu/packages/netcdf/

NOTE: The HDF4 library has partial netCDF compatibility code in it.
This code is incompatible with the full netCDF library. For this
application,
the HDF4 library must be built with the option -DHAVE_NETCDF. See the
HDF4
INSTALL documentation for how to do this. Also, the netCDF code in
HDF4 is
based on a relatively old version (2.x) of netCDF, and has
incompatibilites
with more recent (3.x) version of netCDF at the website above. They
can be
fixed by applying the hdf4patch patch file in the converter source
files.


Building

Once the libraries are installed, hand-edit the Makefile and change
the values of:

HDF - the directory where HDF4 is installed

HE2 - the directory where HDF-EOS4 is installed

HDF5 - the directory where HDF5 is installed

HE5 - the directory where HDF-EOS5 is installed

NETCDF_ROOT - the directory where netCDF is installed, probably
/usr/local
or the libsrc subdirectory of your netCDF distribution
directory


Once all the preliminaries above are complete, you should be able to
build the program by typing 'make'. The resulting binary
hdfeos2netcdf depends only on the libraries mentioned above - it can
be freely copied and installed anywhere that can access those
libraries. The program can optionally be built with verbose tracing
enabled (potentially useful if the program appears to be misbehaving);
do that by saying 'make -DTRACE_OUTPUT=1'.

Converting

hdfeos2netcdf takes two file name arguments: the name of the HDF-EOS5
input file and the name of the netCDF output file. If the converter
encounters an error it can't recover from, it prints an error message
and exits; otherwise, it creates the output file and exits quietly.

How it Works

Overall Structure

HDF-EOS files contain {grids/swaths} which contain fields;
fields are multi-dimensional data containers.

Attributes are strings of characters or vectors of numbers;
files and {grids/swaths} and fields may have attributes.

Dimensions are properties of {grids/swaths}.


netCDF files contain variables;
variables are multi-dimensional data containers.

Attributes are vectors of characters or numbers;
files and variables may have attributes.

Dimensions are properties of files.

The main problem in converting HDF-EOS files to netCDF is that extra
level of structure in HDF-EOS files: a netCDF file is roughly
equivalent to a single HDF-EOS swath or grid. This leaves two basic
choices when converting an HDF-EOS file:

* Have one netCDF output file for each swath or grid.

* Have one netCDF output file, but preserve the swath or grid level by
encoding its name into the variable and dimension names.

The first approach is simpler, but can require data duplication which
might make the total size of the output file to be much larger than
the original input file.

The second approach is more complex, and may confuse some netCDF
software (which might recognize a dimension named "longitude", but
almost certainly won't recognize "Grid34_longitude").

Both approaches are defensible choices. This converter uses the
second approach, partly for portability reasons - if multiple output
files are allowed, the program must seperate them by, say, encoding
the swath or grid name into their file names, a platform-dependent
operation.

Specific Swath/Grid Conversion Problems

HDF-EOS swaths and grids have both data and geolocation metadata, so
to produce a useful result, a converter has to somehow translate both
into a netCDF equivalent.

Swath files have data fields and geolocation fields. Geolocation
fields are N-dimensional arrays whose dimensions are mapped to
corresponding dimensions of data fields. In the simplest case, the
mapped dimensions are the same size, so a geolocation field's entries
map one-to-one to data field entries. In more complex cases, the
dimensions are different sizes, and entry offset and increment
information is also required to map a data field entry to a
corresponding geolocation entry.

netCDF has no native notion of mappings between data variables and
geolocation variables. The CF conventions have a way of doing this
using a "coordinates" attribute, but it only really works when the
data and geolocation variables are the same size, which is rare for
HDF-EOS Swath datasets. The converter currently compromises by
preserving the data and geolocation fields as netCDF variables,
setting the appropriate "coordinate" attribute to relate the
variables,
and recording the dimension mapping metadata in file properties.
Generating a real CFgeolocation variable would be better, but would
require multi-dimensional interpolation/resampling

Grid files are both easier and harder to translate. They're simpler
in that they have a guaranteed mapping to a rectalinear geolocation
grid, which can be translated to netCDF as an appropriate coordinate
variable, at a small cost in storage - HDF-EOS Grid files just store
mappings for the upper left and lower right corners of a grid, and
support the obvious linear mapping, whereas to get that mapping in a
netCDF file it is necessary to store a vector the size of the entire
dimension.

They're harder to translate in that the geolocation dimensions have
generic names of "XDim" and "YDim". To find out what those dimensions
really mean, HDF-EOS has a gridinfo API call that reveals which
projection is in use (lat/long, UTM, etc.). The converter tries to
change the dimension names on output to names likely to be meaningful
to netCDF software ("latitude"/"longitude" for geographic projections,
"northing"/"easting" for UTM projections, etc.). It can't do this
100% of the time, because it is possible for grids to have different
sizes and shapes, but to use the same projection. To get around this,
the converter names the first set of dimensions for a given projection
in the obvious manner ("latitude", "longitude") and remaining sets get
names with the name of the grid prepended ("Grid123_latitude",
"Grid123_longitude").

Back to FAQ list