Dapper In-situ Data Conventions

Version 0.1 October 2006
Joe Sirott NOAA/PMEL

Introduction

This document describes a set of conventions that were developed by the EPIC group at NOAA/PMEL for remotely accessing in-situ geophysical data through the OPeNDAP network DAP 2.0 protocol (pdf). These converntions are implemented by the Dapper server

Geophysical in-situ data consists of a set of measurements of physical quantities that are sampled at irregular space and time coordinates. Examples include time series of weather data collected by weather stations, CTD measurements of ocean salinity and temperature as a function of depth, and radiosonde measurements of atmospheric temperature and pressure as a function of height.

It is difficult to model in-situ data with the the conventions that already exist for gridded data (e.g. COARDS and CF). We have developed conventions that, we believe, are more natural for in-situ data and we hope that these conventions will facilitate greater sharing of in-situ data in the scientific community.

Data representation

Each Dapper in-situ dataset is represented as a DAP 2.0 two level nested sequence. This sequence consists of one outer and one inner sequence, where the inner sequence is contained by the outer sequence. The outer sequence contains a latitude, longitude, unique identification number, and either a depth or height variable (z) or a time (t) variable. The inner sequence contains all of the measurement variables (temperature, salinity, etc.) and a t variable (if z is defined in the outer sequence), or a z variable (if t is defined in the outer sequence).

For example, time series data from weather stations can be represented as the following OPeNDAP DDS:

Dataset {
    Sequence {
        Float32 lat;
        Float32 lon;
        Float32 elev;
        Int32 _id;
        Sequence {
            Float32 visibility;
            Float32 max_wind_gust;
            Float32 dewp;
            Float64 time;
            Float32 slp;
            Float32 temp;
            Float32 wind_speed;
            Float32 max_temp;
            Float32 max_sustained_wind_speed;
            Float32 min_temp;
            Float32 precip;
        } time_series;
    } location;
    ...
} gsod_time_series;

and CTD measurements can be represented as:

Dataset {
    Sequence {
        Float32 lat;
        Float64 time;
        Float32 lon;
        Int32 _id;
        Sequence {
            Float32 depth;
            Float32 temp;
            Float32 salinity;
            Float32 pressure;
        } profile;
    } location;
    ...
} northAtlantic;

Variables

Dapper uses the COARDS conventions for coordinate and data variable attributes.

Dataset

Each dataset contains an outer sequence and an optional constrained_ranges structure.

Outer sequence

The outer sequence must have have an Int32 variable named _id and this variable must have a unique value for each entry of the outer sequence. The _id variable is used in an OPeNDAP constraint expression to retrieve one or more inner sequences from a server without reading through the entire dataset.
The outer sequence must have two variables that specify the x (longitude) axis and y (latitude) axis of the dataset. These variables are identified by the axis attribute and have values of "X" and "Y" respectively.
The outer sequence must have one of either a z (depth or height ) or t (time) variable. If the outer sequence has a t variable then the inner sequence will have a z variable. If the outer sequence has a z variable, then the inner sequence will have a t variable. The variable is identified by the axis attribute and will have a value of "Z" for a z axis and "T" for a t axis.
The x,y,and z axes must have a Float32 or Float64 type. All of the spatial axes must have the same type.The time axis must always be Float64. For maximum compatibility with existing applications, it is recommended that the time axis be Float64 and the time variable represents time as the number of milliseconds since the Unix epoch (1/1/1970).
The sequence may have an attributes variable (of type Structure) that contains per-profile or per-time series metadata as a set of name-value pairs. For example, a weather station might have a station id and station name:
```
    Structure {
        String station_id;
        String station_name;
    } attributes;
```
All members of the structure must have a String type.
The sequence may have a variable_attributes variable of type Structure that contains per-profile or per-time series metadata for a specific data variable. A structure with the name of the data variable is nested in the this structure and it, in turn, contains the metadata. For instance, the following:
```
    Structure {
        Structure {
            Float64 valid_range[2];
        } depth;
    } variable_attributes;
```
could be used to store the min/max values of each depth variable for a dataset of CTD measurements.

Inner sequence

The inner sequence must have either a z or t axis variable, as determined by the outer sequence.
The inner sequence contains all of the measured variables. Each variable must be either a Float32 or a Float64.

Constrained ranges

The dataset may have a variable that contains the minimum bounding hypercube (in x,y,z,t) of an OPeNDAP constraint expression after the constraint has been applied. The variable is named constrained_ranges and has a Structure type:
```
    Structure {
        Float32 lon_range[2];
        Float32 lat_range[2];
        Float32 depth_range[2];
        Float64 time_range[2];
    } constrained_ranges;
```
The bounds of the x,y,z,and t axes are stored in the lon_range, lat_range, depth_range, and time_range variables respectively, and each of these variable has the same type as the corresponding axis variable.

Global attributes

Dapper global attributes are stored in a NC_GLOBAL attribute structure to maintain compatibility with OPeNDAP to netCDF translators. The following attributes are mandatory:

Attribute	Type	Value	Description
max_profiles_per_request	Int32		Maximum number of time series or profiles that the server will return in one request
total_profiles_in_dataset	Int32		The total count of all the time series or profiles contained in the dataset
Conventions	String	"epic-insitu-1.0"	Specifies that the server conforms to the EPIC/Dapper conventions
lon_range	Float64[2]		Longitude range that contains all of the profiles or time series in the dataset
lat_range	Float64[2]		Latitude range that contains all of the profiles or time series in the dataset
depth_range	Float64[2]		Depth range that contains all of the profiles or time series in the dataset
time_range	Float64[2]		Time range that contains all of the profiles or time series in the dataset

Variable attributes

The Dapper server supports the COARDS conventions for variable attributes (e.g. units, missing_value, long_name, etc.). In addition, each coordinate variable must have a string attribute named axis that has a value of "X","Y","Z",or"T". There should only be one of each value of axis specified in the DAS.

Constraints

The OPeNDAP specification requires that an OPeNDAP server support all constraints on all variables. We have found this to be difficult to implement in practice, especially with large databases. We instead have more relaxed requirements:

Projection constraints

A server must support hyperslab projections, but only on the outer sequence variable.
A server must never return more than max_profiles_per_request rows in the outer sequence. This helps to limit denial of service attacks and/or timeout errors from servers handling large databases.

Selection constraints

A server must support all selection constrains on coordinate variables.
A server must support the equality constraint on the _id variable.
A server may support other constraints on other variables.

Examples

DDS

Following is a full DDS for a large in-situ collection of ocean profiles:

Dataset {
    Sequence {
        Float32 lat;
        Float64 time;
        Float32 lon;
        Int32 _id;
        Sequence {
            Float32 temp_qc;
            Float32 salinity_qc;
            Float32 oxygen;
            Float32 depth;
            Float32 temp;
            Float32 chlorophyll;
            Float32 salinity;
            Float32 oxygen_qc;
            Float32 pressure_qc;
            Float32 pressure;
            Float32 chlorophyll_qc;
        } profile;
        Structure {
            String title;
            String data_origin;
            String source;
            String Conventions;
            String country_code;
            String cruise;
            String cruise_number;
            String cast_number;
        } attributes;
        Structure {
            Structure {
                Float32 valid_range[2];
            } depth;
        } variable_attributes;
    } location;
    Structure {
        Float32 lon_range[2];
        Float32 lat_range[2];
        Float32 depth_range[2];
        Float64 time_range[2];
    } constrained_ranges;
} northPacific;

DAS

Following is the DAS for a large in-situ collection of ocean profiles:

Attributes {
    NC_GLOBAL {
        Int32 max_profiles_per_request 15000;
        Int32 total_profiles_in_dataset 1707616;
        String version "1.1.0";
        String owner "";
        String contact "";
        String Conventions "epic-insitu-1.0";
        Float64 lon_range 100.0, 289.180000305176;
        Float64 lat_range 1.20000004244503E-5, 90.0;
        Float64 depth_range 0.0, 10860.900390625;
        Float64 time_range -1.1432412E12, 1.10778048E12;
    }
    location._id {
        String long_name "sequence id";
        Int32 missing_value 2147483647;
        String units "";
    }
    location.lat {
        String units "degree_north";
        String long_name "latitude";
        Float32 missing_value NaN;
        String axis "Y";
    }
    location.profile.temp_qc {
        String units "";
        String long_name "temperature quality flag";
        Float32 missing_value NaN;
    }
    location.profile.salinity_qc {
        String units "";
        String long_name "salinity quality flag";
        Float32 missing_value NaN;
    }
    location.time {
        String units "msec since 1970-01-01 00:00:00 GMT";
        String long_name "time";
        Float64 missing_value NaN;
        String axis "T";
    }
    location.profile.oxygen {
        String units "ml/l";
        String long_name "oxygen";
        Float32 missing_value NaN;
    }
    location.profile.depth {
        String units "m";
        String long_name "depth";
        Float32 missing_value NaN;
        String axis "Z";
    }
    location.lon {
        String units "degree_east";
        String long_name "longitude";
        Float32 missing_value NaN;
        String axis "X";
    }
    location.profile.temp {
        String units "degc";
        String long_name "temperature";
        Float32 missing_value NaN;
    }
    location.profile.chlorophyll {
        String units "ug/l";
        String long_name "total chlorophyll";
        Float32 missing_value NaN;
    }
    location.profile.salinity {
        String units "psu";
        String long_name "salinity";
        Float32 missing_value NaN;
    }
    location.profile.oxygen_qc {
        String units "";
        String long_name "oxygen quality flag";
        Float32 missing_value NaN;
    }
    location.profile.pressure_qc {
        String units "";
        String long_name "pressure quality flag";
        Float32 missing_value NaN;
    }
    location.profile.pressure {
        String units "dbar";
        String long_name "pressure";
        Float32 missing_value NaN;
    }
    location.profile.chlorophyll_qc {
        String units "";
        String long_name "total chlorophyll quality flag";
        Float32 missing_value NaN;
    }
}