The NOAA GEO-IDE UAF ERDDAP
Easier access to all of NOAA's data |
Brought to you by NOAA NMFS SWFSC ERD |
The URL specifies what you want: the dataset, a description of the graph or the subset of the data, and the file
type for the response.
griddap request URLs must be in the form
http://upwell.pfeg.noaa.gov/erddap/griddap/datasetID.fileType{?query}
For example,
http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.htmlTable?sst[(2007-10-21T00:00:00)][0][(-75):100:(75)][(180):100:(360)]
Thus, the query is often a variable name (e.g., sst),
followed by a set of [] for each of the variable's dimensions
(for example, [time][altitude][latitude][longitude]),
with your desired (start):stride:(stop) values within each [].
Details:
Data fileTypes | Description | Info | Example |
---|---|---|---|
.asc | View OPeNDAP-style comma-separated ASCII text. | info | example |
.csv | Download a comma-separated ASCII text table (line 1: names; line 2: units; ISO 8601 times). | info | example |
.csvp | Download a .csv file with line 1: name (units). Times are ISO 8601 strings. | info | example |
.das | View the data's metadata via an OPeNDAP Dataset Attribute Structure (DAS). | info | example |
.dds | View the data's structure via an OPeNDAP Dataset Descriptor Structure (DDS). | info | example |
.dods | OPeNDAP clients use this to download the data in the DODS binary format. | info | example |
.esriAscii | Download an ESRI ASCII file (for lat lon data only; lon must be all below or all above 180). | info | example |
.fgdc | View the data's FGDC .xml metadata. | info | example |
.graph | View a Make A Graph web page. | info | example |
.help | View a web page with a description of griddap. | info | example |
.html | View an OPeNDAP-style HTML Data Access Form. | info | example |
.htmlTable | View a .html web page with the data in a table. Times are ISO 8601 strings. | info | example |
.iso19115 | View the data's ISO 19115-2/19139 .xml metadata. | info | example |
.json | View a table-like JSON file (missing value = 'null'; times are ISO 8601 strings). | info | example |
.mat | Download a MATLAB binary file. | info | example |
.nc | Download a NetCDF-3 binary file with COARDS/CF/THREDDS metadata. | info | example |
.ncHeader | View the header (the metadata) for the NetCDF-3 file. | info | example |
.odvTxt | Download time,latitude,longitude,otherVariables as an ODV Generic Spreadsheet File (.txt). | info | example |
.tsv | Download a tab-separated ASCII text table (line 1: names; line 2: units; ISO 8601 times). | info | example |
.tsvp | Download a .tsv file with line 1: name (units). Times are ISO 8601 strings. | info | example |
.xhtml | View an XHTML (XML) file with the data in a table. Times are ISO 8601 strings. | info | example |
ArcGIS
.esriAsc
- ArcGIS is a family of Geographical Information Systems (GIS) products from ESRI:
ArcView, ArcEditor, and ArcInfo.
If you have ArcGIS 10 or higher, you can download a subset of a dataset in a .nc file and open it in ArcGIS.
If you have ArcGIS 9.x or lower:
in ERDDAP, choose the .esriAscii file type to save a latitude longitude subset of data for just one point in time.
The file's extension will be .asc. This file format was designed by ESRI to transfer coverage data between computers.
Then, import the data file into your ArcGIS program:
Shapefiles - Sorry, ERDDAP currently does not distribute grid data as shapefiles.
Ferret
is a free program for visualizing and analyzing large and complex gridded
datasets. Ferret should work well with all datasets in griddap since griddap is
fully compatible with OPeNDAP. See the
Ferret documentation.
Note that the griddap dataset's OPeNDAP base URL that you use with Ferret's
set data, for example, http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day ,
won't ever have a file extension at the end.
IDL -
IDL is a commercial scientific data visualization program. To get data from ERDDAP
into IDL, first use ERDDAP to select a subset of data and download a .nc file.
Then, use these
instructions
to import the data from the .nc file into IDL.
JSON .json
files are widely used to transfer data to JavaScript scripts running on web pages.
Griddap will format the data in a flat, table-like structure in the .json file.
JSONP
.json
- Requests for .json files may now include an optional jsonp request
by adding &.jsonp=functionName to the end of the query.
Basically, this just tells ERDDAP to add functionName( to the beginning of the
response and ")" to the end of the response.
If originally there was no query, leave off the "&" in your query.
MATLAB
.mat
users can use griddap's .mat file type to download data from within MATLAB.
Here is a one line example:
load(urlwrite('http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.mat?sst[(2007-10-21)][0][(-75):50:(75)][(180):50:(360)]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=Rainbow|C|Linear|0|32|', 'test.mat'));The data will be in a MATLAB structure. The structure's name will be the datasetID
imagesc(erdBAssta5day.longitude, erdBAssta5day.latitude, squeeze(erdBAssta5day.sst), [0 32]) set(gca, 'YDir', 'normal')The numbers at the end of the first line specify the range for the color mapping.
There are also Matlab
Xtractomatic scripts for ERDDAP,
which are particularly useful for
getting environmental data related to points along an animal's track (e.g.,
GTOPP data).
ERDDAP stores datetime values in .mat files as "seconds since 1970-01-01T00:00:00Z".
To display one of these values as a String in Matlab, you can use, e.g.,
datastr(cwwcNDBCMet.time(1)/86400 + 719529)
86400 converts ERDDAP's "seconds since" to Matlab's "days since". 719529 converts
ERDDAP's base time of "1970-01-01T00:00:00Z" to Matlab's "0000-01-00T00:00:00Z".
NetCDF
.nc
- Requests for .nc files return the requested subset of the dataset in a
standard, version 3, 32-bit, .nc file.
If you are using
NetCDF-Java,
don't try to directly access an ERDDAP dataset or subset
as a .nc file. (It won't work, mostly because that .nc file isn't a static, persistent file. It is a
virtual file.) Instead, use one of these two options:
In both cases, you can then do what you want with the nc object, for example,
request metadata or request a subset of a variable's data.
.ncHeader
- Requests for .ncHeader files will return the header information (text) that
would be generated if you used
ncdump -h fileName
on the corresponding .nc file.
Ocean Data View .odvTxt
- ODV users can download data in a
ODV Generic Spreadsheet Format .txt file
by requesting griddap's .odvTxt fileType.
The dataset MUST include longitude, latitude, and time dimensions.
Any longitude values (0 to 360, or -180 to 180) are fine.
After saving the resulting file (with the extension .txt) in your computer:
OPeNDAP Libraries - Since ERDDAP is an
OPeNDAP-compatible data server,
you can use
any OPeNDAP client library, such as
NetCDF-Java, NetCDF-C, NetCDF-Fortran, NetCDF-Perl,
Java-DAP2,
Ferret, or the
Pydap Client,
to get data from an ERDDAP griddap dataset.
When creating the initial connection to an ERDDAP griddap dataset from any OPeNDAP library:
For example, with the NetCDF-Java library, you can use:
NetcdfFile nc = NetcdfDataset.openFile("http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day", null);
(don't use NetcdfFile.open; it is for local files only) or
NetcdfDataset nc = NetcdfDataset.openDataset("http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day");
(NetcdfFiles are a lower level approach than NetcdfDatasets. It is your choice.)
Once you have the nc object, you can request metadata or a subset of a
variable's data.
Pydap Client
users
can access griddap datasets via ERDDAP's standard OPeNDAP services.
See the
Pydap Client instructions for accessing gridded data.
Note that the name of a dataset in ERDDAP will always be a single word,
e.g., erdBAssta5day in the OPeNDAP dataset URL
http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day
and won't ever have a file extension (unlike, for example, .nc for the
sample dataset in the Pydap instructions).
R Statistical Package -
R is an open source statistical package for many operating systems.
In R, you can download a NetCDF version 3 .nc file from ERDDAP. For example:
download.file(url="http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.nc?sst[(2007-10-21)][0][(-75):50:(75)][(180):50:(360)]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=Rainbow|C|Linear|0|32|", destfile="/home/bsimons/test.nc")Then import data from that .nc file into R with the RNetCDF, ncdf, or ncdf4 packages available
download.file(url="http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.csv?sst[(2007-10-21)][0][(-75):50:(75)][(180):50:(360)]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=Rainbow|C|Linear|0|32|", destfile="/home/bsimons/test.csv") test<-read.csv(file="/home/bsimons/test.csv")There are also R Xtractomatic scripts for ERDDAP, which are particularly useful for getting
Making an Image File with a Graph or Map of Gridded Data
If a griddap request URL specifies a subset of data which is suitable for making
a graph or a map, and the fileType is an image fileType, griddap will return an image
with a graph or map.
griddap request URLs can include optional graphics commands which let you
customize the graph or map.
As with other griddap request URLs, you can create these URLs by hand or have a
computer program do it. Or, you can use the Make A Graph web pages, which simplify
creating these URLs (see the "graph" links in the table of
griddap datasets).
The fileType options for downloading images of graphs and maps of grid data are:
Image fileTypes | Description | Info | Example |
---|---|---|---|
.geotif | View a GeoTIFF .tif image file (for lat lon data only; lon must be all below or all above 180). | info | example |
.kml | View a Google Earth .kml file (for lat, lon data only). | info | example |
.smallPdf | View a small .pdf image file with a graph or map. | info | example |
View a standard, medium-sized .pdf image file with a graph or map. | info | example | |
.largePdf | View a large .pdf image file with a graph or map. | info | example |
.smallPng | View a small .png image file with a graph or map. | info | example |
.png | View a standard, medium-sized .png image file with a graph or map. | info | example |
.largePng | View a large .png image file with a graph or map. | info | example |
.transparentPng | View a .png image file (just the data, without axes, landmask, or legend). | info | example |
Image Size - ".small" and ".large" were ERDDAP's original system for making
different-sized images. Now, for .png and .transparentPng images (not other
image file types), you can also use the
&.size=width|height
parameter to request
an image of any size.
.transparentPng - The .transparentPng file type will make a graph or map without
the graph axes, landmask, or legend, and with a transparent (not opaque white)
background. This option can be used for any type of graph or map.
When &.draw= is set to anything other than surface, the default image size
is 360x360 pixels. Use the &.size=width|height parameter
to specify a different size.
When &.draw= is set to surface, ERDDAP makes an image where each
data point becomes one pixel, which may result in a huge image. If the request
takes too long or fails (perhaps for an odd apparent reason, like a Proxy Error)
either use a stride (see below) value greater than 1 (e.g., 5) for the x and y axis
variables, or use the
&.size=width|height parameter
to restrict the image size,
so that the image isn't huge.
Incompatibilities
Some results file types have restrictions. For example, Google Earth .kml is only
appropriate for results with longitude and latitude values. If a given request is
incompatible with the requested file type, griddap throws an error.
Command Line Downloads with curl
If you want to download a series of files from ERDDAP, you don't have to request each file's
ERDDAP URL in your browser, sitting and waiting for each file to download.
If you are comfortable writing computer programs (e.g., with C, Java, Python, Matlab, r)
you can write a program with a loop that imports all of the desired data files.
Or, if are comfortable with command line programs (just running a program, or using bash or tcsh
scripts in Linux or Mac OS X, or batch files in Windows), you can use curl to save results files
from ERDDAP into files on your hard drive, without using a browser or writing a computer program.
ERDDAP+curl is amazingly powerful and allows you to use ERDDAP in many new ways.
On Linux or Mac OS X, curl is probably already installed as /usr/bin/curl.
On Windows, or if your computer doesn't have curl already, you need to
download curl
and install it. To get to a command line in Windows, use "Start : Run" and type in "cmd".
("Win32 - Generic, Win32, binary (without SSL)" worked for me on Windows XP and Windows 7.)
Instructions for using curl are on the
curl man page and in this
curl tutorial.
But here is a quick tutorial related to using curl with ERDDAP:
curl -g "http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.png?sst[%282010-09-01T12:00:00Z%29][][][]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=|||||" -o BAssta5day20100901.pngThe erddapUrl must be percent encoded.
curl "http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.png?sst%5B%282010-09-[01-05]T12:00:00Z%29%5D%5B%5D%5B%5D%5B%5D&.draw=surface&.vars=longitude|latitude|sst&.colorBar=|||||" -o BAssta5day201009#1.png
Using [start:stride:stop]
When requesting dimension (axis) variables or data variables, the query may
specify a subset of a given dimension by identifying the [start{{:stride}:stop}]
indices for that dimension.
Or, if you change the fileType in the URL from .png to .graph, you can see a Make A Graph
web page with that request loaded:
http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.graph?sst[(2007-07-01):(2007-10-21)][0][(29)][(225)]&.draw=linesAndMarkers&.vars=time|sst|&.marker=1|3&.color=0xFF9900&.colorBar=|C|Linear|||
That makes it easy for humans to modify an image request to make a similar graph or map.
Or, if you change the fileType in the URL from .png to a data fileType (e.g., .htmlTable), you can download the data that was graphed:
http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.htmlTable?sst[(2007-07-01):(2007-10-21)][0][(29)][(225)]&.draw=linesAndMarkers&.vars=time|sst|&.marker=1|3&.color=0xFF9900&.colorBar=|C|Linear|||
Or, if you change the fileType in the URL from .png to .graph, you can see a Make A Graph
web page with that request loaded:
http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.graph?sst[(2007-10-21)][0][(-75):50:(75)][(180):50:(360)]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=Rainbow|C|Linear|0|32|
Or, if you change the fileType in the URL from .png to a data fileType (e.g., .htmlTable),
you can download the data that was mapped:
http://upwell.pfeg.noaa.gov/erddap/griddap/erdBAssta5day.htmlTable?sst[(2007-10-21)][0][(-75):50:(75)][(180):50:(360)]&.draw=surface&.vars=longitude|latitude|sst&.colorBar=Rainbow|C|Linear|0|32|