If you are using Navigator 4.x or Internet Explorer 4.x or Omni Web 4.x , this site will not render correctly!

Vis > data > netcdf > NetCDF Utilities

NetCDF Utilities

W	R	D	E	X	C	V	(alphabetically)
							cfchecker	Climate & Forecast (CF) compliance checker
							DDI	data extraction/combination, metadata editing, conversion
							gennet	given a netCDF file, generates FORTRAN program to read it
							grads2nc	a GrADS to netCDF converter
							gribtonc	a GRIB to netCDF converter (part of Unidata's " decoders")
							MexCDF	interface for MATLAB (for MATLAB 4, MATLAB 5, MATLAB 6)
							MexEPS	interface for MATLAB
							ncap	(NCO) Arithmetic Processor
							ncatted	(NCO) Attribute Editor
							ncbo	(NCO) Binary Operator (includes ncadd, ncsubtract, ncmultiply, ncdivide)
							nccat	utility to concatenate like files along the "record" dimension
							nccatm	same as *nccat*, but will concatenate multiple files
							nccmp	compare metadata and data between two files quickly
							ncdiff	extract, difference variable(s) from two files
							ncdump / ncgen	netcdf <--> ASCII translators
							ncea	(NCO) Ensemble Averager
W	R	D	E	X	C	V
							ncecat	(NCO) Ensemble Concatenator
							ncextr	extract a subset of a netCDF file
							ncflint	(NCO) File Interpolator
							NCIR	high level (FORTRAN) NetCDFInterface Routines (shells)
							ncks	(NCO) "Kitchen Sink" to print, extract, crop, combine netCDF data
							ncpdq	(NCO) Permute Permute Dimensions Quickly
							ncra	(NCO) Record Averager
							ncrcat	(NCO) Record Concatenator
							ncrename	(NCO) variable/dimension/attribute renamer
							ncstat	calculate min/max, mean along any or all dimensions
							nctools	stand-alone programs that mimic netCDF routines
							ncclamp	clamps values in-place using comparison operators
							nc_to_vis5d	script that converts NetCDF to Vis5d via Ferret
							ncwa	(NCO) Weighted Averager
							ncx.py	Python script that extracts arbitrary slices for a single dimension
							ncymdhms	from filename and time-index, get year, month, day, etc
							nrecreset	forcibly reset the number of records in a file
							ReadNC	module for SGI/NAG Iris Explorer
							ScientificPython	a collection of Python modules with NetCDF support
W	R	D	E	X	C	V

Key

W	write
R	read
D	dump
E	edit
X	extract / calculate
C	combine
V	convert

If you need addtional functionality (perhaps something you see in Unidata's collection of user-contributed software), let us know and we will try to accomodate you.

NetCDF Interface Routines

DESCRIPTION :: This is a set of locally (GFDL) written FORTRAN "shell" routines intended to minimize the work required to read and write netCDF files. The "surface area" of contact between the user and netCDF is reduced to a limited number of FORTRAN routines for which the user need only satisfy an argument list. There is no need to make the (many) low-level netCDF API calls. In addition, the use of these routines helps insure a consistent structure which also complies with the COARDS conventions for writing netCDF files, including the use of the Unidata UDUNITS package to ensure a standard representation of units for all coordinate axes and data variables.; Also included are 2 prototype scripts which go through all the motions for compiling and linking some test/demo programs. These scripts work on a T90, T3E, and our SGI's.; (deprecated) Performance can be an issue on the Cray platforms. Beginning with netCDF version 2.4, the use has control over the Cray FFIO layer. Some guidelines for optimizing performance on Crays is available.
PLUSES :: - insulates the user from the details of the netCDF API; - performs many error checks on metadata to insure COARDS compliance; - insures a level of consistency among files produced

MINUSES :: - obviously, not as flexible, complete, or efficient as one could make it with pure API calls for a specific application

HOW TO INVOKE :

see Getting_Started

the source code for the interface routines are available via FTP

When compiling on a Cray, do not use the "-Ra" option, as a compiler bug causes some routines to fail when they call PUTENV.

HINTS :

Take a look at the sample codes referred to in "Getting_Started".
Keep an eye out for changes and developments in "What's New".
Be aware of the known bugs in the previous version and in the current version.
GrADS 1.6 has a nominal capability to read netCDF, but units for the time axis must be specified as "<units> since <basetime>"; i.e., GrADS cannot handle a simple "days" units. Also, GrADS is inherently unable to handle more than 1 horizontal grid (e.g., a B grid), regardless of how many grids are defined in the netCDF file.

INFO / DOCUMENTATION:

man page (none yet)
COARDS Conventions
Available in Unix-compressed format
NetCDF User' s Guide

ncdump

DESCRIPTION :: ncdump generates an ASCII representation of a specified netCDFfile on standard output. The ASCII representation is in a language called CDL ("CDF Description Language'") that is quite readable and can serve as input to ncgen, a program that can generate a binary netCDF file from a CDL file. Hence ncgen and ncdump can be used as inverses to transform the data representation between binary and ASCII representations. See the ncgen man page for a description of CDL and netCDF representations.

PLUSES :: - fastest/easiest way to find out what a netCDF file contains

MINUSES :: - dumping data (optional) this way can result in a lot of output (ASCII)

HOW TO INVOKE :

show only the header information


        ncdump -h <file>

show only the data, no header information (use custom ncdump found below)

        alias ncdump /home/rsz/util/netcdf/{linux,irix}/ncdump
        ncdump -d <file>

shows the header and values of coordinate variables


        ncdump -c <file>

shows header, coord's, and data for selected variables


        ncdump -v var1[,var2] <file>

HINTS :

To avoid massive unintentional dumps to your screen, I suggest you create an alias:


        alias ncdump ` \ncdump -h \!* '

To use the data only option (GFDL & AOS Only, others download below):

        alias ncdump /home/rsz/util/netcdf/{linux,irix}/ncdump

INFO / DOCUMENTATION:

man page
NetCDF User' s Guide
URL/Web info
C Source Code for ncdump with -d data only option.

As an example, to rename a dimension in a netCDF file, use ncdump to get a CDL version of the netCDF file, edit the CDL file to change the name of the dimensions, and use ncgen to generate the corresponding netCDF file from the edited CDL file.

INFO / DOCUMENTATION:

gennet

DESCRIPTION :: gennet examines a specified netCDF file and generates a FORTRAN program to read it.

PLUSES :: - a very quick way to create a code to read almost any file

MINUSES :: - kind of mindless:; - defines massive arrays, as if you will read in everything at once; - reading in anything less than an entire variable requires understanding of the netCDF API; - obviously, you have to add code to actually do something with the data; - no error/consistency checks, as in readnc.F (part of the locally written shell routines in the directory /home/jps/pub/netcdf/ )

HINTS :

Give it a try, especially if you have a small file to read. Also, you might want to try it just to get idea of what the netCDF API looks like.

INFO / DOCUMENTATION:

URL : FTP

ncks

DESCRIPTION :: ncks extracts a subset of the data from in.nc and either prints it as ASCII-readable text to the screen, or writes (or pastes) it to out.nc, or both. ncks combines a generic netCDF printer (like a formatted ncdump),extractor (like ncextr), hyperslabber (the nccut specification), and combiner (partial ncpaste implementation) into one command. ncks and ncrename (available from the same FTP site) are intended to perform many common tasks for the rapid manipulation and analysis of netCDF data.; ncks will append variables and attributes from the input file to the output file if the output file is a pre-existing netCDF file whose relevent dimensions conform to dimension sizes of in.nc. The append features of ncks are intented to provide a rudimentary means of adding data from one netCDF file to another, conforming, netCDF file.

PLUSES :: - Can be used to merge data into a file (but - must be conformal)
MINUSES :: - dumping data (optional) this way can result in a lot of output (ASCII); - When naming conflicts exists between the two files, data in out.nc is usually overwritten by the corresponding data from in.nc. Thus it is recommended that the user backup out.nc in case valuable data is accidentally overwritten.; - output isn't very pretty, but the appearance is partially under user's control of the user with -s switch

HOW TO INVOKE :

The latest build is located in
```
        /net2/rsz/{linux,irix}/nco/bin
```
First, do an "ncdump -h" to get basic info on file contents;
To print some data to screen :

ncks -d dim1,min1,max1 [-d dim2...]] -v var1[,...] -c -F -m in.nc out.ncwhere dim1, dim2,... are dimension names, min1/max1 are limits (integers imply indices, floating point values imply coordinates), and var1[, var2,...] is the variable(s) to extract

To extract a subset of variables into another file ("-h" turns off the "history" attribute)

ncks -v var1[,...] -c -h -F in.nc out.nc

To extract selected hyperslabs of variables to another file :

ncks -d dim1,min1,max1 [-d dim2...]] -v var1[,...] -c -h -F in.nc out.nc

In the above examples, if output file exists,

the relevent dimensions must conform to the dimensions of in.nc
you will be prompted about whether to overwrite the existing file or append to it
if you don't want to be prompted, add the argument "-A" to the command line

If you don't want ncks to alphabetize the variables in the output file, add "-a" to the command line

INFO / DOCUMENTATION:

NCO Homepage (including link to on-line User's Guide)

ncymdhms

DESCRIPTION :: One common problem with netCDF files is determining the time, in a useful format, corresponding to a given slab of data. This is because netCDF time is generally stored with units such as "hours since 01-01-1970 00:00:00.0" and values like "34,244". Given a netCDF filename and a time index (ie, slice), the utility ncymdhms returns a message translating the time of that slice into year, month, day, hour, minute, second (get it? YMDHMS?).; A FORTRAN-callable version of ncymdhms is also available.
PLUSES :: - handy one liner useful at the command line
MINUSES :: - if a file has more than one time-dimension, the first one encountered which is REAL*8 and has legitimate units for "time" is used

HOW TO INVOKE :

Couldn't be easier (use the appropriate platform designation below):


        ~jps/pub/netcdf/util/bin/{sgi;linux;hpcs}/ncymdhms file.nc itindex

itindex

You can also get a usage message by typing ncymdhms and hitting return

HINTS :: The source code for ncymdhms is in /home/jps/pub/netcdf/util/src/ncymdhms.F, which could be useful in constructing time-stamps within your own application. See the comments at the top for the other code modules needed.

INFO / DOCUMENTATION :

nctools

DESCRIPTION :: The nctools are a collection of stand-alone C programs that allow netCDF users to manipulate netCDF files from the command line. Each nctool mimics a subroutine in the netCDF library in usage and function. The usage syntax follows the netCDF Users Guide - if you type the program name with no arguments, you get the usage and an example. You will need to be familiar with the terminology/components of a netCDF file in order to make the most of these tools. The changes made by any of the tools are applied "in place"; ie, changes are made to the file specified - there is no "output" file.

Here is the list of the (29) tools :
PLUSES :: - handy one liners useful at the command line (and for scripts), especially for renaming or deleting items, and for concatenating files; - (see also nccat, another incarnation of the same concatenation tool, unfortunately by the same name)
MINUSES :: - output of variable values is ASCII, one element per line, which can get voluminous; - concatenation works only for file which are identical in variable names and dimensions and; have a leftmost (C-wise and CDL-wise) unlimited dimension; - most tools will require making a copy of the file, so can be ponderous, especially if several operations are required; see ncrename for a utility which will do multiple rename's at once

HOW TO INVOKE :

To get a usage message, simply type the name of the tool and hit return

HINTS :

INFO / DOCUMENTATION :

ncrename

DESCRIPTION :: This utility, part of the NCO utilities written by Charlie Zender at UCAR, renames dimensions, variables, and attributes of a netCDF file. A number of objects can be renamed with one invocation, thus saving reopening and making a new copy of the file for each instance, such as would be necessary with the nctools. You will need to be at least minimally familiar with the terminology/components of a netCDF file in order to make the most of this utility.
PLUSES :

multiple objects can be renamed with one invocation, avoiding multiple reopening/copying file
can create new output file, or changes can be made "in place"

MINUSES :

Known Bugs:

- dimension names are not checked to see if they could possibly conflict with existing variable names before renaming is attempted

use caution - the results could be serious

You cannot change the attribute of one particular variable (unless it is uniquely named); all occurrences of the attribute of a given name will be renamed wherever it occurs
Renaming a dimension does not automatically rename its assocaited coordinate variable (see Hints below)

HOW TO INVOKE :

The latest build is located in

        /net2/rsz/{linux, irix}/nco/bin

To get usage message at any time, enter "ncrename" with no arguments.

ncrename [-a ...] [-D lev] [-d ...] \ [-l path] [-O] [-p path] [-R] [-r] [-v ...] in.nc [out.nc]

-a old_att,new_att        Attribute's old and new names

-d old_dim,new_dim        Dimension's old and new names

-v old_var,new_var        Variable's old and new names

-D lev                    Debugging level

-l path                   Local storage path for remotely-retrieved files

-O                        Overwrite existing output file, if any

-p path                   Path prefix for all input filenames

-R                        Retain remotely-retrieved files after use

-r                        Program version and copyright notice

in.nc                     Input file name(s)

out.nc                    Output file name

If "out.nc" is the same as "in.nc", changes are made "in place".

Every old name must exist in the file, unless the name is preceded by ".", in which case it will be changed if present, otherwise ignored

HINTS :: - ncrename will not rename an attribute if an attribute by that name is already attached to the variable.; - If you want to rename an axis, be sure to rename both the dimension and the associated coordinate variable (eg, "-d xx,lon -v xx,lon"); - See also nctools

INFO / DOCUMENTATION :

NCO Homepage (including link to on-line User's Guide)

DDI

DESCRIPTION :

The Data and Dimensions Interface (DDI) transfers data between files, formats, and visualization environments. It works within the specifications of each supported format to promote compatibility. DDI provides the following capabilities:

Browsing through the contents of files written in a variety of formats
Randomly selecting data variables
Rearranging variables into the desired form
Modifying the value of certain variable attributes
Saving variables into new files
Feeding variables directly to a visualization environment

DDI can also execute as an integrated module in IRIS Explorer or AVS, but we have noticed some problems with that incarnation (can't handle the "record" dimension), and we have a fairly robust netCDF module for Explorer which is a little easier to use (ReadNC).

PLUSES :: - fairly robust ability to select variables, sample/crop, combine with others, output to a file

MINUSES :: - does not transfer the global metadata to the output file; - does not preserve the "record" dimension - "record" axis is output as a non-record axis; - DDI does not seem to be under very active development - little progress since 2000

INFO / DOCUMENTATION :

Reference manual PS

README

ncextr

DESCRIPTION :: This netCDF operator extracts a subset of the data from in.cdf and writes it to out.cdf. One can specify either the subset to appear in the output, or the subset to exclude from the output. If no options are specified, then this operator copies the input netCDF to the output netCDF, deleting superfluous dimensions.; ncextr is largely a subset of ncks's functionality.

PLUSES :: -

MINUSES :: -

HOW TO INVOKE :

When logged onto the HPCS (IRIX):
```
        /usr/local/bin/ncextr 
```

Subset of selected variables into a new file :


        ncextr -v var1[,...] -c in.cdf out.cdf

Subset, ecluding variables, into a new file :


        ncextr -v var1[,...] -x v -c in.cdf out.cdf

HINTS :

INFO / DOCUMENTATION :

man page
Local Documentation
URL/Web info : N/A (deprecated)

ncstat

DESCRIPTION :: This netCDF operator computes a statistical summary over one or more dimensions an input netCDF file and writes the result to a new output netCDF file. The statistics inlclude min, max, and mean.
PLUSES :: - easy way to get averages (vertical, zonal, time, etc.); - calculation is done for all variables employing the selected axis(axes)

MINUSES :: - the axis along which the average is taken is removed; ie, the resultant field has no coordinate location along the averaged axis, so you won't be able to distinguish averages over different ranges (eg, 90-180 deg_E vs. 180-270 deg_E); - A bug causes spurious bulls-eyes at intervals along left edge of rows of points.; - Does not assign any "weighting" to the points; ie, all points are weighted equally, even if they are unevenly spaced; - Does not account for "missing_value"'s when computing statistics;

HOW TO INVOKE :

On the HPCS (IRIX):
```
        /usr/local/bin/ncstat
```

Compute a mean along the longitude axis :


        ncstat -d lon -s mean in.cdf out.cdf

HINTS :

Alternatives:

ncra
FAN
Ferret
MATLAB

INFO / DOCUMENTATION :

man page

nrecreset

DESCRIPTION :: This netCDF operator allows the user to forcibly reset the number of records in a netCDF file. The main use for this utility is when a job terminates abnormally and does not close the file properly, in which case the data may be present in the file, but the metadata will indicate too few (or no) actual records.
PLUSES :

Easy
Does not need to make a copy of the file, which makes it very fast

MINUSES :

Modifies the file in-place, so use caution
Obviously, one needs to know (or have a good idea) of what value to to use. Presumably, one would get this from other information output by the job which created the file.

HOW TO INVOKE :

On a Linux workstation or the HPCS (IRIX):
```
        /usr/local/bin/nrecreset 
```
Usage:
```
        nrecreset file.nc nrecs
```

HINTS :

INFO / DOCUMENTATION :

ncra

DESCRIPTION :: This netCDF utility computes an average along the record (time) dimension of one or more input netCDF files and writes the result to an output netCDF file. It is part of the NCO utilities written by Charlie Zender at UCAR. There are a number of options, but some of them seem more specific to NCAR's computing facility. Those that are more generally useful are described below.
PLUSES :

provides an easy way to get averages over the time dimension, particularly if spread over multiple files
the calculation can be limited to specific variables and/or specific sub-sections

MINUSES :

integration does not use the trapezoidal rule, so that points along the record dimension are treated as if the are equally spaced/weihgted

Bug

Like all the NCO utilities which perform mathematical operations, arithmetic with packed data (ie, 8- or 16-bit binning) is not supported. The operation will proceed, but will be performed on the packed data, not their appropriate expanded representation. The result is not meaningful.

HOW TO INVOKE :

The latest build is located in

        /net2/rsz/{linux, irix}/nco/bin

To get a usage message, type ncra with no arguments; an abbreviated version is shown below (Note: "Append" appends variable to file - it does not append to the end of the record dimension):

ncra [-A] [-O] [-F] [-c] [-D lev] \

     [-d ...] [-p path] [-r] \ 

     [-v ...] [-x] \ 

     in.nc [...] out.nc



-A                   Append to existing output file, if any

-O                   Overwrite existing output file, if any

-F                   Fortran indexing conventions (1-based) for I/O

-v var1[,var2[...]]  Variables to process

-x                   Extract all vars EXCEPT those specified with -v

-d dim,[min][,[max]] Dimension's limits in hyperslab

-p path              Path prefix for all input filenames

-r                   Program version and copyright notice

-c                   Coordinate variables will all be processed

-D lev               Debugging level

in.nc...             Input file name(s)

out.nc               Output file name

Compute a time mean for several files:


        ncra file1.nc file2.nc file3.nc outfile.nc

Same as above, but select only variable Var1 :


        ncra -v Var1 file1.nc file2.nc file3.nc outfile.nc

Same as above, but select only 3 horizontal slabs of variable Var1 :


        ncra -v Var1 -d height,3,5 file1.nc file2.nc file3.nc outfile.nc

HINTS :

To get a usage message at any time, type ncra with no arguments
Note: the "Append" option does not append along the record axis - it is intended to append variables to an existing file
(deprecated) When running on a Cray, be sure you set the environment variable NETCDF_FFIOSPEC, and do not make it too large if you are reading lots of files or your memory usage will blow up; the following should be sufficient in almost all cases :

              setenv NETCDF_FFIOSPEC cachea:112:2

INFO / DOCUMENTATION :

NCO Homepage (including link to on-line User's Guide)

ncdiff

DESCRIPTION :: This netCDF operator subtracts the variables in one netCDF file from those in another file. It is part of the NCO utilities written by Charlie Zender at UCAR. There are a number of options, but some of them seem more specific to NCAR's computing facility. Those that are more generally useful are described below.
PLUSES :: - Right now, this is one of the few utilities available for performing mathematical operations on netCDF files/data

MINUSES :
PLUSES :
HOW TO INVOKE :
HINTS :
HINTS :
HINTS :
DESCRIPTION :: ScientificPython is a collection of modules for the Python language. It's most important module to GFDL is Scientific.IO.NetCDF , which allows reading and writing NetCDF files. The interface is much simpler in every respect as compared to the NetCDF Toolbox for MATLAB. This includes modifying attributes, data, and dimensions. Plus, Python's local memory is only constrained by hardware, unlike MATLAB, which has a 1+ GB built-in limit, so even the largest NetCDF files can be manipulated.
PLUSES :
MINUSES :
HINTS :
DESCRIPTION :: A Python script that extracts arbitrary slices for a single dimension for any number of variables in a NetCDF file. The dimensionality is preserved for every sliced variable. The input file's grid ranks can be arbitrary.

As an example, this can be used to extract only specific pressure levels for variables defined on a (time, level, lat, lon) grid.
PLUSES :
MINUSES :
HOW TO INVOKE :
HINTS :
INFO / DOCUMENTATION :
DESCRIPTION :: ncclamp is a command-line tool for NetCDF files that allows you to replace values in-place by specifying the variable, the old value to be replaced, the new value, and a comparison operator. The change is applied across all of the variable's dimensions, including the record dimension if any.

In a less than obvious way, the same could probably be achieved using mask conditions with NetCDF Operators, but this tool greatly simplifies the syntax and I/O.
PLUSES :
HOW TO INVOKE :
HINTS :
INFO / DOCUMENTATION :

smaller bigger reset

ncdims	- lists names of all dimensions
ncdimvars	- lists names of dimensions that are also variables
ncmetasize	- returns byte size of file's header (metadata)
ncndimvars	- returns the number of dimensions that are also variables
ncinqvar	- returns id, type id, type, type length, # dims., dim. ids, dim. lengths, dim. names & # attributes
ncnondimvars	- lists names of variables that are not dimensions
ncnonrecvars	- lists names of variables that don't use a record dimension
ncnrec	- returns the number of records
ncnnondimvars	- returns the number of variables that aren't dimensions
ncrec	- returns the record dimension name
ncrecoffset	- returns byte offset for the record data section
ncrecsize	- returns byte size of a single record
ncrecvars	- lists names of variables that use a record dimension
ncnonrecoffset	- returns byte offset to the non-record data section (same as ncmetasize)
ncvaroff	- returns byte offset & length of a variable

Geophysical Fluid
Dynamics Laboratory

NetCDF Utilities

NetCDF Interface Routines

ncdump

ncgen

gennet

ncks

ncymdhms

nctools

ncrename

DDI

ncextr

ncstat

nrecreset

ncra

ncdiff

nccat / nccatm

ncrcat

gribtonc

grads2nc

MEXCDF for MATLAB 4 (for MATLAB 5, see below)

MexCDF and NetCDF Toolbox for MATLAB 5 (for MATLAB 6, see below)

MexCDF and NetCDF Toolbox for MATLAB 6

MEXEPS

cfchecker

nc_to_vis5d

Introspection Tools

nccmp

ScientificPython

ncx.py

ncclamp