Welcome to the netCDF Operator (NCO) Homepage
Current NCO version is 3.9.5 last updated on Monday, 20-Oct-2008 04:01:25 UTC
|
The temporary NCO logo…
Have a better one?
Send it in!
|
NCO News & Announcements
- September 24, 2008: Extensive ncap2 documentation available
- August 8, 2008: NCO Wiki is up!
- June 11, 2008: NCO overview paper published in EMS
- May 22, 2008: SWAMP Talk at CCGRID08 in Lyon, France
- May 11, 2008: NCO 3.9.5 released
- March 4, 2008: NCO 3.9.4 released
- December 10, 2007: SWAMP Poster IN11B-0469 at Fall AGU
- December 7, 2007: NCO 3.9.3 released (and SWAMP!)
- December 4, 2007: NCO scaling paper published in IJHPCA
- August 29, 2007: NCO 3.9.2 released (_FillValue support)
- August 27, 2007: SWAMP site spins-off from NCO
- June 29, 2007: NCO 3.9.1 released (netCDF4 deflation support)
- May 26, 2007: NCO 3.9.0 released (netCDF4 atomic type support)
- May 2, 2007: SWAMP Talk at GPC, Paris, France
- April 20, 2007: NCO 3.2.0 released (with i386/x86_64 debs and RPMs)
- March 29, 2007: NCO 3.1.9 x86_64 RPMs released
- March 10, 2007: NCO 3.1.9 released (with i386/x86_64 debs and i386 RPMs)
- February 22, 2007:
Talk
on SWAMP at OPeNDAP Developer's Workshop, Boulder, Colorado
- January 15, 2007:
Talk and
Extended Abstract
on NCO architecture and Scaling at AMS IIPS, San Antonio, Texas
- January 15, 2007:
Talk and
Extended Abstract
on SWAMP at AMS IIPS, San Antonio, Texas
- December 9, 2006:
Poster on NCO
Scaling at Fall AGU, San Francisco, California
- December 9, 2006:
Poster on
SWAMP at Fall AGU, San Francisco, California
- December 6, 2006: NCO 3.1.8 released
- November 11, 2006: NCO 3.1.7 released
- September 18, 2006: NCO 3.1.6 released
- August 29, 2006: NCO 3.1.5 released
- July 7, 2006: Submitted manuscripts that describe NCO and its scaling
- June 7, 2006: NCO 3.1.4 released
- May 30, 2006: NCO 3.1.3 released
- April 21, 2006: NCO 3.1.2 released
- January 30, 2006: NCO 3.1.1 released
- December 2, 2005: NCO 3.1.0 released—All arithmetic operators support MPI
- October 28, 2005: NCO 3.0.3 released
- October 17, 2005: NCO Parallel Benchmarks now available.
- August 22, 2005: NCO 3.0.2 released
- June 10, 2005: NCO 3.0.1 released
- March 24, 2005: NCO 3.0.0 released
- September 7, 2004: NCO 2.9.9 released—ncpdq now packs data quietly
- August 21, 2004:
National Science Foundation (NSF)
funds the NCO→SDO Project!
- August 7, 2004: NCO 2.9.8 debuts ncpdq operator—performs arbitrary dimension re-ordering and reversal
- March 4, 2004: Submitted to NSF the first
proposal
for Institutional funding for NCO.
- Past NCO News & Announcements
What is NCO?
The netCDF
Operators, or NCO, are a suite of programs known as operators.
Each operator is a standalone, command line program which is executed at
the UNIX shell-level like, e.g., ls or mkdir.
The operators take netCDF
files as input, then perform a set of operations (e.g., deriving new
data, averaging, hyperslabbing, or metadata manipulation) and produce a
netCDF file
as output.
The operators are primarily designed to aid manipulation and analysis of
gridded scientific data.
The single command style of NCO allows users to manipulate and analyze
files interactively and with simple scripts, avoiding the overhead
(and some of the power) of a higher level programming environment.
The NCO User's Guide illustrates their use
with examples from the field of climate modeling and analysis.
Note that the “averagers”
are misnamed because they perform many non-linear operations as well,
e.g., total, minimum, maximum, RMS:
The operators are as general as
netCDF
itself: there are no restrictions on the contents of the
netCDF
file(s) used as input.
NCO's internal routines are completely dynamic and impose no limit on
the number or sizes of dimensions, variables, and files.
NCO is designed to be used both interactively and with large batch jobs.
The default operator behavior is often sufficient for everyday needs,
and there are numerous command line (i.e., run-time) options, for
special cases.
NCO works well on most modern operating systems, including:
Apple OS X, *BSD, Cray UNICOS, DEC Tru64, IBM AIX, HPUX,
Linux, Microsoft Windows, NEC Super UX, SGI IRIX, and Sun
Solaris.
How to Contribute
In March, 2000, NCO became an Open Source project hosted by
SourceForge.net.
This facilitates collaboration, code contributions, and support.
There is a TODO list about 200 items long!
No matter what your programming background there is a task you can help with.
From re-organizing the TODO list itself, to improving this cheesy
webpage, to documentation, to designing and implementing new features
and interfaces, we need your help!
The NCO project homepage
contains mailing lists, discussion forums, and instructions to make
contributing to NCO easy.
As of June, 2003, you may donate money to NCO via Paypal.
Your donation will be distributed to NCO developers as an incentive
for them to tackle TODO items on a
pay-per-feature basis.
These homepage donation counters track the influence of your monetary
donations on NCO development:
Donations received between 20030624 and 20070531: US$60.00.
NCO features "incentivized" by these donations: None.
NCO→SDO Project
The National Science Foundation
NSF IIS-0431203
funds our
NCO→SDO Project,
“SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data”
as part of the
Science and Engineering Informatics (SEI) program.
We appreciate the proposal reviewers for and staff of the Divisions of
Information and Intelligent Systems (IIS)
and
Shared Cyberinfrastructure (SC)
in the
Directorate for Computer and Information Science and Engineering (CISE).
NCO development was completely voluntary and without institutional
support from its inception until September 2004.
Hence this NSF support has dramatically changed the scale and pace of
NCO/SDO development.
This URL,
http://nco.sf.net#prp_sei,
points to the most up-to-date information on the NCO/SDO proposal.
The NSF project provides the human and technical resources to analyze,
design, and implement novel advanced methods for distributed
data analysis and reduction (DDRA) of geophysical data.
These powerful methods will be “grid-enabled”, and we will
demonstrate their efficacy with analyses of distributed climate
prediction and observation datasets.
On the human side, recruited full-time help for specific areas of SDO,
alongside our current and future volunteers.
We have a
full-time programmer
to help design, implement, and release major code changes.
We have a
graduate student researcher
pursuing innovative Ph.D. research in using
“grid-aware”
distributed memory,
shared memory, and
client-server software
technologies to improve DDRA of climate datasets.
Another graduate student helps us design, implement, and release major
code changes.
Contact us if you are seriously interested in tackling these
problems on a full-time, part-time, or pay-per-feature basis.
On the institutional and hardware side, this project connects the
Earth System Modeling Facility (ESMF)
to the
California Institute for Telecommunications and Information Technology (Cal-IT2)
OptIPuter at the
San Diego Supercomputer Center.
These supercomputers will dedicate a TB of storage each to
OPeNDAP-served climate simulation
datasets for DDRA.
Once the proof-of-concept SDO DDRA is complete, we will attempt a DDRA
intercomparison of the internationally distributed
global climate change simulations
that reside on the
Earth System Grid.
We expect our analyses to yield otherwise undiscovered patterns among
different climate prediction scenarios for the same model, and among
different climate models.
Finally we still encourage you to e-mail NCO/SDO endorsements to
Charlie “my surname is zender” Zender
with Subject: “NCO/SDO Proposal Endorsement”.
An endorsement can be a sentence or a page, describing how NCO/SDO
benefits your work or research.
The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO/SDO proposal advocacy.
“What future proposals?” you ask, “Didn't you just get funded?”
Yes, we are funded to optimize and “grid-enable” NCO.
There are still many desirable and unexplored methods for distributed data analysis
and reduction (DDRA) that would benefit geosciences.
For example, have you tried analyzing multiple NASA satellite datasets in HDF5 format?
Need I say more?
If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.
Publications and Presentations
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2008), Compiling the uncompilable: A case for shell script compilation, Submitted to ACM Trans. Softw. Engin. Method..
BibTeX
PDF (© 2008 by Springer-Verlag)
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2008), Efficient Clustered Remote Data Analysis Workflows using SWAMP, Submitted to Earth Sci. Inform..
BibTeX
PDF (© 2008 by ACM)
-
Wang, D. L., C. S. Zender, and S. F. Jenks: Cluster Workflow Execution of Retargeted Data Analysis Scripts, in Cluster Computing and the Grid, 2008. CCGRID '08. 8th IEEE International Symposium on. Lyons, France, 19 22 May, 2008. IEEE Computer Society, 449–458, doi:10.1109/CCGRID.2008.69.
BibTeX
PDF (© 2008 by IEEE)
-
Zender, C. S. (2008), Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO), Environ. Modell. Softw., 23(10), 1338–1342, doi:10.1016/j.envsoft.2008.03.004.
BibTeX
PDF (CSZ)
PDF (EMS) (© 2008 by Elsevier Ltd.)
-
Wang, D. L., C. S. Zender and S. F. Jenks (2007): A System for Scripted Data Analysis at Remote Data Centers. Presented at the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 10–14, 2007. Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN11B-0469.
BibTeX
PDF (DLW)
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side parallel data reduction and analysis, in Advances in Grid and Pervasive Computing, Second International Conference, GPC 2007, Paris, France, May 2–4, 2007, Proceedings. IEEE Lecture Notes in Computer Science, vol. 4459, edited by C. Cérin and K.-C. Li, pp. 744–750, Springer-Verlag, Berlin/Heidelberg.
BibTeX
PDF (DLW)
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2007),
Server-side Parallel Data Reduction and Analysis.
Presented by D. Wang to the International Conference on Grid and
Pervasive Computing (GPC),
Paris, France, May 2–4, 2007.
PDF
-
Zender, C. S., and H. J. Mangalam (2007), Scaling Properties of Common Statistical Operators for Gridded Datasets, Int. J. High Perform. Comput. Appl, 21(4), 485–498, doi:10.1177/1094342007083802.
BibTeX
PDF (CSZ)
PDF (IJHPCA) (© 2007 by SAGE Publications)
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2007),
Server-side Data Reduction and Analysis with Script Workflow Analysis for Multi-Processing.
Presented by D. Wang to the OPeNDAP Developer's Workshop,
Boulder, CO, February 21–23, 2007.
PDF
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2007),
Server-side Data Reduction and Analysis with Script Workflow Analysis for Multi-Processing.
Presented by D. Wang to the 23rd AMS Conference on Interactive Information and
Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology,
San Antonio, TX, January 14–18, 2007.
PDF
-
Wang, D. L., C. S. Zender, and S. F. Jenks (2007),
DAP-enabled Server-side Data Reduction and Analysis,
Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology,
Paper 3B.2., January 14–18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA.
BibTeX
PDF
-
Zender, C. S., and D. L. Wang (2007),
High performance distributed data reduction and analysis with the netCDF Operators (NCO).
Presented to the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, January 14–18,
San Antonio, TX, January 14–18, 2007.
PDF
-
Zender, C. S., and D. L. Wang (2007),
High performance distributed data reduction and analysis with the netCDF Operators (NCO),
Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology,
Paper 3B.4., January 14–18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA.
BibTeX
PDF
-
Wang, D. , Zender, C. S., and S. F. Jenks (2006):
Server-side netCDF Data Reduction and Analysis.
Presented by D. Wang to the Fall Meeting of the American Geophysical Union,
San Francisco, CA, December 5–9, 2006.
Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0826.
PDF
-
Zender, C. S., H. J. Mangalam, and D. L. Wang (2006):
Improving Scaling Properties of Common Statistical Operators for Gridded Geoscience Datasets.
Presented to the Fall Meeting of the American Geophysical Union,
San Francisco, CA, December 5–9, 2006.
Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0827.
PDF
Version Summary
New releases receive unique CVS tags and their tarballs are stored
here on SourceForge and
here on my server.
You can also retrieve the source of tagged versions directly
with CVS.
- NCO/SDO 3.9.6: (In progress)
Memory management to improve threading;
ncatted workaround for netCDF4 _FillValue;
disable OpenMP with netCDF4/HDF;
no math on type NC_CHAR;
ncap2 documentation,
#include files,
incomplete gamma functions,
missing value propagation;
- NCO/SDO 3.9.5:
Multi-slabbing and -X everywhere;
ncecat -M;
netCDF4-enabled builds on Cygwin;
ncap2 builds on Solaris;
- NCO/SDO 3.9.4:
File consanguinity;
ncks: CF auxiliary coords, LZ stats;
ncap2: masks, index arrays, print attributes, OpenMP, atan2(), pow();
- NCO/SDO 3.9.3:
SWAMP release;
ncecat -u;
wget URLs;
GCC 4.2;
ncap2 OpenMP;
- NCO/SDO 3.9.2:
_FillValue supplants missing_value;
UDUnits2 support;
ncap2 type-conversion support and printing;
netCDF4 configure fixes;
Change licenses to GPL3 and FDL1.2;
- NCO/SDO 3.9.1:
Support lossless netCDF4/HDF deflation;
Improve Intel compiler support;
ncap2 for/while loops;
various and sundry build fixes;
- NCO/SDO 3.9.0:
Support netCDF4 types (ubyte, ushort, uint, int64, and uint64);
Control ncks metadata copying;
Fix ncbo packed valued bug (upgrade recommended);
Improve IRIX 6.5 support;
- NCO/SDO 3.2.0: More forgiving exclusion list (-x -v var_nm) handling; Fix rmssdn normalization; Support Mac OS X on Intel; Documentation bugfixes;
- NCO/SDO 3.1.9: ncap2 in RPM packages; Quieter output; AIX configure supports ncap2; Update to DAP for netCDF 3.6.2; Fix nc[erw]a for coordinate min/max/ttl; ncecat allows files to differ in record dimension size;
- NCO/SDO 3.1.8: Support _FillValue with compile-time switch; Debian package synchronized, includes ncap2
- NCO/SDO 3.1.7: ncap2 “double-parsing”, array initialization, supercedes ncap
- NCO/SDO 3.1.6: Support stride in all hyperslabbing operators; change more WARNINGs to INFOs
- NCO/SDO 3.1.5: New ncap2 array and hyperslab features; change some WARNINGs to INFOs, add Pathscale and update PGI and Intel compiler support
- NCO/SDO 3.1.4: Fix ncbo memory problem; report timer results
- NCO/SDO 3.1.3: ncap2 automatically propagates attributes to new/derived variables; NCO handles record variables with zero records; support CF coordinates convention; ncwa -b retains degenerate dimensions; build fixes for Cray T3E, Solaris, Cygwin.
- NCO/SDO 3.1.2: ncap2 control structures (conditionals, loops, etc.); keep non-processed data packed in output file; binary operators ncbo and ncflint work when only one variable has missing_value attribute; change ncra and ncea to always average coordinate variables
- NCO/SDO 3.1.1: NCO builds (again) with C++ compilers; ncap2 development code; server-side extensions
- NCO/SDO 3.1.0: MPI operators are bug-free; fix ./configure for MPI and netCDF4; MRV algorithm speeds up ncwa 5—10 times; ncap random number generator rnd_nbr()
- NCO 3.0.3: Eliminate all critical read regions (accelerates all SMP operators); builds with netCDF4 alpha; read NETCDF4_CLASSIC (i.e., HDF) files automatically; write NETCDF4 files on request; benchmark results quantify parallelism.
- NCO 3.0.2: Fewer critical read regions; all known memory leaks plugged; better Intel compiler support; SFTP protocol support; FTP protocol supports .netrc-based passwords; better model grid-property-awareness (e.g., lat_bnds); ncatted ncks, ncrename support metadata header padding with --hdr; ncbo supports heterogeneous variable ordering; ncatted bugfix
- NCO 3.0.1: benchmarks; ncks printing tweaks; memory cleanup; ncap double-parse speedup and exponentiation bugfix; thread operators by default
- NCO 3.0.0: New ncks -P print switch; ncap optimizations; -Z option supports 64-bit offsets (requires netCDF 3.6); NEC SX build improvements; support GCC 3.4.X; nco_put_vara() overloads in libnco_c++
- NCO 2.9.9: ncpdq packs/unpacks entire files; ncbo is threaded
- NCO 2.9.8: ncpdq released; ncwa --msk_sng consolidated mask switch; C99 compiler required
- NCO 2.9.7: Read input file lists from stdin; write nco_input_file_list attributes; add nco_put_att<int,short,long> functions to libnco_c++; large file support (LFS); write nco_openmp_thread_number attribute
- NCO 2.9.6: Support -o fl_out syntax; 64-bit ABI for x86_64; Debian builds libnco_c++
- NCO 2.9.5: ncflint handles missing_value data better, is commutative when weights are equal
- NCO 2.9.4: ncecat works on pure scalar files; change to cvs.sf.net; AIX GCC builds
- NCO 2.9.3: ncra and ncea now work correctly when missing_value equals zero or any intermediate sums
- NCO 2.9.2: ncap S/V, S%V, and S^V operations fixed (S = scalar, V = variable)
- NCO 2.9.1: ncwa works again on packed NC_FLOAT data (broke when?)
- NCO 2.9.0: Complete ncap modulo operator (V%S) and generic exponentiation (V^S), scalar divided by variable (S/V); workaround stack memory problem with GCC extensions; fix make check
- NCO 2.8.8: --enable-[optimize/debug]-custom options for GCC work again (broke in 2.8.6)
- NCO 2.8.7: ncap packing works again (broke in 2.8.4); Cygwin builds work again
- NCO 2.8.6: AIX and IRIX builds work again; custom GCC builds with -Werror
- NCO 2.8.5: Solaris make install works; ncap changes attribute values correctly; Support DODS 3.4+, deprecate 3.3-; add --enable-[optimize/debug]-custom options to ./configure
- NCO 2.8.4: Index-based hyperslabs work again (broke in 2.7.3, upgrade highly recommended); ncap peak memory usage reduced still more; support for AMD Opteron x86_64 architecture
- NCO 2.8.3: Fix big ncap memory leak; fix regressions; remove scary packing warnings
- NCO 2.8.2: ncbo handles distinct missing_values correctly; improve error diagnostics
- NCO 2.8.1: “Variable wildcarding”—extended regular expressions in -v var,... lists—support in all operators
- NCO 2.8.0: New “binary” operator ncbo, obsoletes ncdiff; Debian integration
- NCO 2.7.4: Autotools build support for AIX; fix UDUnits confusion with floating point hyperslab specifications
- NCO 2.7.3: Support UDUnits (and some CF) conventions, e.g., ncks -d wvl "1 picometer","1 furlong" -d time,"2001-03-19 06:00:0.0","2001-03-19 18:00:0.0"
- NCO 2.7.2: Printing (ncks -H) plays well with multi-slabbing
- NCO 2.7.1: (Packaged) Fallback to B. Sittler's getopt() for long option support; store fixed, packed variables correctly with ncap.
- NCO 2.7.0: Support for GNU/POSIX long options; ncks introduces multi-slabbing (= multiple hyperslabs) capability
- NCO 2.6.6: Fix and document ncrename attribute renaming; add type conversion functions to ncap
- NCO 2.6.5: Cray, Sun build improvements
- NCO 2.6.4: Mac OS X, NEC, SGI build improvements
- NCO 2.6.3: Mac OS X builds work seamlessly; documentation fixed
- NCO 2.6.2: Major improvements in autotools support; ncap supports user-defined dimensions
- NCO 2.6.1: Minor fix to unpacking support
- NCO 2.6.0: All arithmetic operators (ncap, ncdiff, ncea, ncflint, ncra) support unpacking
- NCO 2.5.6: Handle missing_value correctly for integer variables in ncra, merge autoconf build
- NCO 2.5.5: Handle files without existing global “History” attribute correctly
- NCO 2.5.4: Add support NEC-SX support; update Cray support; small build changes
- NCO 2.5.3: Fix native builds under MS Windows with Cygwin ncap; hint to ncwa
- NCO 2.5.2: Add math functions to ncap; hint to ncwa
- NCO 2.5.1: ncatted deletes all attributes of att_var when att_nm not specified
- NCO 2.5.0: man(1) pages from Debian distribution
- NCO 2.4.6: Free (speech) sort routines
- NCO 2.4.5: Arithmetic speedups
- NCO 2.4.4: 64-bit AIX builds
- NCO 2.4.3: Handle non-standard “History” attributes
- NCO 2.4.2: ncap supports nesting files with #include
- NCO 2.4.0: ncks supports writing native binary output with -b switch
- NCO 2.3.1: ncap builds on all Unices
- NCO 2.3.0: Library redesign
- NCO 2.2.4: ncap fully supports packing and unpacking variables
- NCO 2.2.3: ncap supports manually unpacking variables stored with scale_factor and add_offset
- NCO 2.2.2: fix rare ncwa crash with weights
- NCO 2.2.1: fix ncap divide and fmodf bugs
- NCO 2.2.0: (RPM available) 20020203: ncap public debut (has division error, please upgrade)
- NCO 2.1.3: ncap implements LHS casting
- NCO 2.1.2: C++ API to netcdf; libnco_c++ open for business
- NCO 2.1.0: ncap full lexer/parser operational
- NCO 2.0.3: small fixes to 2.0.0 (e.g., ncks printing)
- NCO 2.0.2: Add packing/unpacking capabilities based on scale_factor and add_offset convention
- NCO 2.0.1: Add user-configurable multi-threaded capability to arithmetic-intensive operators ncwa, ncea, and ncra
- NCO 2.0.0: (RPM available) 20010507: Switched all internal calls to netCDF 3.x API
- NCO 1.2.2: 20010211: Allow weights with
ncwa in min/max/ttl operations; easier building with HDF4.
ncrename can now work on specific variables rather than whole file.
ncks has -q option for cleaner printing
- NCO 1.2.1: 20000828: Added type-conversion of integers to floating point before arithmetic; fixed broken -v option
- NCO 1.2: 20000730: Added non-linear operations to ncra/ncea and ncwa: min,max,total,rms,rmssdn,sqrt. Added DODS compliance.
- NCO 1.1.48: 20000515: Potentially serious bugfix to ncdiff, recommend upgrade
- NCO 1.1.47: 20000510: Improved ncra and ncrcat to allow lazier specifications of input files
Download NCO Source and Binaries
NCO Source Code
The best way to acquire NCO sources is with CVS.
The second best way is to download the source and documentation as a compressed tarfile:
-
nco-3.9.5.tar.gz
(3.1M compressed tar-file)
MD5(src/nco-3.9.5.tar.gz)= 8201af4c7eac5136e5729291a2e835d2
SHA1(src/nco-3.9.5.tar.gz)= 175a53b4e25d97ec5d07e7c297eb0f5435653467
NCO's browsable
CVS Repository
contains up-to-the-minute sources and is the easiest way to stay
synchronized with NCO features.
Retrieving NCO requires some familiarity with GNU development tools,
especially CVS and
Make.
Sourceforge provides
generic instructions
for accessing their CVS servers.
First log into the NCO CVS server:
cvs -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco login
Hit return when asked for the CVS password.
This login procedure is only required the first time you access the CVS server.
Then you may retrieve any NCO distribution you wish.
Usually you wish to retrieve a recent tagged (i.e., released) version.
This command retrieves and places NCO version 3.9.5 (which is
tagged as nco-3_9_5 due to CVS rules) into local directory
nco-3.9.5:
cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -r nco-3_9_5 -d nco-3.9.5 nco
This command retrieves the current (“bleeding edge”)
development version of NCO into a local directory named nco:
cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -kk nco
Track changes to the development version using
cd nco;cvs update -kk
One difference between running a "tagged" release
(e.g., nco-3_9_5) and the development version is that the
tagged release operators will print a valid version number (e.g.,
3.9.5) when asked to do so with the -r flag
(e.g., ncks -r).
The development version simply places today's date in place of the
version.
Once the autotools builds are working more robustly, the confusion
over versions should largely disappear.
Developer NCO Source Documentation
Automated source documentation, created by the
Doxygen tool is available.
Some developers find this documentation helpful, as it can clarify code and data
relationships in the code.
Please be aware that the documentation may be slightly inaccurate and
infrequently updated. Comments and suggestions are certainly welcomed on the
forums.
NCO Binaries and OS-specific Packages
NCO developers are too short-handed to provide pre-built binary
executables for all platforms.
We do attempt to provide (theoretically) platform-independent sources
in the most common UNIX packaging formats (Debian and RPM).
Moreover, the source tarballs are always up-to-date, and work on our
development systems
(Ubuntu-flavored Debian GNU/Linux for x86 and x86_64)
and in our high-end scientific computing environments
(AIX 5 for Power4/5).
Below there are also links to binaries and packages created by
volunteers and organizations who port NCO to other platforms.
We welcome volunteers willing to perform regular regression testing
and porting of NCO to other platforms.
Debian and
Ubuntu
GNU/Linux homepages for NCO.
‘aptitude install nco’ installs the standard NCO for your Debian-compatible OS.
- Debian package for most recent NCO release (install with, e.g., ‘dpkg --install nco_3.9.5-1_i386.deb’):
- nco_3.9.5-1_amd64.deb (954K): Executables AMD64-compatible (last updated Monday, 12-May-2008 13:41:41 UTC)
- nco_3.9.5-1_i386.deb (951K): Executables i386-compatible (last updated Monday, 12-May-2008 14:04:17 UTC)
- nco_3.9.5-1.dsc (409 ): Description (last updated Monday, 12-May-2008 14:01:03 UTC)
- nco_3.9.5-1_i386.changes (724 ): Changes since last deb package (last updated Monday, 12-May-2008 14:04:17 UTC)
- nco_3.9.5.orig.tar.gz (1.5M): Upstream Source (last updated Monday, 12-May-2008 14:01:01 UTC)
- nco_3.9.5-1.diff.gz (724 ): Debian patch to upstream source (last updated Monday, 12-May-2008 14:04:17 UTC)
Thanks to Daniel Baumann, Francesco Lovergine, Brian Mays, Rorik
Peterson, and Matej Vela for packaging NCO for Debian over the years.
- Our most recent NCO RPMs (install with, e.g., ‘yum install nco-3.9.5-1.fc7.i386.rpm’):
- nco-3.9.5-1.fc7.i386.rpm (790K): Executables for i386/Fedora Core 7-compatible environments (last updated Monday, 12-May-2008 12:56:53 UTC)
- nco-3.9.5-1.fc7.src.rpm (1.5M): Source (last updated Monday, 12-May-2008 12:56:52 UTC)
- nco-3.9.5-1.x86_64.rpm (952K): Executables for x86_64/CentOS 5-compatible environments (last updated Monday, 04-Aug-2008 19:31:27 UTC)
- More information on x86_64 RPMs available here
From about 2004–2006, NCO was part of the Fedora Extras distribution (thanks to Ed Hill).
Then NCO RPMs in Fedora Core release
lagged our upstream RPMs by no more than 6–8 months.
A new volunteer is needed to update NCO to 3.9.x and maintain it in the Fedora distribution.
Searching rpmfind.net for NCO yields these
NCO RPMs:
- nco-0-3.1.5-3.fc6.html: Official Fedora Core 6 NCO binary package (contains all executables) homepage
- nco-devel-0-3.1.5-3.fc6.html: Official Fedora Core 6 NCO development package (contains libraries and headers) homepage
- Fink packages for NCO: Currently NCO 3.9.2. Maintained by Alexander Hansen.
- MacPorts infrastructure for NCO:
Portfile
for NCO 3.9.5. Maintained by Takeshi Enomoto.
Documentation and User's Guide
View the NCO User's Guide in any of these formats:
- DVI Device Independent (kdvi, xdvi)
- HTML Hypertext (any browser)
- Info GNU Info (M-x Info, emacs)
- PDF Portable Document Format (acroread, evince, kpdf, xpdf)
- Postscript Printing (ghostview, kghostview)
- TeXInfo Documentation Source code (emacs)
- Text Plain text (more)
- XML Extensible Markup Language (firefox)
nco.texi is the most up-to-date.
Files nco.dvi, nco.ps, and nco.pdf are
the most complete and contain all the mathematical formulae (typeset
with TeX) missing from the screen-oriented formats.
The screen-oriented formats—nco.html,
nco.info, nco.txt, and nco.xml—contain
all the documentation except the highly mathematical sections.
Wiki:
- As of August, 2008, NCO has a Wiki.
The Wiki contains a Cookbook
of NCO examples not found in the manual.
Volunteers are encourage to add their own “NCO Hacks” to this cookbook!
Other documentation:
- This abbreviation key unlocks the mysteries of the source code abbreviations and acronyms.
- Unidata netCDF for Developers Workshop NCO examples
README/ChangeLog/TODO
Files containing useful information about the current NCO distribution:
- README Platforms and software requirements
for a successful NCO installation
- ChangeLog Up-to-date history of
NCO changes
- TODO An unordered list describing the features and
fixes we currently have in mind for NCO.
FAQ: Frequently Asked Questions
These questions show up almost as frequently as my mother.
But they are more predictable:
- I still have questions, how do I contact the NCO project?
The NCO project has various Q&A and discussion forums described
below.
- Where can I find prebuilt NCO executables?
Pre-built executables of some versions of NCO for the operating
systems described above (Debian-compatible
GNU/Linux, Fedora/RedHat GNU/Linux, Gentoo GNU/Linux, and
Mac OS X).
If you have an account at UCI or SDSC, try the locations in my
personal home directories listed below (AIX, IRIX).
At NCAR, first try /usr/local/bin (AIX, IRIX).
Otherwise, you are on your own.
- Does NCAR support NCO?
The NCAR CISL Technical Consulting Group (TCG) supports NCO like other
community software packages such as lapack.
No other NCAR division has coordinated NCO support.
The NCAR CISL-suported executables are in /usr/local/bin.
If you notice problems with the NCO installation on SCD machines,
or if you would benefit from a more recent release or patch, then
inform the relevant system administrators, e.g.,
NCAR User Support.
If you have a comment, suggestion, or bug report, then contact the
developers as described below.
- Is there an easy way to keep up with new NCO releases?
Subscribe to the
nco-announce
mailing list.
This list is for NCO-related announcements, not for questions.
nco-announce is very low volume, averaging one message every few months.
Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)
Users at UC Irvine, NCAR, and SDSC may find pre-built,
almost up-to-date NCO executables in the following locations.
These executables are unsupported and are almost
guaranteed to hail from different NCO versions and thus to
behave slightly differently.
(They are built from the main trunk, not a tagged version, and
currently it is impossible to tell exactly what version they are).
Moreover they may be compiled with debugging enabled and thus may be
slower than executables optimized for production use.
Users at NCAR should try the supported executables in
/usr/local/bin first.
- ESS soot.ess.uci.edu (Linux 2.6.19): /home/zender/bin/LINUX
- ESS esmf.ess.uci.edu (AIX 5.1.0.0): /home/zender/bin/AIX
- ESS sand.ess.uci.edu (Linux 2.6.19): /home/zender/bin/LINUXAMD64
- CGD network (Solaris 2.7): /home/zender/bin/SUNMP
- SCD blueice.ucar.edu (AIX 5.3): /usr/local/bin
- SCD bluevista.ucar.edu (AIX 5.3): /usr/local/bin
- SCD tempest.ucar.edu (IRIX 6.5): /usr/local/bin
Benchmarks
We are benchmarking NCO in SMP (e.g., AIX) and MPI (e.g., Linux) environments.
The full NCO Benchmarks (in PDF form) are available
here.
Please send us your suggestions for improved configurations.
The NCO distribution includes all the custom benchmarking software,
and is designed to allow you to contribute your own benchmarks.
Contact us if you're interested.
Current tests include data stored locally and across OPeNDAP servers.
We will benchmark NCO with new features (e.g., netCDF4) and hardware
(e.g., parallel filesystems) as time permits.
There are two parallel versions of NCO—one uses
OpenMP on SMP (including
hyperthreaded) machines and the other uses the
Message Passing Interface (MPI)
on distributed architectures.
OpenMP requires a compiler that supports it.
Commercial compilers which we have used include those from
IBM
and the Portland Group.
Intel's Linux compiler is free (beer).
The MPI operators will run with most MPI implementations, including LAM, MPICH, and MPICH2.
LAM installs easily with our own testing environment (latest Ubuntu).
Build-from-Source Requirements
Best Practices:
- Install
ANTLR 2.7.x (required for ncap2),
GSL (required for ncap2),
netCDF (absolutely required),
OPeNDAP (enables network transparency), and
UDUnits (allows dimensional unit transformations)
before building NCO.
If possible build these all with the same compiler (and switches).
NCO is mostly written in C99, and although you can mix and
match compilers, this is often difficult in practice and is not recommended.
The exception is ncap2 which is written in C++.
ANTLR, OPeNDAP, and NCO must be built with the same C++ compiler
to properly resolve the C++ name-mangling.
NCO does not yet support newer ANTLR versions because the
ANTLR 3.x C++ interface is incomplete.
Hints to Build-from-Source:
Generic Build Hints:
- The two NCO build mechanisms:
NCO has two build mechanisms: (1) GNU configure and
(2) a custom Makefile (nco/bld/Makefile)
designed for hand-editing.
Try the traditional GNU configure method first, i.e.,
./configure;make;make check;make install.
This method involves no hand-editing of makefiles, but trouble-free
builds with it are less common than we would like.
The file nco/configure.eg contains a table which describes the
success you can expect building NCO with the configure
mechanism.
Please contribute patches required to get the configure
mechanism working properly for NCO on your platform.
The regressions archive
contains gory details of successful (and failed) builds for many platforms.
All platforms continue to work fine with the nco/bld/Makefile
method: cd ~/nco/bld;make all;make test.
Disadvantages are that nco/bld/Makefile (1) Usually requires
editing to fix your platforms paths; (2) Does not support shared
libraries; and (3) Requires a degree from Hogwarts to understand.
- Build Chain Version Incompatibilities:
NCO requires GNU make version ≥ 3.78.1 to support target-specific variable values in bld/Makefile.
NCO most often fails to build because ncap fails to build.
Two of the most common reasons that ncap fails to build are
(1) GNU bison is out-of-date (upgrade bison)
and (2) --enable-optimize-* is specified to
configure (do not specify it).
- C99-compliance & old compilers:
NCO versions ≥ 2.9.8 requires compilers which
support the 1999 ISO C-Standard called C99.
Currently, the only C99 feature NCO takes advantage of is restricted
pointers.
To build NCO versions ≥ 2.9.8 without a C99-compliant
compiler, simply undefine the restrict type qualifier
before compiling NCO.
Do this, e.g., with CPPFLAGS='-Drestrict='.
More details on C99 compatibility are in the header file
nco/src/nco/nco.h.
- Old netCDF libraries:
NCO versions ≥ 3.0.0 require netCDF versions
≥ 3.6 to take advantage of the new 64-bit offset file type.
The library and include directories (e.g., /usr/lib and
/usr/include, respectively) for netCDF version ≥ 3.6
must be on the path searched by the compiler, and must precede the
locations of any older netCDF installations.
To build NCO versions ≥ 3.0.0 with older netCDF versions
< 3.6, simply define the pre-processor token
NC_64BIT_OFFSET to 0 before compiling NCO.
Do this, e.g., with CPPFLAGS='-DNC_64BIT_OFFSET=0' or
USR_TKN='-DNC_64BIT_OFFSET=0' if you use configure
or bld/Makefile, respectively.
Users attempting to build NCO with OPeNDAP versions ≤ 3.6.x may
need this workaround because OPeNDAP did not fully support netCDF
version 3.6.x functionality until OPeNDAP versions ≥ 3.7.0.
(The indicator for this problem is a build error like
"nco_fl_utl.c", line 30.20: 1506-045 (S) Undeclared identifier
NC_64BIT_OFFSET.)
All modern NCO versions require netCDF versions ≥ 3.5.0 to
support nc_type NC_NAT in NCO source code.
NCO versions 3.9.0 and greater (May, 2007) require netCDF versions
≥ 3.6.0 to resolve the nc_inq_format() function.
- ncap build hints:
ncap is a very sophisticated and somewhat fragile program.
ncap requires flex to build correctly.
AT&T lex will not work because it does not understand
EOF rules that ncap_lex.l uses.
bison version 1.29 (the default version on early
Mac OS X) is buggy and does not parse ncap_yacc.y
correctly; previous and later versions of bison work
correctly.
bison version 1.25 will not build ncap on Solaris.
Please upgrade bison to the latest version (e.g., 1.875+)
before reporting ncap build problems.
If ncap keeps you from building NCO, and you do not need
ncap, configure with --disable-ncap (and know that
ncwa will fail to build as a result).
- Build Hints with OPeNDAP/DODS:
NCO versions 3.1.3— support automatic building and linking
for OPeNDAP versions 3.6.x and
GCC versions 4.0+.
NCO versions 3.0.1—3.1.2 support automatic building and linking
for OPeNDAP versions 3.5.x and
GCC versions 3.4+.
NCO versions 2.8.5—3.0.0 support automatic building and linking
for OPeNDAP versions 3.4.x and
GCC versions 3.3+.
NCO versions ?.?.?—2.8.4 support automatic building and linking
for DODS versions 3.2.x and
GCC versions 3.2+.
No other version combinations are supported.
- libnco_c++ build hints:
libnco_c++ requires a good C++ compiler to build correctly.
Few people need libnco_c++, so the workaround is not to build it.
This is done by configuring with --disable-nco_cplusplus.
Platform-Specific Build Hints:
- AIX build hints:
NCO versions 2.8.7+ do not build with autotools under AIX when
--enable-optimize-* is selected.
The workaround is to not use --enable-optimize-*.
NCO versions 2.9.7+ do not build shared libraries with AIX xlc
compilers because of silly AIX library dependencies activated
by OpenMP directives.
The workaround is to configure with --disable-shared.
- IRIX build hints:
The most common problem with building on SGI IRIX is and out-of-date toolchain.
- Mac OS X build hints:
Mac OS X users report success building NCO without patches so
long as they (1) Install the latest version of bison,
(2) Disable shared libraries and (3) Disable extended regular
expressions. This is done with
./configure --disable-regex --disable-shared
Building NCO with shared libraries on Mac OS X on may be possible.
To try, first re-build and re-install the netCDF library using
CFLAGS=-fno-common.
Apparently netCDF, unlike most codes, does not set this automatically.
Then build NCO normally.
Mac OS X ships with a broken version of the POSIX cut
utility which causes many NCO regression tests to appear to fail.
In fact the NCO commands succeed, but the cut output fails.
The workaround is not to heed the regression tests, or to install a
working cut, e.g., from the GNU coreutils package.
- Solaris build hints:
All recent NCO versions build with autotools on Solaris 2.7—2.9.
The most common problem with building on Solaris with
cc/CC is that bison may be out-of-date.
Install an up-to-date bison (e.g., 1.875+).
Solaris 2.10, i.e., Solaris10, had the audacity to change the
prototype of the ctime_r() function and so does not build
NCO 3.0.1.
The fix is to upgrade to the current NCO version.
- SX build hints:
NEC SX does not understand the S_ISLNK macro,
preferring instead the (non-POSIX) S_IFLNK macro.
This macro appears in the file
nco_fl_utl.c.
The patch I wrote for this may be broken.
Will someone with NEC SX please send us a working patch?
- Windows build hints:
Building on Microsoft Windows is possible with Cygwin.
The resulting executables will behave normally on local files,
but will not have most of NCO's network functionality
(e.g., scp, OPeNDAP).
See the User's Guide
for more details.
Known Problems
Problems with current release:
Platform-specific Run-time Problems:
- Float-valued intrinsic arithmetic functions in ncap on AIX:
ncap versions through 3.9.5 have a bug that causes all float-valued
intrinsic math functions to fail under AIX.
Float-valued math functions are the ISO C99 functions, e.g.,
cosf(), fabsf(), logf().
The user does not invoke these functions directly—
the user always specifies the generic function name, e.g.,
cos(), abs(), log().
NCO automatically calls the native single precision (i.e.,
float-valued) math functions when the generic function argument
is a native float (e.g., naked constants like 1.0f or
variables stored as NC_FLOAT).
Double precision arguments cause NCO to invoke the standard
(double-valued) form of the generic function, e.g., cos(),
fabs(), log().
Problems with older releases:
Generic Run-time Problems:
- Index-based hyperslab problems:
NCO versions 2.7.3—2.8.3 have a nasty bug that causes
index-based hyperslabs, e.g., -d lat,1, to
behave like value-based hyperslabs, e.g., -d lat,1.0 under
some conditions.
Unfortunately, the incorrect answers generated may be hard to notice!
This problem was most often enountered by users trying to assemble
monthly averages using the stride feature of ncrcat.
One common symptom is that the time-offset of the output file is
incorrect.
Versions 2.7.3—2.8.3 are buggy and should be upgraded ASAP.
Re-do any data-processing that used index-based hyperslabbing with
these versions of NCO.
Operator-specific Run-time Problems:
- Arithmetic problems with ncap division, modulo, and exponentiation:
ncap versions < 3.0.1 incorrectly exponentiate
variables to variable powers (V^V).
We recommend that all ncap users upgrade.
ncap versions up to 2.9.1 incorrectly handle division,
modulo, and exponentiation operations of the form S/V,
S%V, and S^V where first operand (S) is
scalar (i.e., either typed directly in the ncap script or
converted from an attribute) and the second operand (V) is
a full variable (i.e., stored in a file or computed by ncap).
Instead of the requested quantity, ncap returned
V/S, V%S, and V^S.
In other words ncap treated some non-commutative operations
as commutative. This is now fixed.
The
V/V, V%V, V^V,
V/S, V%S, V^S,
S/S, S%S, and S^S operations were never
affected.
We recommend that all ncap users upgrade.
- Incorrect ncbo output for packed input:
ncbo versions ???—3.2.0 incorrectly write differences
of packed input. This only affects packed variables.
- Problems with ncflint and missing_values:
The algorithm ncflint used to perform interpolation in
versions up to 2.9.4 was not commutative.
It returned the weighted valid datum when the other datum was
missing_value, or it returned missing_value,
depending on the order the input files were specified.
As of version 2.9.5, ncflint always returns
missing_value when either input datum is
missing_value.
Possible future implementations are discussed
here.
- Problems with ncra and ncea when missing_value = 0.0:
The algorithm ncra and ncea used to perform
arithmetic in versions up to 2.9.2 breaks if missing_value
is 0.0.
Why, you ask?
Running average (or total, etc.) algorithms must initialize the answer
to 0.0.
This is done since the sum accumulates in place as ncra and
ncea proceeds across records and files.
(Normalizing this accumulation by the total number of records is the
last step).
The old algorithm compared both the current running average and the
new record to the missing_value.
If either comparison matched, then nothing accumulated for that
record.
This zero-initialization led to a state where it was impossible
to ever recognize valid data.
As a result nothing accumulated and the answer was always zero.
The record and ensemble averages would also fail (in a non-obvious)
way whenever an intermediate sum equalled missing_value.
The chances of the latter event ever happening are exceedingly
remote.
The new algorithm compares only the new record to the
missing_value.
This fixes both problems and is faster, too.
- Packing problems with ncwa:
NCO versions ???—2.9.0 have a bug that causes ncwa
to fail (produce garbage answers) when processing packed
NC_FLOAT data. Version 2.9.1 fixes this problem.
This problem may have been noticed most by
OPeNDAP users since many
netCDF climate datasets served by
OPeNDAP are packed
NC_FLOATs.
Upgrade to 2.9.1 if you use ncwa on packed data.
- Packing problems with ncap:
NCO versions 2.8.4—2.8.6 have a bug that causes the ncap
intrinsic packing function pack() to fail.
Version 2.8.7 fixes this problem.
People:
(Please communicate with us using the project forums rather than contacting us individually)
Contacting the Project:
Contact us through the Project Forums (rather than personal e-mail)
so other users can benefit from and contribute to our exchange.
Let us know how NCO is working for you—we'd like to hear.
Have you read the documentation and browsed the
forums to see if your question/comment has been reported before?
Please read the Guide's suggestions for productive
Help Requests and Bug Reports.
- Where should I ask my questions on how to use NCO?
On the Help site.
- Where should I post suggestions/comments on NCO features and usage?
On the Discussion site.
- Where are NCO development and bug-squashing discussed?
At the Developer
site.