/ Announce / Developer / Discussion / Help / Manual / Homepage / Project / Source / Wiki / [ANTLR] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]

Welcome to the netCDF Operator (NCO) Homepage

Current NCO version is 3.9.5 last updated on Monday, 20-Oct-2008 04:01:25 UTC

The temporary NCO logo…
Have a better one?
Send it in!

NCO News & Announcements

What is NCO?

The netCDF Operators, or NCO, are a suite of programs known as operators. Each operator is a standalone, command line program which is executed at the UNIX shell-level like, e.g., ls or mkdir. The operators take netCDF files as input, then perform a set of operations (e.g., deriving new data, averaging, hyperslabbing, or metadata manipulation) and produce a netCDF file as output. The operators are primarily designed to aid manipulation and analysis of gridded scientific data. The single command style of NCO allows users to manipulate and analyze files interactively and with simple scripts, avoiding the overhead (and some of the power) of a higher level programming environment. The NCO User's Guide illustrates their use with examples from the field of climate modeling and analysis. Note that the “averagers” are misnamed because they perform many non-linear operations as well, e.g., total, minimum, maximum, RMS:

The operators are as general as netCDF itself: there are no restrictions on the contents of the netCDF file(s) used as input. NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases. NCO works well on most modern operating systems, including: Apple OS X, *BSD, Cray UNICOS, DEC Tru64, IBM AIX, HPUX, Linux, Microsoft Windows, NEC Super UX, SGI IRIX, and Sun Solaris.

How to Contribute

In March, 2000, NCO became an Open Source project hosted by SourceForge.net. This facilitates collaboration, code contributions, and support. There is a TODO list about 200 items long! No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The NCO project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

As of June, 2003, you may donate money to NCO via Paypal. Your donation will be distributed to NCO developers as an incentive for them to tackle TODO items on a pay-per-feature basis.

These homepage donation counters track the influence of your monetary donations on NCO development:
Donations received between 20030624 and 20070531: US$60.00.
NCO features "incentivized" by these donations: None.

NCO→SDO Project

The National Science Foundation NSF IIS-0431203 funds our NCO→SDO Project, “SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data” as part of the Science and Engineering Informatics (SEI) program. We appreciate the proposal reviewers for and staff of the Divisions of Information and Intelligent Systems (IIS) and Shared Cyberinfrastructure (SC) in the Directorate for Computer and Information Science and Engineering (CISE). NCO development was completely voluntary and without institutional support from its inception until September 2004. Hence this NSF support has dramatically changed the scale and pace of NCO/SDO development. This URL, http://nco.sf.net#prp_sei, points to the most up-to-date information on the NCO/SDO proposal.

The NSF project provides the human and technical resources to analyze, design, and implement novel advanced methods for distributed data analysis and reduction (DDRA) of geophysical data. These powerful methods will be “grid-enabled”, and we will demonstrate their efficacy with analyses of distributed climate prediction and observation datasets. On the human side, recruited full-time help for specific areas of SDO, alongside our current and future volunteers. We have a full-time programmer to help design, implement, and release major code changes. We have a graduate student researcher pursuing innovative Ph.D. research in using “grid-aware” distributed memory, shared memory, and client-server software technologies to improve DDRA of climate datasets. Another graduate student helps us design, implement, and release major code changes. Contact us if you are seriously interested in tackling these problems on a full-time, part-time, or pay-per-feature basis.

On the institutional and hardware side, this project connects the Earth System Modeling Facility (ESMF) to the California Institute for Telecommunications and Information Technology (Cal-IT2) OptIPuter at the San Diego Supercomputer Center. These supercomputers will dedicate a TB of storage each to OPeNDAP-served climate simulation datasets for DDRA. Once the proof-of-concept SDO DDRA is complete, we will attempt a DDRA intercomparison of the internationally distributed global climate change simulations that reside on the Earth System Grid. We expect our analyses to yield otherwise undiscovered patterns among different climate prediction scenarios for the same model, and among different climate models.

Finally we still encourage you to e-mail NCO/SDO endorsements to Charlie “my surname is zender” Zender with Subject: “NCO/SDO Proposal Endorsement”. An endorsement can be a sentence or a page, describing how NCO/SDO benefits your work or research. The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO/SDO proposal advocacy. “What future proposals?” you ask, “Didn't you just get funded?” Yes, we are funded to optimize and “grid-enable” NCO. There are still many desirable and unexplored methods for distributed data analysis and reduction (DDRA) that would benefit geosciences. For example, have you tried analyzing multiple NASA satellite datasets in HDF5 format? Need I say more? If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.

Publications and Presentations

Version Summary

New releases receive unique CVS tags and their tarballs are stored here on SourceForge and here on my server. You can also retrieve the source of tagged versions directly with CVS.

Download NCO Source and Binaries

NCO Source Code

The best way to acquire NCO sources is with CVS. The second best way is to download the source and documentation as a compressed tarfile:

NCO's browsable CVS Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Retrieving NCO requires some familiarity with GNU development tools, especially CVS and Make. Sourceforge provides generic instructions for accessing their CVS servers. First log into the NCO CVS server:

cvs -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco login

Hit return when asked for the CVS password. This login procedure is only required the first time you access the CVS server. Then you may retrieve any NCO distribution you wish. Usually you wish to retrieve a recent tagged (i.e., released) version. This command retrieves and places NCO version 3.9.5 (which is tagged as nco-3_9_5 due to CVS rules) into local directory nco-3.9.5:

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -r nco-3_9_5 -d nco-3.9.5 nco

This command retrieves the current (“bleeding edge”) development version of NCO into a local directory named nco:

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -kk nco

Track changes to the development version using

cd nco;cvs update -kk

One difference between running a "tagged" release (e.g., nco-3_9_5) and the development version is that the tagged release operators will print a valid version number (e.g., 3.9.5) when asked to do so with the -r flag (e.g., ncks -r). The development version simply places today's date in place of the version. Once the autotools builds are working more robustly, the confusion over versions should largely disappear.

Developer NCO Source Documentation

Automated source documentation, created by the Doxygen tool is available. Some developers find this documentation helpful, as it can clarify code and data relationships in the code.

Please be aware that the documentation may be slightly inaccurate and infrequently updated. Comments and suggestions are certainly welcomed on the forums.

NCO Binaries and OS-specific Packages

NCO developers are too short-handed to provide pre-built binary executables for all platforms. We do attempt to provide (theoretically) platform-independent sources in the most common UNIX packaging formats (Debian and RPM). Moreover, the source tarballs are always up-to-date, and work on our development systems (Ubuntu-flavored Debian GNU/Linux for x86 and x86_64) and in our high-end scientific computing environments (AIX 5 for Power4/5). Below there are also links to binaries and packages created by volunteers and organizations who port NCO to other platforms. We welcome volunteers willing to perform regular regression testing and porting of NCO to other platforms.

Debian and Ubuntu GNU/Linux

Fedora and CentOS GNU/Linux

Gentoo GNU/Linux

Mac OS X/Darwin

Documentation and User's Guide

View the NCO User's Guide in any of these formats:

nco.texi is the most up-to-date. Files nco.dvi, nco.ps, and nco.pdf are the most complete and contain all the mathematical formulae (typeset with TeX) missing from the screen-oriented formats. The screen-oriented formats—nco.html, nco.info, nco.txt, and nco.xml—contain all the documentation except the highly mathematical sections.

Wiki:

Other documentation:

README/ChangeLog/TODO

Files containing useful information about the current NCO distribution:

FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

Users at UC Irvine, NCAR, and SDSC may find pre-built, almost up-to-date NCO executables in the following locations. These executables are unsupported and are almost guaranteed to hail from different NCO versions and thus to behave slightly differently. (They are built from the main trunk, not a tagged version, and currently it is impossible to tell exactly what version they are). Moreover they may be compiled with debugging enabled and thus may be slower than executables optimized for production use. Users at NCAR should try the supported executables in /usr/local/bin first.

Benchmarks

We are benchmarking NCO in SMP (e.g., AIX) and MPI (e.g., Linux) environments. The full NCO Benchmarks (in PDF form) are available here. Please send us your suggestions for improved configurations. The NCO distribution includes all the custom benchmarking software, and is designed to allow you to contribute your own benchmarks. Contact us if you're interested.

Current tests include data stored locally and across OPeNDAP servers. We will benchmark NCO with new features (e.g., netCDF4) and hardware (e.g., parallel filesystems) as time permits.

There are two parallel versions of NCO—one uses OpenMP on SMP (including hyperthreaded) machines and the other uses the Message Passing Interface (MPI) on distributed architectures. OpenMP requires a compiler that supports it. Commercial compilers which we have used include those from IBM and the Portland Group. Intel's Linux compiler is free (beer). The MPI operators will run with most MPI implementations, including LAM, MPICH, and MPICH2. LAM installs easily with our own testing environment (latest Ubuntu).

Build-from-Source Requirements

Hints to Build-from-Source:

Known Problems

Problems with current release: Problems with older releases:

People:

(Please communicate with us using the project forums rather than contacting us individually)

Contacting the Project:

Contact us through the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.

SourceForge.net Logo
/ Announce / Developer / Discussion / Help / Homepage / Manual / Project / Source / Wiki / [ANTLR] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]