Data Management

 

Program Directors: Karl Kaufmann & Steven Paton
In 1991, The Environmental Science Program contracted the services of a full-time data manager to manage data from the terrestrial programs of the ESP. In 1993 a second data manager was added to manage the data of the Marine programs. For the next 10 years both data managers managed their respective data sets in parallel. Although both programs were managed according to common principals and goals, different approaches were sometimes taken to solve similar issues. Beginning in 2004, efforts have been undertaken to unify the two programs. This effort is ongoing and should be completed by 2005. This web site is the first major result this undertaking.
Diiferent problems require different solutions. Several different approaches to the management of the ESP data have and are currently being used. The management of the biological and physical data is primarily done with Visual Foxpro (VFP), a commercial database program for PC microcomputers. This program was chosen because it offers a well regarded program development environment and is well suited to on-the-fly data analysis.  This program has become the de facto database management program among the main database managers at STRI.

A few terrestrial data sets are also stored in Excel spreadsheets.

Data manager write programs for data aquisition, storage and analysis, and oversee the acquisition and archiving of project data. Some older programs are written in older, DOS versioins of Foxpro and dBase, are in the process of being upgraded to Visual Foxpro.

All Marine program data are managed by the marine data manager. Only some of the terrestrial programs are directly managed by the terrestrial data manager (physical monitoring, plant reproduction, mammals). The remainder of the terrestrial program PI's manage their own data, but are required to periodically provide the data manager with a copy of their data as well as related metadata.
VFP programs have been developed to coordinate the acquisition of physical data from the instruments in the field into our database and to archive the resulting data files. Sensors record data into loggers which then transfer up to several months of data into text files over a serial interface. Each logger produces a text file with a format unique to its manufacturer. These files must be translated into database files, screened for errors, and then appended to the appropriate main data file.
To facilitate this process, and to reduce the chances of technician error,  Foxpro sub-programs have been written to handle as many routine chores as possible. The program automatically senses the format of a text data file from a logger, applies the proper translation routine, and creates a raw data file. If the sensor requires a calibration correction, the program prompts for the sensor ID, looks up the proper calibration correction and applies it to the data.
It then looks up the expected ranges for each type of data and marks suspicious data in the raw data file.Once those checks have been performed data are transferred into the main data file. The program ensures that the data is going into the proper main data file by checking a list of main data files and the corresponding raw data files and backup files. The particular file, whether from the primary or backup sensor, that was used in the main file is then recorded so that the origin of any datum can be traced if necessary.
With this program, the chief technician can handle the entire process, from downloading the data to screening the data to appending the relevant files to the main data file. It was written so that new monitoring equipment could be quickly and easily added to the system by writing a translation program specific to the particular format used by the new sensor and then adding a record with the ranges for bad data to the proper file. Once this is done, all of the other features for browsing, screening, graphing, updating and documenting the data become available to the user. This kind of flexibility allows the addition of new sites and new equipment with a minimum of extra programming and training.