Data Quality Program
Noteworthy Information
- Send in Data Quality Findings
- Data Quality Health and Status (DQ HandS)
- Data Quality Problem Report (DQPR)
- Problem Identification Report/Corrective Action Report/Data Quality Report (PIF/CAR/DQR)
- Instrument-Specific Data Quality Information and Guidance (Instrument Pages)
- ARM Data Archive (Archive)
- Value-Added Products and Quality Measurement Experiments (VAP/QME)
Introduction
One of the goals of the ARM Program is to provide data streams of quality suitable for scientific research. Maintaining data quality for a program of the size and complexity of ARM is a significant challenge - efforts toward this end have matured and evolved over the life of the program.
The ARM Program Data Quality (DQ) Office was established in July 2000 to help coordinate the continued evolution and implementation of efforts to assure the quality of the data collected by its field instrumentation. The DQ Office has the responsibility for ensuring that quality assurance results are communicated to 1) data users so that they may make informed decisions when using the data, and 2) ARM's Site Operations and Engineers to facilitate improved instrument performance and thereby minimize the amount of unacceptable data collected.
Toward these goals, the DQ Office, instrument mentors, site scientists, and others in ARM help review and assess ARM's data streams and write and submit appropriate Data Quality Reports (DQRs) as needed. These groups also work closely with the ACRF site operators to impart information about data quality that will initiate troubleshooting and/or corrective maintenance activity. Work is also done to devise and implement schemes to flag questionable data as they are processed into ARM data files. To view ARM's file hierarchy structure, please read about the Data Level Documentation.
In addition, the DQ Office participates in the collection and presentation of documentation on the data quality program, such as the web page you are now reading, to help achieve consistency of presentation within the ARM Program and to make this information available and useful to data users. This includes ARM's web pages describing its instruments and measurements. Standardization of the quality assurance procedures that are applied to data is also an ongoing process in order to establish data quality baselines and protocols for each instrument that are consistent across the three research sites.
The results of routine data stream checking, often on an instrument by instrument basis, allow us to identify where known problems and questionable data are so that we may communicate this information to data users and the ARM infrastructure. We can also communicate about how we made these determinations. We in general will deem unflagged data as good for most purposes. Higher-level data stream intercomparisons and the generation of value added products can augment the routine checking by giving us an idea of the relative utility of data streams, and in essence, tell us "how good" the good data might be. These higher-level checks can also point out deficiencies that are not necessarily detectable within individual data stream checks. The creation of the higher level products also provides the user community with heavily-screened, best-estimate data sets ready for use in high-level scientific research.
The sections within this page provide general description and documentation of the various parts of ARM's Data Quality Program. This page, along with instrument- and measurement-specific information on data quality that is contained within the instrument web pages, represents a Quality Control Manual for the ARM Program. These will be living documents that will allow us the ability to provide updates, changes, and notices of progress.
Data Quality Health and Status - Working Display of Current and Recent Quality Control Results
The Data Quality Office has produced a web area dedicated to displaying data quality "health and status" results in the near real-time so that those involved in quality assurance activities can monitor data quality. It is known as "DQ HandS".
- Data Quality Health & Status (DQ HandS)
Within this site you can find color characterizations of file flags on both a daily and hourly basis, a diagnostic plots browser, an interactive plotting capability (NCVweb), and links to supporting information such as instrument and maintenance logs and various reports related to the data quality effort. NCVweb allows for easy plot manipulation and the display of file headers, variable details and statistics, and listings of the data values themselves. It also allows you to create an ASCII copy of the data file on your own computer.
- Interactive Data Plotting Tool (NCVWeb)
Instrument Web Pages - Specifications and Expectations
The ARM Program has created a series of detailed web pages that describe the specifications of its instruments. A fundamental of quality control is a "statement of expectations." Quality is the measure of how closely something conforms to an expectation. Without an expectation, a quality assessment is not possible. Thus, the instrument pages represent ARM's statement of expectations for its instruments, the baseline against which the observations can be compared. These web pages include our current understanding of the measurement systems and their quirks and deficiencies, including common problems encountered or inherent to the measurement.
These pages are also the place for you to find specific information about the data quality for each instrument. For a quick summary of ARM instruments at each field site, view the Instrument Location Table or try the alphabetized list. To learn about the data the ARM instruments collect, see measurements.
Automated Quality Control - Flagging Contained Within Data Files
Automated Quality Control refers to data quality checks that are applied when data streams are processed by ARM, generally during ingest into the site data systems. For most ARM instruments at present, data are checked for violations of simple physical limits (minimum, maximum) and maximum change (delta) on each data field. Samples that exceed these criteria are flagged and these flags are included in the netCDF files. Some instrument data files contain the results of more sophisticated quality checks. The results of automatic flagging are shown as color codes in the ARM Data Browser.
Information about flagging is included in the header (known as a Data Object Design or DOD) of each netCDF file. The data user needs to carefully read and consider this header information and any quality flags when processing their data.
The format in which the ARM Program stores quality control information in data files changed in spring 2001 to improve the consistency of data representation across the three fixed ARM research sites.
Quality Control Applied by Instrument Mentors, Site Scientists, and the Data Quality Office - "Value-Added" Checking
Instrument Mentors are charged with developing the technical specifications for instruments procured for the ARM Program. The instrument mentor then tests and operates the instrument system either at his or her location or at one of ARM's three research sites. In addition, the mentor works with data system personnel on data ingest software requirements. Data ingest involves the conversion of data streams to the International System of Units (SI), as well as the acquisition of parameters that can be used to monitor instrument performance (e.g., monitoring an instrument's output voltage for a 5-V power supply or the continuity of the wire in a hot-wire anemometer). Data collection and ingest, then, are the focus of this first level of data quality assurance. Quality at this level is monitored routinely by instrument mentors and site operators. Instrument mentors provide all calibration, operations, and maintenance documents and lists of spare parts to site operations. Typically, the mentor provides additional detailed documentation and hands-on training so that appropriate support can be offered by site operators.
A next level of mentor data quality assurance involves beta release of data streams from individual instruments. The mentor receives the data from the instrument to determine whether the technical specifications of the instrument are being met. When the mentor is satisfied that the instrument is functioning properly and the technical specifications have been met, the data are formally released to the Science Team and other data users. After this release, the instrument mentor is charged with reviewing and reporting on the state of the instrument's data on a monthly basis. Mentors report their findings to data users via a monthly summary report and Data Quality Reports. Mentors also provide the DQ Office with guidance on how to perform daily-to-weekly data quality inspections and assessments. Instrument mentors are key players in the problem resolution process.
The DQ Office provide the first line of defense for routine data inspection and assessment activities for data collected from all ARM sites by performing daily to weekly checks of the data. The DQ Office also extracts when possible data quality information from Value-Added Products and Quality Measurement Experiments to further evaluate the quality of the instrument data streams. The main goal of this inspection and assessment is to provide quick notification to site operators and mentors whenever irregularities in data quality are found to minimize the amount of unacceptable data collected at a field site.
DQ Office inspections are based on automatically generated tables of automated QC results, near real-time diagnostic plots, and supporting evidence such as site operations maintenance information and access to various reports. Data streams from different instruments are often compared and the quality of the data is assessed based on guidelines developed or prescribed by instrument mentors. When problems are encountered, mentors, site operators and site scientists are notified to start the problem resolution process and/or to do more investigation. The DQ Office issues weekly instrument assessment reports to instrument mentors. Problem reports are issued when action needs to be taken to solve a problem.
Site Scientist data quality assessment efforts involve more in-depth evaluation of both individual and multiple sets of data streams, as needed, to address a data quality issue specific to the site. These efforts both complement and augment those of instrument mentors and the DQ Office. Site scientists may also perform research on topics related to site data quality issues.
Data Quality Reporting - Notification to Problem Solvers and Data Users
A Data Quality Problem Report (DQPR) allows the DQ Office, instrument mentors, and site scientists to submit problems to site operations and track the problem resolution process. Oversight of the process is performed by the site scientist. A DQPR is closed when the problem is solved, with the final act being the writing of a Data Quality Report (DQR). If a problem cannot be solved within a reasonable amount of time (usually 30-45 days) or is catastrophic in nature, a Problem Identification Form (PIF) is submitted, with the problem then coming to the attention of the ARM Problem Review Board. The end result of PIF solution is also a DQR.
A Data Quality Report (DQR) is a written statement about the quality of data in a particular data stream. The information could be quite simple (e.g., stating an instrument system was turned off and the data do not exist) or quite complex (e.g., providing detailed analyses and equations that should be used to adjust the instrument's data). At present, when a person orders ARM data from the ARM Data Archive, all DQRs written on the desired data streams are attached to the order. DQRs are typically written by the appropriate instrument mentor. These DQRs, as described earlier, provide a quality designation beyond that of automated flagging.
Value-Added Products and Quality Measurement Experiments - Second Generation Data Streams
Many of the scientific needs of the ARM Program are met through the analysis and processing of existing data products into "value-added" products or VAPs. Despite extensive instrumentation deployed at the ARM sites, there will always be quantities of interest that are either impractical or impossible to measure directly or routinely. Physical models using ARM instrument data as inputs are implemented as VAPs and can help fill some of the unmet measurement needs of the Program. Conversely, ARM produces some VAPs not in order to fill unmet measurement needs, but instead to improve the quality of existing measurements. In addition, when more than one measurement is available, ARM also produces "best estimate" VAPs. A special class of VAP called a Quality Measurement Experiment (QME) does not output geophysical parameters of scientific interest. Rather, a QME adds value to the input data streams by providing for continuous assessment of the quality of the input data based on internal consistency checks, comparisons between independent similar measurements, or comparisons between measurement with modeled results, and so forth.
Data Plots - Online Visualization of Data Streams
Visualizations of collected data of varying sophistication are available in ARM. Some of these data plots are intended, and are more useful, for operational diagnostic purposes, while others are useful for scientific inquiry. Please see data plots web page for more information.
Data User Notes - General Online Guidance for the Data User
Several situations may arise during instrument operation that can affect the quality of the data, but may not be flagged or otherwise corrected--the user needs to be aware of them. Some of these instances may be documented in "general" DQRs and will be attached to data orders.
These instances are, however, documented within the instrument handbook on the instrument web pages. Such user advice can be found under the topic "User Notes and Known Problems" and sometimes under "Frequently Asked Questions."
The data user is urged to read and heed such information when available! Research site-specific issues, based on the vagaries of measurements in diverse locations such as the tropics, mid-latitudes, and polar regions, will be called out in these websites or in general DQRs, if they exist.
Calibration and Maintenance Information - Instrument Performance
The ARM Program collects and stores information on calibration and maintenance procedures, the results of applying such procedures, and general information about the results of site maintenance visits. Some of this information is available within the instrument handbook of the instrument web pages.
Locations for information specific to each research site is also given below. The types of information generated by each research site, and how they are displayed, varies.
- NSA Operations Management Information System (OMIS), (includes preventative and corrective maintenance reports, sonde pre-launch logs, and instrument inventory)
- SGP Operations Management Information System (OMIS), (includes engineering logs, preventative and corrective maintenance reports, calibration reports, sonde status reports and launch logs, site status, vegetation observations, instrument inventory, and operations procedures)
- TWP System Event Log, (contains information about actions performed on instrument systems)
Continuous Quality Improvement Program - Never Ending Improvement
In 1998, SGP Site Operations implemented a Continuous Quality Improvement Program (CQIP). The CQIP is a field inspection program designed to routinely evaluate field collection sites from the scientific, engineering, operational, and safety perspectives. It is anticipated that a CQIP-type activity will be extended to the TWP and NSA research sites.
The CQIP consists of a cycle of planning, implementation, proficiency checking, and analysis, and is embodied in periodic site visits, or audits. These audits are made by the CQIP team, comprised of a SGP Site Scientist representative, the Argonne National Laboratory Environmental Safety and Health Director, the SGP Site Operations Safety Officer, and the SGP Instrumentation and Facilities Manager. The diversity in background of this team allows it to assess the sites from the aforementioned perspectives, both for the sites themselves and the instrumentation deployed on them. The audits include documented inspections and observations of site grounds, instruments, equipment, maintenance procedures, technician proficiency, and other work quality measures. Data collected during the audits are analyzed, and improvements in the work process are developed and inserted into the planning process for implementation. Continued audits and checks provide feedback to the quality cycle by assessing the effects of the improvements. The continuous assessment provides for evolution and improvements in instrument performance and maintenance, site maintenance, and planning for future site development.
Data Quality Program Personnel - Who to Contact?
Data Quality Program Office:
Randy A. Peppler
ARM Data Quality Office Manager
120 David L. Boren Blvd., Suite 2100
Norman, OK 73072-7304
Phone: (405) 325-6667
Fax: (405) 325-3098
Kenneth E. Kehoe
ARM Data Quality Office Specialist
120 David L. Boren Blvd., Suite 2100
Norman, OK 73072-7304
Phone: (405) 325-8983
Fax: (405) 325-3098
Instrument Mentors: Please see instrument mentors web page.
NSA Site Scientist Team: Scott Richardson, NSA Site Scientist Home Page, NSA Operations Page
SGP Site Scientist Team: Daniel Hartsock, SGP Site Scientist Home Page, SGP Operations Page—restricted access
TWP Site Scientist Team: Chuck Long, TWP Program Office Home Page
