Report of the EOS Data Quality Assurance (QA) Workshop

--Bob Lutz (rlutz@ltpmail.gsfc.nasa.gov), Hughes STX Corp.

On November 6, 1996, an EOS Data Quality Assurance (QA) Workshop was convened at the Goddard Space Flight Center. Approximately 60 people attended, including Principal Investigators (PIs) or their representatives from Instrument Teams (ITs) of the AM-1 and PM-1 platforms, SAGE-III, and TRMM; representatives from the DAACs and several Interdisciplinary Science (IDS) Teams; and numerous Earth Science Data and Information System (ESDIS), EOS Project, and NASA Headquarters personnel. The Workshop continued an effort that the QA Scientist (Bob Lutz) had been pursuing for the past two years under the auspices of the ESDIS Science Office (H. K. Ramapriyan).

Bob Lutz opened the meeting, welcomed the participants, and provided logistical information.

Skip Reber (EOS Deputy Senior Project Scientist and Acting Earth Observing System Data and Information System [EOSDIS] Project Scientist) spoke next, defining the following goals for the workshop:

• to have each Instrument Team (IT) present its QA methodologies;

• to share thoughts among the Project, ITs, and EOSDIS Core System (ECS) on QA;

• to agree that data need to be published with error assessments; and

• to agree to some form of common nomenclature and format.

H. K. Ramapriyan followed Reber's introductory remarks. He stated that while it is understood that quality assessment of data products is the responsibility of the respective instrument teams, the QA planning is being coordinated by the ESDIS Science Office to ensure that:

• the ESDIS Project understands the requirements that the ITs have on EOSDIS;

• the IT requirements are provided by the ITs in time to meet the overall schedules of the system development;

• the QA Plans of each of the ITs are made available to the others and to the other users of EOSDIS; and

• EOSDIS is developed to provide the services needed to meet the ITs' QA needs.

Bob Lutz then presented the agenda of the workshop. He also explained that prior to the workshop, each AM-1 team had been provided copies of the QA Plans that had been received by the ESDIS Project from the other teams and E-mailed a list of questions/issues that they were requested to cover in their presentations. In addition, all IDS teams had been solicited through a questionnaire (one-third responded) for their desired requirements on QA parameters from the AM-1 teams.

The morning session and a portion of the afternoon discussion were devoted to instrument team QA presentations. (It will be noted in the following report, that the instrument teams are using different terminology for the same functions. The issue of developing a common nomenclature is still being resolved.)

Erika Geier presented the CERES QA Plan, which had been submitted to ESDIS prior to the Workshop. The CERES QA plan is divided into automated QA done at the DAAC and manual QA done at the Scientific Computing Facilities (SCFs). Automated QA is performed by the production software. Errors will be handled in 2 ways: 1) A flag is set when problems occur. 2) In the case of a fatal error, "alarm scenarios" are activated. Every granule of an archival data product will have associated with it at least one Quality Control (QC) report. Manual QA is performed by the Data Management and Science Team, and the QA method is based on that used for the Earth Radiation Budget Experiment (ERBE). They have alarm/stop conditions if they do not want data production to continue. CERES expects metadata support for the Science Product Access Code.

Paul Bailey summarized the MOPITT QA Plan, which also had been supplied for review before the workshop. MOPITT QA activities are limited to what can be done operationally as the data products are produced. QA Analysis components include DAAC Product Generation Executive (PGE) components and SCF components. DAAC PGE components include on-line checks and Exception Logs, post processing of Exception Logs, and Data Product (DP) Summaries. SCF components include manual granule review, retrieval of Exception Logs and DP granules, and the transmission of the Granule Accessibility Code and metadata to the DAACs. Bailey indicated that there will be no QA functions performed by DAAC personnel, and he provided estimates of SCF staffing.

The MODIS QA Plan consists of four separate QA Plans coordinated by Al Fleig (not present at the workshop). The four plans are for Level 1B, Land, Atmosphere, and Ocean products. (Note : though a representative from Level 1B was not at the workshop, a detailed QA Plan for Level 1B had been provided to ESDIS for distribution and review.)

David Roy discussed the MODIS Land (MODLAND) QA Plan, which was distributed at the meeting. QA Activities are to be performed at the proposed MODLAND data processing facility (Land Data Product Evaluation--LDOPE facility). QA procedures which are used by algorithm developers, DAAC personnel, and data product users are executed at run and post-run time. For science data run time, QA information is stored in mandatory and optional QA planes. MODLAND said that product metadata summaries are stored in the mandatory QA plane and core metadata are stored in the mandatory ECS QA flags. Post-run-time QA will be applied to each product. Estimates of data volume QA-ed and staffing were also provided.

The MODIS Atmosphere QA Plan was presented by Allen Chun. MODIS Level 2 atmospheric products, spatial resolution, and run-time QA flags were presented. (This Draft QA Plan was submitted to ESDIS in March 1997.)

Bob Evans reviewed the MODIS Oceans QA Plan. The MODIS Oceans QA methodology is based on a series of steps that include use of engineering data and flags from the Level 1 product to determine if a pixel is suitable for processing (i.e., sensor and ancillary data valid). Level 2 flags are in part shared between Sea Surface Temperature (SST) and color (observation geometry, spatial/spectral cross-talk, polarization angle, and mirror incidence angle), together with state of the atmosphere. Evans reported that each ocean product will use the first flag set to determine product-specific flags. The Level 3 processing is conducted separately for each ocean product and uses a binning scheme, which preferentially keeps the highest quality data. At Level 3 the flags are combined into 4 quality levels.

Ralph Kahn and Barbara Weymann discussed the MISR QA Plan, which was received by ESDIS in January 1997. Prior to the Workshop, an article had been published (The EarthObserver, January/February 1996), describing their methodology. Science QA parameters are developed in concert with data product generation. Errors in the PGE processing result in "fatal alarms." DAAC operators monitor these alarms and may examine images. They contact the SCF on any anomalies found. Activities required are currently being negotiated between the LaRC DAAC and MISR and will be documented in the MISR Operations Agreement. MISR has also developed error-processing scenarios. They will use the "pull scenario" for data to be sent to the SCF for evaluation. Data transfer rates were also provided.

Craig Leff summarized the ASTER QA Plan, which was submitted in advance of the workshop. ASTER performs automated QA within the PGEs at the DAAC during production, and manual QA of every nth product (TBD) at the SCF. Future plans may include participation of the DAAC staff in science QA. A flowchart was presented depicting ASTER QA operations. Alert information is generated, if necessary, for every PGE, with the alert log periodically archived for Science Team review. Each output pixel range is to be checked, and bad data will not be replaced with marker values. QA data planes will only be used for data products at Level 2 and above. Estimated ASTER data flows between the DAAC and the SCF, and the staffing required, were discussed.

Jim Stobie described the Data Assimilation Office (DAO) QA approach. A formal QA Plan will be submitted at a future date. DAO uses instrument data (including information about the instrument errors) to produce its assimilated products. The assimilation process is a series of cycles. Each cycle involves:

1. Running a model-generated forecast based on the output from the previous cycle. This results in a first guess.

2. The first guess is then compared with observations and adjusted accordingly. The amount of adjustment depends on the quality and quantity of the observations. High-quality observations receive more weight than low-quality observations. The assimilation process depends on accurate error statistics for both the observations and the first-guess model. DAO also reported that important by-products of the assimilation process are updates to the model and instrument error statistics.

Concluding the instrument team presentations, Vanessa Griffin (speaking for Steve Goodman) reviewed TRMM-LIS QA methodology.

LIS QA parameters will include: false alarm rate, event probability, and summary statistics. For manual QA, errors are flagged for improper dataset assembly or problems with the science data or platform time/ephemeris. LIS processing will be performed by the LIS SCF operations team.

Laurie Glaze, a representative of the Volcanology IDS Team, discussed the team's QA information needs. The Volcanology Team uses datasets from several instrument teams (ASTER, MODIS, and MISR) and will need to understand QA parameters from each IT. This team's minimum requirements for QA data include QA data parameters for each scene and QA parameters per pixel, with a major concern being the flagging of bad, missing, or interpolated data. The team is also concerned with how much confidence there is in radiance data. The Volcanology Team concluded that they would like to have access to QA Plans, as they are developed, and instrument team test datasets, with QA parameters included, to modify their algorithms. (A general area of concern was voiced during this presentation. Realizing that some AM-1 products may have as many as 20 QA parameters, IDS teams that use numerous products may face a considerable task of interpreting hundreds of QA parameters.)

ECS support for QA activities was then reviewed by Karl Cox. General services provided by ECS for QA include:

• ability to acquire data and information for QA;

• accept and store QA information;

• update QA metadata attributes; and

• ability to alert operational staff to suspend operations when QA failure occurs

The Science Data Model QA attributes were then described. They represent the metadata in the inventory tables. There are separate Automatic, Operational, and Science QA flags for each granule, with explanation fields associated with each. The Automatic Flags are set by the PGE, and the Science flag is set by the SCF. Comments from members of the ITs during this discussion indicated that additional QA support is needed from ECS. Specifically, this includes a method for batch updates by the SCFs of QA metadata and the allowance of multiple sets of flags/statistics per granule for different parameters. (Note: ECS has agreed to fulfill both of these requirements.) It was recognized during this session that a more-effective dialogue needs to be established between ECS and the ITs concerning QA requirements.

Bob Lutz and Skip Reber led a discussion on QA data dependencies amongst the AM-1 teams. Each team provided material on what information is needed from another IT for its dataset production. A conclusion was reached that identification of specific QA parameters was needed from the data producers before the final B.0 QA Plans are due. A subsequent discussion was then held to come to an agreement on definitions of content, structure, and code QA parameter descriptions, so that appropriate dates could be set for the delivery of these parameters.

At the conclusion of the meeting, an Action Item list was generated. The most important of these Action Items was:

• An interactive dialogue between ECS and the ITs needs to be established to communicate IT QA requirements.

To work this issue and several of the issues below, a small QA Working Group has been formed. The group includes representatives from the ITs, ECS, ESDIS, and the DAACs. To specifically address this issue, all ITs were requested to scrub their requirements (ECS functions and services) needed for QA. These requirements were compiled and forwarded to ECS. ECS has since provided a response to these requirements, which was communicated to the ITs via the QA Working Group. A QA Working Group meeting has been scheduled for the summer of 1997 to discuss these ECS responses, any unresolved QA Workshop Action Items, and B.0 QA operational scenarios.

Other Action Items included :

• The ability of the SCF Metadata Update Tool to identify granules is limited. ECS needs to provide granule identification within the tool and to allow this tool to function in concert with the Subscription and Data Ordering Tools (i.e., linked functionality of the tools). (ECS has conceptually agreed to fulfill this requirement. The QA Working Group will provide guidance to ECS on this issue.)

• IDS teams, ITs, and non-ECS users need a tool to view and display QA information in a convenient and user-friendly manner. (A Tiger team of the Client Design Working Group has been working the general issue of the displaying [and grouping] of core and product-specific metadata on the Client.)

• The QA Working Group will address and come to closure on a common method of presenting and publishing QA information/terminology amongst the ITs, i.e., common nomenclature. (This issue will be discussed and resolved at the QA Working Group meeting).

• A QA Home Page will be developed by the ESDIS Science Office to allow easy access to IT Draft QA plans and QA information. (This has been established--see end of article.)

• All instrument teams will supply information on QA parameters that are prioritized in terms of content by March 20, 1997. Code parameters will be delivered by July 1, 1997. The collection of these parameters will be disseminated to all ITs, via the Working Group, for comment. (As this information is known, it will be disseminated. This is an on-going Action Item.)

• Though it is realized that Science QA is an Instrument Team responsibility, some ITs have indicated that they will negotiate with their DAACs to share QA responsibilities. Agreements are needed on the division of these DAAC/SCF QA functions. (This item is in the process of being completed by the ITs and their respective DAACs and will be discussed at the QA Working Group Meeting.)

Several issues, such as the interaction of validation on QA activities, were not discussed during the workshop due to time constraints. A follow-on workshop is expected to be conducted during the winter of 1997-1998, after the ITs develop their final B.0 QA Plans, to address these and other QA issues that may arise.

The workshop agenda, minutes, and action items are found within the ESDIS Science Homepage (http://spsosun.gsfc.nasa.gov/Science/QA_Nindex. html) under the heading Quality Assurance. For additional information regarding the topic of Quality Assurance, please contact the author (rlutz@ltpmail.gsfc.nasa.gov).