Model Evaluation

Updates

You will need Adobe Reader to view some of the files on this page. See EPA's PDF page to learn more.

Background

In 2004, at the direction of the EPA Administrator (PDF), (2 pp, 86KB) the CREM commissioned a study by the National Research Council (NRC) Board of Environmental Studies and Toxicology on the use of models in environmental decision making. The objective of the study was to provide advice on the development of guidelines for model development, use and evaluation and a vision for model selection and use at EPA. The committee considered cross-disciplinary issues related to model use, performance evaluation, peer review, uncertainty, and quality assurance/quality control.

The NRC study, released in the summer of 2007 provides a set of specific recommendations aimed at enhancing the development, use, and evaluation of environmental models for better decision making. The report is meant as a guide for the selection and use of models in the regulatory process.

The NRC Panel identified several fundamentals of models in the regulatory process:

Models always constrained by computational limitations, assumptions, and knowledge gaps
Environmental models can never be completely “validated” in the traditional sense. But they can be “evaluated”
Regulatory models are typically used to describe important, complex, and poorly characterized problems
Models in the regulatory process best seen as tools providing inputs, as opposed to “truth generating machines”

These and other features prompted the panel to recommend that evaluation of a regulatory model should continue throughout the life of a model. In particular, model evaluation should not stop with the evaluation activities that often occur before the public release of a model but should continue throughout regulatory applications and revisions to the model. For all models used in the regulatory process, the panel felt the Agency should begin by developing a life-cycle model evaluation plan commensurate with the regulatory application of the model (for example, the scientific complexity, the precedent-setting potential of the modeling approach or application, the extent to which previous evaluations are still applicable, and the projected impacts of the associated regulatory decision).

The CREM Guidance on the Development, Evaluation and Application of Environmental Models (PDF) (95 pp, 3.8MB)was published for public comment in August 2008. This guidance provides recommendations for effective use of models in environmental decision making. The draft guidance incorporates recommendations from Agency white papers, EPA’s Science Advisory Board reports, and peer reviewed literature. It includes recommendation on model development, evaluation and application.

The guidance describes model evaluation as the process for generating information over the life cycle of the project to help determine whether a model and its analytical results are of a quality sufficient to serves as the basis for a decision. However, the guidance document does not address all of the issues identified in the NRC report on life-cycle model evaluation. Therefore, the EPA modeling community will be gathering to discuss our response to the NRC Report.

Workshop on Life-Cycle Evaluation of Models: Response to the NRC

A national workshop is planned to engage the modeling community across EPA on the recommendations of the NRC on enhancing regulatory modeling activities through conduction evaluations of models throughout their life-cycle. Such evaluation encompasses all stages of the modeling process from conceptualization through development and implementation, and includes retrospective analysis of models after they have been used to inform a policy-decision.

This workshop will identify the current practices related to life-cycle evaluations of models across EPA’s Program Office and Regions. This workshop is an opportunity for the EPA modeling community to participate in discussions on the future of model evaluation at EPA.

Model Evaluation

Updates

Local Navigation