![[logo] US EPA](https://webarchive.library.unt.edu/eot2008/20090513064031im_/http://www.epa.gov/epafiles/images/logo_epaseal.gif){kind=logo}
Report Development
Basic Information Links
Chapters
The 2008 ROE was developed by EPA’s Office of Research and Development, working in collaboration with EPA’s program and Regional offices as well as external partners. This appendix describes the key elements of the 2008 ROE development process.
- Laying the Foundation
- Indicator Development
- Indicator Peer Review and Public Comment
- ROE Review
- References
Laying the Foundation
EPA published its Draft Report on the Environment in June 2003 and invited feedback. The Agency received comments from several sources:
- The Science Advisory Board (SAB) reviewed the 2003 Draft ROE Technical Document in March 2004, issuing draft comments shortly afterwards and publishing final comments in December 2004.1
- Through February 2004, the public provided comments on the 2003 Draft ROE Technical Document via EPA’s online public comment system.2
- Stakeholders commented on the 2003 Draft ROE Public Report during six dialogue sessions across the nation in 2003 and early 2004.3
In January 2004, the EPA Administrator requested that work begin to develop the next version of the ROE. Exhibit B-1 shows the organizational structure for development of EPA’s 2008 ROE.4 A standing ROE Work Group took the lead in all phases of development. The group included five theme leads, each responsible for development of a particular chapter of the 2008 ROE, plus representatives of EPA Regions and other relevant EPA offices. During the development process, the theme leads coordinated with other federal agencies and organizations involved in indicator development or data collection. An Environmental Indicators Steering Committee, composed of senior managers from across the Agency, oversaw development of the ROE. The Steering Committee reviewed Work Group activities and draft products.
The ROE is based on three components:
- A series of fundamental questions about the condition of the nation’s air, water, and land; about human exposure and health; and about the condition of ecological systems. These are questions that the Agency considers to be of critical importance to its mission.
- An indicator definition.
- Criteria against which indicators are evaluated to ensure that they are useful, objective, transparent, and scientifically reliable.
The first step in developing the 2008 ROE was to review and refine the 2003 Draft ROE version of these components:
- Questions. Over 100 EPA specialists from across the Agency were convened in the five ROE theme areas: air, water, land, human exposure and health, and ecological condition. Each theme team was charged with considering feedback and refining the ROE questions. The questions were finalized after review by the Environmental Indicators Steering Committee in 2004.
- Indicator definition and criteria. The 2003 Draft ROE indicator definition and criteria were refined for the 2008 ROE using an iterative process that included input from EPA specialists and review by the Environmental Indicators Steering Committee. Care was taken to ensure that the criteria were consistent with requirements of EPA’s Information Quality Guidelines.
Indicator Development
Once the questions, definition, and criteria were refined, the next step was to identify and develop indicators to answer the questions.
- 2003 Draft ROE indicators were screened against the 2008 ROE indicator definition and criteria. Many 2003 Draft ROE indicators were proposed for the 2008 ROE; some were withdrawn; and some were combined into other indicators (see Appendix C for details).
- Ideas for new indicators were solicited from across EPA, other federal agencies, and organizations. Newly proposed indicators were screened for their ability to meet the indicator definition and criteria and for their value in answering the ROE questions.
- For each indicator that passed screening, three components were developed: text describing the indicator, a graphic or table displaying the indicator data, and a metadata form that documents the data source and quality (see Box B-1).
The 2008 ROE development team worked with staff at other departments, agencies, and private organizations that originally developed indicators or provided indicator data to ensure that indicator graphics, data, and quality assurance information were up to date and accurate. Indicators were reviewed by the Environmental Indicators Steering Committee.
Box B-1. Questions Addressed in the 2008 ROE Metadata Forms
- Describe the physical, chemical, or biological measurements upon which this indicator is based. Are these measurements widely accepted as scientifically and technically valid? Explain.
- Describe the sampling design and/or monitoring plan used to collect the data over time and space. Is it based on sound scientific principles? Explain.
- Describe the conceptual model used to transform these measurements into an indicator. Is this model widely accepted as a scientifically sound representation of the phenomenon it indicates? Explain.
- For which ROE question(s) is this indicator used? To what extent is the indicator sampling design and monitoring plan appropriate for answering the relevant question(s) in the ROE?
- To what extent does the sampling design represent sensitive populations or ecosystems?
- What, if any, are the established reference points, thresholds, or ranges of values for this indicator that unambiguously reflect the state of the environment?
- What documentation clearly and completely describes the underlying sampling and analytical procedures used?
- To what extent is the complete data set accessible, including metadata, data-dictionaries, and embedded definitions? Are there confidentiality issues that may limit accessibility to the complete data set?
- Are the descriptions of the study or survey design clear, complete, and sufficient to enable the study or survey to be reproduced? Explain.
- To what extent are the procedures for quality assurance and quality control of the data documented and accessible?
- What statistical methods, if any, have been used to generalize or portray data beyond the time or spatial locations where measurements were made (e.g., statistical survey inference, no generalization is possible)? Are these methods scientifically appropriate?
- What uncertainty measurements or estimates are available for the indicator and/or the underlying data set?
- To what extent do uncertainty and variability impact the conclusions that can be inferred from the data and the utility of the indicator?
- Describe any limitations, or gaps in the data that may mislead a user about fundamental trends in the indicator over space or over the time period for which data are available.
Indicator Peer Review and Public Comment
Once the full suite of proposed indicators was assembled, all indicators were independently peer-reviewed by nationally recognized experts to ensure that they were scientifically sound and properly documented, met the indicator definition and criteria, and were useful for answering the questions posed in the ROE. Two rounds of review were conducted:
- At a workshop in July 2005, 21 experts reviewed the initial set of 88 proposed indicators.
- In November 2005, nine experts reviewed 11 indicators that were new or had been substantially revised since the July 2005 review.
The peer review, organized by a contractor, was conducted following the Office of Management and Budget’s (OMB’s) specifications for peer review of “Highly Influential Scientific Assessments” as specified in OMB’s “Final Information Quality Bulletin for Peer Review.”5 The reviewer selection criteria, list of reviewers, charge to reviewers, and reviewer comments can be found in the peer review summary report.6 EPA announced the peer reviews in the Federal Register and also posted the proposed indicators on a Web site for public comment. Key questions addressed during the review are listed in Box B-2. After the peer review and public comment period, EPA revised and finalized the indicators. EPA’s responses to reviewer and public comments are available at EPA’s ROE Web site: http://www.epa.gov/roe.
Box B-2. Charge Questions for Peer Review of the Proposed 2008 ROE Indicators
- Indicate the extent to which you think the proposed indicator is appropriate, adequate, and useful for evaluating ________.a
- Indicate the extent to which you think the proposed indicator makes an important contribution to answering the specific ROE question it is intended to answer.
- To what extent do you think the indicator meets the indicator definition?
- To what extent do you think the indicator meets each of the indicator criteria?
- Do you have any suggestions for more effective graphic presentation of the data?
- Provide any additional comments, suggestions, or concerns regarding the indicator that you have not already noted earlier. In particular, note any limitations to the indicator.
- Select one: Overall, this indicator (1) ___ should be included in the ROE; (2) ___ should be included in the ROE with the modifications identified above; or (3) ___ should not be included in the ROE.
- Do any of the proposed indicators clearly seem to be more appropriate, adequate, or useful for evaluating ________a than others? Do any seem to be more important than the others for answering the question(s) they are intended to answer?
- Are there any additional national-level indicators that make an important contribution to answering one of the ROE questions in your topic area, but were not proposed for the ROE, that you would recommend? As you consider this question, consider the list of indicators presented in the 2003 ROE that EPA does not intend to carry forward to the 2008 ROE, along with EPA’s rationale for withdrawing them. If you disagree with EPA’s rationale and feel any of these indicators should be included in the ROE, please so indicate in your response to this question, along with your rationale for why they should be included.
a This part of the charge varied according to theme area as follows:
- Air: “our nation’s air and therefore useful for contributing to an overall picture of our nation’s air”
- Water: “our nation’s waters and for contributing to an overall picture of our nation’s waters”
- Chemicals on land: “trends in chemicals used on land and their effects on human health and the environment”
- Land wastes: “trends in wastes and their effects on human health and the environment”
- Human health: “human health and for contributing to an overall picture of human health”
- Ecological condition: “ecological conditions and therefore useful for contributing to an overall picture of ecological conditions”
ROE Review
Concurrent with indicator development, EPA’s ROE team, working with specialists across the Agency, developed the text elements of the 2008 ROE. The final indicators were incorporated into the text to produce the full 2008 ROE. This draft document was reviewed internally at EPA, externally by other federal agencies and OMB, and externally by SAB (including public comment on the federal docket). EPA revised the document based on comments and, after the third review, finalized it for publication.
References
1 U.S. Environmental Protection Agency. 2004. EPA’s Draft Report on the Environment (ROE) 2003: An advisory by the ROE Advisory Panel of the EPA Science Advisory Board. Science Advisory Board. EPA/SAB/05/004. http://yosemite.epa.gov/sab/sabproduct.nsf/18246BED9FB52FE085256F6A006BC3C1/$File/SAB-05-004_unsigned.pdf
2 U.S. Environmental Protection Agency. 2004. E-docket for Draft Report on the Environmental Technical Document. Docket Number: OEI-2003-0030. http://www.regulations.gov/fdmspublic/component/main?main=DocketDetail&d=EPA-HQ-OEI-2003-0030
3 U.S. Environmental Protection Agency. 2004. Summary report of the National Dialogue on the EPA Draft Report on the Environment 2003. Office of Environmental Information. http://www.epa.gov/Envindicators/docs/National_Dialogue_Summary_Report.pdf
4 An additional organizational element, the Indicators Work Group, was added to the process as the indicators were being finalized for the July 2005 peer review. The Indicators Work Group provided coordination between the ROE Work Group and the Environmental Indicators Steering Committee.
5 Office of Management and Budget. 2004. Final information quality bulletin for peer review. December 16, 2004. http://www.whitehouse.gov/omb/memoranda/fy2005/m05-03.pdf
6 U.S. Environmental Protection Agency. 2006. Report of the peer review of proposed ROE07 indicators. Office of Research and Development. http://cfpub.epa.gov/eroe/index.cfm?fuseaction=main.peerReview