Best practice approaches for characterizing, communicating, and incorporating scientific uncertainty in decisionmaking

Also available:
CCSP Synthesis and Assessment Products. Four-page background document (dated September 2007). In addition, it is available as a PDF file and can be ordered in hardcopy from the GCRIO Online Catalog

Reviewers

Name:  Eric Holdsworth / William L. Fang
Organization:  Edison Electric Institute (EEI)
Mailing Address: 701 Pennsylvania Avenue, N.W., Washington, D.C. 20004
Phone: (202) 508-5617 /  (202) 508-5103
E-mail:  eholdsworth@eei.org  /  bfang@eei.org

Name: Michael MacCracken, Ph.D.
Organization: Climate Institute
Mailing Address: 1785 Massachusetts Ave., NW, Suite 100, Washington DC 20036
Phone: 301-564-4255
E-mail: mmaccrac@comcast.net
Areas of expertise: Climate change, climate impacts, and assessment

General Comments

General Comments from Eric Holdsworth / William L. Fang

According to the draft (p. 2, lines 10-13), the Strategic Plan for the Climate Change Science Program (CCSP) defines uncertainty (apparently for climate change purposes) as “[a]n expression of the degree to which a value (e.g., the future state of the climate system) is unknown” [footnote 1] (emphasis added). 

The draft adds:

The level of certainty in the projections of climate change and its effects has emerged as a central issue in the public discourse, reinforcing the need to evaluate current methods and to define best practices for assessing uncertainty.  The scientific community -- which includes researchers from academia, government, and the private sector, as well as scientific and operational agencies – are looked to by policymakers, decision makers, and the media for “answers” (or insights) about trends, rates, impacts, and adaptation options related to climate change. [footnote 2]

The climate research community has taken steps in recent years to explain the nature of uncertainty in their assessment efforts.  The Intergovernmental Panel on Climate Change (IPCC), the largest international climate assessment effort, recognized the need for a more formal, decision analysis-based treatment of uncertainty in Chapter 11 of its report on Climate Change 1995:  The Science of Climate Change (McBean et al., 1995).  In response to this need, recommendations for reporting uncertainty were developed for the authors of the IPCC Third Assessment Report (TAR) and the ongoing Fourth Assessment Report (AR4).

(Emphasis added.)[footnote 3]

The draft then states (p. 3) that the intention of Synthesis and Assessment Product (SAP) 5.2 is “to further develop this topic through the synthesis, assessment, and communication of what is known about the character of uncertainty (as it applies to climate), and to address some potential approaches to decision making under uncertainty.”  It adds that the “report will address uncertainty dimensions that are inherent to the full spectrum of decision support activities, ranging from the conduct and communication of research to the actual consideration and use of scientific knowledge and information products in decision making.”  It emphasizes that the SAP “is designed to address two distinct purposes and audiences” as follows (p.3):

One purpose of the report is to synthesize and communicate the current state of understanding about the characteristics and implications of uncertainty related to climate change and variability to an audience of policymakers with an interest in developing a fundamental understanding of the issue. . .An increased awareness and understanding of the characteristics of scientific uncertainty as applied to climate is a critical step in this effort.

The second purpose is to provide recommendations for best practices for characterizing, analyzing and communicating uncertainty for scientists, science managers, and technical operational entities involved in conducting research and assessments, and producing climate information in the context of decision support, based on a thorough, state-of-the-art assessment of the current state of understanding.

(Emphasis added.)

As to both purposes, we question why the intended “audiences” are apparently limited, in one case only to “policymakers” who are interested “in developing a fundamental understanding of the issue,” and in the other to “scientists, science managers, and technical operational entities.” [footnote 4] The purpose should be to broaden the audiences to include, among others, decision makers and affected representatives of, for example, industry, business, agriculture, workers and non-governmental organizations, as well as the media and the general public.

• Response: The initial Prospectus language was not intended to preclude interest in this largely methodological paper on the part of a broader suite of stakeholders.  The text in the final Prospectus has been modified to reflect the breadth of potential readers (page 3, line 13, and page 3, lines 34-35).

Lastly, the draft states that the SAP “will address” a series of questions (p. 4) “in the context of climate change and variability.”  While the listed questions are probably useful, there is no indication that there must be “balance” in scientific and other assessments, reports, etc. in order to ensure that uncertainty is treated on a par with other aspects of the assessments.  Unfortunately, since that is not always the case, we request the addition of such an indication.

• Response: This important topic falls under one of the themes identified in the Prospectus: challenges associated with estimating uncertainty.  This comment will be passed along to the author team.

General Comments (Michael MacCracken, Climate Institute)General Comments (Michael MacCracken, Climate Institute)

First General Comment

The prospectus for this product seems much closer to describing what is very likely to be a very interesting review paper on the subject rather than what might be considered a broadly based community assessment. The reason I say this is that the author team that is identified seems to me too narrowly drawn, both in a disciplinary sense and in an institutional sense. I would have thought, for example, that some actual decision-makers and policy makers would have been included, not just as contacts that this author team will make, but as intimately involved in how this important issue is described and addressed; in particular, I would have expected to see some government officials from the states  (or even from the City of Aspen, which is in the process of completing its own, quite focused assessment) and Congress (and the 2002 hearing at which Senator Kerry asked questions based on the National Assessment lexicon might be an example of a case to evaluate the effectiveness of the lexicon as a communication tool), some representatives from the business and insurance communities, some representatives from the environment and private sector NGO community, etc. I would also like to have seen a stronger representation of the physical scientist community, for example of some of those like Dr. T. M. L. Wigley who are working with new ways to present the scientific information. I would also like to have seen a bit wider involvement of those in the community, this author set being quite weighted to those who have or have had a CMU connection. I also think it would help to have drawn from some of the regional assessment leaders who actually worked very closely with stakeholders (Prof. Ann Fisher of PSU comes to mind—and she had a proposal in to do some surveying about how the lexicons used as part of the IPCC and US National assessments are affecting how the scientific results are being perceived, so she would be well-qualified). In making these comments, I do not mean to take anything away from those listed—only that I would think the group should be broadened to ensure that there is an encompassing review of what has been attempted and a wide set of input on what is needed.

• Response: The team of contributing authors was selected by the lead author, Dr. Granger Morgan, for their expertise in the methodological issues associated with uncertainty in the context of climate.

The lead author has explicitly solicited feedback on and input to this effort from other members of the scientific community.  In addition, efforts will be made to include a decision maker on the National Research Council’s (NRC) review committee for SAP 5.2, and decision makers will be invited to attend one of the NRC’s review meetings for SAP 5.2  and interact directly with members of the author team.  Finally, input from the broader community of scientists, decision makers and the media and general public can be provided via the 45 day public review period of the report (see Timeline on page 8 of the final Prospectus; page 8, lines 10-12).

Second General Comment

There is one nominee to the contributing author list that I was surprised to see there, and that is Dr. Thomas Wilbanks. I say this not because of any limitation in his credentials, but because the prospectus indicates that the NRC will conduct a review of the draft report and Dr. Wilbanks is chair of the very NRC committee that would likely handle that review. While he could presumably opt out of that process, his participation in the NRC level review seems to me a very important one (the CCSP is going to the NRC for review because it can draw upon such experts as Dr. Wilbanks), likely more important than his serving as an author on this assessment, especially as he is also listed as an author on CCSP 4.6, which is another very important and likely time-consuming assessment.

• Response: Dr. Tom Wilbanks, a contributing author to SAP 5.2, will not have a role in the NRC review of the document. The NRC will establish a committee under the auspices of the Board on Atmospheric Sciences (BASC), with assistance from the Committee on the Human Dimensions of Global Change.  The NRC has assured NOAA that all policies regarding potential conflict of interest will be followed, and that Dr. Wilbanks will not have a role in the process.

Third General Comment

I would hope that one aspect of the assessment effort of this team would be to address the rather unfortunate definition of uncertainty used in the CCSP Strategic Plan and included in the write-up on page 2, lines 12-13. It is simply not clear that one can have degrees of something being “unknown”—it would be interesting to know if this definition makes any sense to the expert community, much less to the public. I also think the phrase “level of certainty” on line 24 is poorly stated—one can have a level of confidence, perhaps, in, for example, a range of a number, but having levels of certainty is not understandable. Given these limits, I really think it essential that this panel be asked to start from scratch in putting together their report, going back to the very beginning and defining ever term in ways that are acceptable and understandable to both the expert community and to the broad stakeholder community.

• Response:  The final Prospectus text has been modified to reflect a request to NRC to consider the clarity of terms used SAP 5.2 (page 7, line 5).

Fourth General Comment

I would hope that the charge to the panel would include asking for recommendations on research needs to help improve our understanding of this area. This topic generally was a recommended research area in the US National Assessment, and, as far as I know, much too little research has been done on how the various lexicon-based approaches have worked (or not worked), how the various approaches of the regional and sectoral teams in communicating with stakeholders worked, how the assessments by such other groups as the Pew Center and UCS are being received, how terms are being perceived across groups and disciplines, etc. Just one example of the type of problem I am seeing—at a recent briefing sponsored by EESI, pollsters Thomas Brewer and Clay Ramsey described results of their surveys of public opinion on the climate change issue. As one example, Brewer’s results showed something like 60+% acceptance by Republicans and 80+% by Democrats that climate change posed a threat to future generations, and, along with other answers, they concluded that there was a “consensus” about climate change among the American public. A later question by Ramsey, however, asked if viewpoints would change if there were a scientific consensus on the issue. Clearly, given that the nations of the world have agreed on the issue unanimously over three assessment cycles, there is some inconsistency on defining what a consensus means in various fields, indicating that the issue of uncertainties (or better, levels of confidence) are not being effective4ly conveyed. Research to better understand what the situation is might well be very helpful.

• Response: SAP 5.2 is not intended to comprehensively identify research needs related to uncertainty but rather to focus on best practice approaches for characterizing, communicating and incorporating scientific uncertainty in decision making related to climate.  While the identification of research needs is critical to planning in all areas of research, such an effort would require a different approach than one focused on of best practices.

Fifth General Comment

In expressing level of confidence to have in the results of the CCSP (and international) research effort on climate change, it would also be useful if this panel were to evaluate the issue of how uncertainties link together across various aspects of the science. While the CCSP research strategy mentions uncertainties throughout the document, there is no indication of how uncertainties on one aspect of the science are related to uncertainties on any other aspect, and there is no discussion of an integrated effort to determine how different uncertainties feed together to generate an overall uncertainty—basically, the CCSP strategy presents no metrics for uncertainties (i.e., no definitions of “level of certainty” or “degree to which a value is unknown” as this prospectus calls for) and no way of relating them together to provide some indication of what is important and of the value of information. It will be important for this panel to thus not only to cover the issue of how uncertainty can be conveyed about individual findings, but how the uncertainties on different aspects can be inter-related and what the context is for making such evaluations—how this can be bias-free instead of politically based, for example. It would be helpful if the final prospectus thus ensured the full breadth of what is needed, not just covering “the degree to which a value is unknown,” but the context for such an evaluation and what level can be expressed about confidence when considering how uncertainties in all the various values (plural) affect some of the general conclusions about climate change.

• Response: The topic of the propagation of uncertainty falls under the discussion of a topic already identified in the Prospectus: how uncertainty is analyzed and applied in analyses of adaptation options.  This comment will be passed along to the authors.

Specific Comments

Specific Comments from Eric Holdsworth /  William L. Fang, Edison Electric Institute

First Specific Comment  Page 5, line 43 to page 6, line 13

In general, the draft states that SAP products “will be developed in consultation with a diverse group of stakeholders.”  It adds that the members of the SAP 5.2 team “are engaged in the current discourse related to uncertainty” and that they “interact frequently on this topic with their scientific colleagues at workshops, conferences,” etc. “as well as decision makers.”  However, since the SAP 5.2 “timeline. . .is currently under development,” there is no “review process” established yet.  As far as we can determine from the explanation of the review process on page 6, the “team” will largely interact with their “colleagues” in the scientific community and the lead authors of “other” SAPs.  There is no apparent means for the team to interact with other stakeholders, such as business, industry, etc., through workshops or conferences  so that the team can impart its thinking to such stakeholders and receive feedback.  We urge that such an opportunity be afforded.

• Response: Non-scientific stakeholders (including decision makers with various public and private organizations and industries, the media and the general public) will have an opportunity to provide comments on and feedback to SAP 5.2 through the 45 day public review period (see timeline in final Prospectus).  In addition, the NRC will invite a number of decision makers/stakeholders to participate in one of its review meetings and to interact directly with the authors of SAP 5.2 (page 6, lines 10-12).

Second Specific Comment  Page 6, line 33 to page 7, line 35

The draft indicates that NOAA, as the lead agency for the SAP, “will develop and oversee a review process that satisfies the SAP guidance issued by the CCSP, and is consistent with the Information Quality Act.”  However, the draft fails to explain why this process is not included in the draft along with the applicable timeline.  Therefore, the draft is incomplete, particularly since the “review process” is a critical element of the draft Prospectus.

• Response: A timeline that outlines the major milestones in the review process is included in the final Prospectus (page 8).

Separate from the above unstated “review process,” the draft states that NOAA “will submit a draft of SAP 5.2 to the National Academies’ National Research Council (NRC) for expert scientific review.”  It adds (p. 3) a list of questions that it asserts “are likely to be addressed” in the NRC review.  A number of the questions are too simplistic and therefore not very useful (e.g., questions 1 and 4).  In the case of question 5, there is a suggestion that whatever the SAP concludes, etc. should not deviate from “approaches” for “addressing uncertainty” that have been “embraced” by other organizations, such as the IPCC and the NRC.  If the SAP’s conclusions are to be “consistent” with such “approaches,” we question why the SAP is even necessary or appropriate.  The SAP team should be looking at such approaches independently of what has occurred previously and should not be bound by past approaches.

• Response: The text in review question #5 has been modified for clarity (page 7, lines 18-20).

As to the NRC review, we do not understand the need for it.  The draft does not articulate why NOAA should submit the draft to the NRC for so-called “expert scientific review,” nor does it explain what period of time will be given to the NRC for this review.  It is our understanding that the SAP team is being chosen because of its expertise.  We question why it is appropriate for the NRC to engage in such review.

• Response: The CCSP Synthesis and Assessment Products (SAPs) are considered highly influential scientific assessments and must conform to the requirements established by the Information Quality Act (IQA).  The subsequent guidance from the Office of Management and Budget (Peer Review Bulletin) calls for certain standards of review for these types of documents.  All of the CCSP SAPs are considered to fall into this category, and, as such require a certain standard of review;  an NRC review satisfies this requirement.

In addition, we are concerned that the NRC will have an opportunity to review the draft before public review, and thus the public will not have an opportunity to see or comment on the NRC review before the SAP team considers and addresses it in the final report.

• Response: The NRC review will occur prior to the public review of the report in order to ensure that the product submitted for public review has been shaped by scientific considerations.  However, the public will have the opportunity to access the NRC review of the report prior to the 45 day public review period.