Home Library Synthesis and Assessment Products Product 5.2: Best practice approaches for characterizing, communicating, and incorporating scientific uncertainty in decisionmaking Public Review Comments on Draft Prospectus |
Also available: |
ReviewersName: Eric Holdsworth / William L. Fang Name: Michael MacCracken, Ph.D. General CommentsGeneral Comments from Eric Holdsworth / William L. FangAccording to the draft (p. 2, lines 10-13), the Strategic Plan for the Climate Change Science Program (CCSP) defines uncertainty (apparently for climate change purposes) as “[a]n expression of the degree to which a value (e.g., the future state of the climate system) is unknown” [footnote 1] (emphasis added). The draft adds:
(Emphasis added.)[footnote 3] The draft then states (p. 3) that the intention of Synthesis and Assessment Product (SAP) 5.2 is “to further develop this topic through the synthesis, assessment, and communication of what is known about the character of uncertainty (as it applies to climate), and to address some potential approaches to decision making under uncertainty.” It adds that the “report will address uncertainty dimensions that are inherent to the full spectrum of decision support activities, ranging from the conduct and communication of research to the actual consideration and use of scientific knowledge and information products in decision making.” It emphasizes that the SAP “is designed to address two distinct purposes and audiences” as follows (p.3):
(Emphasis added.) As to both purposes, we question why the intended “audiences” are apparently limited, in one case only to “policymakers” who are interested “in developing a fundamental understanding of the issue,” and in the other to “scientists, science managers, and technical operational entities.” [footnote 4] The purpose should be to broaden the audiences to include, among others, decision makers and affected representatives of, for example, industry, business, agriculture, workers and non-governmental organizations, as well as the media and the general public.
Lastly, the draft states that the SAP “will address” a series of questions (p. 4) “in the context of climate change and variability.” While the listed questions are probably useful, there is no indication that there must be “balance” in scientific and other assessments, reports, etc. in order to ensure that uncertainty is treated on a par with other aspects of the assessments. Unfortunately, since that is not always the case, we request the addition of such an indication.
General Comments (Michael MacCracken, Climate Institute)General Comments (Michael MacCracken, Climate Institute)First General CommentThe prospectus for this product seems much closer to describing what is very likely to be a very interesting review paper on the subject rather than what might be considered a broadly based community assessment. The reason I say this is that the author team that is identified seems to me too narrowly drawn, both in a disciplinary sense and in an institutional sense. I would have thought, for example, that some actual decision-makers and policy makers would have been included, not just as contacts that this author team will make, but as intimately involved in how this important issue is described and addressed; in particular, I would have expected to see some government officials from the states (or even from the City of Aspen, which is in the process of completing its own, quite focused assessment) and Congress (and the 2002 hearing at which Senator Kerry asked questions based on the National Assessment lexicon might be an example of a case to evaluate the effectiveness of the lexicon as a communication tool), some representatives from the business and insurance communities, some representatives from the environment and private sector NGO community, etc. I would also like to have seen a stronger representation of the physical scientist community, for example of some of those like Dr. T. M. L. Wigley who are working with new ways to present the scientific information. I would also like to have seen a bit wider involvement of those in the community, this author set being quite weighted to those who have or have had a CMU connection. I also think it would help to have drawn from some of the regional assessment leaders who actually worked very closely with stakeholders (Prof. Ann Fisher of PSU comes to mind—and she had a proposal in to do some surveying about how the lexicons used as part of the IPCC and US National assessments are affecting how the scientific results are being perceived, so she would be well-qualified). In making these comments, I do not mean to take anything away from those listed—only that I would think the group should be broadened to ensure that there is an encompassing review of what has been attempted and a wide set of input on what is needed.
Second General CommentThere is one nominee to the contributing author list that I was surprised to see there, and that is Dr. Thomas Wilbanks. I say this not because of any limitation in his credentials, but because the prospectus indicates that the NRC will conduct a review of the draft report and Dr. Wilbanks is chair of the very NRC committee that would likely handle that review. While he could presumably opt out of that process, his participation in the NRC level review seems to me a very important one (the CCSP is going to the NRC for review because it can draw upon such experts as Dr. Wilbanks), likely more important than his serving as an author on this assessment, especially as he is also listed as an author on CCSP 4.6, which is another very important and likely time-consuming assessment.
Third General CommentI would hope that one aspect of the assessment effort of this team would be to address the rather unfortunate definition of uncertainty used in the CCSP Strategic Plan and included in the write-up on page 2, lines 12-13. It is simply not clear that one can have degrees of something being “unknown”—it would be interesting to know if this definition makes any sense to the expert community, much less to the public. I also think the phrase “level of certainty” on line 24 is poorly stated—one can have a level of confidence, perhaps, in, for example, a range of a number, but having levels of certainty is not understandable. Given these limits, I really think it essential that this panel be asked to start from scratch in putting together their report, going back to the very beginning and defining ever term in ways that are acceptable and understandable to both the expert community and to the broad stakeholder community.
Fourth General CommentI would hope that the charge to the panel would include asking for recommendations on research needs to help improve our understanding of this area. This topic generally was a recommended research area in the US National Assessment, and, as far as I know, much too little research has been done on how the various lexicon-based approaches have worked (or not worked), how the various approaches of the regional and sectoral teams in communicating with stakeholders worked, how the assessments by such other groups as the Pew Center and UCS are being received, how terms are being perceived across groups and disciplines, etc. Just one example of the type of problem I am seeing—at a recent briefing sponsored by EESI, pollsters Thomas Brewer and Clay Ramsey described results of their surveys of public opinion on the climate change issue. As one example, Brewer’s results showed something like 60+% acceptance by Republicans and 80+% by Democrats that climate change posed a threat to future generations, and, along with other answers, they concluded that there was a “consensus” about climate change among the American public. A later question by Ramsey, however, asked if viewpoints would change if there were a scientific consensus on the issue. Clearly, given that the nations of the world have agreed on the issue unanimously over three assessment cycles, there is some inconsistency on defining what a consensus means in various fields, indicating that the issue of uncertainties (or better, levels of confidence) are not being effective4ly conveyed. Research to better understand what the situation is might well be very helpful.
Fifth General CommentIn expressing level of confidence to have in the results of the CCSP (and international) research effort on climate change, it would also be useful if this panel were to evaluate the issue of how uncertainties link together across various aspects of the science. While the CCSP research strategy mentions uncertainties throughout the document, there is no indication of how uncertainties on one aspect of the science are related to uncertainties on any other aspect, and there is no discussion of an integrated effort to determine how different uncertainties feed together to generate an overall uncertainty—basically, the CCSP strategy presents no metrics for uncertainties (i.e., no definitions of “level of certainty” or “degree to which a value is unknown” as this prospectus calls for) and no way of relating them together to provide some indication of what is important and of the value of information. It will be important for this panel to thus not only to cover the issue of how uncertainty can be conveyed about individual findings, but how the uncertainties on different aspects can be inter-related and what the context is for making such evaluations—how this can be bias-free instead of politically based, for example. It would be helpful if the final prospectus thus ensured the full breadth of what is needed, not just covering “the degree to which a value is unknown,” but the context for such an evaluation and what level can be expressed about confidence when considering how uncertainties in all the various values (plural) affect some of the general conclusions about climate change.
Specific CommentsSpecific Comments from Eric Holdsworth / William L. Fang, Edison Electric InstituteFirst Specific Comment Page 5, line 43 to page 6, line 13In general, the draft states that SAP products “will be developed in consultation with a diverse group of stakeholders.” It adds that the members of the SAP 5.2 team “are engaged in the current discourse related to uncertainty” and that they “interact frequently on this topic with their scientific colleagues at workshops, conferences,” etc. “as well as decision makers.” However, since the SAP 5.2 “timeline. . .is currently under development,” there is no “review process” established yet. As far as we can determine from the explanation of the review process on page 6, the “team” will largely interact with their “colleagues” in the scientific community and the lead authors of “other” SAPs. There is no apparent means for the team to interact with other stakeholders, such as business, industry, etc., through workshops or conferences so that the team can impart its thinking to such stakeholders and receive feedback. We urge that such an opportunity be afforded.
Second Specific Comment Page 6, line 33 to page 7, line 35The draft indicates that NOAA, as the lead agency for the SAP, “will develop and oversee a review process that satisfies the SAP guidance issued by the CCSP, and is consistent with the Information Quality Act.” However, the draft fails to explain why this process is not included in the draft along with the applicable timeline. Therefore, the draft is incomplete, particularly since the “review process” is a critical element of the draft Prospectus.
Separate from the above unstated “review process,” the draft states that NOAA “will submit a draft of SAP 5.2 to the National Academies’ National Research Council (NRC) for expert scientific review.” It adds (p. 3) a list of questions that it asserts “are likely to be addressed” in the NRC review. A number of the questions are too simplistic and therefore not very useful (e.g., questions 1 and 4). In the case of question 5, there is a suggestion that whatever the SAP concludes, etc. should not deviate from “approaches” for “addressing uncertainty” that have been “embraced” by other organizations, such as the IPCC and the NRC. If the SAP’s conclusions are to be “consistent” with such “approaches,” we question why the SAP is even necessary or appropriate. The SAP team should be looking at such approaches independently of what has occurred previously and should not be bound by past approaches.
As to the NRC review, we do not understand the need for it. The draft does not articulate why NOAA should submit the draft to the NRC for so-called “expert scientific review,” nor does it explain what period of time will be given to the NRC for this review. It is our understanding that the SAP team is being chosen because of its expertise. We question why it is appropriate for the NRC to engage in such review.
In addition, we are concerned that the NRC will have an opportunity to review the draft before public review, and thus the public will not have an opportunity to see or comment on the NRC review before the SAP team considers and addresses it in the final report.
|
|