Search Options
Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us

SECY-00-0035

February 11, 2000

• PURPOSE:
• BACKGROUND:
• DISCUSSION
• Who is the Public?
• Measurement Techniques
• RECOMMENDATION
• RESOURCES:
• COORDINATION:

PURPOSE:

The purposes are to discuss public confidence and measuring public confidence, to provide the Commission with the staff's assessment of conducting a survey to measure public confidence, and to obtain the Commission's guidance on whether or not to conduct such a survey.

BACKGROUND:

Public confidence has been defined in the past in the context of improving communication with the public. Communication with external stakeholders was discussed in the Strategic Assessment and Rebaselining Initiative as Direction Setting Issue 14, Public Communications Initiatives. As a result of the Commission decision on this issue, public confidence was included in several arenas and strategies in the NRC's Strategic Plan. As a result, a range of actions has been and continues to be implemented as part of the staff's activities to implement the Strategic Plan.

Examples of actions to improve public confidence include public meetings with stakeholders, workshops with members of the public and/or licensees, increased use of focus group meetings, annual two-day media workshops to provide a better foundation in NRC practices to reporters from around the country, a plain language primer on the agency's new reactor oversight program, increased opportunity for public participation in meetings, and outreach or coordination meetings to better understand external stakeholder interests and comments. These actions are being conducted by various NRC Offices related to regulatory initiatives such as the new reactor oversight and inspection program, and issues involving materials and reactor licensees. While these activities have provided the opportunity for improved communication with external stakeholders, the staff has not attempted to measure the effect of these interactions on public confidence.

In a related activity, the staff is preparing an update to the agency strategic plan as required by the Government Results and Performance Act (GPRA). The draft Nuclear Reactor Safety chapter of the Strategic Plan was issued and public comments were requested on August 6, 1999. The chapter contained four performance goals and strategies to increase public confidence. The other arenas in the draft Strategic Plan, Nuclear Materials Safety, Nuclear Waste Safety, and International Nuclear Safety Support, also include increasing public confidence as a performance goal.

A public workshop was held on August 20, 1999, which included representatives from the Union of Concerned Scientists, the Government Accounting Office, the Vermont Department of Public Service, the Nuclear Energy Institute, and the Institute of Nuclear Power Operations. This workshop was held to solicit input on the draft chapter. Public confidence and measurement of public confidence were addressed in written comments and verbal comments during the stakeholder meeting. A summary of these comments was provided to the Commission in an October 21, 1999, letter from Jesse L. Funches, Chief Financial Officer, "Stakeholder Comments on the Nuclear Reactor Safety Chapter of the Strategic Plan." Stakeholder comments related to public confidence were summarized in this letter which stated that "[o]ne significant and unanimous comment from the stakeholders was that NRC needs to benchmark the "Increase Public Confidence" goal using a public survey instrument." As part of the review of these comments the Executive Council considered such a survey. The letter further stated that "[t]he Executive Council believes such survey should address all of NRC's regulatory responsibilities, not just reactors." Finally, the letter stated that "[t]he staff will assess the feasibility of such a survey and provide the Commission with its assessment."

This paper provides the result of the staff's assessment and recommendations related to conducting surveys to measure public confidence in NRC's regulatory activities.

DISCUSSION

The Strategic Plan includes strategies and performance goals related to improving public confidence. These goals state that we will (1) increase and (2) measure changes in public confidence in the NRC. In order to achieve these goals, a first step is to define a baseline measure of public confidence. The results of these measures would be used to assess the effectiveness of staff activities in this area. The process of measuring public confidence can be complicated by how we define the term "public" in context of the NRC's regulatory activities. While discussed, this term has not been clearly defined in the strategic plan. The definition is central to the development and implementation of measurement tools to ensure that the proper populations and issues are included, and to set expectations for the survey as well as uses for its results.

Who is the Public?

We commonly use the term "external stakeholders" to refer to populations outside the agency affected by NRC actions and programs. The groups that comprise external stakeholders have a broad range of knowledge and understanding of the NRC. Since only specific segments of external stakeholders typically have direct interactions with the NRC, it may be beneficial to consider the differences among external stakeholders. For purposes of this discussion, one segment of external stakeholders can be defined as "indirect stakeholders." The indirect segment includes members of the general public, and external stakeholders who have limited or no interactions with the NRC. Another segment can be defined as "direct stakeholders." The direct segment includes stakeholders who have frequent interaction with the NRC such as licensees, licensee employees, and some State, Local or Federal government representatives.

In general, stakeholders in the direct segment have a better understanding of who we are and what we do, as opposed to the indirect segment who may have little or no knowledge of the NRC. This distinction is important when considering the types of activities which could be implemented by the staff to increase and subsequently measure the effectiveness of communications with external stakeholders. For example, if our efforts were intended to address the general public or indirect segment, the activities would be more general such as outreach efforts to inform this group of NRC's mission, responsibilities and activities. The external NRC Web site is an example of an outreach effort. Different types of efforts would be considered to address the concerns of stakeholders in the direct segment. Activities which affect the perceptions of both segments would be considered as part of the development of the measurement method.

This discussion of stakeholders was provided to illustrate the segments which could be included in an effort to measure stakeholder views on or confidence in the NRC. The segments could be divided into smaller groups depending upon the information that is intended to be included in the scope of the measurement. Identifying the types of questions and participants that would provide meaningful data can be challenging. If the segment to be measured does not understand the mission of the NRC, the information is more likely to reflect this lack of information or understanding about the NRC. Measurement of this group's views is likely to provide meaningful information about some NRC activities but may not provide useful information in evaluating agency performance in activities such as inspection and licensing. Additionally, not all segments of the public described above have the same functional relationship or frequency on interaction with the NRC. For these reason, the staff will need the support of contractors that have in-depth experience in developing appropriate measurement techniques.

Measurement Techniques

A primary consideration of any technique is to determine which external stakeholder perceptions or views should be measured, and the willingness of the participants to take part in the measurement activity. After determining the population and issues of interest, the next decision is to choose a measurement technique or method. Surveys and interviews offer the advantages of being direct measurement techniques. In the case of public confidence, an initial survey could be conducted which would define or create an initial baseline measure of public confidence. Follow up surveys would be conducted to determine if public confidence was changed as a result of staff activities. In order to create a broad baseline or measure of public confidence in NRC activities, numerous surveys would need to be developed and conducted.

The staff evaluated three options for measuring public confidence: 1) develop and conduct a survey with support provided by a commercial contractor; 2) participation in the American Customer Satisfaction Index (ACSI); and 3) a system or process for collecting and evaluating feedback. In considering these alternatives, the staff met with two commercial contractors and representatives of the ACSI. The staff also contacted three other agencies with a regulatory function to ask them about their survey practices. These three agencies participated in the ACSI as well as using commercial contractors for other surveys. For each of the options discussed below, the questions would be tailored to the issues that the NRC believes will provide insight on our ethics, credibility, and actions.

1)  Conventional Survey by a Commercial Contractor

This option involves working with a contractor to develop and implement a survey that would be designed specifically for the NRC. Based upon discussions with other federal agencies with a regulatory mission, this is a frequently used option. Some agencies have used this technique for years, to evaluate program performance and customer expectations. The survey data is often kept within the agency, to help focus on improvements to various agency functions. The agencies used a combination of commercial contractors and in-house expertise to develop and conduct the surveys. One agency is planning to evaluate their progress on their strategic plan performance goals through surveys.

To evaluate this option further, the staff met with two commercial contractors. Preliminary meetings with commercial contractors suggest that during survey development, commercial contractors would meet with focus groups to learn how best to design the survey. The contractors would probably start with a limited segment of the public and focus on a limited number of issues or questions. For example, they might suggest surveying the public living near Yucca Mountain to solicit their views on NRC's actions concerning high-level waste. Both contractors indicated that the recommended sample size for a survey of this type was about 2,000 individuals.

The timing of a survey performed by a commercial contractor would be affected by the Census blackout. This blackout period runs from March through June 2000. Based on the blackout dates, a survey could take place after June 2000. Completion of the survey and analysis of the data is expected to take approximately four to five months. The cost of this type of survey is estimated to be $100 - 200K.

The primary advantage of using a conventional survey instrument is the flexibility. The survey instrument can be developed to meet the specific needs of the NRC. The primary disadvantages are the cost and the time necessary to develop and implement the survey.

2)  The American Customer Satisfaction Index (ACSI)

The same agencies that were contacted about their experience with conventional surveys had participated in the American Customer Satisfaction Index (ACSI). Their participation was a direct result of Executive Order 12862, September 11, 1993, "Setting Customer Service Standards," which directed that the standard of quality for government services equal that of business.

The ACSI is produced by a partnership of the National Quality Research Center at the University of Michigan Business School, Arthur Andersen, and the American Society for Quality. The ACSI partnership was awarded the contract to measure customer satisfaction for high impact agencies by the General Services Administration. High impact agencies were designated by the National Partnership for Reinventing Government (NPR) because of their breadth and visibility of interactions with the public. The index currently measures satisfaction, and the drivers and outcomes of satisfaction, for 170 private sector companies, two types of local government services, the U.S. Postal Service, and 30 Federal customer segments. Among the agencies surveyed were agencies with regulatory functions such as EPA , FAA , FEMA , OSHA , and FDA .

An ACSI survey would have a standard sample size of 250 individuals per "segment." Agencies work with ACSI to identify the stakeholder segment to be surveyed, develop a list of possible participants in the segment, and tailor the questions about agency activities. Rather than developing issue-specific questions, the index provides a database of questions that can be tailored to customer satisfaction activities of concern to the agency. The index focuses on customer satisfaction and includes questions which solicit information on activities such as processes (for obtaining services), accessibility of information, and customer service from agency personnel. There is a difference in the outcomes for the surveys conducted for the private and public sectors. The measured outcome for the private sector companies is customer loyalty. For government agencies, it is the trust of its customers.

As discussed in the ACSI's December 1999 report to the General Services Administration, ACSI uses a model that ties customers' evaluations of quality to satisfaction, and then explains the effects of satisfaction on customer complaints and on an objective of importance to a company or government agency. "For most government agencies the objective is some form of customer/user trust, typically measured in terms of confidence in the agency, and either future reliance on agency services or future compliance with agency regulations." This report also discussed the difference in survey results between regulatory agencies and agencies that provide direct benefits to members of the public. The report stated that "Not unexpectedly... satisfaction is highest among customer segments that receive a direct benefit from an agency and lowest for customer segments subject to regulation by agencies such as the Federal Aviation Administration and the Occupational Safety and Health Administration."

Although the index is not as flexible as a survey developed for the NRC by a commercial contractor, it is less expensive. Preliminary discussions indicate that the cost would be approximately $35K per segment. Costs would be directly proportional to the number of segments surveyed. There would also be some time and effort saved in developing the survey questions. The ACSI would be an interagency effort, not a commercial contract.

The visibility of the results would be another aspect of the ACSI that would differ somewhat from conventional surveys. For the agencies that participated in the 1999 ACSI surveys, the NPR expected the agencies to list the results and follow up actions on an agency-sponsored web page. The NRC would likewise be expected to publish the results of the survey on our web site and develop and implement an action plan to respond to deficiencies in customer service noted in the index results.

The primary advantages of participating in this index are 1) the ASCI survey is an established and accepted measurement tool, 2) can be conducted for specific segments in a timely manner (approximately 4 to 6 months) at reasonable cost, 3) allows the identification of specific actions in response to stakeholder concerns and comments, 4) contracting through an interagency agreement will require less resources than using a commercial contract, 5) the survey facilitates comparing the performance of government agencies and private sector companies, and 6) OMB has issued Paperwork Reduction Act waivers for ACSI surveys conducted for Federal agencies and is in the process of issuing a blanket waiver for additional ACSI surveys for Federal agencies.

The disadvantage of participating in this index, as expressed by the agency representatives who discussed the ACSI with the staff, is the potential confusion caused by comparison of one agency's results to other Federal and private sector companies.

3)  System for Collecting and Evaluating Feedback

This option is a less direct and more qualitative method of measuring public confidence. This option would be a continuation of staff activities augmented by a method to collect, record and analyze data. As noted above, the staff has initiated a range of activities which have increased both the number of opportunities for and effectiveness of interactions with external stakeholders. These activities include meetings and focus group discussions which include comments and feedback to the staff on NRC actions. As a result of stakeholder feedback, the staff has a better understanding of the issues of concern to these stakeholders and continues to make changes to address these issues. While this process is iterative, the staff believes that the interactions are positive and are resulting in improved public understanding and confidence in the NRC.

One approach to augment the current process would be to request feedback or comments as part of each or selected interactions with external stakeholders. This could include a more publicly visible and documented approach to solicit, collect, evaluate and provide responses to stakeholder feedback. The conclusions reached based upon this option would be limited, in that, only a small segment of stakeholders participate in these interactions. Typically, participants in these interactions include customers which have specific interests and or issues to discuss. As a result, the comments and feedback would tend to be focused and the conclusions may be difficult to generalize. On the other hand, this group is typically more informed and knowledgeable of NRC activities and could provide more meaningful information related to programmatic activities.

In order to implement this option, a process which includes defining the questions to be asked, collecting and evaluating the responses would need to be developed. This process could be developed with the assistance of a contractor. Following a review of NRC's interactions with stakeholders, a contractor could make recommendations concerning what indirect methods would be most applicable or appropriate measures of public confidence.

RECOMMENDATION

In order to provide a baseline measure of public confidence, the staff recommends that a survey be conducted. Additionally, based upon the experience other agencies have had with the ACSI, and the relatively low cost of this survey, the staff recommends that the ACSI technique be used to establish the baseline measurement of public confidence.

Following Commission approval the staff will work with a contractor to identify the individuals to be included in the survey and to prepare the questions in the survey. Since there are various existing methods for NRC licensees and industry representatives to provide comments to the staff (direct stakeholders), the initial survey participants could be limited to members of the general public (indirect stakeholders). The survey would not include NRC licensees, industry representatives or individuals associated with the nuclear industry. The results of this approach could be used to determine what changes to consider related to interactions with and information provided to the general public.

For example, a survey could include members of the public living in the emergency planning zone of a commercial power reactor. This survey could provide information and insights that would describe this group's understanding of NRC and its regulatory programs. The results would be useful to determine the awareness of ongoing NRC activities and assess the methods currently used to communicate with the group and determine what changes could be made. The results would provide information which would better characterize this group's desire for or expectations concerning information that could be provided by the NRC. Another example could be a survey to determine the public's views on the information contained in and ease of use of the NRC's Web site. The results of this survey would be used to determine what changes would be made to the Web site to enhance its usefulness.

Based upon the experience and knowledge gained by conducting this initial baseline survey, the staff will evaluate the usefulness of the information, get feedback from the stakeholders who participated in the survey and make a recommendation to the Commission regarding subsequent surveys, including surveys that would include direct stakeholders.

RESOURCES:

If the Commission adopts the staff recommendation of using the ACSI technique (Option 2), costs would be directly proportional to the number of segments surveyed. The resources required to conduct each segment are approximately $35K and can be reprogrammed within existing agency resources.

COORDINATION:

This plan has been coordinated with the Office of Public Affairs (OPA). The Office of the Chief Financial Officer (OCFO) and the Office of the Chief Information Officer (OCIO) have no objections. The Office of the General Counsel (OGC) has no legal objection to this paper.