| |
Search Options | |||
| Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us | |
|||
|
POLICY ISSUE SECY-06-0074 March 31, 2006
To present the results of the staff's annual self-assessment of the Reactor Oversight Process (ROP) for calendar year (CY) 2005. The CY 2005 self-assessment results indicate that the ROP has been effective in meeting the program goals and achieving its intended outcomes; however, areas needing further improvement remain. The self-assessment identified the ROP as successful in being objective, risk-informed, understandable, and predictable, and in ensuring safety, openness, and effectiveness. The effective implementation of the various ROP processes demonstrates that the ROP achieved its intended outcomes in CY 2005. The staff continued to focus on stakeholder involvement and to improve various aspects of the ROP as a result of feedback and lessons learned. In particular, the staff implemented several additional ROP improvements advocated by the Commission, and incorporated several changes based on independent reviews by the Davis-Besse Lessons Learned Task Force (DBLLTF) and the Office of the Inspector General (OIG). The staff also made numerous improvements based on feedback from internal and external stakeholders. Stakeholder feedback, provided through an external survey and various other mechanisms, was generally positive. Stakeholders raised specific concerns about the effectiveness of the performance indicator (PI) program and the consistency and timeliness of the significance determination process (SDP), and suggested other improvements to the ROP. The ROP met most of its 54 performance metrics, with the exception of three PI metrics, one inspection metric, two SDP metrics, and one assessment metric. As part of the self-assessment effort, the staff identified issues and actions in the key ROP program areas of PIs, inspection, SDP, and assessment. The timeliness and efficiency of the process for resolving interpretations in PI guidance has improved, but the staff and many stakeholders remain concerned about the ability of the current set of PIs to contribute to the identification of declining performance in a timely manner. The staff implemented several changes to the inspection program to address recommendations from independent evaluations and from a detailed analysis of the scope and level of effort of each of the inspection procedures. The staff continues to focus on improving SDP timeliness and has made significant progress in implementing the SDP Improvement Plan, though timeliness again fell short of expectations. The staff expects SDP timeliness to further improve based on recent process enhancements. The staff also made additional enhancements in the assessment program during CY 2005, particularly in the area of cross-cutting issues, and expects further improvement in CY 2006 based on program revisions to more fully address safety culture. Although progress has been made in CY 2005, the staff will continue to improve the ROP based on lessons learned and stakeholder feedback. The staff will actively solicit input from the NRC's internal and external stakeholders and will evaluate potential program improvements via the ongoing self-assessment process. On February 24, 2000, the staff issued SECY-00-0049, "Results of the Revised Reactor Oversight Process Pilot Program." The resulting staff requirements memorandum (SRM), issued on March 28, 2000, approved initial implementation of the ROP as recommended by the staff. The initial implementation of the ROP began on April 2, 2000. In a followup SRM issued on May 17, 2000, the Commission directed the staff to report on the ROP results after the first year of implementation. The staff did so and documented the results in SECY-01-0114, "Results of the Initial Implementation of the New Reactor Oversight Process," issued June 25, 2001. SECY-01-0114 also noted the staff's intention to continue to perform an annual self-assessment of the ROP. Accordingly, the staff has issued an ROP self-assessment Commission paper each year before the Agency Action Review Meeting (AARM) and has briefed the Commission on the self-assessment results following the AARM. This paper provides the results of the sixth annual self-assessment of the ROP. The staff performed the CY 2005 self-assessment in accordance with Inspection Manual Chapter (IMC) 0307, "Reactor Oversight Process Self-Assessment Program." As noted in IMC 0307, the ROP is a regulatory framework that includes licensee PI data, NRC inspection activity and determination of inspection finding significance, and licensee performance assessment. The ROP self-assessment program evaluates the overall effectiveness of the ROP through its success in meeting its preestablished goals and intended outcomes. In accordance with IMC 0307 and as noted in SECY-05-0070, "Reactor Oversight Process Self-Assessment for Calendar Year 2004," security and safeguards activities are no longer included in this self-assessment except where specifically noted. In response to the staff's briefing on the results of the AARM on May 25, 2005, the Commission directed the staff to take several actions in the SRM dated June 30, 2005. The Commission directed that in addition to efforts described in the ROP self-assessment, the staff should consider further improvements to PIs to provide the NRC with better indicators of performance to help focus inspection resources. The Commission also directed the staff to (1) make further efforts to clarify the guidance on substantive cross-cutting issues, (2) continue to emphasize the importance of effectively implementing a good licensee corrective action program as it participates in conferences, workshops, and meetings with licensees, and (3) ensure that the Mitigating System Performance Index (MSPI) process is as transparent as possible to external and internal stakeholders when it is implemented. The ROP self-assessment process uses program evaluations and performance metrics to determine its success in meeting the goals and intended outcomes of the ROP. The ROP is considered effective if it meets the program goals and achieves the intended outcomes. The ROP's seven goals include the four specific program goals of being objective, risk-informed, understandable, and predictable, as well as the three applicable performance goals listed in the NRC's Strategic Plan (ensuring safety, openness, and effectiveness). The intended outcomes of the ROP, which help form its basis and are incorporated into the various ROP processes, include appropriately monitoring and assessing licensee performance, identifying performance issues through NRC inspection and licensee PIs, determining the safety significance of identified performance issues, adjusting resources to focus on significant performance issues, evaluating the adequacy of corrective actions for performance issues, taking necessary regulatory actions for significant performance issues, communicating inspection and assessment results to stakeholders, and making program improvements based on stakeholder feedback and lessons learned. During the sixth year of ROP implementation (CY 2005), the staff conducted numerous activities and obtained data from many diverse sources to ensure that it performed a comprehensive and robust self-assessment. Data sources included the ROP performance metrics described in IMC 0307, recommendations from independent evaluations, comments from external stakeholders in response to a Federal Register notice, insights from internal stakeholders based on the ROP internal feedback process, and feedback received from stakeholders at various meetings, workshops, and conferences. The staff also applied the direction and insight provided by the Commission through several SRMs. The staff analyzed this information to gain insights regarding the effectiveness of the ROP in fulfilling its program goals and intended outcomes. The staff has concluded that the ROP was effective in CY 2005 based on meeting the seven program goals, although the staff continues to experience challenges in certain ROP areas and recognizes the need for further improvement. The staff further believes that successful implementation of the various ROP processes has demonstrated that the ROP achieved its intended outcomes in CY 2005. As a result, the staff has concluded that the NRC has appropriately monitored operating nuclear power plant activities and focused agency resources on significant performance issues in CY 2005, and that plants continue to receive a level of oversight commensurate with their performance. The staff identified issues and needed actions in the key program areas of PIs, inspection, SDP, and assessment, as discussed in the following paragraphs. In addition, the staff assessment included discussions of ROP communication and training activities, ROP self-assessment and independent evaluations, ROP resources, and resident inspector (RI) demographics. As noted in the pertinent sections of this paper, the staff has also included several enclosures with additional detail to support the staff's assessment and conclusions. ROP Program Area Evaluations The staff performed evaluations in each of the four key program areas of the ROP—PI program, inspection program, SDP, and assessment program. The results are summarized below. Enclosures 1 through 4 to this paper discuss these four ROP program areas in more detail, respectively. Enclosure 5 PI Program—Although the PI program continues to provide the NRC with objective indicators regarding plant performance, and in some areas has focused licensee attention and contributed to improved performance, the staff and some public stakeholders remain concerned with the capability of the current PIs to contribute to the identification of declining performance. The staff engaged senior industry management to explore possible actions to address issues with the PI program and plans to continue analyzing potential improvements, including adjusting PI thresholds, adding or removing PIs, and clarifying PI definitions. Three of the seven PI performance metrics failed to meet their established criteria in CY 2005. These metrics failed based on an increase in reporting deficiencies and discrepant PIs, and the negative perception, particularly by public interest groups, regarding the usefulness of the PIs in ensuring plant safety and the existence of an appropriate overlap between the PIs and the inspection program. The staff continued to work with industry in CY 2005 and plans to replace the safety system unavailability PIs with the MSPI in April 2006. The timeliness and efficiency of the frequently asked question process for resolving interpretations in PI guidance has improved over prior years based on effective program changes. Inspection Program—The inspection program continued to improve during the sixth year of ROP implementation. In particular, the staff implemented several changes to address recommendations from the OIG audit of the baseline inspection program, the DBLLTF, and a detailed analysis of the scope and level of effort of each of the inspection procedures (IPs). In response to the Commission's direction, the staff completed a pilot program to improve the effectiveness of the engineering design inspections. The staff plans to conduct the revised engineering inspections at all plant sites by the end of CY 2007 and to reevaluate the inspections after CY 2007 based on lessons learned during implementation of the revised IP. The regions completed the required baseline inspection program for CY 2005. Although feedback on the inspection program was generally favorable, the staff's self-assessment of inspection findings, internal and external comments, and other independent reviews of the ROP indicated that additional adjustments may be warranted in the level of resources applied to some of the baseline IPs. The staff plans to further refine and formalize the process to ensure alignment of inspection resources to include consideration of industry performance. All but one inspection program performance metric met their criteria during CY 2005. The failed temporary instruction (TI) timeliness metric has prompted the staff to review the reasons for untimely completion of TIs and recommend possible solutions in CY 2006. Significance Determination Process—Improvements in SDP efficiency and effectiveness continued during CY 2005. The staff made significant advances to complete the objectives of the SDP Improvement Plan and improve SDP timeliness, although additional actions are needed. The most notable achievements in CY 2005 included issuing Revision 2 of the risk-informed inspection notebooks and the accompanying pre-solved tables, issuing a new SDP addressing maintenance-related findings, further revising the fire protection SDP, and increasing NRC management attention towards improving SDP timeliness. Recent SDP process changes, including the creation of the new Planning SDP/Enforcement Review Panel (SERP) and the use of best available information, are intended to achieve efficiencies and further improve the process. The planning SERP is expected to improve the early identification and time management of potentially complex and/or moderate to high risk significance issues. The Planning SERP, convened early in the process, will reach consensus on the scope of the evaluation to be performed, the schedule on which the evaluation will be completed, and who will perform the evaluation. Two of the eight SDP performance metrics failed to meet program expectations. Final significance determinations were not timely, and the SDP was not perceived to yield an appropriate and consistent regulatory response across all ROP cornerstones. The staff continues to believe that relative parity has been achieved among the cornerstones based on the potential impact on public health and safety and the designated NRC response to specific findings. SDP timeliness improved in CY 2005, and the staff expects additional progress from the changes noted above and the enhanced training regimen associated with each new SDP and SDP revision. The staff will continue to monitor planned SDP improvements and developments via the SDP Improvement Plan. Assessment Program—During CY 2005, the staff made additional improvements to the assessment program, as reflected in revisions to IMC 0305, "Operating Reactor Assessment Program," and IMC 0350, "Oversight of Reactor Facilities in a Shutdown Condition Due to Significant Performance and/or Operational Concerns." In particular, the staff revised the guidance to (1) further clarify the development and treatment of substantive cross-cutting issues, (2) provide better definitions of the human performance and problem identification and resolution bins, (3) clarify the exit criteria for substantive cross-cutting issues, and (4) incorporate recommended improvements by the Davis-Besse Oversight Panel. All but one of the performance metrics in the assessment area met its established criteria or goals in CY 2005. The exception was the increase in the number of Action Matrix deviations. The staff evaluated the deviations for potential program changes and incorporated one. Additional program changes based on deviations may result from the staff's efforts to more fully integrate safety culture into the ROP. The staff developed an approach, with involvement of internal and external stakeholders, to enhance the treatment of cross-cutting areas in the ROP and in supplemental procedures to more fully address safety culture. The staff expects to complete the safety culture enhancements to the ROP in CY 2006. ROP Communication and Training Activities The staff continued to focus on stakeholder involvement and open communication about the ROP. The staff used a variety of communication vehicles to ensure that all stakeholders have access to ROP information and results, and have an opportunity to participate in the process and provide feedback. The staff continued to conduct monthly public meetings with external stakeholders and conducted a survey of external stakeholders to actively solicit and analyze stakeholder feedback regarding the effectiveness of the ROP. The staff also continued the ongoing internal feedback process, held biweekly telephone conferences and frequent meetings with internal stakeholders, and visited several sites and each region to give regional staff and management the opportunity to discuss ROP implementation and provide feedback. The responses from the survey of external stakeholders were similar in content to previous years, as were the number and distribution of the responses. Specifically, 9 of the 21 responses were from utilities, while 4 were from State agencies, 7 were from the public, and 1 was from an anonymous respondent. Overall, the responses were generally positive, but they raised specific concerns about the effectiveness of the PI program, the consistency and timeliness of the SDP, and other perceived needed improvements to the ROP. Enclosure 6 To address a prior concern that the staff had been unresponsive to the external stakeholder survey comments, the staff consolidated the comments by question during CY 2005 and provided a comprehensive response to each question from last year's survey. Since the staff received positive feedback on the consolidated response from several stakeholders, and to ensure openness and the continued positive perception that the NRC is responsive to the public's inputs and comments on the ROP, the staff plans to prepare a similar consolidated response to this year's survey. This paper, the annual ROP performance metric report, and the consolidated response to the CY 2005 external survey will be posted to the ROP Web page and sent along with an acknowledgment letter to each survey respondent. The staff also continued its efforts to improve inspector training. During CY 2005, the staff developed and distributed the first survey on inspector training effectiveness, and the responses to the survey were favorable. The staff is further reviewing the survey results and will consider and implement program changes based on recommendations and insights gained. The staff developed and implemented Web-based read-and-sign training on substantive cross-cutting issues, documenting findings in inspection reports, and revisions to the SDP guidance. The staff continued to maintain the ROP Web pages to ensure that they remain useful tools for communicating accurate and timely ROP information to all stakeholders. In an effort to increase inspector efficiency, the staff developed a pocket reference guide that provides field observation best practices, implemented a Web-based knowledge management tool, and continued to issue the inspector newsletter. In summary, the staff continues to seek and implement improvements to the ROP based on feedback and insights from all stakeholders. Enclosure 6 ROP Self-Assessment Metrics and Independent Evaluations The staff revised the ROP self-assessment program, as defined in IMC 0307, to support the new safety performance measures of the NRC's Strategic Plan, to better define the ROP goals and intended outcomes, and to consolidate and clarify several of the performance metrics. The staff conducted the CY 2005 ROP self-assessment in accordance with the revised IMC 0307 and associated performance goals. In addition to the ROP self-assessment program, several independent evaluations have been performed in the past few years, most notably those by the Office of Management and Budget, OIG, and the DBLLTF. These evaluations have generally provided favorable results, but have also suggested potential areas of improvement. This paper addresses several recommendations from these independent evaluations. In CY 2006, the staff expects to receive a report from the General Accounting Office on ROP implementation and will address any recommendations as necessary. In order to gain further efficiencies, and because the comments and staff analysis have tended to repeat the same themes from year to year, the staff is considering a change in the frequency of the external survey to every other year, consistent with the internal survey. As such, one year's ROP performance metrics and self-assessment would include survey inputs and analysis from internal stakeholders, and the following year would include external survey inputs and analysis. Regardless, internal and external feedback will be considered each year based on continuous feedback during meetings, the feedback process, and other venues as described in Enclosure 6 Annual ROP Performance Metrics—The staff performed its annual self-assessment of performance metrics for CY 2005 in accordance with the recent revision to IMC 0307. The majority of metrics met their established criteria; however, some metrics in each of the ROP program areas did not. The program area evaluations in Enclosures 1 through 4 discuss the staff's corrective actions to address these issues. In addition to the specific program area metrics, there are 17 overall ROP metrics, each of which met the established criteria. Two of these metrics failed to meet their criteria during last year's evaluation, but, as a result of staff corrective actions and increased positive perception, they met their criteria this year. Specifically, these metrics gauge whether the public perceives the NRC to be responsive to its inputs and comments, and whether the public perceives that the ROP results in unintended consequences. The annual performance metric report is publically available through ADAMS (reference ADAMS Accession No. ML060590135). DBLLTF Recommendations—In 2005, the staff completed the necessary enhancements to the ROP based on the implementation of DBLLTF action items. These changes will enhance the NRC's ability to detect declining plant performance, including the specific issues that were identified at the Davis-Besse plant. The staff issued its final status report to the Commission on October 4, 2005, detailing the changes made to address the DBLLTF recommendations. On November 1, 2005, the staff conducted the last public Commission briefing on the status of DBLLTF recommendations. As noted in Enclosure 2 OIG Audit Activity—The staff successfully resolved all 10 recommendations from the OIG audit of the baseline inspection program (OIG-05-A-06, issued December 22, 2004). The staff made changes to the inspection program to close all but three of the recommendations by the end of CY 2005 as discussed further in Enclosure 2 SDP Task Group Recommendations—The agency established the SDP task group in 2002 to complete an independent and objective review of the SDP. The SDP task group developed 30 recommendations generally aimed at improving the Phase 2 evaluations using the risk-informed inspection notebooks. The staff has revised the SDP guidance or other portions of the ROP to incorporate 27 of the task group's recommendations. The staff is tracking the three remaining recommendations using the SDP Improvement Plan. Regulatory Impact—The staff receives and evaluates feedback from licensees on an annual basis as part of the regulatory impact process, established in 1991 based on Commission direction to develop a process for obtaining feedback from licensees and reporting the feedback to the Commission. Over the past year, the staff received feedback from 91 reactor licensees on 253 issues. The staff also received feedback from the Regulatory Information Conference in March 2005. Of the comments received, 83 percent were favorable and 17 percent were unfavorable. The comments fell into three main categories—formal communication with licensees, inspector performance, and security and safeguards activities. Enclosure 7 Industry Performance Trends—The NRC collects and monitors industrywide data to assess whether the nuclear industry is maintaining the safety performance of operating plants. The NRC also uses these industry-level indicators to provide feedback on the ROP. In CY 2005, the staff continued to implement and further develop the Industry Trends Program (ITP). The staff added several new indicators based on a consolidation of the data submitted by licensees to support the ROP PIs. The staff also revised the long-term trending methodology to be influenced less by historical plant performance. An annual Commission paper that complements this document reported the results of the ITP, along with any actions taken or planned. The results of the ITP will also be reviewed at the AARM. ROP Resource Analysis As in the 2004 inspection cycle, all four regions completed their baseline inspections in CY 2005 with the allocated regional resources without the need for the coping measures experienced in CY 2002 and 2003. The inspection effort expended for the ROP has increased steadily during the 2003, 2004, and 2005 inspection cycles. Overall staff effort in 2005 was 5.4 percent higher than in 2004. The increased inspection effort in 2005 was most likely the result of increased regional inspection activity due to additional requirements that have been imposed on the inspection staff in recent years, including implementation of the DBLLTF and OIG recommendations and other program improvements. A recent reevaluation of ROP resource needs indicated that the regional inspection budget should increase by 14 full-time equivalent (FTE) staff members. The FY 2007 and FY 2008 inspection budget requests include these additional resources. The staff recognizes that inspection resources in FY 2006 may be strained. However, the staff anticipates that baseline inspections and other elements of the ROP will be completed as they have been during the past year and will redirect existing resources as necessary. Although the ROP has resulted in improved inspection effectiveness, any efficiency gains that may have been achieved since ROP implementation have likely been offset by the additional requirements that have been imposed on the inspection staff. As discussed in other sections of this paper, a number of initiatives currently underway may improve program effectiveness. These initiatives include a realignment of resources allocated to the individual baseline IPs, revised engineering design inspections, pilot implementation of the "unique site" budget models, continued SDP improvements, and implementation of the MSPI program. Enclosure 8 Resident Inspector Demographics and Site Staffing As directed in an SRM dated April 8, 1998, the staff developed measures to monitor and trend RI demographics and reports the data and analyses to the Commission on an annual basis. The staff also developed a site staffing metric in response to a DBLLTF recommendation, which has been included with the annual demographics analysis. The 2005 data indicate that the experience levels of both RIs and SRIs are relatively high, the RI and SRI staffing levels are generally good, and the staffing turnover rate was not excessive. Enclosure 9 Prior Commitments—The staff made six commitments in the CY 2004 ROP self-assessment Commission paper to improve the efficiency and effectiveness of the ROP. The following summarizes the staff's actions and status to address these commitments:
New Commitments—As described in this paper, the staff plans the following significant actions or activities to improve the efficiency and effectiveness of the ROP in CY 2006:
The staff will include the status of these commitments and other program improvements noted in this paper in the CY 2006 ROP self-assessment Commission paper. The self-assessment results in CY 2005 indicate that the ROP has been effective in meeting the seven program goals and achieving its intended outcomes, although the staff continues to experience challenges in certain areas and recognizes the need for further improvement. The ROP was successful in being objective, risk-informed, understandable, and predictable, and in ensuring safety, openness, and effectiveness. The ROP achieved its intended outcomes in CY 2005, as demonstrated by the successful implementation of the various ROP processes. The NRC has appropriately monitored operating nuclear power plant activities and focused agency resources on significant performance issues in CY 2005, and plants continue to receive a level of oversight commensurate with their performance. The staff maintained its focus on stakeholder involvement and continued to improve various aspects of the ROP as a result of feedback and lessons learned. Based on its CY 2005 self-assessment, the staff intends to focus on the following areas during CY 2006:
The staff will also continue evolutionary improvements to various aspects of the ROP. This paper describes a number of program improvement activities. The budget requests through FY 2007 include the resources required to maintain the ROP and develop and implement improvements as part of the overall ROP refinement and management effort. These resources include all regional, Office of Nuclear Reactor Regulation, Office of Research, and Office of Nuclear Security and Incident Response efforts for ROP refinement, management, and performance assessment activities within the scope of the current budget requests. Currently, the staff estimates that approximately 60 full-time equivalent (FTE) staff members and $1.8 million will be needed for FY 2006 and 60 FTE and $850,000 will be needed for FY 2007. No additional resources are needed for FY 2006 and FY 2007 for these activities. Planned actions to improve the ROP will be prioritized and scheduled to remain within allocated resources. The Office of the General Counsel has reviewed this Commission paper and has no legal objections to its content. The Office of the Chief Financial Officer has reviewed this Commission paper for resource implications and has no objections.
|
|
Privacy Policy |
Site Disclaimer |
provides a consolidated listing of implementation issues in each program area with the status of each issue. In addition, the annual ROP performance metric report, available through the Agencywide Documents Access and Management System (ADAMS), provides the data and staff analysis for each of the program area metrics (reference ADAMS Accession No. ML060590135).