| |
Search Options | |||
| Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us | |
|||
SECY-97-072
April 2, 1997
| FOR: | The Commissioners |
| FROM: | L. Joseph Callan /s/ Executive Director for Operations |
| SUBJECT: | STAFF ACTION PLAN TO IMPROVE THE SENIOR MANAGEMENT MEETING PROCESS |
PURPOSE:
To inform the Commission of the Staff's plans to address the recommendations made by the Arthur Andersen Company to improve the Senior Management Meeting process.
BACKGROUND:
The Senior Management Meeting (SMM) is an important part of the NRC process for evaluating licensee performance. Decisions regarding the appropriate level of regulatory attention to plants exhibiting performance problems are made at the semiannual SMM(1). Throughout the process, the information about the performance of the plants is provided primarily by the regions and the Office for Nuclear Reactor Regulation (NRR). The Office for Analysis and Evaluation of Operational Data (AEOD) also provides independent insights based on the NRC Performance Indicators. The Office of Enforcement (OE) and the Office of Investigations (OI) provide additional insights.
The SMM process is described in detail in Attachments 1 and 2 of SECY-96-093,(2) and in Management Directives 8.13 and 8.14.
In a Staff Requirements Memorandum (SRM) dated June 28, 1996, the Commission directed the staff to assess the SMM process and evaluate the development of indicators that can provide a basis for judging whether a plant should be placed on or deleted from the watch list. In response to the Commission request, a study was completed with the assistance of a contractor (Arthur Andersen, Limited Liability Partnership). The contractor's report was issued December 30, 1996, and the staff briefed the Commission on the results on February 18, 1997.
The report contained the following findings and recommendations:
This paper fulfills the staff's commitment to submit recommendations for improvements to the SMM process, including the staff's plans to address the recommendations of the Arthur Andersen report. The paper also responds to the principal points of an SRM dated March 14, 1997.
DISCUSSION:
ARTHUR ANDERSEN RECOMMENDATIONS
The recommendations in the Arthur Andersen report address two areas: the SMM information base and the SMM process. The first area involves the inputs to the SMM decision process, including performance indicators and the criteria used by the senior managers. The second area involves the roles of SMM participants, the method of reaching consensus, the presentation of information, and the documentation of meeting results.
SMM Information Base:
The staff has developed a detailed draft action plan to address the Arthur Andersen recommendations regarding the information base for the SMM. The plan includes provisions for the development of technical products, peer review by ACRS, and public participation. A draft milestone chart is included as Attachment 2. The fully developed products from this plan are scheduled to be available for trial use and evaluation for the June 1998 SMM. Preliminary inputs will be made available for the January 1998 SMM.
Trend plots and economic indicators based on the Arthur Andersen report will be supplied for the SMM screening meetings scheduled for April 1997. The trend plots and economic indicators will be provided for the purpose of enhancing the information base of the SMM screening meetings, but will not be used as part of the formal decision process during the June 1997 SMM.
The plan addresses the Arthur Andersen recommendations as follows:
Recommendation 1:
Recommend assessing leading indicators, such as management and operational effectiveness on an ongoing basis.
Action:
The staff has initiated development of a process to assess leading indicators, such as management and operational effectiveness, on an ongoing basis. Arthur Andersen has been contracted to assist in the development of methodologies for measuring management and operational effectiveness. The staff will develop characteristics, measures, and indicators based on insights from existing NRC inspection programs, ongoing NRC research, and industry evaluation techniques.
Management measures to be considered include problem identification and resolution, work processes, management involvement/control, and communications. Operational measures to be considered will address the quality of maintenance, engineering, operations, and plant support functions. After identifying appropriate measures, the staff will determine how to best integrate the assessment of the measures into the agency's processes. In addition, the staff will evaluate the need for staff training in this area and develop any required inspection guidance.
A status report on the pilot study will be made available to the January 1998 SMM. However, initial input to the SMM decision process is expected to begin with the June 1998 SMM.
Recommendation 2:
Recommend re-engineering assessment information to better support SMM decision-making process.
Action:
The staff has initiated a program to develop a plant performance template and evaluation criteria for use in the process of identifying candidate plants for formal action in the context of the SMM. In addition to the management and operational effectiveness measures discussed above, the template will use a validated algorithm for analysis of performance indicators. The algorithm will be primarily based upon the current set of performance indicators, but may include other indicators judged to have a relation to plant safety performance. To the extent possible, the indicators chosen for use in this process should be related to nuclear safety and regulatory performance; should be based on information readily available to NRC; should not be subject to manipulation; should be comparable among licensees; should reflect a range of performance; should be independent of each other; and should be leading (that is, the indicators should reveal declining trends in operational safety performance before the decline is manifest in a serious operational event).
The algorithm will utilize the best attributes of the Arthur Andersen trend plots and the algorithm used by the Idaho National Engineering and Environmental Laboratory in the current AEOD SMM screening process. The algorithm will be judged by whether correlations exist between the algorithm over a time frame preceding past SMMs with plants designated for discussion at those SMMs.
The new algorithm will correct known weaknesses in the Arthur Andersen trend plots. Provisions will be made to accurately reflect performance of plants in extended outages. Plants will be compared against similar plants rather than the industry average. The staff also intends to examine the use of an absolute standard rather than a sliding scale based on current industry average. Finally, the staff will examine the algorithm to evaluate the extent to which it gives consistent results for units of multiple plant sites.
Trend plot input to the April 1997 SMM screening meetings will be developed based on the existing Arthur Andersen algorithm. These will be provided for the purpose of enhancing the information base of the meeting, but will not be part of the formal decision process at the June 1997 SMM. Trend plots based on the new algorithm will be provided for the June 1998 SMM. For subsequent meetings, the algorithm will be revised, as appropriate, to include risk-based indicators.
Recommendation 3:
Recommend shift from subjective to objective factors.
Action:
Input to the SMM already includes a significant amount of objective information (verifiable qualitative or quantitative observations, information, records or statements of fact). This includes the Plant Performance Review (PPR) information, Plant Issues Matrices (PIM) and NRC performance indicators. Actions in response to recommendations 1 and 2 will add significant new objective information in a more structured format.
Recommendation 4:
Recommend restricting format and volume of information provided to senior managers.
Action:
The staff will continue its efforts to improve the style and content of presentation in order to assist senior managers in assimilating the information and focusing on performance issues.
The PPR summaries and the PIM are used to develop background information for the SMM. Recent revisions to the PPR Guidance have enhanced the clarity and usefulness of PPR information. In addition, utilization of the PIM, which is a synopsis of significant objective safety performance information helps to assist the senior managers in focusing on performance issues.
As the validated trend plots and other improvements mentioned elsewhere in this paper become available, their use will further enhance the availability of summarized and scrutable information for decision-making.
The plant performance template that has been included in Management Directive 8.14 provides a useful framework for summarizing performance information. The template will be revised as necessary to include insights from the Arthur Andersen report and additional efforts will be made to reduce the volume of information provided.
Recommendation 5:
Recommend continued effort to improve information access through automation.
Action:
NRC will continue efforts to improve information access through automation. NRR has several initiatives underway to develop, evaluate, and implement information management systems that are intended to improve inspection report preparation, documentation, and agency-wide utilization of data. Considerable additional work, including coordination with the Chief Information Officer (CIO) organization and training of field personnel, remains to be completed before the implementation can begin.
Recommendation 6:
Recommend using new economic indicators outside the SMM process.
Action:
The staff recognizes that economic stress can be used as an early warning of potential operational stress. The staff is also mindful of the impact deregulation and utility reorganization may have on licensee performance. Consequently, the staff has initiated an effort to define and develop indicators of economic stress at both the plant and corporate level. In the future, the staff will examine indicators of economic stress and monitor plants to determine what affect, if any, economic stress has on the individual facility.
The staff agrees that economic indicators should not be used as part of the basis for formal action within the SMM decision process, because economic stress does not necessarily predict changes to operating performance. Consequently, while the economic indicators will be made available for the SMM, they will not be factored into the plant performance template.
The staff will provide input to the April 1997 SMM screening meeting using the economic indicators suggested in the Arthur Andersen report. These will be provided for the purpose of enhancing the information base of the meeting, but will not be part of the formal decision process for the June 1997 SMM. For the January 1998 SMM, the staff will provide an evaluated set of economic indicators. Management will assess the most effective use of economic indicators at that time.
SMM Process:
Some improvements to the SMM process were accomplished during the January 1997 SMM. These included:
Efforts to improve the SMM process will be continuous. Attachment 1 is the March 11, 1997, memorandum from the Director of NRR to the principal SMM participants, outlining some improvements to the SMM screening meeting process for the June 1997 SMM. The staff will address the Arthur Andersen recommendations as stated below.
Recommendation 7:
Recommend attaining better balance in participants' roles in the decision process and consideration of consensus decision-making techniques.
Action:
Efforts undertaken in the January 1997 SMM to enhance the participation and integration of the views of each senior manager will be continued. The senior managers are expected to be active participants in the discussion of facilities as well as to present and explain the information developed by their office for the SMM.
Further improvements will be implemented in the June 1997 SMM cycle. As outlined in Attachment 1, the manner in which AEOD and OE will discuss their information at the screening meeting will be revised. Following the Regional Administrator's summary of plant performance, the Director of NRR will solicit from key participants recommendations whether to discuss a plant at the SMM and the basis for the recommendation.
Consensus has always been a goal of the SMM process, although this has not always been achieved. Efforts to facilitate consensus among the senior managers were initiated at the January 1997 SMM through the use of "Pro and Con" charts and was evidenced by the enhanced participation by the attendees. Further effort to clearly define the role and increase the active participation of all SMM participants will improve the chances of reaching consensus.
The staff has concluded that a facilitator is not needed. The EDO will continue to preside over future SMMs. The Director of NRR will facilitate discussion and be responsible for assuring that the relevant views of all participants are elicited and considered in the decision process.
Recommendation 8:
Recommend presenting information in a rigorous and structured manner.
Action:
The recently issued Management Directive 8.14, "Senior Management Meeting," includes guidance for presenting information for each plant discussed. As mentioned previously, some improvements were implemented for the January 1997 SMM. In response to recent SRMs, the staff is considering revisions to these documents directed towards designing the SMM process, including the "screening meetings," to be as objective as possible and providing a visible connection between facility performance and the ensuing decisions. In addition, the actions described in response to other recommendations (developing an improved trend plot algorithm, management and operational performance effectiveness measures, and a plant performance template) will further advance the rigor and structure of the SMM information presentation.
Recommendation 9:
Recommend developing a better process for compiling the public record of SMMs.
Action:
In response to recommendations in the Arthur Andersen report, the minutes of the January 1997 SMM were enhanced to better reflect the basis for decisions. Further efforts will be made to improve the minutes of the SMMs to ensure that key decision points are captured and reflected in the public record. The staff has concluded that with these improvements, it will not be necessary to transcribe the meetings.
RESPONSE TO THE MARCH 14, 1997 SRM
The staff's plans for addressing the March 14, 1997, SRM is as follows:
Item 1:
The staff should specifically consider the extent to which objective performance information can be used in the decision process, with new indicators being phased in as appropriate
Action:
As discussed above in response to Arthur Andersen recommendation 1, the staff has initiated development of a process to assess leading indicators, such as organizational and operational effectiveness, on an ongoing basis.
In response to Arthur Andersen recommendation 2, the staff has initiated a program to develop a validated algorithm for use in identifying candidate plants for discussion at SMMs. The new algorithm will correct known weaknesses in the Arthur Andersen trend plots. Provisions will be made to accurately reflect performance of plants in extended outages. Plants will be compared against similar plants rather than the industry average. The staff also intends to examine the use of an absolute standard rather than a sliding scale based on current industry average. Finally, the staff will examine the algorithm to evaluate the extent to which it gives consistent results for units of multiple plant sites.
To the extent possible, the indicators chosen for use in this process should be related to nuclear safety and regulatory performance; should be based on information readily available to NRC; should not be subject to manipulation; should be comparable among licensees; should reflect a range of performance; should be independent of each other; and should be leading. These include the AEOD Performance Indicators. As new information, such as risk-based performance indicators, become available subsequent to the June 1998 SMM they will be phased into the algorithm.
Initial input to the SMM decision process is expected to begin with the June 1998 SMM. However, some information from a pilot study will be made available to the January 1998 SMM. For the April 1997 SMM screening meetings, trend plots will be developed using the existing Arthur Andersen algorithm, but will not be part of the formal decision process at the June 1997 SMM.
Item 2:
The staff should consider the use of a "plant performance" template to help display the objective performance information.
Action:
In Management Directive 8.14, the staff proposed a template which detailed a consistent set of categories for assessing plant performance. The staff intends to continue implementation of this template while the other evaluation methods described in this paper are developed and implemented. The staff is developing guidance on how the "pro/con" charts should be developed using the plant performance evaluation template.
The staff action plan includes development of a plant performance template that includes all aspects of plant operation as a method of structuring and presenting plant performance information. The Arthur Andersen Integrated Performance Model will provide much of the basis for the revised template.
Item 3:
The staff should clearly explain and describe the decision-making process, especially with regard to one which starts with a "rebuttable presumption."
Action:
The staff will delineate a decision-making process using objective data to arrive at the SMM decisions. The "rebuttable presumption" as proposed to be applied in the SMM process will include development of criteria for action and a decision process to support or rebut the position suggested by the criteria.
Item 4:
The staff should design the "screening meetings" to be as objective as possible in selecting facilities to be discussed in the SMM.
Action:
The staff is committed to assuring that the selection process for SMM discussion plants is as objective as possible. The staff actions in response to Arthur Andersen recommendations 1 and 2 will provide a more objective information base for the screening meeting decisions. Interim measures to improve objectivity are outlined in Attachment 1. They include greater participation from AEOD, OE, OI, and others in the screening meetings, greater use of objective data including the Arthur Andersen trend plots, and a commitment to seek wider participation in the screening meeting decisions.
Item 5:
Regarding the SMM itself, a connection must be shown between facility performance information and the ensuing decisions, such as the use of the "plant performance" template referred to above.
Action:
At the January 1997 SMM, the senior managers began the use of "pro/con" charts that summarize facility performance information in terms of reasons for and against taking action, such as issuing trending letters or placing plants in Category 2 or 3. The staff intends to continue this practice, and is developing clearer guidance on how the charts should be developed using the SMM plant performance evaluation template that is part of Management Directive 8.14.
As a result of efforts described in this paper, the SMM decision process will be enhanced through the use of objective information and criteria and reduced emphasis on subjective factors. Moreover, the minutes of future meetings will contain a more complete and scrutable record of the basis for decisions.
Item 6:
Consistency should be sought between the SMM decisions and decisions which are reached in other evaluative processes.
Action:
The staff believes that innovations implemented in recent SMM cycles have led to greater consistency in the information base available to the SMM and other NRC evaluative processes. Improvements expected from the development efforts in progress will further enhance this consistency. The improvements in the decision process, such as greater participation on the part of senior managers, increased emphasis on consensus decision-making, the use of objective criteria, and reduced emphasis on subjective factors, will also foster consistency.
However, the staff believes that complete consistency cannot, and should not, be the goal of the SMM. The purpose of the SMM is to bring together the varied knowledge and perspectives of all NRC offices. It is anticipated that there will be instances in which a consensus view of this group will differ from the results of a SALP board, which consists primarily of regional and NRR participants. While we should strive to promote consistency through the use of consistent information, criteria and processes, we must be open to the possibility that fresh insights and perspectives will occasionally lead to differing conclusions.
SUMMARY:
The recommendations provided by Arthur Andersen address two areas, the SMM information base and the SMM process.
Efforts are underway to develop new objective measurements and methodologies and to improve the SMM information base. Development of these products will involve considerable staff and contractor effort. The effort requires significant time to produce a quality product and subject it to peer review by the ACRS and public comment. Input to the SMM process from the new performance trend plots and methodologies to measure management and operational effectiveness will begin with the June 1998 SMM.
Some improvements to the SMM process were accomplished during the January 1997 SMM. Efforts to improve the SMM process will be continuous and will address the Arthur Andersen recommendations as stated above.
Some actions can and will be taken to address these concerns by the next SMM. Specific improvements planned for the April 1997 SMM screening meetings include:
The Director of AEOD is assigned the responsibility to periodically monitor and report to the Commission the staff's overall progress in addressing these recommendations.
| L. Joseph Callan Executive Director for Operations |
| CONTACT: | Alan L. Madison, AEOD (301) 415-6412 |
| Attachments: | 1. | Memo to Multiple Addressees from S. Collins, dated 03/11/97, June 1997 Senior Management Meeting Screening Meeting Schedules |
| 2. | Plant Performance Effectiveness Review Process Chart |
1. There are three categories of Watch List plants: Category 3 are shutdown plants requiring Commission authorization to startup and that the NRC will monitor closely; Category 2 plants are those authorized to operate that the NRC will monitor closely; and Category 1 plants are those removed from the Watch List.
2. U.S. Nuclear Regulatory Commission, "Guidance for Senior Management Meeting and Plant Evaluation Processes," Commission Paper SECY-96-093, May 1, 1996.