1 UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION *** BRIEFING ON ANALYSIS OF QUANTIFYING PLANT WATCH LIST INDICATORS *** PUBLIC MEETING *** Nuclear Regulatory Commission Commission Hearing Room 11555 Rockville Pike Rockville, Maryland Tuesday, February 18, 1997 The Commission met in open session, pursuant to notice, at 2:39 p.m., the Honorable SHIRLEY A. JACKSON, Chairman of the Commission, presiding. COMMISSIONERS PRESENT: SHIRLEY A. JACKSON, Chairman of the Commission KENNETH C. ROGERS, Member of the Commission GRETA J. DICUS, Member of the Commission EDWARD McGAFFIGAN, JR., Member of the Commission NILS J. DIAZ, Member of the Commission . 2 STAFF AND PRESENTERS SEATED AT THE COMMISSION TABLE: JOHN C. HOYLE, Secretary of the Commission KAREN D. CYR, General Counsel EDWARD JORDAN, Deputy Executive Director for Regulatory Effectiveness, Program Oversight, Investigations, & Enforcement DR. DENWOOD ROSS, Acting Director, AEOD RICHARD BARRETT, Deputy Director, Incident Response Division, AEOD IRA GOLDSTEIN, Arthur Andersen, Partner, Federal Industry KAREN VALENTINE, Arthur Anderson, Senior Manager, Office of Government Services LOUIS ALLENBACH, Senior Management Consultant KATHRYN KELLY, Senior Consultant AARON LIEBERMAN, Senior Consultant . 3 P R O C E E D I N G S [2:39 p.m.] CHAIRMAN JACKSON: Good afternoon, ladies and gentlemen. I am pleased to welcome members of the NRC Staff to brief the Commission on the Arthur Andersen Assessment of the Senior Management Meeting Process and Information Base. The assessment was performed to ascertain how the Senior Managers can improve the timeliness and thoroughness of its plant safety assessments. The Senior Management meeting process is intended to facilitate the early identification of plants which require increased regulatory attention The Commission has indicated previously its belief that there is room for improvement in the Senior Management Meeting decisionmaking process. These improvements relate to making the process more scrutable, using objective data with well-defined decisions criteria. The objective ultimately should be to attain a clear, coherent picture of performance at operating reactor facilities. I understand that copies of the slide presentation are available at the entrances to the meeting room, so unless my fellow Commissioners have any opening comments, Mr. Jordan, please proceed. . 4 MR. JORDAN: We changed on you from the last meeting. CHAIRMAN JACKSON: That's right. MR. JORDAN: Thank you very much, Chairman Jackson, Commissioners. With me at the table are Dr. Denny Ross, Acting Director of AEOD and Rich Barrett, Deputy Director, Division of Incident Response, who provided direct management oversight of this effort. Seated behind us are some of the Arthur Andersen personnel who conducted the assessment. Ira Goldstein is the partner in charge of Arthur Andersen's Federal Industry -- CHAIRMAN JACKSON: Raise your hand high. Thank you. MR. JORDAN: Thank you -- Federal industry work. Karen Valentine is the Senior Manager of the Office of Government Services. Louis Allenbach is the Senior Management Consultant. Kathryn Kelly is a Senior Consultant. Aaron Lieberman is a Senior Consultant. They are available to respond to specific questions about their work that NRC Staff are unable to answer. . 5 The Arthur Andersen study of the Senior Management Meeting grew out of a discussion at the June 25th, 1996 periodic Commission meeting on operating reactors in fuel cycle facilities. At that meeting the Commission raised a number of questions about improving the information base of the Senior Management meeting in order to make the Senior Management Meeting decisions more objective, consistent and timely. Following the issuance of an SRM on June 28th, 1996, the responsibility for this assessment was assigned to AEOD by the Executive Director for Operations. The AEOD staff decided to conduct an independent assessment of the Senior Management Meeting process using a contractor with extensive experience in management consulting and performance indicators. Arthur Andersen Consulting was selected for this responsibility, using a streamlined process to select from a list of GSA approved contractors. For the four-month period of the study the AEOD staff provided Arthur Andersen with the information and access they needed in order to provide a creditable assessment. The NRC Senior Advisory Panel was created to review and comment on the NRC Staff proposed statement of work and to provide input at key milestones in the study. The Advisory Panel consisted of myself, Jim Milholland, Dave . 6 Morrison, Stu Ebneter, and Frank Miraglia. The report you have received and will be briefed on today represents the views of Arthur Andersen Consulting. The NRC Staff has begun an aggressive effort to evaluate the recommendations and develop implementation options. The NRC Staff recommendations will be presented in a Commission paper which we plan to forward in the end of March, this year. The briefing this afternoon is intended to review the findings and recommendations of the Arthur Andersen report without providing NRC Staff views, and that is normally difficult but Rich, I will ask you to begin the presentation, please. CHAIRMAN JACKSON: Commissioner? COMMISSIONER McGAFFIGAN: Just one comment, because we won't come back to this. I would like to commend you for the process that you went through, the streamlined procurement process, I think to get this study in this timeframe. I think that is, whether it was AEOD or working I'm sure with Procurement shop, the strategy of going to the GSA approved list, getting a contract with the appropriate qualifications and getting them on board rapidly, that's very refreshing because it often times takes a lot longer to get this sort of study. . 7 MR. JORDAN: I intended to give Admin the credit for assisting us in that effort. Thank you. MR. BARRETT: Good afternoon, Chairman Jackson, Commissioners. If I could have Slide 2, please. Our intention this afternoon is to simply go through the content of the Arthur Andersen report including the methodology that they used in preparing the report and also to present their findings about the outcomes of past Senior Management Meetings as well as their findings and recommendations regarding the information that we have used in the past and the information we might use in the future for Senior Management Meeting decisions, and the process we use for making these decisions. As Mr. Jordan pointed out, we will briefly at the end talk about the schedule for the Staff's evaluation of the recommendations and for development of options for implementation. Slide 3, please. I think Mr. Jordan has already pretty well gone over the chronology of the study. I would like to point out however one thing I think is of interest. The original Staff requirements memorandum concentrated on the development of indicators that could form a more objective basis for Senior Management Meeting . 8 decisions. After Arthur Andersen came on board and began to review the written documentation from the Senior Management Meeting they made the recommendation to us that we expand the study so that we also look at the process itself because their feeling was that a great deal of what was happening in the Senior Management Meeting was related to the process we used and that to have a full examination of a step toward a more objective measures required us to look at the process. The Staff evaluated that recommendation and concurred with it, so the contract was modified at that point and we went forward with the fuller scope of work. If I could have Slide 4, please. Arthur Andersen assigned nine professional to this task. As Mr. Jordan mentioned, it was led by a partner of the firm as well as two senior managers of Arthur Andersen. In addition, they involved part-time two of their senior staff with extensive experience in utility finances as well as nuclear operations, some experience in nuclear operations, and four very capable staff members who worked primarily almost full-time throughout the course of the study. The methodology they used was quite thorough in my opinion. They first of all did a very thorough review of the written record of the senior management meeting from 1992 to 1996. . 9 That included the briefing books, which are supplied to the Senior Managers prior to the meeting, the Minutes that are published after the meeting, and the transcripts of the Commission briefings that are given after each one of the meetings. Based on their review of the written record they developed an extensive database. This was a database of the characteristics and measures that were most often cited as being the basis for the decisions, the performance characteristics and performance measures. In fact, they ended up with a database of 1700 records, which provided a great deal of insight into the bases that we have used in the past for these decisions. Secondly, Arthur Andersen conducted over 30 interviews of three types -- interviews with NRC Senior Managers who have past experience with the Senior Management Meeting, both Headquarters Managers and Regional Representatives from all of the regions; we interviewed Resident Inspectors and their immediate supervisors in the Regional office; and we interviewed five senior utility executives at the Vice President, Nuclear level. Now the purpose of these interviews was different in each case. In the case of the interviews with the NRC Senior Managers, what we were trying to get there was an understanding of how the Senior Management Meeting process . 10 works, because it was very important for Arthur Andersen to understand that and of course they had no opportunity to attend the meeting. Also, to understand in the opinion of the people who participated what were the most important factors in shaping the decisions that have been made in past Senior Management Meetings and also to understand what the roles of the various participants in the meetings are and finally to see if these Senior Managers had any suggestions for process improvements or if they felt that there were any plants that if they had a chance to go back and look again they might have treated differently -- so that was the purpose of interviewing the NRC Senior Managers. The purpose of interviewing the Resident Inspectors and their immediate supervisors was for Arthur Andersen to get a sense of how information that is fundamental to the Senior Management Meeting performance assessment process, how it is first gathered and how it is developed an analyzed as it moves up through the chain of events and then becomes part of our assessment, performance assessment processes, such as the SALP and the Senior Management Meeting. Finally, the purpose of interviewing the utility executives was to get a sense of how much they use performance indicators in evaluating their own plants and . 11 how they make use of performance indicators. Also, we wanted to get a sense of what their understanding of the Senior Management Meeting process was from an outsider's perspective, so we conducted over 30 interviews. Third, the third aspect of their methodology was to create what they call a performance trend model, and we will actually show you an example of the performance trend model later in this presentation and we'll discuss it in great detail, but the purpose of the performance trend model was to demonstrate how indicators could be used in making decisions related to the Senior Management Meeting, indicators that are already available to the NRC Staff and are already developed in the processes that we have ongoing -- and how criteria could be used in conjunction with those indicators to inform the process of decisionmaking. COMMISSIONER McGAFFIGAN: Could I ask -- MR. BARRETT: Sure. COMMISSIONER McGAFFIGAN: -- maybe it's appropriate to wait until later, but the indicators that are available to the Staff, are they also available to the public? If you went through our documents, could you make one of the charts that you are going to show us later from . 12 the publicly available information? MR. BARRETT: At the moment, seven of the nine indicators we use are routinely made available to the public. These are the NRC performance indicators. The two other indicators, which were related to our enforcement and to numbers of allegations are not, I believe, routinely made available to the public although I don't believe there is any problem with making them available to the public. CHAIRMAN JACKSON: But we do have those indicators ourselves and we make use of them. MR. BARRETT: We do, yes. CHAIRMAN JACKSON: Or we at least trend them at this stage. MR. BARRETT: We trend seven of the nine and the other two I believe are just used internally within the offices that they are developed in. CHAIRMAN JACKSON: I see, but the information base for developing indicators with them exists. MR. BARRETT: It exists, yes. Okay. It was not a great deal of effort for Arthur Andersen to develop these charts with the information. The fourth item they did was to create an . 13 integrated performance model, and again we will look at the integrated performance model, but the purpose of the integrated performance model was to illustrate how different types of information could be used at various stages in the performance assessment process, and again we will discuss that in some detail. Finally, Arthur Andersen developed a process map, and part of that was developed -- this is a process that describes how the NRC gathers information of various types, how we analyze it, and use it in various processes such as enforcement, the SALP process, and other processes leading up to the Senior Management Meeting. We don't plan to go into detail today on that process map, but it is available in the report in Appendices 3 and 4. If I could have Slide 5, please. Arthur Andersen drew some conclusions about the past record of the Senior Management Meeting with regard to identifying poorly performing plants and with regard to taking formal action. CHAIRMAN JACKSON: Let me ask this question. Did the use of the Arthur Andersen performance trend charts identify any plants with poor performance which had not been identified for discussion or vice versa? MR. BARRETT: If you looked at the Arthur Andersen . 14 performance trend plots and the criteria that they developed as a straw man criteria, there would be plants that would come up that were not on the list and were not discussed. CHAIRMAN JACKSON: Would those have been ones that upon discussion with Senior Managers or utility execs might be agreed that should be on the list but were not, or -- or was there any agreement that any that had previously been placed on the list should not have been? MR. BARRETT: There were no cases where plants had been placed on the list where there was agreement among anyone interviewed that it should not have been placed on the list. There were cases of plants, there were two cases of plants that have been on the list where you could not have identified those performance problems purely on the basis of indicators. You would have to have looked at other information to identify those as problem plants. With regard to whether there were plants that should have been on the list according to the charts that were not on the list in the past. Yes, there were. There were some that based on these, on this particular chart with these criteria, would have been identified. I would say that, and Arthur Andersen would say this, that these particular indicators and these particular criteria are not meant to be the set of indicators and . 15 criteria and their recommendation to the NRC is that they use the insights from the study to go in and do a systematic look at indicators and criteria to come up with the ones that we feel are the true indicators of performance. CHAIRMAN JACKSON: So when you come back with the paper in March, you intend to have identified what those indicators really should be? MR. JORDAN: Yes, Step 1. We will never have the final answer but we will have an improved list -- CHAIRMAN JACKSON: With improved criteria or refined criteria? MR. JORDAN: Yes. CHAIRMAN JACKSON: Dr. Ross? DR. ROSS: I was going to say we were cautioned, and this is in the report, in Appendix 2, page 2 -- in fact, they felt strongly enough about it that they put it in italics. They said that "The stress of our recommendations" -- of course, meaning the Arthur Andersen recommendations -- "lies in the methodology, not in the numbers reported in the methodology. The NRC should first conduct a review of the selected performance indicators to be used when analyzing performance trends and then turn its attention to formalizing a methodology such as the one proposed to categorize plants." And I think that is what we need to do. . 16 CHAIRMAN JACKSON: Okay. Commissioner McGaffigan, did you have a comment? COMMISSIONER McGAFFIGAN: I'll come back. CHAIRMAN JACKSON: Okay. MR. BARRETT: I will come back to that question, your question, in a moment. First of all, with regard to the outcomes, in general the Arthur Andersen concluded that for plants that had performance problems the NRC has identified them for discussion and that that was a fairly favorable result. In addition, they concluded that plants that had been put on the Watch List in the past had been placed there appropriately, that the NRC has not been in the habit of over-reacting in terms of putting plants on the Watch List. Arthur Andersen also concluded, however, that the NRC, the Senior Management Meeting has sometimes been slow in taking formal actions in terms of trending letters or Watch List designation and that NRC outcomes, Senior Management Meeting outcomes, appear to be inconsistent. That is to say that plants with apparently similar performance have had experienced different outcomes. Now if I could get back perhaps to a more full discussion in answer to your question, the basis for the Arthur Andersen's conclusions was really the entire scope of the information they looked at. . 17 We have no base truth here. We really don't have anything with any fundamental principle we can go back to and say based on this fundamental principle we now know which plants should have been on the list or should not have been on the list, so we had to use the preponderance of information that was available, and the information that was available, first of all, was their review of the written record. When Arthur Andersen reviewed the written record of the Senior Management Meeting, their staff formed certain impressions about the severity and the duration of performance problems and based on that they came to preliminary views about which plants seemed to deserve to be put on the Watch List or deserved to get trending letters. The second source of information that was used were the interviews. In the interviews with our own Senior Managers, many of them expressed in hindsight the views that certain plants probably should have been treated differently, so that was a second source of information. Finally, the performance trend model was developed and was run for 109 plants and there were many cases where the results of the model did not comport with the results of the Senior Management Meeting. The conclusions that Arthur Andersen drew are . 18 based on a confluence of those three sources of information where consistency could be seen in all three sources of information. CHAIRMAN JACKSON: Since we are talking about the Arthur Andersen assessment, and since we have the benefit of having this team sit here, I am going to ask whoever is the senior-most person on that team to offer to give us any further illumination you might wish to provide. MR. GOLDSTEIN: Madam Chairman, I'm Ira Goldstein. I am the partner responsible for this engagement and, indeed, for our government work. I think the summary that the staff has given to you is an accurate reflection of the work that we did and of the conclusions that we drew. I would focus for one moment on the general perception, as I think Mr. Barrett said, that the correct set of indicators had been looked at, that a great deal of information had always been collected and that if there was one indication of change that we concluded should be focused on, it was the extent to which discussions occurred that led to watch list placements somewhat later than our model would indicate could have been the case. The other conclusion, if you will, that I would focus on is the balance between the objective indicators and subjective judgments. Our belief, as I think the staff and the staff of the NRC has always believed that ultimately . 19 judgment must prevail. What we found was, as we looked at the outcomes of the senior management meetings, that some of the performance indicators could be used in a more objective way as indicators that could lead to what we would like to call a presumptive judgment and that is that the model can give you some indication that there might be a presumption that a particular plant could be appropriate for the watch list, subject to rebuttal in a discussion. Our recommendation secondly focused on that. Thirdly, we also provided some recommendations relating to the breadth and depth of the discussion in the meeting and that we felt and I think the staff has expressed sympathy with this perception that expanded participation and expanded independent debate within that meeting could lead to a fuller discussion of those indicators. So with those three points of focus, I certainly believe that the recounting that you hear is an accurate reflection of what we reported. CHAIRMAN JACKSON: Thank you. MR. BARRETT: If I could have slide six -- COMMISSIONER ROGERS: Before you leave this slide, I wonder if you could just clarify what you mean by "most"? I see the word "most" appearing here a couple of times and I just want to get a little feeling, particularly about the second bullet. Most NRC senior manager utility executives . 20 agreed that plants on the watch list were appropriately placed. How large was the disagreement there? MR. BARRETT: I wouldn't say there was disagreement. There was really a question of those people who addressed the question and those people who did not. I don't recall and perhaps Arthur Andersen recalls, but I don't recall anyone saying that, disagreeing with that statement. It was just a question of which people addressed it and which people did not. COMMISSIONER ROGERS: All right. If that's what you found. Is that what you did find in your interviews, folks from Andersen? MR. GOLDSTEIN: Yes. COMMISSIONER ROGERS: All right. MR. BARRETT: If I could have slide six now? The next three slides relate to findings and recommendations of the Arthur Andersen study with regard to the information base, the information we use for the current senior management meeting decisions. And Arthur Andersen made some favorable conclusions about our information base which I think are very heartening. First of all, one of their first impressions was that the NRC has a wealth of information available to us, a wealth of information that is directly applicable to the . 21 assessment of performance and directly applicable to safety and they don't always find that when they go out to assess organizations. So there were no significant gaps that we need to go out and start new major programs to develop new information. They also concluded that the performance characteristics that have been used in past senior management meeting decisions are indeed related to safety and are related to risk, so again a very positive, positive finding. Arthur Andersen did identify what they considered were conditions, however, related to how information was handled and how information is used. First of all, they concluded that the NRC focuses on events, tends to focus on events or major problems that occur at plants and then, based on those events, take a retrospective look at the plant, looking for the root causes and quite frequently finding the root causes in problems with management effectiveness and operational effectiveness. And what Arthur Andersen basically is recommending is if we continue to focus on events in this way, we are going to be identifying performance problems later than we could. If, on the other hand, we had an ongoing systematic program for assessing management effectiveness and operations effectiveness, that we would have a program that . 22 identified performance problems earlier and would give licensees more of an opportunity to turn these problems around before they become significant to safety. CHAIRMAN JACKSON: That actually raises an interesting question in my mind. The question is, is that to say that the NRC does not assess management and operational effectiveness on an ongoing basis or that that assessment occurs but in the senior management meeting decisions it is not focused on? And those are separate issues. So I don't know if you want to speak to it or the Arthur Andersen rep wants to speak to it or both. MR. BARRETT: I think Arthur Andersen would say that the management and operations effectiveness are clearly focused on in most of the major programs, especially the inspection program at the NRC. For instance, operations effectiveness is a key focus of the SALP process. What they are saying, basically, is that we need to have a more systematic and structured way of developing management effectiveness and operations effectiveness information in a way that better feeds the senior management meeting process. So it's a question of how information is handled and how it's used. DR. ROSS: The retrospective might be the key word in terms of what are leading versus lag, and more focus on the second bullet might produce leading indicators, which is . 23 the main lesson to extract from this. CHAIRMAN JACKSON: Okay, but there is a separate question that underlies this and that is the question of is there anywhere in our plant assessment processes that we focus on management and operational effectiveness as leading indicators? That's the first question, that's part A. And part B is, if the answer is, yes, are we saying that it is not used as such in the senior management meeting process? So that's question one. Or is it that we don't assess it? I mean, those are two separate questions. You see, we do SALP, we do plant performance reviews, we do this, we do that. And the question is, do we focus on management and operational effectiveness at those levels but on an ongoing basis but it doesn't propagate to the senior management meeting. Or are we saying that we don't, anywhere in our program, focus on an ongoing basis on assessing management and operational effectiveness and those are two separate kinds of things. MR. JORDAN: Right. I think I can try to answer that. Certainly the discussions in the senior management meeting talk both about management and the SALP process provides data input evaluations on operational effectiveness. So they are both present. . 24 In terms of having the data assimilated in a way that is more easily used by the senior managers, I believe that's the focus. So there are assessments but the structure and collection of the information is not conducive to use and we do, in fact, extract much of our information about management effectiveness from things that happen as opposed to a more I would say overview of capabilities. And that is sort of historic. In the past, when we try to look at capabilities, the industry itself was critical of the NRC going in as a paragon of management skills and knowledge and not looking at performance because it really is an idea of management performance. So the staff has been cautious, I believe, in assessing management in terms of their capabilities as opposed to their performance. CHAIRMAN JACKSON: Mr. Goldstein looks as though he has an itch. Mr. Goldstein, I think when you sit down, we would like you to sit in a green chair after this. MR. GOLDSTEIN: A green chair? CHAIRMAN JACKSON: here at the table. MR. GOLDSTEIN: At the table, okay. My wife points out every morning when I pick my ties that I am close to color blind so that as we wave over the chair -- . 25 CHAIRMAN JACKSON: Well, it turns out that your tie matches the chair. [Laughter.] MR. GOLDSTEIN: I will mention that when I get home. CHAIRMAN JACKSON: She set you up. MR. GOLDSTEIN: She has done that before, Madam Chairman. Let me reinforce something that Mr. Jordan said and maybe even extend it a little further. We have put into the report what we call an integrated performance model that speaks very directly to the issue you are raising and indeed, as Mr. Jordan said, management effectiveness is discussed and is looked at and there is a great deal of discussion in the record of management effectiveness. But as one looks at risk and resource allocation, the closer you get to an actual performance failure, the more difficult it is to do something constructive and the more the risk goes up and the more resources it takes to fix the problem. We like to view the levels of indicators as four groupings. Furthest from the event is economic stress. If you could see that, that would give you some more distant indication. Management effectiveness, perhaps next. Operational effectiveness, getting closer. And then . 26 ultimately, performance results. I would respond to your question by saying the discussions of management effectiveness appear to be triggered in the senior management meeting by results events, by performance events as opposed to being a leading edge of that type of performance. CHAIRMAN JACKSON: Okay, thank you. Commissioner Diaz? COMMISSIONER DIAZ: Would it be fair to say that as important as propagating an assessment of operating effectiveness would be to negatively bias events that might not have significance as part of this operational effectiveness rather than propagating the event, the event in a continuously amplified basis. Would you think it's -- would you like me to restate that? MR. GOLDSTEIN: I would appreciate it. I am having trouble understanding the relationship you're drawing. COMMISSIONER DIAZ: Okay, there are two issues in here. One is we take an event and that event might tend to dominate the process and then the actual assessment of operational effectiveness might not propagate and be properly amplified through the system to give it its importance. My point is that, as we look at the indicators, it . 27 might be as important to negatively bias an event, okay, that might not have full safety significance and comparable impact on operational effectiveness and it is to amplify properly those components that do have operational significance on safety. MR. GOLDSTEIN: I agree. It could be. But as Mr. Jordan pointed out, it is much more difficult to assess in any objective way management effectiveness and so I would be cautious about using it as an amplifier or as a reduction because it's a much softer indicator. COMMISSIONER DIAZ: But I didn't say management. I took the word "management" out. I said operational effectiveness and event-related response. MR. GOLDSTEIN: Easier to deal with, no question. COMMISSIONER DIAZ: Thank you. CHAIRMAN JACKSON: Commissioner McGaffigan? COMMISSIONER McGAFFIGAN: Madam Chairman, this goes back to a question you asked earlier about were there plants that the performance trend model would indicate should have been discussed and weren't discussed and there clearly were some. If you also apply the decision criteria suggested, there are plants that should have been on lists and weren't on lists. The thing that seems to, you know, bearing in mind that italicized wording in Appendix 2, but the difference . 28 between some of those plants and I've asked staff about one, they said, oh, yeah, that's one of our lower quartile plants, they limp along but they didn't have an event. And so they can look quite bad on the Arthur Andersen performance indicators over a very extended period of time, one case a decade, but not be on the list because they don't have an event. They are adequate. They are getting SALP 3's and occasional 2's but they aren't trending downward. And that's -- one of the insights you get from the Arthur Andersen report, I think, is the relative importance of events in sort of focusing us and I don't know. I mean, at a previous meeting, Commissioner Rogers, we talked, you know, about adequate -- we were getting the SALP, a 3 trending downward or trending upward, what is a 3, a three is adequate. But what is a watch plant list? A watch plant, a plant deserving to be on the watch list is a -- I'm not sure we yet have the right criteria for. But that isn't going to be decided today. It's just that we get a lot of insight from looking at the 108 plants, not all of which are in the report, and seeing, you know, comparing those judgments. CHAIRMAN JACKSON: Well, we seem to be able -- that everyone might know that there is a plant that is, as you would say, limping along and it is as if, well, we can't do anything unless it has an event and so we are event . 29 triggered. And then there is the potential that if one is event triggered, if we are event triggered, can overreact to an event at the same time. And so it's an interesting issue, so I am interested to see how you are going to suggest you are going to deal with it. But Mr. Barrett came prepared to give his presentation so let's let him continue. MR. BARRETT: Well, let me move on with some of the other problems that were identified with the information base. Arthur Andersen made the finding that the information for the assessment was inconsistent from plant to plant and from region to region. And what they mean by that is that in the past, in the written record, information that appeared to be important for one plant was not mentioned for other plants. For instance, SALP. Sometimes SALP was very important in the discussion for one plant, not very important for another plant and, in other cases, the results seemed to be even inconsistent with the SALP. Of course, SALP is a lagging indicator but nevertheless there were examples of that where information seemed to be used in an inconsistent manner. Arthur Andersen recommends reengineering the information, the way in which we deal with information . 30 again. And we will talk about their integrated model in a while. By the way, there has been some discussion recently about improvements that were made in the most recent senior management meeting in the way in which information was organized and presented and so while I was not at the meeting, that sounds like perhaps an improvement in that respect. CHAIRMAN JACKSON: So is the inconsistency the inconsistency in the information or inconsistency in its use and application? MR. BARRETT: It's the information, in this particular case, in what information is brought to the table. CHAIRMAN JACKSON: So that's in its use? MR. BARRETT: It's use, yes. CHAIRMAN JACKSON: All right. MR. BARRETT: If I could have slide seven. I can move through some of these others with regard to information. Arthur Andersen found that the decision process is highly subjective and that there is -- the process minimally values objective indicators. Now, when they refer to subjective information, I think it is important to understand what they mean. Information can be unquantified . 31 or unquantifiable and still be objective. It can still be observable, it can still be inspectable. When they refer to subjective information, they are referring to information that can be viewed quite differently by two observers and the examples that they most frequently cite are the fact that the written record from 1992 to 1996 frequently emphasizes the importance of personnel changes and reorganizations that have been made recently at a plant and improvement plans that have been developed. Arthur Andersen considers these to be subjective information, information that we really can't evaluate a priori and that this information appears to keep -- to carry very high weight in the senior management meeting process. Conversely, with regard to performance indicators which have been available to the NRC for quite some many years, the indications that they have from the interviews are that not very many, in fact very few if any of the senior managers interviewed, identified the performance indicators as primary decision criteria for the senior management meeting decisions. And Arthur Andersen also observed they actually attended all of the January 1997 screening meetings, and their observations were that while the performance indicators were mentioned, they were not focused on. So that the bottom line of all of this is that objective indicators appeared to be minimally valued in past . 32 senior management meeting discussions. CHAIRMAN JACKSON: Excuse me, Mr. Barrett, I want to ask Mr. Goldstein, what do you mean when you say that the personnel changes, reorganization or improvement plans are subjective as opposed to objective? MR. BARRETT: Two observers can watch the change in leadership. One can draw the conclusion that it will focus the organization more directly in the correct direction and another person can determine that it's a step backwards because the new leader does not have experience in nuclear safety. Valuing that change as positive or negative will be a subjective assessment. CHAIRMAN JACKSON: I see. COMMISSIONER ROGERS: Don't you have a little difficulty here when you are talking about assessing management effectiveness and at the same time trying to find objective measures to do that? The kinds of things that we are touching upon here relate to judgment calls about management decisionmaking and therefor potential effectiveness and isn't this really an area where it is very difficult to have it both ways, to get away from subjective measures or subjective judgments and yet judge management effectiveness at all levels? Now, I mean, at a lower level it is easier to do than at the higher level in the organization to judge . 33 management effectiveness and it seems to me that that's a very thorny area to get into. It is one that Mr. Jordan touched on, why we haven't gone further in that direction in the past. And certainly I would like to hear, you know, any thoughts you may have sometime on that issue because it is central to overall safety and yet it is the most difficult one for us to deal with. MR. GOLDSTEIN: I believe that Mr. Jordan will be presenting the integrated performance model that we put in the report as well as the proposal we made for use of harder indicators in the meetings and, if I could, I think it would be more effective for me to wait until after that and then use those to answer your question. COMMISSIONER ROGERS: Fine. Fine, very good. CHAIRMAN JACKSON: Mr. Barrett? MR. BARRETT: Yes. I would like to take a shot at that question, though. There are things that happen every day at nuclear power plants which are objective indications of the effectiveness of the organization and perhaps organizational effectiveness is a better term than management effectiveness. I think part of the challenge as we evaluate options for implementing the Arthur Andersen recommendation will be to find objective ways of -- objective, observable, inspectable findings that indicate how effective the organization is and the management, as . 34 opposed to behaviors which, of course, are -- management behaviors, which are subjective. Let me move on. Another finding of the Arthur Andersen assessment was that the mass of unprioritized information inundates senior managers. Many of the managers we interviewed cited the large volume of information in the briefing books and also many of them talked about the difficulty in assimilating the information as it's presented by the regional administrator. The numerous examples that are put on the table that, after a while, the listener begins to lose context and so that the Arthur Andersen recommendation is that we pay more attention to the formatting of information and the volume of information that is presented to senior managers so that they can get a better context of what it all means. Analyze the information and present it in such a way that conclusions might be more evident. Have a consistent structure and order of presentation of information so that problems can be put in context and plants can be compared with plants previously discussed. And I should point out and Arthur Andersen points this out that there already has been a lot of progress in this area over the past several senior management meetings with some of the information, management strategies such as the plant issues matrix, which is good. . 35 If I could have slide eight, please? This is the final slide on information issues. One of the issues that they noticed was that a great deal of manual effort goes into assimilating performance information here at the NRC. And without going into a lot of detail, their recommendation is that we could have a process that would be much more efficient and have a much better sharing of information if we continue to improve information access through automation. And the agency, as you know, has some efforts in place to improve our availability of information, making sure that information is available in standard formats that is available electronically to everyone who wants to use it for whatever purpose. So this is an area that Arthur Andersen feels would really help us to be more efficient and more effective in our assessments. And, finally -- CHAIRMAN JACKSON: Mr. Gillespie, when we were talking about the reactor oversight program last week, talked about some activities having to do with automating things along the line beginning with inspection and various other inputs. These beginning efforts that you are talking about, is that what you are speaking of? MR. BARRETT: Yes, that would definitely be apropos. CHAIRMAN JACKSON: And then a question I have is . 36 how proprietary is that system to just NRR's use as opposed to in fact being accessible and/or compatible with other systems? MR. BARRETT: I am not in a position to answer that question. I don't know enough about that system. MR. JORDAN: It is an NRC system. It would be available to the regions and other managers. CHAIRMAN JACKSON: Will it be available to other parts of the agency not in NRR? MR. JORDAN: Yes. CHAIRMAN JACKSON: I mean not in just the reactor part of the business? MR. JORDAN: Yes. CHAIRMAN JACKSON: Okay. MR. BARRETT: The final finding regarding information base has to do with economic stress. I don't think it is any secret to anyone that there is a concern about economic stress due to deregulation of the industry and Arthur Andersen has made the finding that the NRC needs to keep an eye on this kind of stress because economic stress can be a cause of performance problems. On the other hand, they caution us that economic stress cannot -- is not necessarily a predictor of problems. Economic stress can be handled by some organizations, quite nicely, in fact. In some cases, can actually lead to an . 37 improvement in performance. So they are not recommending that we use economic stress in the context of the senior management meeting as an indicator that would be used in the decisionmaking process. They are rather recommending that we have a process and have a system available whereby we can choose economic indicators, track those indicators and use them as a way of nominating plants for perhaps a little extra oversight that we can see, keep an eye on whether economic stress as it is indicated does indeed have an impact on performance as time goes by. CHAIRMAN JACKSON: Well, isn't it also true that excessive expenditures of money can also be an indicator of organizational ineffectiveness. It doesn't necessarily mean -- what you are really saying is that you can't track dollar expenditures to organizational effectiveness. MR. GOLDSTEIN: I agree. I think what we are saying is variations too far away from the norm ought to catch your attention but, as Mr. Barrett said, we would not put them into a model as one of a quantity of more formalized indicators but one ought to go find out why that's happening and keep an eye on it is really what we are trying to say. MR. BARRETT: We are also saying it is not necessarily the absolute value of an indicator. It is the trend over some period of time that may be more important to . 38 look at. COMMISSIONER McGAFFIGAN: If they want to go back to slide 17, they might want to flash up there briefly, that shows what the economic indicators proposed by Arthur Andersen are. CHAIRMAN JACKSON: Why don't we come to that and I will offer them an opportunity to speak to it. But I assume that's why on the next -- on page 9 that economic stress is an ellipse and not a rectangle; is that right? MR. BARRETT: Well, I'll say yes. But I will say there is no plan to go at any point in the presentation to slide 17, so -- CHAIRMAN JACKSON: But I'm saying you do now. MR. BARRETT: Okay. COMMISSIONER ROGERS: I would like to come back to it myself. MR. BARRETT: Okay, fine. COMMISSIONER DICUS: I can save my question until we come back. We have questions. MR. BARRETT: All right. Let's go to -- I think I've lost track of where I am. CHAIRMAN JACKSON: Page 9. MR. BARRETT: Slide 9, yes. Slide 9 is a conceptual representation of the . 39 approach that we have already talked about to a certain extent here. It's an approach for using four levels of information in a coordinated way for assessments and from the right-hand side as I look at it to the left-hand side, you're getting information that has a greater and greater value in terms of getting more and more warning of impending performance problems. On the right-hand side, the bar there is called results and what that really refers to is the occurrence of significant events or other issues that might be viewed as having a direct impact on safety. You can certainly catch performance problems using this type of indicator but this is going to catch performance problems at a point where they are going to have a higher safety implication and it is going to take more resources on the part of the utility to reverse the trend. These kind of indicators typically are the kinds that we have used in terms of significant events or severe accident precursors, SCRAMs, safety system failures. These are occurrences that actually have safety significance. If you are looking for a more timely, ongoing type of assessment, Arthur Andersen would ask you to move to the left one block to operations effectiveness and get an ongoing systematic way of looking at operations effectiveness in a way that can be presented to the senior . 40 management meeting. Operations effectiveness refers to sort of those categories that we use in the SALP process, the operations program, the maintenance program, engineering and the other plant support programs. We already have a large program to inspect in these areas. What Arthur Andersen is proposing is that we need a systematic approach to assessing performance in these areas. If you want a still more timely systematic way of looking at performance that will give you earlier warning, earlier indication, management effectiveness or, as I would prefer to call it, organizational effectiveness should be looked at in a systematic way. These are issues such as the ability of the licensee to do self-assessment, the ability to identify problems and resolve those problems, the ability to coordinate and control work, the quality of procedures and procedural adherence and issues of this type that are sometimes referred to as soft issues. Again, we look at these but quite frequently it is a retrospective look in the wake of an event. Arthur Andersen would like us to look at it in an ongoing way and in a consistent and systematic way. And finally, on the far left, we have economic stress which, as I said before, can cause performance problems and may be an early indicator and certainly should . 41 be watched by the NRC. But, again, as I said, it is not recommended for use in the senior management meeting itself. Why don't we pull up 17. Slide 17, which is a backup slide. While we are waiting for slide 17, I think an important point to make here, and I think this is something that the Arthur Andersen people make quite frequently, is that we shouldn't be looking necessarily for the magic set of indicators. There are any number of good indicators that can tell us about performance degradation. It is important that we look at a spectrum of indicators and understand that we are looking at indicators that are somewhat independent of each other, but there is no magic set that is going to tell you the answer. And the five that they have given us here are five that they are proposing as being ones that certainly have promise but, again, they are recommending that the NRC do a systematic look and see which ones that we are interested in. The first one here is operating costs per kilowatt hour. Apparently, this is a measure that is quite frequently used by utilities for their own internal look at the operating effectiveness of a nuclear unit or any unit for that matter. And it is certainly an indication of the competitiveness of a particular unit in a market, especially . 42 a competitive market where price is important. And if a plant is not competitive, that may well be an indication it will be experiencing economic stress in the future. Debt to equity ratio is more of a measure of the overall health of the company, especially a publicly traded company, obviously. As debt to equity ratio rises, that can be a negative measure on the overall strength of the company and, again, perhaps a leading indicator of stress coming down the road. The next two, operating cost trends and capital spending trends, are much more directly related to the way in which the plant is operated. Capital spending trend, of course, the indication is from past experience that capital spending is one of the first things that's sacrificed when a plant, when a company is undergoing economic stress and, according to Arthur Andersen, this is one that may be a good indicator of more immediate economic stress that a plant is experiencing because of economic stress at a higher level in the corporation. COMMISSIONER McGAFFIGAN: How do you treat steam generator replacements within that because that is sort of a big lump that pops up that isn't necessarily a good indicator other than that they want to continue to operate for a while or whatever. MR. BARRETT: Yes, that's -- clearly, a lot of . 43 these indicators are -- there are many, many things that happen in the life of a nuclear power plant. Just an outage, for instance, which has to be taken into account. And any of the indicators, even in the ones we currently use, and certainly big expenditures like that, we would have to look at these things in a smart way. CHAIRMAN JACKSON: Well, the signal may be in trend since the steam generator replacement is a delta function. MR. BARRETT: It's a delta function and it might actually lead to loss of capital spending elsewhere as they try to squeeze that in or it might not. With regard to operating costs, Arthur Andersen said that we should simply look at the trend in operating costs. Either an increase in operating costs or a decrease in operating costs should be looked at because it may -- we should try to understand the underlying reason for that change. And, finally, one that kind of surprised me but maybe it shouldn't have, is the percent of utility generating capacity from nuclear. According to the Arthur Andersen report that it's the opinion of their experts that they have consulted that stress is greater on a utility that has a high percentage of nuclear units, whether those units are performing poorly or performing well. Nevertheless, . 44 there is more economic stress on a utility that has a high percentage of nuclear units. So that is the rationale in a nutshell as to the five that are proposed here but, as I said before, Arthur Andersen is urging us to take an independent look at all the indicators including economic stress indicators. CHAIRMAN JACKSON: You had a question that you wanted to ask, Commissioner Rogers? COMMISSIONER ROGERS: Just I do think -- I was listening for it and I think I heard it. And that is that really it's changes that you have to be watching that trigger your attention and that if something is changing you better understand why it's changing, could be going up or down and either one could be good or bad, depending upon the reason for that. Operating costs per kilowatt hour, generally speaking, low is good but if you just try and reduce your costs to get that down and you're not looking at the best way to do that but just in a shortcut way, that's bad. So, you know, it seems to me that what you are telling us is watch for changes and try to understand what they are and then use that as a way of screening or calling attention to plants that you might want to look at more closely, but not by themselves are determinant of whether somebody will go on a watch list or not. . 45 CHAIRMAN JACKSON: Commissioner? COMMISSIONER DICUS: I have a question. If we were to incorporate this in some fashion into the overall decisionmaking process, in your view do you think that the NRC staff has the resources to do this? And perhaps even on a couple of these, the expertise to be able to effectively evaluate them? MR. JORDAN: Maybe I could answer by saying that these five that are listed are commercially available. The staff would not have to do any collection of information. They are part of the financial community. In terms of NRR does have persons that are involved in the review of the financial capability of utilities, a limited number. The object here would not be to affect the decision process but to, in engineering terms, if there is stress there may be strain so if the presence of the stress is causing safety strain then there would be communication to the staff to be watchful for safety strain. So it would be a sensitization and so it would be one of the earliest measures that one could become concerned about but not as a basis for decision. CHAIRMAN JACKSON: It wouldn't be a decision trigger but I like the word of sensitization. Please. MR. BARRETT: If I could have slide 10? . 46 Arthur Andersen made some findings and recommendations regarding the process that we use. The first one is a very positive one, namely that they feel that the process is logically sound. They did take a look at our process from front to back, bottom to top, and they feel that we are using -- we have good processes for gathering information. It is a logical progression of analysis and we have the right people involved in the senior management meeting. Among the negative findings Arthur Andersen made are, first of all, they feel that -- they concluded that the senior management meeting process is dominated by the regional administrator and the basis for that, first of all, is that much of the information is developed in the region. Secondly that a lot of this conclusion came from the interviews that were conducted. Clearly NRC managers in general tend to defer to the regional administrator's greater depth of first-hand knowledge about the plants and certainly that -- there is a certain amount of reasonableness to that for sure. They found that in interviews at the meeting, while it involves many people, in the past at least it has tended to be dominated by the regional administrator, the EDO and the director of NRR and, among the three of those, the deference is to the regional administrator. . 47 The regional administrator is the principal presenter at the meeting and the observation of Arthur Andersen regarding their experience with the screening meetings in January 1997, which they attended, was that the regional administrator tended to act as a gatekeeper for other participants and other information. So the process is dominated by the regional administrator and the role of some of the other senior managers is unclear. So the recommendation that they make is that there be a better balance among the participants, that the NRC should strive to elevate the importance of independent sources of information such as AEOD's event information and enforcement information from OE, information about investigations and allegations, that we try to elevate the importance of these independent sources of information and also that we consider a consensus building process, some sort of techniques for consensus building. One of the things that they suggested was the possible use of a facilitator for the meeting. I should note that in the January 29 meeting, there was a fair bit of discussion about more discussion among the various participants, a greater amount of participation in the January 1997 meeting than has been experienced in the past. Slide 11. . 48 One of the most important findings of the Arthur Andersen assessment is that we have no clear criteria for various levels of formal actions and that they view that as a very important thing. We will discuss in a little while the issue of objective criteria. They found that the presentation of information at the meeting is not balanced in structure, again coming back to some of the things we said before. The regional administrator presents his list of problems and at the last senior management meeting apparently also the list of strengths for each plant and the weight of this information dominates all subsequent discussion. The finding is that there is not sufficient weight given to events and other types of information and indicators and they are recommending a more rigorous and structured presentation. That objective information be put on the table first in a scrutable and compelling format and that it be used as a rebuttable presumption. That the objective information presents a case for some action and then the discussion can be either to reinforce that case or to rebut it for the rest of the meeting. COMMISSIONER DIAZ: Who actually makes the rebuttal? Has that been considered? MR. BARRETT: Anyone who is at the meeting who has information that is relevant. . 49 COMMISSIONER DIAZ: You are not planning on separating teams? MR. BARRETT: There was no specific mention of teams, no. CHAIRMAN JACKSON: If in fact you are talking about having the objective information presented in a way that it forms the basis of or generates a rebuttable presumption, aren't you in some sense really getting at the screening meetings themselves? Because how plants come forward or that is a rebuttable presumption that a plant be discussed for inclusion in the watch list has to flow from somewhere, you know, in order for it to get put on the table. And really it is at that screening meeting level that a lot of the -- essentially the bias in the system occurs, whether it is either to put a plant onto the table for discussion coupled with the discussion itself in the meeting but it sounds like what you are saying is the discussion follows what essentially has flowed out of that regional discussion. Or to not put a plant onto the table for discussion. MR. BARRETT: I think you are absolutely right. There was no discussion of that in the Arthur Andersen report but I think you're right. This recommendation does push the process back into the screening meeting. COMMISSIONER McGAFFIGAN: To some degree it is . 50 because if you go to chart 15, which is another one of the backup slides you're probably not planning to use, it really goes to the Chairman's question in that the first two bullets are the screening meetings. Select discussion plans using trend charts and decision criteria for input using evaluation sheets and trend charts. Those are the two places where the rebuttable presumption using the decision criteria and the trend charts get put together really by staff long before the meeting. Then you have the discussion. Then they suggest places that they go away from the rebuttable presumptions, the accepted rebuttals, that that also be documented to the Commission. So I think that chart sort of answers the report, has at least some glimpse of that. MR. BARRETT: Yes, it does. You're right, absolutely right. CHAIRMAN JACKSON: It's really like a three-part process. It's what comes up through a prior -- whatever prior process there is, screening. Then there is the actual process in the meeting and then there is the documentation and public presentation of whatever the results are. So there are those three distinct phases and pieces. MR. BARRETT: Arthur Andersen also found that stakeholders do not understand the process and the outcomes of the senior management meeting, that our discussions with . 51 utility executives, there was a fair bit of consensus that they were not clear on what it takes to get on the problem plant list or off the list and they are not clear about what the process is by which we make the decision. Arthur Andersen feels that we must do a better job of communicating to the Commission, to the public and to the industry and they are recommending that we more fully document the public record at the senior management meeting. They are recommending that we consider publishing transcripts of the meeting or at least that we publish a more complete and accurate set of minutes at the meeting, so that there can be a better understanding of what we decided and why. COMMISSIONER DIAZ: You could probably add stakeholders and one commissioner right here. CHAIRMAN JACKSON: You're a stakeholder, commissioner. COMMISSIONER DIAZ: Oh. CHAIRMAN JACKSON: We know the commissioner is a special beast but we are all stakeholders. [Laughter.] MR. BARRETT: If I could go to slide 12? I would like to talk a little bit about the trend plots before we actually put one up there. The Arthur Andersen trend plots basically show how . 52 NRC information can be used, could be used, along with some reasonable criteria to greatly inform the decisions of the senior management meeting. The model tracks the performance of a plant against nine indicators in this particular case, although Arthur Andersen, as Mr. Ross earlier said, wants us to go back and do a systematic look at which indicators we want to use. Takes those nine indicators, including the nine -- including the seven performance indicators of the NRC plus an indicator of civil penalties and an indicator of the number of allegations that a plant has experienced. When a plant exceeds twice the average value for the industry in any given indicator, then that becomes a hit, twice the average for the industry, that's a hit. And if it -- and that only has to exist for one quarter. Hits accumulate. They accumulate for four quarters and there is a four-quarter running sum of hits that a plant carries with it. CHAIRMAN JACKSON: Is each quarter weighted the same? MR. BARRETT: Yes. CHAIRMAN JACKSON: And that particular averaging, was that rooted in anything in particular or was it arbitrary? MR. BARRETT: It was arbitrary. . 53 MR. GOLDSTEIN: The concept of using a rolling average -- CHAIRMAN JACKSON: No, I know that. The issue is how much -- what do you roll over. You know -- MR. GOLDSTEIN: How many quarters? CHAIRMAN JACKSON: Exactly. MR. GOLDSTEIN: No. I think enough so that you can pick up changes and drop them in a timely fashion. You don't want it too long. CHAIRMAN JACKSON: All right. MR. BARRETT: One of the assertions of the Arthur Andersen study is that performance does not change precipitously at the plants. It takes time for a plant's performance to degrade and it takes time for it to recover. CHAIRMAN JACKSON: Right. No, I agree with all that. Part of the reason I bring that up is commissioners have raised the point in the past or questions relative to SALP and how it ties into the senior management meeting process and the SALP covers a certain period of time that is on the order of 18 to 24 months and that is the reason why I asked the particular question about the number of quarters over which you do the rolling average. MR. BARRETT: So at any given point on the graph is the sum of the hits for four quarters and for any four quarters, the maximum number of hits you could have is 36, . 54 four times nine. What I think is important about this particular model is not necessarily the details of it but two things really. First of all, it is predicated on the idea that if a plant is experiencing true performance problems it is going to show up not in one indicator but in a variety of indicators so you should be looking at a number of hits and that you should be looking at it over an extended period of time, not just for one quarter. COMMISSIONER McGAFFIGAN: Could I ask a question about the comment that NEI made after our briefing on the senior management meeting? The heart of their comment was you could be on the watch list today and would have been a top quartile plant a decade ago and sort of built into these performance indicators, and maybe it's fair to ask Arthur Andersen, if the trend overall in performance indicators is an improvement, being twice as bad as the industry average and therefore deserving a hit, it could be quite a bit better today than it was a decade ago. And so if there is continued improvement and I know in recent years there has been a sort of leveling off in the performance indicators but if you have a declining trend then you are potentially holding people to a moving target. Is that a fair criticism of your model or -- MR. GOLDSTEIN: I think as Mr. Barrett explains . 55 further, I believe this is in his explanation, the action that we would propose that you take would be related to not solely whether you have an accumulated number of hits above a certain amount but, more importantly, to the trend over time. That you would -- that a few quarters of growth would lead to a discussion. Reduction over time would lead to a step to take you or a rebuttable presumption that you be taken off the watch list, so that our focus is on the -- is on the trend over time as an indicator of risk, even if your number of hits is higher than the average. You still, perhaps, should be moving down the level of risk that the Commission uses. MR. BARRETT: Let me add a word to that -- MR. JORDAN: I think the answer to your question is, yes. MR. BARRETT: One of the things we might consider is actually fixing the criteria. Rather than comparing to an industry average, compare to some fixed value and it might be the industry average. MR. JORDAN: But we are responding to this model and this model would facilitate a rising standard and compare plants against a rising standard. This is an intriguing model but we are not trapped by it; I think it is a useful concept. COMMISSIONER McGAFFIGAN: But that is a possible . 56 problem with this model? MR. JORDAN: Yes, correct. COMMISSIONER McGAFFIGAN: As long as performance indicators continue to improve in the industry, you would be continuing to -- you would be moving against a moving target. I don't know what numbers in 1987 would get you a hit but it's probably now, it would put you in the lowest quartile. MR. BARRETT: I suspect that still one SCRAM would get you a hit. COMMISSIONER McGAFFIGAN: One SCRAM would get you a hit today whereas in '87 it might -- MR. BARRETT: Yes, because the industry average would be less than half of a SCRAM per quarter. And there are a number of indicators where that would be the case. So it is not a fatal flaw in the model but it is something that you would need to fix if, you know, we went forward. COMMISSIONER ROGERS: It is a question of establishing some calibration for it, which is what you have suggested might be a way to do it, and some absolute number. And the other one is, you know, the obvious problem with it and, you know, it's a bad thing to be below average. I mean, you just can't be below average. . 57 [Laughter.] COMMISSIONER McGAFFIGAN: They didn't say you just had to go below average, you had to be twice as bad as the average. COMMISSIONER ROGERS: I know, but there is always going to be somebody twice as bad. CHAIRMAN JACKSON: Well, we are all scientists and engineers here for the most part and we all know that what you normalize to is always the critical thing. MR. BARRETT: Okay, well, let's move on to the next slide which is an actual -- which is a performance trend plot for an actual plant that was graphed from 1987 to 1996. The curve with the diamonds represents the four- quarter sum of hits for the actual plant. The squares represent the industry average number of hits which ranges from about five to six if you look on the right-hand scale. Just to help you understand this, you can see the peak there of the diamonds is 16 hits in that particular quarter of 1991. And again, the maximum number you could possibly have would be 36 hits. So, for this particular plant, plant A, it ran along at about the industry average or better than the industry average until 1991 when it took a turn for the worse, peaking at 16 hits in the fourth quarter of 1991 and then moving along through 1995 at . 58 roughly that level. On the left-hand margin, you will see the action levels from one to five where a five is equivalent to being a category three plant shutdown requiring Commission action to allow them to restart. Action step four would be a watch list plant. Three would be a trending letter and two would be a discussion plant and one would be a plant that should be removed from the list. The yellow bars represent the actual NRC actions with respect to this plant. It was discussed several times starting in 1991 and was placed on the watch list by action of the senior management meeting in January of 1996. The green -- they turned out blue there, don't they? Well, anyway, they're green when you're up closer. The green bars are the criteria or the actions that would be indicated by the Arthur Andersen criteria. And they would have said that this plant would get a trending letter in 1992 and then be placed on the watch list in 1993. This is a plant that would illustrate a case where Arthur Andersen would say the NRC was slow to take formal action and this was a plant that many NRC managers during the interviews said they believed in retrospect might have gone on the list earlier. CHAIRMAN JACKSON: What triggered the action in the first quarter of '96? . 59 MR. BARRETT: The action on the part of the NRC? CHAIRMAN JACKSON: Right. MR. BARRETT: I -- CHAIRMAN JACKSON: It's just the way it happened? COMMISSIONER McGAFFIGAN: That might identify the plant, which -- CHAIRMAN JACKSON: Never mind. We're not supposed to be discussing these guys. That's right. COMMISSIONER McGAFFIGAN: Could I ask, another problem with performance indicators that comes up when you look at some of the charts, and I go away from this plant but if you are shut down, it's hard to get SCRAMs so you eliminate one category of hits. Now, if you're shut down, you also may be getting plenty of additional inspectors finding problems which gives you hits. But how do you -- have you thought through, and maybe this is a March 31 question, how you are going to deal with normalizing the performance indicators to things like what -- whether the plant is in a shutdown condition or not and that sort of thing? MR. JORDAN: Clearly, this scheme has limitations with respect to plants that are not operating and so it simply doesn't work right for that and so there are a number of conditions that for the March presentation -- we have to look at the independence of the indicators, relative . 60 weighting that one applies, the plant condition, whether a rising standard is embedded in it. So there are a lot of parameters that we have to consider when we come back to say, okay, here is closer to the ideal. But I think the model that they provided is a real thought provoker and has a lot of merit to it but we have to look further. CHAIRMAN JACKSON: What was the indicator you used for allegations and for enforcement action? Just numbers? MR. JORDAN: Yes. MR. BARRETT: Just number of allegations. I believe it was number of civil penalties. COMMISSIONER DIAZ: Should this plant -- MR. BARRETT: Excuse me, it's dollars of civil penalties. CHAIRMAN JACKSON: Dollars of civil penalties. COMMISSIONER DIAZ: Should this plant have remained on the watch list if it was placed on the watch list after it broke down? MR. BARRETT: Yes. As you can see, the green bar there would not have indicated that they met the criteria for removal. The Arthur Andersen model also has criteria for removal. COMMISSIONER DIAZ: Right. Another quarter would have done that at that performance? . 61 MR. BARRETT: Possibly. I would say, yes, because that would be three quarters consecutively below the industry average. COMMISSIONER DIAZ: So it would be four quarters below the industry average. CHAIRMAN JACKSON: Mr. Goldstein? MR. GOLDSTEIN: I would like to avoid focusing too much on this model and these indicators. The objective of the engagement was to probe on the issue of objective versus subjective decisions at the senior management meeting. This is one model. There are many, many others that can be used and also replicated by the industry each plant in its own behalf to tell how it will fare under a set of objective criteria. A lot of indicators have been put on the table here and these may be the right nine. I'm not sure that they should all be weighted equally. Dollars are used for the indication of enforcement action. Maybe it should be number of enforcement. The key point is that models can be created that can track historically and that is a test that has to be done, and for which sensitivity analyses have to be done and the time frame we had in this engagement neither did we conduct some of the usual validity checks that have to do with the sensitivity of the model to things like changing . 62 the number of quarters and so forth. The one thing I guess I would urge is that the individual elements of whatever model is picked not be tested against an ironclad standard but be viewed as a starting point. It will take years to refine the right model that gives you both the right objective standard and some flexibility but the term "continuous improvement" in my business is one way we try to convey to clients that it is better to start and even if you're refining as you go along it, at least in this environment, can be an improvement. MR. JORDAN: I'd make one comment. We have a remarkable historical record that we can use to benchmark against. The variables that occur in the plant in terms of objective measures and how their performance of those plants has actually changed over time so the validation, subsequently, can be reasonably powerful. CHAIRMAN JACKSON: Okay. MR. BARRETT: If I could have slide 14, I would like to wrap this up. In summary, Arthur Andersen concluded that we have a logical process but that there are findings and recommendations regarding the information and the process itself that can greatly improve the way in which we conduct our assessments. We do not intend to implement the findings until . 63 we have developed a staff consensus on what the right options are to go forward and until we have had policy guidance from the Commission but we will be preparing a Commission paper which we expect to forward on March 31 and we will proceed following Commission guidance. The Commission paper will deal with options for the process changes that have been recommended by Arthur Andersen and also options and plans for development of the leading indicators and the integrated process, the integrated information system that is proposed by Arthur Andersen. In the meantime, we would expect that there might be incremental changes implemented at the June 1997 senior management meeting, mostly those that might relate to process changes. It is a much more difficult challenge to address the types of issues that have to be gotten over in order to develop the information changes and we would expect that those would be implemented on a trial basis in January of 1988. So that concludes my presentation. If you have any further questions, I would be happy to try to answer them. CHAIRMAN JACKSON: Commissioner Rogers? COMMISSIONER ROGERS: Well, I think the report is an extremely interesting one and I think that a number of suggestions that have come out of it have been really very . 64 good. It is a question of details on things like the model and whatnot and I think that the disclaimers that have been made have been appropriate, don't get too hung up on it right now but it is a very interesting and possibly quite powerful approach. A couple of points about the report. One is I think you did say, I don't remember the pages now but I know I read it carefully at one time at any rate and noticed that you were emphasizing the importance of risk. But I really didn't see anything much about risk in the report and I wondered what you had in mind there, whether you were talking about really a kind of qualitative judgment of risk or something more mathematically defined, such as we would come up with with a probabilistic risk assessment. And so what is your concept of how we ought to fold risk into this process? MR. GOLDSTEIN: Are you asking me? COMMISSIONER ROGERS: Yes. MR. GOLDSTEIN: We learned early in the engagement that the NRC has and we reviewed them, quantitative standards that you use for what would be acceptable events, the kind of radiation problems that would occur immediately proximate to the plant and further out and those members of the team who are anchored in risk issues for the nuclear . 65 industry rapidly translate that into performance integrity and assuring that the integrity of the plant and the protection against some major operational failure is their translator into risk. I could contrast it to FAA. We do a great deal of work for FAA where, although certainly a serious crash is a disaster, it is not of the same magnitude. And so the concept of risk isn't defined as zero defects; in fact, FAA has a specific policy about refining designs as a result of recurring air failures. Our industry people in working with us here seem to be very comfortable that the operating concept of risk that you use and that we therefore could use is a zero defect avoiding of operating failures and that is the -- we did not go past that line to challenge the quantitative models that you use, translate that into probability of failure. COMMISSIONER ROGERS: Well, I'm nodding my head. That just means I heard you; I don't necessarily agree with that definition. MR. BARRETT: I would like to add a few words on that subject. COMMISSIONER ROGERS: Yes. MR. BARRETT: We did in the process of this study inform Arthur Andersen on the NRC's model of risk in terms . 66 of its quantitative model of risk being consequences times frequency and the major factors that tend to drive risk, which is initiating events, failure probability and equipment failures and common cause failures. And we developed a qualitative model that relates those to the types of things that tend to be assessed in the context of the senior management meeting and that writeup is actually Appendix 1 of the report, which was developed by the NRC staff and given to Arthur Andersen. But there was no intention and there is no intention of trying to make a quantitative assessment of risk based on performance. In the future, we have under development risk- based indicators which, as they become available, as the information becomes available to develop those indicators, I could see that we could move those indicators into the model, either to supplement the indicators we currently are using or perhaps even to replace indicators that we are currently using. But, basically, the answer to your question is it is a qualitative rather than quantitative connection. COMMISSIONER ROGERS: Just one more point, I think, before I get out of here and let other people have their say. I think this suggestion with respect to consensus decisionmaking and the idea of a rebuttable presumption on the part of -- as a starting point for an . 67 analysis I think is extremely interesting and I wonder if, you know, there could be some more specific mechanisms discussed for doing that, not necessarily right here today. But I think if this process is to be one that is clearly defensible and transparent to the public, then I think we have to be pretty clear on exactly how we are going to get to an end point starting with a rebuttable presumption and a consensus decisionmaking process, just exactly what that means. CHAIRMAN JACKSON: Mr. Jordan, you were going to make a comment? MR. JORDAN: Rich covered my comment extremely well. CHAIRMAN JACKSON: Commissioner Dicus? COMMISSIONER DICUS: Nothing further, thank you. CHAIRMAN JACKSON: Commissioner Diaz? COMMISSIONER DIAZ: Let's see, I've got one, two, three, four, five. I'm going to throw them all away and go back to zero defect. I'm going to throw all my questions away. This zero defect of operational failure which you said is the basis on which you developed your performance indicators, could you explain what an operational failure is? Is that a core meltdown or is that control rods falling in or is that a leaking pump? What is an operational . 68 failure? MR. GOLDSTEIN: We didn't develop the performance indicators. The indicators that are here are the indicators -- the seven indicators that the staff already uses and that we are putting in the model. Those are what we did use. COMMISSIONER DIAZ: Zero defect. MR. BARRETT: Well, I don't know that. Nuclear power plants, as you well know probably as well as I or better, are very complex machines and they are designed to be somewhat forgiving of failures here and there so with redundancy and diversity so -- COMMISSIONER DIAZ: We don't base performance indicators on zero defects, do we? MR. JORDAN: No. In the context you asked it, I believe, what the Arthur Andersen report was saying was that the NRC is adverse to risk and I would say in terms of a severe accident, it is unacceptable to have a severe accident. So that would be the connotation that I would put on their comment. COMMISSIONER DIAZ: Okay. So a connotation is a severe accident that has significant impact on the health and safety of the public versus, you know, the plant shutting down because he has a bad seal on a pump. MR. JORDAN: Correct. Correct. . 69 COMMISSIONER DIAZ: So there is a very important difference in there. Okay, thank you. CHAIRMAN JACKSON: Well, we, I mean, I would imagine, be hard pressed to prevent, you know, a seal on a reactor coolant pump from failing. The question is, do we pick up things ahead of time to not get to the severe accident scenario. Commissioner McGaffigan? COMMISSIONER McGAFFIGAN: Just one comment. I do think this was a remarkable effort over the last six months and commend Admin for working with you at the start, as I said, and I think the result is one of the best pieces of work if not the best piece of work I have seen in the six months I have been here. That said, I would like to ask a question and that is while this has been going on the General Accounting Office is looking at exactly this set of issues. Are we sharing all of our analysis and everything with GAO? How are we trying to deal with being open and candid with the Congress via the GAO? MR. JORDAN: Certainly, the information that has been developed is being made available or has been made available to the GAO. They are aware of the effort and have interviewed or are beginning to interview our staff. COMMISSIONER McGAFFIGAN: So they have a sense . 70 that we are struggling with the exact same set of issues that they have been tasked to look at? MR. BARRETT: They have conducted a number of interviews of not only the people who worked with me as NRC staff on this but they have also interviewed a number of the Arthur Andersen panelists on the study. We have provided them an early copy of the report prior to public release. We have tried to be as -- COMMISSIONER McGAFFIGAN: Do we have a sense of the timing? Will they -- the March 31 meeting where you are going to tell us at least your preliminary views as to how to deal with the report and what we might be able to adopt, is that compatible, will that be ahead of GAO or will they run ahead of that? Will they be able to wait and see what you are proposing to us? MR. JORDAN: We don't know what their schedule is. We will find out and communicate with GAO. CHAIRMAN JACKSON: Yes, Mr. Goldstein? MR. VALENTINE: Let me just answer that because I met with them twice. One thing we did have the advantage of is both Ira and I used to work at GAO so we sort of -- COMMISSIONER McGAFFIGAN: This looked like a GAO report. MR. VALENTINE: Well, I hope it didn't look completely like a GAO report. But we met with them and I . 71 think one thing about GAO that I have found since I came over to Arthur Andersen, we generally do things a little quicker than GAO, so they are not going to be ready by March 31 with a detailed report but they are sort of interested in what's going on here. They are very aware of what is going on and as much as they could be supportive, they were supportive. CHAIRMAN JACKSON: Thank you. I would like to thank the staff as well as the representatives from Arthur Andersen for a very informative briefing. I think what we can say is that the Arthur Andersen report indicates that there is a relationship between existing NRC indicators and plant performance and I believe the staff should continue to evaluate to what extent the existing indicators can be used to characterize plant performance and you have kind of spoken to it, Mr. Barrett, yourself that if the current set of indicators are inadequate in the sense that they are not fully risk informed, then the assumption is that the staff is exploring the development of new indicators and will phase them in as appropriate. We have already talked about using management or, as you said, organizational effectiveness as well as risk- based indicators and I think those are very important. The thing that has kind of been woven through this . 72 but it seems needs more direct focus is the issue of the screening meetings which feed the senior management meetings and having them be as objective as possible. And a question I would like to leave you with is whether the performance indicators are perhaps better used at that point in terms of developing the rebuttable presumptions about the plants and having the meetings themselves focus on the kinds of process improvements that you mentioned. And there was a plant performance template that had been developed or was being developed for use in that meeting and it would be useful to know what intent you intend to make of that. Then speaking further about the senior management meeting itself, the scrutability of the framework and the process, the process and the framework for decisionmaking should display the connection, I think, that exists between the plant performance data and what the actual ensuing decisions are. And, as I said, it seemed that you had moved along the lines of developing a plant performance template to help do that. And I think the Commission would be very interested in your establishing a consistency and if the consistency already exists then establishing the evidence of it, of the consistency between the senior management meeting decisions and decisions that are reached in our other evaluative processes. And here we are talking about the SALP process, the plant performance reviews and the . 73 inspection reports. We had a briefing last week on the reactor oversight program. It spoke to that. We have had a discussion here about the performance indicators and their uses. And we are speaking to it but we have to see the connection in actual fact and so I think that's very important. So unless there are any further comments by the commissioners, we are adjourned. [Whereupon, at 4:21 p.m., the briefing was adjourned.]
Privacy Policy |
Site Disclaimer |