1 1 UNITED STATES OF AMERICA 2 NUCLEAR REGULATORY COMMISSION 3 - - - 4 BRIEFING ON 5 SENIOR MANAGEMENT ASSESSMENT PROCESS 6 FOR OPERATING REACTORS 7 - - - 8 PUBLIC MEETING 9 Nuclear Regulatory Commission 10 One White Flint North 11 Rockville, Maryland 12 Friday, September 19, 1997 13 The Commission met in open session, pursuant to 14 notice, at 1:30 p.m., Shirley A. Jackson, Chairman, 15 presiding. 16 COMMISSIONERS PRESENT: 17 SHIRLEY A. JACKSON, Chairman of the Commission 18 GRETA J . DICUS, Commissioner 19 NILS J. DIAZ, Commissioner 20 EDWARD McGAFFIGAN, JR., Commissioner 21 STAFF PRESENT: 22 JOHN C. HOYLE, Secretary of the Commission 23 STEPHEN BURNS, Deputy General Counsel 24 25 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 2 1 PRESENTERS: 2 JOSEPH CALLAN, EDO 3 SAM COLLINS, Director, NRR 4 R. WILLIAM BORCHARDT, Chief, Inspection Program 5 Branch, NRR 6 MALCOLM KNAPP, Acting Director, RES 7 THOMAS MARTIN, Director, AEOD 8 RICHARD BARRETT, Deputy Director, Incident 9 Response Division, AEOD 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 3 1 P R O C E E D I N G S 2 [1:30 p.m.] 3 CHAIRMAN JACKSON: Good afternoon, ladies and 4 gentlemen. I'm pleased to welcome members of the NRC staff 5 to brief the Commission on their activities vis-a-vis 6 improvements to the senior management meeting process, and 7 in particular the staff's plans for and the results of an 8 integrated review of the NRC assessment process for 9 operating commercial nuclear reactors. 10 As we all know, the senior management meeting 11 process is intended to facilitate the early identification 12 of plants which require increased regulatory attention. 13 The Commission previously has indicated its belief 14 that there is room for improvement in the senior management 15 meeting decision-making process. These improvements relate 16 to making the process more scrutable and using objective 17 data with well defined decision criteria. The objective 18 should be to obtain a clear, coherent picture of performance 19 at operating reactor facilities. 20 The staff will describe its current activities to 21 support these objectives as well as the disposition of the 22 Arthur Andersen recommendations. 23 The staff also should describe any incremental 24 improvements to the process that have already been 25 accomplished. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 4 1 Staff assessment processes other than the senior 2 management meetings include the systematic assessment of 3 licensee performance (SALP), the plant performance reviews 4 (PPRs), and the plant issues matrix (PIM). 5 The staff will discuss the strengths and 6 weaknesses of each of the processes and its plans for 7 conducting an integrated review of the NRC assessment 8 process for operating commercial reactors. 9 I understand that copies of the slide presentation 10 are available at the entrances to the room, and unless my 11 colleagues have any comments they would like to make, 12 Mr. Callan, please proceed. 13 MR. CALLAN: Thank you, Chairman. I am not going 14 to try to repeat Hugh Thompson's virtuoso performance this 15 morning and personally walk you through all these slides. I 16 intend to turn the discussion over to Sam Collins. 17 I will say, though, as noted by the introduction, 18 two offices have involvement in this effort, the Office of 19 NRR and AEOD. NRR, of course, has the lead and hence I'm 20 going to ask Sam to lead the discussion. 21 MR. COLLINS: Good afternoon. As you will see 22 during the presentation today and hear from members of the 23 staff, the development of the information base for the 24 senior management meeting process and the integrated review 25 are currently being performed in parallel and somewhat ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 5 1 independently. There is broad office participation and 2 involvement in this activity. Along with the members at the 3 table here today, we have a representative of the regions, 4 Mr. Ellis Merschoff from Region IV, who is in attendance, 5 representing the region and has been involved in the 6 process, as I mentioned. 7 CHAIRMAN JACKSON: And you have Research 8 represented. 9 MR. COLLINS: Yes. 10 NRR is committed to coordinating these two efforts 11 so that a new assessment process can be developed and 12 implemented in a timely manner. We believe that the 13 evaluation tools that are being developed by AEOD for the 14 senior management meeting process are tools that can be 15 applied equally to any new assessment process. 16 It is important to realize that there is more to 17 these efforts than simply developing the new process. It 18 will also take a concerted effort and a significant effort 19 devoted to developing these new management directives both 20 procedurally and with staff training. That has been 21 outlined previously to the Commission. 22 These include involving and achieving full support 23 and cooperation of the stakeholders, including the industry, 24 to ensure that successful implementation on an improved 25 process is a success. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 6 1 At this time I would like to turn the briefing 2 over to Mr. Rich Barrett. 3 MR. BARRETT: Thank you, Sam. 4 If I could have slide 6, please. 5 Good afternoon, Chairman, Commissioners. Today's 6 briefing deals with the development of an objective set of 7 indicators and standards for senior management meeting 8 decisions. We are looking to produce early indications of 9 performance problems and to promote consistency in 10 decision-making and scrutability of the basis for those 11 decisions. 12 This is a program which the Commission first 13 requested in June of 1996 and which the Commission has 14 repeatedly endorsed in subsequent SRMs. 15 The work I will describe has principally involved 16 AEOD staff in both the Incident Response Division and the 17 Safety Programs Division. 18 Significant support for this effort has been 19 provided by the Office of Research, primarily from their 20 experts in risk assessment, human performance, and 21 organizational effectiveness. 22 AEOD has employed the Idaho National Engineering 23 and Environmental Laboratory for statistical support and 24 Arthur Andersen Consulting for independent assessment. 25 The work has been overseen by NRR, who will be our ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 7 1 principal customer, and has had the benefit of continuous 2 regional involvement and regional oversight. 3 Slide 8, please. 4 I want to begin by reviewing the information that 5 NRC has available for performance assessment and by showing 6 how it relates to the products that we are developing. 7 On the left of this slide is a list of our 8 principal information sources, including the inspection 9 program, licensee event reports, and several other areas 10 listed here. I should point out that there are many other 11 sources of information that I have not listed. 12 From these sources we derive two types of 13 objective data as shown in the middle column. First there 14 are indicators, which are quantitative measures of 15 performance, such as the number of safety system failures, 16 or quantitative measures of economic stress, such as the 17 cost per kilowatt hour of electricity generated. 18 The other type of objective information are what 19 we are calling issues. These are qualitative findings but 20 nonetheless objective. 21 An example of an issue that might appear in the 22 plant issues matrix of a region would be an observation on 23 the part of an inspector that a licensee failed to restore a 24 system to its proper configuration following maintenance. 25 The category called issues represents the majority ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 8 1 of the information we have available to us. 2 We are developing methods to structure all of this 3 information, the issues and the indicators, to make it 4 scrutable, and we are developing criteria and guidelines to 5 assist senior managers in making decisions which are 6 objective and consistent. 7 The three methods we are developing or the three 8 products we are developing are shown in the right-hand 9 column. They are the performance template, the performance 10 trend methodology, and the economic trend methodology. 11 CHAIRMAN JACKSON: Are licensee self-assessments 12 folded into this at all? 13 MR. COLLINS: Licensee self-assessments could be 14 folded into this. We haven't done anything explicit at this 15 point. 16 MR. CALLAN: A fundamental ground rule, Chairman, 17 is that the information be publicly available; it has to be 18 in the docket file. To the extent that licensee 19 self-assessment insights are captured, and there are various 20 ways of doing that, either through inspection reports, by 21 the licensee submitting it to us under a cover a letter, 22 which happens from time to time, to the extent that that 23 information gets into the docket, then it is eligible to be 24 used in this process. 25 CHAIRMAN JACKSON: Are the results or issues that ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 9 1 arise from those self-assessments ever captured in 2 inspection reports? 3 MR. CALLAN: They are. We have specific 4 inspection modules. I happen to have memorized one 5 inspection, 40501 -- don't ask me why I remember that -- 6 which is specifically directed at NRC follow-up of licensee 7 self-assessments to independently validate and verify the 8 effectiveness of them. That particular inspection module 9 essentially requires that we provide the significant 10 insights of that self-assessment into the docket, if not the 11 actual self-assessment. That's a judgment that has to be 12 made. 13 I'm guessing here. I think probably more often 14 than not the path of least resistance is for the licensee to 15 submit the self-assessment to us. 16 MR. BURNS: Chairman Jackson, might I add 17 something here? An issue that has come up over the years, 18 and in fact I think was an issue with Northeast Utilities in 19 terms of some early assessments, was their status as public 20 documents. At times licensees may submit them to us with a 21 request for a treatment as proprietary information under 22 FOIA Exemption 4, and there is legitimate treatment. There 23 is a body of law that is developed basically out of the 24 medical arena and other arenas which permits that type of 25 proprietary treatment. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 10 1 I think what Joe is saying is important in terms 2 of the public nature, but there may be a tension in terms of 3 some licensees with respect to their willingness to allow or 4 wanting their self-assessments in the public domain. There 5 has been some dialogue even more recently on those types of 6 things as well. 7 MR. CALLAN: It is probably worth further 8 digression here. This is a very important issue. There is 9 another degree of tension, and that tension has to do with 10 the chilling effect that a requirement from us for a 11 licensee to place self-assessments in the docket, the impact 12 of that requirement on the licensee's willingness to bare 13 their soul, so to speak, and to provide brutal critical 14 self-assessments. We are always concerned about that 15 aspect. 16 CHAIRMAN JACKSON: What is the nexus between that 17 and the inspection module? 18 MR. CALLAN: The inspection module that I 19 mentioned is a module that was created about three years ago 20 that enables us, under very restricted conditions, to 21 embrace a licensee's self-assessment, to take credit for it, 22 so to speak, to leverage our resources. In order to do 23 that, we have specific requirements levied. We have to do 24 some independent verification, validation; portions of the 25 self-assessment, if not the entire self-assessment, should ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 11 1 be made publicly available. If certain conditions are met, 2 we would then use those insights, capture them as our own 3 and act on them. 4 CHAIRMAN JACKSON: So they actually would play 5 into helping to develop the performance issues part of the 6 objective? 7 MR. CALLAN: Absolutely. In fact, let me give you 8 one example of kind of an extreme case. At the Cooper 9 Nuclear Station we used in effect a licensee's 10 self-assessment that we validated. The validation team 11 leader is behind me, Ellis Merschoff. That self-assessment 12 was put on the docket. We used that, and it was very 13 insightful. We have a similar effort, as you know, ongoing 14 at the Clinton Station. That's an extreme case. There are 15 lesser cases than that. 16 MR. BARRETT: Let me move on to slide 9. I would 17 like to talk for a little while about the performance trend 18 methodology. 19 Our performance trend method has been developed in 20 response to specific Commission guidance which is listed in 21 detail in slides 9 and 10. For instance, we have gone back 22 and done a complete reevaluation of each candidate 23 indicator. We actually looked at over 50 candidate 24 indicators. We looked at them from the perspective of 25 objectivity, their ability to resolve plant performance ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 12 1 issues, face validity, and also statistical correlation with 2 past senior management meeting results. 3 We have evaluated different ways of combining 4 indicators to produce trend plots. We have settled on two 5 ways, which I will be showing you in just a little while. 6 We have examined the use of various time periods 7 over which to aggregate and integrate the data. We have 8 looked at different weighting schemes, and we have also 9 looked at the issue of using fixed standards for comparison 10 versus floating standards. Again, we'll discuss that in a 11 moment as well. 12 In response to the Commission's request, we are 13 having this methodology peer reviewed by the ACRS. We plan 14 to submit it for public comment, at which time I am sure we 15 will also get a fair bit of industry feedback, and we plan 16 to hold a public workshop on it. In fact actually all of 17 this is going to be applied to the template and the economic 18 indicators as well. 19 We have done a preliminary benchmark of our 20 preliminary methods here against past senior management 21 meeting results, and I will show you some of that. 22 Also we plan, in accordance with Commission 23 guidance, to do a trial test of this methodology in the 24 January 1998 and the June 1998 senior management meeting 25 cycles. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 13 1 CHAIRMAN JACKSON: I think Commissioner McGaffigan 2 has a question. 3 COMMISSIONER McGAFFIGAN: It may be more in the 4 form of a comment, because I've said it before at meetings 5 on this subject. The statistical correlation with past 6 senior management meeting results worries me a little bit 7 because there is a presumption that those results were 8 correct, that we got the right discussion plants, that we 9 got the right trending letters, that we got the right people 10 on the list, and obviously we have been criticized that we 11 haven't done that. So I'm not sure whether statistical 12 correlation with past senior management meeting results is a 13 good thing or a bad thing. You're going to have to convince 14 me. 15 MR. CALLAN: ACRS has made the same point, 16 Commissioner. 17 COMMISSIONER McGAFFIGAN: In fact I made the same 18 point in a letter in today's reader. 19 MR. BARRETT: I think the point is a very good 20 one. I can only say a couple of things regarding that. 21 First of all, the Arthur Andersen report did say that from 22 the perspective of the discussion plants that every 23 indication from their study was that we had done a good job 24 of identifying plants for discussion, whereas we may have 25 been slow in putting plants on the watch list, for instance. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 14 1 The correlations we did were against the discussion plants. 2 COMMISSIONER McGAFFIGAN: I don't have the report 3 fresh in mind, but one of the examples -- they had sort of 4 three charts in their report, as I recall, and one of them 5 was a plant that we had on the watch list and we took it 6 off, and then its performance deteriorated worse than it 7 ever had been by their indicators, and we never discussed it 8 again. 9 The trouble with making a general statement is, 10 yes, maybe in general we got the right discussion plants, 11 but one of the three cases that they chose to highlight was 12 one where, whichever plant it was, we did not discuss it at 13 a time when their model would have called for at least 14 discussion if not getting it back on the list. That's my 15 recollection of one of the three examples they highlighted. 16 MR. BARRETT: That's correct. One of the things 17 that we did to try to test this question of whether the 18 discussion plant list was a good list -- Again, there is 19 always a certain amount of circularity here, because there 20 is no ground truth, there are no tablets in stone that say 21 which plants are good and which plants are not. We took 22 some of our indicators and we did what we call a clustering 23 analysis. We took indicators that we felt have face 24 validity, that is to say, indicators that we thought of and 25 by themselves were good indicators of performance. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 15 1 We went through a statistical process that I don't 2 fully understand and I wouldn't even attempt to describe 3 today to see if those of and by themselves in combinations 4 would nominate certain plants for discussion without any 5 reference to past discussion lists. 6 It turned out that the ones that they nominated in 7 these combinations correlated pretty well with past 8 decisions. Again, it is still circular and I don't think we 9 ever get the ground truth. 10 COMMISSIONER McGAFFIGAN: One question rather than 11 a comment. There are some proprietary indicators out there 12 and they are used when people come in to talk to 13 Commissioners. One of them is the WANO overall performance 14 indicator. I understand it's proprietary, but I understand 15 we also have access to it through the arrangements we have 16 with INPO. Have we looked at the proprietary indicators and 17 said, wow, one of these is so good in predicting -- I'm 18 sorry if I preempted your question. 19 COMMISSIONER DICUS: That's all right. Go ahead. 20 COMMISSIONER McGAFFIGAN: Have you looked at 21 propriety indicators, in particular the WANO overall 22 performance indicators? 23 MR. BARRETT: I don't know the answer to that 24 question. 25 CHAIRMAN JACKSON: Or looked at the correlation of ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 16 1 the Arthur Andersen prediction to what the WANO indicators 2 might say? 3 MR. CALLAN: Let me weigh in here. We met a 4 couple weeks ago. As you know, the senior managers had 5 their annual public meeting with INPO. INPO made a strong 6 point that there are over 400 plants internationally that 7 use the WANO indicators, and all the plants in the U.S. are 8 included in that number. They wanted the NRC to take a hard 9 look at those indicators so that we don't create our own set 10 and cause ambiguity and confusion. This is a request that 11 they made a couple weeks ago. 12 We need to do that. We need to take a hard look 13 at the WANO indicators and make a deliberate decision yes or 14 no, but at least I think we are obligated to do that. 15 WANO, as you know, also comes up with a figure of 16 merit, which is a single percent number. You probably 17 heard, Commissioner, the same thing I've heard from time to 18 time, which is that some utility executives believe that to 19 be the best indicator out. 20 COMMISSIONER McGAFFIGAN: Yes, I heard that 21 yesterday, which is why I asked the question. 22 MR. CALLAN: So I think we are obligated to take a 23 look at that very close. 24 CHAIRMAN JACKSON: Commissioner Dicus, did you 25 have an additional question? ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 17 1 COMMISSIONER DICUS: No. That was my question. 2 MR. BARRETT: We have looked at the individual 3 indicators, I think the complete set of individual WANO 4 indicators in one form or another. 5 Let me move on to page 11. 6 I just want to point out that at this point, based 7 on the work that we have done so far, the eight indicators 8 that you see here are the ones that we are concentrating 9 with the model and with the graphs that I am going to be 10 showing you in just a moment. 11 These are ones that passed the test that I 12 mentioned earlier, the test of face validity, the 13 correlation test. These are indicators that discriminate 14 well between discussion and non-discussion plants. 15 Also, these are indicators that have relatively 16 less subjectivity involved with them; relatively less 17 assessment is involved with these compared with some of the 18 other indicators that were included in the earlier Arthur 19 Andersen work. 20 CHAIRMAN JACKSON: Did you have criteria? What 21 you just described were the criteria you used to cull these 22 out of a larger set? 23 MR. BARRETT: Those and others, yes. 24 CHAIRMAN JACKSON: Thank you. 25 MR. BARRETT: If I could move to slide 12. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 18 1 In the next few slides I would like to just show 2 you a sample of the type of information that we will be 3 providing as a test case of this methodology for the January 4 senior management meeting cycle, which begins later this 5 month with the PPRs. 6 First, what we will be showing is just basically 7 an overview for each of the regions of the information in 8 each plant. 9 In Region D you can readily see which plants the 10 agency might want to look more closely at. Plant 103 is 11 certainly one of them. 12 CHAIRMAN JACKSON: What is the y axis? 13 MR. BARRETT: That axis is the number of hits. 14 This particular model is a threshold type of model and it's 15 a model similar to the Arthur Andersen model in which you 16 look at indicators one at a time and then count up the 17 number of hits. 18 COMMISSIONER DIAZ: Thresholds are wonderful 19 things. They are also very dangerous. 20 MR. BARRETT: They are, yes. 21 COMMISSIONER DIAZ: What is the meaning of the 22 threshold? Does it have special significance? 23 MR. BARRETT: This particular threshold is simply 24 three hits. 25 CHAIRMAN JACKSON: Does that mean that a plant is ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 19 1 a potential discussion plant? 2 MR. BARRETT: That's right. That's a candidate 3 threshold for discussion. That would mean that for three of 4 the eight indicators in this model this plant had a hit. 5 CHAIRMAN JACKSON: In this kind of trend model is 6 more recent data weighted differently than data earlier in 7 the assessment here? 8 MR. BARRETT: Not in this particular model. This 9 particular model carries six quarters of data equally 10 weighted. 11 CHAIRMAN JACKSON: What does regional average 12 mean? 13 MR. BARRETT: This would be for all of the plants 14 in this region there was an average of approximately 1.5 15 hits per plant. 16 COMMISSIONER McGAFFIGAN: In the Arthur Andersen 17 methodology, to get a hit you had to be twice as bad in the 18 indicator as the industry average. Is that carried over 19 here, or do you have multiple thresholds? You have a 20 threshold to get a hit and you have a threshold to be 21 considered for discussion? 22 MR. BARRETT: That's correct. In this particular 23 model it was one standard deviation from the mean of that 24 indicator averaged over six quarters. 25 COMMISSIONER McGAFFIGAN: That raises the question ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 20 1 as to whether you have the threshold at the right level. In 2 order to get hits they have to be worse than industry 3 average, and then you are looking for deviations from 4 industry average. You basically chose three times the 5 industry average or twice the regional average. If you are 6 going to try to get uniformity across the industry, it 7 presumably is the industry average that matters. Why did 8 you choose three as opposed to two, which would force 9 several of the plants, or at least plants 94 and 95, into 10 discussion space? 11 MR. BARRETT: At this point we haven't settled on 12 a threshold. I will show you some results based on this 13 threshold of three. We have a separate set of results based 14 on a threshold of two, and those are not too much different. 15 There are several issues we still need to examine. 16 One of them is whether we should be comparing to an industry 17 average or whether we should be setting a fixed standard. 18 These are still open questions. 19 CHAIRMAN JACKSON: Please go ahead. 20 COMMISSIONER DIAZ: What kind of statistical 21 distribution do you assume to come up with one standard 22 deviation from the mean? Obviously this is not a normal 23 population. There are very few numbers of plants. 24 MR. BARRETT: There was no need to assume a 25 distribution because we could take all the plants and just ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 21 1 calculate the standard deviation. 2 COMMISSIONER DIAZ: I see. Okay. 3 CHAIRMAN JACKSON: These hits are relative to the 4 performance parameters you had shown on page 11? 5 MR. BARRETT: That's correct. 6 CHAIRMAN JACKSON: Okay. 7 MR. BARRETT: Let me move on to slide 13, which 8 shows the performance trend for Plant 103. This is an 9 actual plant, by the way. There is no Region D. That's a 10 conglomeration of real plants but not in any particular 11 region. 12 This again, as I said, is the threshold model. 13 You can see for Plant 103 there is an increasing trend 14 forward in the indicators. That would obviously lead you to 15 ask, what is it that is driving this trend? So what we will 16 be providing in addition to the overall trend will be the 17 trends for the individual indicators, all eight of them. 18 In this particular case I am showing that forced 19 outage rate and safety system failures are certainly 20 contributing to this trend. That would lead you again to 21 further ask, well, what were those safety system failures, 22 how serious were they? We will be also supplying textual 23 information to back this up as to what is driving these 24 indicators. 25 CHAIRMAN JACKSON: Is there any double counting if ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 22 1 you look at forced outage rate and safety system failures? 2 MR. BARRETT: There could be double counting, yes. 3 In fact there probably will be double counting, because some 4 of these indicators relate to cause codes, and those cause 5 codes would relate perhaps to the same safety system 6 failures. That's why the textual information becomes 7 important so you get behind the numbers and understand what 8 is driving this. 9 If we could move on. Just briefly looking at page 10 14, as I mentioned, we do have two models. The second model 11 is a model that is based on a regression fit using similar 12 indicators. This is a regression fit to past discussion 13 plants. It gives a different picture, but still it gives a 14 picture for Plant 103 for what would appear to be degrading 15 performance. 16 COMMISSIONER DIAZ: In here you did weight them 17 more heavily as a function of time, right? The last few 18 periods were weighted heavier than the earlier ones when you 19 did the multiple step regression? If you look at the curve, 20 it would seem to me like weighting is directly proportional 21 to time or maybe even the square of time. 22 MR. BARRETT: We could, but this particular model, 23 the coefficients in the regression analysis were not given 24 any preference to more recent versus -- 25 COMMISSIONER DIAZ: Look at the data and you will ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 23 1 see that it appears -- I don't know -- that the latest point 2 is heavier weighting than the earlier ones. Just by looking 3 at it, but I'm not sure. It does look like it comes up 4 earlier and it rises very, very steep, which might indicate 5 that it's weighted heavily towards the end. 6 MR. BARRETT: I understand your point. I don't 7 believe that's the case, but I could double check that. 8 CHAIRMAN JACKSON: The real question is this sharp 9 crossover at the 942 point. 10 What kind of regression model is this? 11 MR. BARRETT: I can give you the name of it. 12 CHAIRMAN JACKSON: Tell me the name. 13 MR. BARRETT: Logistic regression. 14 CHAIRMAN JACKSON: Okay. 15 MR. BARRETT: Let me move on to slide 15. 16 I'd like to ask you to look at the slide on the 17 monitor, because it has been changed from the slide that we 18 supplied to you earlier. The reason is we felt that we 19 needed to add additional information to this slide to 20 clarify it. The slide that we provided you could easily be 21 misinterpreted, we felt. 22 We have compared both of these models with the 23 results of past discussion lists. What we found is, first 24 of all, that there was what we call an 87 percent agreement. 25 That is to say, there were 79 plants out of 109 which both ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 24 1 the senior managers and the model thought should be 2 non-discussion plants. There were 15 plants which both the 3 senior managers and the model felt should be discussion 4 plants. 5 However, there were four non-discussion plants 6 that the model identified, and there were ten discussion 7 plants per the senior managers that were missed by this 8 model. In other words, this model identified 15 of the 25 9 discussion plants in this particular time period and 10 identified four plants that were not on the discussion list. 11 CHAIRMAN JACKSON: When you say model, you mean 12 the trend model or the regression model? 13 MR. BARRETT: Actually both models gave similar 14 results. 15 We will continue to refine this model, and we 16 think that we can improve the model. It remains to be seen 17 how much we can improve the model. 18 CHAIRMAN JACKSON: What do you think is the more 19 critical error, the ones that were identified by the model 20 that were not discussed, or the other way, the ones that 21 were discussed but not identified by the model? 22 MR. BARRETT: I don't know if I could make a 23 distinction between the two. I've actually gone and looked 24 at a couple of these plants, and we had a couple of 25 discussion plants for which the indicators just don't ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 25 1 indicate anything, and yet we had serious concerns about 2 those plants. 3 I believe you have to look behind each of the two 4 types of errors. 5 MR. CALLAN: I have an opinion on that. I think 6 Bill Borchardt's presentation gets into this, but the role 7 of the indicators is largely to provide a forcing function 8 for the discussion, to force the senior managers to have to 9 face facts, so to speak, and not to rationalize away 10 problems. 11 In that context, I guess you could say a false 12 positive, in other words, the ten discussion plants that 13 were missed by the model, would be a bigger problem. This 14 approach could accommodate false negatives. In other words, 15 the four that perhaps shouldn't be discussion plants that 16 become discussion plants because of the data scatter. The 17 process can accommodate that better than it can accommodate 18 missing ten plants that should be discussed. 19 CHAIRMAN JACKSON: Commissioner McGaffigan. 20 COMMISSIONER McGAFFIGAN: Did you do the obvious 21 check of looking at a lower threshold which would kick in in 22 your model plants 94 and 95? That obviously gives you more 23 plants for discussion. Were those the "right plants," the 24 plants that had been identified for discussion just by 25 varying the threshold from three to two? ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 26 1 MR. BARRETT: We've lowered the threshold. We did 2 at least one such sensitivity analysis. What it did, of 3 course, was identify more plants. It did a better job of 4 identifying plants that were discussed. It also produced 5 more false positives. 6 COMMISSIONER McGAFFIGAN: False positives by your 7 definition because they weren't discussed. 8 MR. BARRETT: Because they weren't discussed. 9 Again, always keeping in mind that we don't have ground 10 truth here. But I will tell you that the two plants that I 11 looked at that I felt were quite important were two plants 12 which we not only discussed but took further action. They 13 were not identified by the model. Those two plants remained 14 outside that universe. So there may well be a residual 15 class of plants that indicators don't pick up. 16 COMMISSIONER DICUS: I noticed on these two models 17 there are three -- I think three -- indicators that are 18 common to both models. Obviously they are going to impact 19 quite a bit. Could you give me a little bit of information 20 on why you did that? 21 MR. BARRETT: We were not trying to keep the 22 indicators separate. In fact, if anything, it might be an 23 ideal case if they were the same but applied in different 24 ways. In the case of the threshold model the emphasis is 25 more on trying to find indicators that you feel each in its ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 27 1 own way has face validity and indicates performance in a way 2 that you understand, whereas in the case of the regression 3 model, of course, you allow the regression to decide which 4 combination of indicators gives you the best results. 5 CHAIRMAN JACKSON: How sensitive are these results 6 to length of observation period? 7 MR. BARRETT: I couldn't tell you numerically. As 8 we got the observation period out towards six quarters, what 9 we found was that we were able to average out some of the 10 fluctuations that you get as a result of shutdowns and 11 things like that. But I don't have numbers as to how 12 sensitive we are. 13 I'd like to move and quickly talk about the 14 economic indicators. In the interest of time, I'd rather 15 not spend too much time on them, but I'd simply like to say 16 that we have performed a similar process with economic 17 indicators to the ones I just described with the performance 18 indicators. 19 We started with a large number of these 20 indicators, some of which were site-specific, some of which 21 were corporate, and we evaluated each one of them using 22 correlations with past senior management meeting 23 discussions. 24 We developed a trending methodology based on the 25 combination of the ones that gave the best results. I don't ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 28 1 know if anybody will be surprised by this, but we found that 2 the site-specific indicators showed a much better 3 correlation with past results than the corporate indicators. 4 We intend to supply economic trend plots similar 5 to the performance plots I just showed both as aggregates 6 and as individual plots of individual indicators for this 7 senior management meeting cycle along with explanatory 8 information so that everyone understands what these economic 9 indicators mean. This will be in mid-October of this year 10 for the January cycle. 11 CHAIRMAN JACKSON: To what extent were the 12 economic indicators leading indicators? 13 MR. BARRETT: The economic indicators were 14 leading. The correlations tended to show that where there 15 was a correlation there was a leading correlation. The 16 analyses were done by using delays in the -- again, if you 17 would like to know more about this, I might ask someone to 18 come to the podium, but they were shown to be somewhat 19 leading. 20 CHAIRMAN JACKSON: Commissioner McGaffigan. 21 COMMISSIONER McGAFFIGAN: Early on in your 22 presentation either you or Sam Collins said that at some 23 point you're going to get this ACRS comment and public 24 comment on all this. How is that going to work? Is that 25 going to work in time? If somebody looks at this from ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 29 1 outside and says you're all wrong here, is that going to be 2 in time to course adjust? 3 MR. BARRETT: The answer to that is yes. We will 4 always be in a position to course adjust. After we have 5 been through the public comment period and after we have 6 been through the trial tests of all these methods, we would 7 plan to come to the Commission -- our current schedule would 8 say the end of next summer -- with recommendations of how to 9 proceed with this information. 10 The ACRS peer review has already started. We met 11 with the ACRS in March of this year and gave them an 12 overview of the program. We met with them earlier this 13 month and showed them a much more detailed version of what 14 you are seeing today. We've gotten a great deal of feedback 15 from them and it's very useful feedback, and we will be 16 factoring it in. 17 We plan to go back to the ACRS in February. We 18 are scheduled to do that. We have an invitation from them 19 to come back at intermediate points to try out some of our 20 thinking as we go along. 21 So we believe that the ACRS comments can be 22 factored in as we go. 23 CHAIRMAN JACKSON: You mentioned what products you 24 are going to have for the January senior management meeting 25 cycle, but will any of this economic data be actually used ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 30 1 in the senior management meeting process? 2 MR. CALLAN: That's an NRR question. 3 MR. BORCHARDT: The intent right now is that we 4 use that information at the screening meeting primarily. If 5 it's found to be exceptionally relevant, it would move 6 forward to the actual senior management meeting, but its 7 primary purpose will be restricted to the screening meeting. 8 CHAIRMAN JACKSON: So it's at that point in the 9 process? 10 MR. BORCHARDT: Yes. 11 MR. CALLAN: Let me throw out a cautionary note on 12 the economic indicators. Even more so than the case of the 13 other indicators, which I think Commissioner McGaffigan 14 questioned because of the circular nature of the validation 15 process, I think the economic indicators are even more 16 subject to that kind of error. We are correlating these 17 indicators with plant performance in a period when they were 18 under economic regulation. So we are correlating the 19 indicators today with plant performance trends against that 20 context. 21 I don't think we can take much comfort from the 22 fact that we are seeing correlation or that we are even 23 seeing that they are leading indicators. I don't think we 24 know enough about the economic deregulation environment to 25 make too many flat statements on that. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 31 1 CHAIRMAN JACKSON: To draw too many conclusions. 2 MR. CALLAN: Right. 3 CHAIRMAN JACKSON: Commissioner Diaz. 4 COMMISSIONER DIAZ: Mr. Callan just answered my 5 question. 6 MR. BARRETT: I'd like to move on to slide 18 and 7 discuss the template. 8 CHAIRMAN JACKSON: Let me ask you a quick 9 question. The data to support the indicators that you have 10 on page 17, the site model indicators, is that publicly 11 available data? 12 MR. BARRETT: Yes, it is. It's information that 13 is available to us either from reports that are required to 14 be sent to the SEC or else monthly operating reports that we 15 get. 16 CHAIRMAN JACKSON: Okay. 17 MR. BARRETT: With regard to the template, the 18 Commission directed the staff to consider use of a plant 19 performance template and suggested that the template be used 20 as a way of making a connection between performance 21 information and the ensuing decisions. 22 You also directed us to show how the template and 23 the trend plots will be used in tandem in the decision 24 process and how the template can include both quantitative 25 as well as qualitative information. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 32 1 Finally, you directed us to more precisely and 2 objectively determine the specific criteria and thresholds 3 for NRC action levels. 4 If I could have slide 20, please. 5 Slides 20 and 21 present the current version of 6 our template. It's a version that was frozen as of July 7 31st, although this is still a work in progress. It 8 embodies a balanced and structured presentation of the key 9 elements that constitute plant performance. 10 I don't plan to go through this template in detail 11 because of the time constraints, but I would like to just 12 describe it in some outline. 13 The major categories in this template mirror the 14 categories in the template of Management Directive 8.14. We 15 decided to adopt those major categories after some 16 deliberation because we felt that it was a good set of 17 categories. We also felt that it was a set of categories 18 that were risk-informed. 19 To develop the subcategories we went back to the 20 record of the past senior management meetings to look at the 21 areas which past senior managers thought were important to 22 performance. 23 We went back to the briefing books for the senior 24 management meetings, to the minutes of the senior management 25 meetings, and to the transcripts of their briefings to the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 33 1 Commission, and we built a database of approximately 1,700 2 specific statements that were made about plants. 3 From those we identified common characteristics 4 that were discussed. We aggregated those down to a couple 5 of dozen categories and then we looked to see if they fit 6 naturally into the major categories of Management Directive 7 8.14. And they did, with one exception. 8 We found that about a third of the judgments that 9 were stated in past senior management meetings related to 10 organizational effectiveness or management, and there was no 11 such category in the management directive template. So we 12 added a major category which we call organizational 13 effectiveness. 14 We took this template and we had it reviewed by 15 NRR and Research. The Office of Research suggested 16 significant changes to it, especially in the areas of human 17 performance and organizational effectiveness, and we 18 implemented those proposed changes. 19 We continue to refine this template, but in the 20 meantime we are using this version of the template for two 21 purposes. One is to define the information which will be 22 the input to this template, and secondly, to define the 23 criteria and decision model that will be based on this 24 template. 25 CHAIRMAN JACKSON: Right. That's what I was going ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 34 1 to point out, that what you have here is almost like a 2 listing of topics, a topical listing. 3 MR. BARRETT: That's correct. 4 CHAIRMAN JACKSON: The real question is what is 5 your hierarchy for decision-making and what are the criteria 6 you are going to use and what are the objective standards 7 that are applied. 8 MR. BARRETT: The ACRS talked about a decision 9 model. 10 CHAIRMAN JACKSON: How far are you away from 11 defining a decision model? 12 MR. BARRETT: The definition of the decision model 13 and the criteria is in its early stages. It's difficult to 14 say. 15 COMMISSIONER DICUS: It's written in pencil on a 16 piece of paper. 17 CHAIRMAN JACKSON: You decided to start it this 18 afternoon. 19 [Laughter.] 20 MR. BARRETT: No, we didn't start this afternoon. 21 We are a lot farther along than that, I'd say. In fact, I'd 22 like to discuss some of the early thinking as we get to that 23 point in a couple of slides. 24 CHAIRMAN JACKSON: Commissioner Diaz. 25 COMMISSIONER DIAZ: I was going to make a comment ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 35 1 and a question on this organizational effectiveness. I have 2 absolutely no problem with any of the categories except the 3 issue of culture. It's kind of a sensitive issue to me. 4 Making a small point in here in a relaxed afternoon, I used 5 to be in a place where people used to talk about having to 6 judge people by their revolutionary conscience. 7 [Laughter.] 8 COMMISSIONER DIAZ: I really don't want to get to 9 that point in our deliberations, and I'm sure that you are 10 very, very sensitive to that. When we assess culture we've 11 got to be very careful that we don't infringe on the freedom 12 of our institutions. 13 CHAIRMAN JACKSON: That's why what your criteria 14 are and what your decision model is is very important in 15 order to guard against that. 16 COMMISSIONER DIAZ: By the way, I flunked the 17 test. 18 [Laughter.] 19 MR. BARRETT: If I could move on to slide 22. 20 I want to talk a little bit about the information 21 that will populate this template. The template for a 22 specific plant will contain the issues that result from the 23 regional PIMs and other sources, as well as the indicators 24 that are appropriate to the various categories and 25 subcategories. We feel that these indicators can be ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 36 1 associated with the various categories. 2 We have actually done a trial application of this 3 approach using the PIMs data from a specific region, and we 4 found that the inspection data fit the template quite well, 5 again with one exception, and that is organizational 6 effectiveness. It's not surprising that we didn't find very 7 many findings related to management and organizational 8 effectiveness because inspectors are not encouraged to look 9 into management issues in the inspection process. 10 So that leaves a question, and the question is, 11 how are we going to populate the organizational 12 effectiveness category? That's a question that we are 13 currently working with the Office of Research. They have 14 some methodologies that we are evaluating to use the issues 15 from the other categories to populate the organizational 16 effectiveness category. 17 In addition to categorizing the issues, what we 18 intend to do is also assign a qualitative risk significance 19 to these issues. That could be a high versus low. 20 Certainly not a quantitative estimate. High/medium/low or 21 high/low. We are working with the Office of Research also 22 on this, to develop simple guidance on how to categorize 23 issues with respect to risk. 24 We are beginning a pilot application of the 25 template. We intend to ask the regional offices beginning ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 37 1 this fall to code new inspection findings as they go into 2 the PIMs, to code them in accordance with this template, 3 excluding the organizational effectiveness category. 4 By doing this we expect that when the June 1998 5 senior management meeting cycle begins, which is March of 6 1998, we will have a database of approximately six months 7 worth of information about these plants, and that will allow 8 us to do a trial application of this template in that senior 9 management meeting cycle. 10 CHAIRMAN JACKSON: Let me reemphasize something. 11 The issue of the organizational effectiveness and really 12 having a data and decision criteria relative to it is very 13 important for the kind of reason that Commissioner Diaz 14 mentioned. As I recall reading the Arthur Andersen report, 15 essentially it indicated that by the time the decisions 16 propagated to the senior management meeting that they were 17 basically anecdotal, that there was a database and there 18 seemed to be more linkage between the inputs and judgments 19 made at earlier stages in the process and things seemed to 20 work okay until you got to the senior management meeting, 21 and then there were a set of decisions that were made that 22 seemed unlinked or disjointed from everything that had gone 23 on before. 24 So it is very important that you develop the 25 criteria and what data needs to feed into that and then to ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 38 1 show what your decision model is that actually then gets 2 promulgated into the actual decisions in the senior 3 management. Otherwise you will always be accused of having 4 a process that is not scrutable. Either way. That you are 5 making arbitrary judgments about management that could be 6 negative without any supporting line and decision process 7 and data, or for those who would be the detractors of the 8 process, that you do it the other way, that you make 9 arbitrary decisions or you give credit for or you weigh 10 management behavior to the positive effect, again without 11 any real objective data. 12 So it is an important issue that I think you need 13 to give some heightened and accelerated attention to to the 14 extent that you can. 15 MR. BARRETT: We will. 16 MR. BORCHARDT: Chairman, if I may. I think the 17 problem is even more difficult than you describe. The 18 current configuration of the inspection program does not 19 right now support the data that Rich is alluding to that he 20 needs to set up that model. If the Commission decides that 21 we are going to go in that direction, there is a wide range 22 of inspection guidance and standards and training for the 23 inspection staff that needs to be completed, and that won't 24 be an easy task. 25 CHAIRMAN JACKSON: None of this is, though. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 39 1 Commissioner McGaffigan. 2 COMMISSIONER McGAFFIGAN: A related point is that 3 much of the information here is really available to the 4 licensee. The PIMs are or soon will be; LERs, obviously; 5 the economic indicator that the Chairman talked about back 6 on page 17. I assume if we are using publicly available 7 data that they can either replicate it or we could just hand 8 it to them. This is what the economic indicator that we are 9 using is. But this other stuff isn't, because it hasn't 10 been captured anywhere. I think that is where the greatest 11 chance of disconnect is. 12 CHAIRMAN JACKSON: Exactly. That's why you really 13 have to give that focused attention in this area, because 14 everybody needs to know what it is. 15 MR. CALLAN: That's one of the basic ground rules. 16 We don't use information that is not publicly available. 17 CHAIRMAN JACKSON: You do make decisions where you 18 make implicit management judgments. 19 MR. CALLAN: Derivative conclusions, right. 20 CHAIRMAN JACKSON: Without a clear decision-making 21 path for how you got there. 22 MR. CALLAN: That's right. 23 CHAIRMAN JACKSON: You can't unequivocally say 24 that you don't use it, because you do use it. 25 MR. CALLAN: Right. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 40 1 CHAIRMAN JACKSON: The real issue is clarifying it 2 and pulling it out and supporting it. 3 COMMISSIONER McGAFFIGAN: The one place where it 4 struck me that it's not publicly available is allegations. 5 How do you use allegations? 6 MR. CALLAN: The data that we use is the 7 information that we provide the licensees and I presume goes 8 in the docket. 9 MR. COLLINS: Yes, which are numbers in comparison 10 without specifics to the issues. 11 MR. CALLAN: But by category. 12 COMMISSIONER McGAFFIGAN: So the only way we use 13 allegations is, are there a lot of allegations at this 14 plant? 15 MR. CALLAN: And how many of them are harassment, 16 intimidation allegations, how many are technical 17 allegations, that sort of thing. 18 MR. MARTIN: We also focus on substantiated 19 allegations, which would have been communicated. 20 CHAIRMAN JACKSON: Right. So it's not just 21 counting the allegations. 22 Commissioner Diaz. 23 COMMISSIONER DIAZ: On slide 22, I imagine that 24 consistent with Commission guidance the assigning of 25 risk-significant is taking an appropriate priority in the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 41 1 process for any kind of categorization that is made of 2 events or issues. 3 MR. BARRETT: We are giving a lot of attention to 4 making this whole process consistent with the agency's 5 initiative on risk-informed regulation. 6 MR. CALLAN: Let me clarify a point. We weigh 7 substantiated allegations heavily, obviously, but the 8 information we provide licensees includes all the 9 allegations we receive, and we tell them how many are 10 substantiated in various categories. So we give them a 11 pretty good set of data. We've been doing that for a couple 12 of years. 13 MR. COLLINS: We have been doing it for a few 14 years on request and we are doing it routinely now. 15 CHAIRMAN JACKSON: But making sure that you 16 maintain the confidentiality of the clients, et cetera, 17 right? 18 MR. MARTIN: In terms of data, your 19 characterization is absolutely correct, but if we 20 substantiate an allegation and then subsequently take 21 enforcement action, it's very clear they have the facts. 22 They may not know that it came about as an allegation, but 23 they know the fact that will appear in the PIM because it's 24 an enforcement item. 25 COMMISSIONER DIAZ: On slide 22, periodic ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 42 1 headquarters audit of implementation, what is that? Is that 2 a new interface you are creating now to audit the 3 implementation of the program? 4 MR. BARRETT: If we provide guidance to the 5 regions, for instance, on how to do the risk significance, 6 we would want to periodically look at samples of it just to 7 make sure that everyone understands the guidance. Perhaps 8 the word "audit" is a bit strong. 9 COMMISSIONER DIAZ: Does it refer to risk 10 significance in itself or to the entire process? 11 MR. BARRETT: The entire process. We want to make 12 sure that everyone understands what the template categories 13 mean and are implementing them uniformly as well as the risk 14 significance. 15 COMMISSIONER DIAZ: Because the senior management 16 provides themselves an evaluation of the entire process, I 17 was concerned that we might not be reevaluating the 18 reevaluation. 19 CHAIRMAN JACKSON: Let me make sure I understand 20 something here about the comment that you kind of made as a 21 sidebar comment, about what you provide to licensees 22 vis-a-vis allegations. There is a sensitivity issue having 23 to do with not revealing people's names. This is in terms 24 of protection of allegers. You don't mean that you just 25 give all information. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 43 1 MR. CALLAN: No. All we do is give them numbers. 2 We'll tell a licensee that in the 12-month period ending the 3 first of September the NRC received 20 allegations of which 4 5 were H&I. 5 CHAIRMAN JACKSON: I just wanted clarity for the 6 record. 7 COMMISSIONER McGAFFIGAN: I'm sorry to beat this 8 dead horse to some degree, but on allegations, isn't there a 9 chance that there will be time lag? I'm not saying that 10 this is bad. I think it's just a fact of life. You could 11 be in a senior management meeting in January and have a 12 bunch of substantiated allegations that since they require 13 enforcement action you have not shared with the licensee 14 because there is a time lag in enforcement. So it might 15 weigh in your decision as to whether the plant deserves to 16 be discussed. 17 MR. CALLAN: There are two facets to allegation 18 data. One facet gives you a window into the organizational 19 climate of the plant. To that extent, whether they are 20 substantiated or not is almost not real important. If you 21 get a lot of allegations of which a lot are H&I, then that 22 deserves close NRC scrutiny: is there a pathology there at 23 that site involving the management climate? 24 The other facet, of course, is what you are 25 getting to, which is the substance, the technical substance, ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 44 1 and is it enforceable or not. That second facet we handle 2 more routinely through the process, but when we call upon 3 the allegation coordinator who attends the senior management 4 meeting, we are really looking for the first set of 5 insights: what does the allegation data tell us about the 6 health of that organization? That's the insight we are 7 looking for. 8 MR. BARRETT: Let me just briefly on slide 23 9 discuss the question of criteria. As I said, the decision 10 criteria development is in an early stage. We believe that 11 both the trend plots and the templates are important to the 12 decision process. The trends are, of course, amenable to 13 strict thresholds and numerical criteria. 14 Interpretation of the template will require more 15 qualitative criteria and will entail some judgment. Some of 16 the factors that we think are important in these criteria, 17 first of all, would be the number and the risk significance 18 of the issues in a particular category or subcategory. 19 Also the significance of issues as they relate to 20 programmatic problems. So if you have a category that has a 21 lot of issues, no particular issue might be risk 22 significant, but as an aggregate they may point to a 23 programmatic problem. That's a precursor of 24 risk-significant activities. 25 Also the relative importance of various ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 45 1 categories. We know that not all six of these categories 2 would be equally important in a decision regarding the watch 3 list. We need to think about which ones are most important. 4 Finally, relationships among the categories. It 5 could very well be that poor performance in a particular 6 category might be mitigated by good performance in another 7 category, or a combination of two categories that have poor 8 performance might have more significance than some other 9 combination. 10 We have to look at this systemically; we have to 11 develop a decision model and develop qualitative criteria 12 around that model so that we can make this a consistent and 13 scrutable process. 14 We are working with the Office of Research and 15 with Arthur Andersen Consulting on this model. 16 CHAIRMAN JACKSON: When do you anticipate that the 17 guidelines will be available? 18 MR. BARRETT: I think that we would have a draft 19 set of guidelines available in the next couple of months. 20 We would certainly want to have them available before the 21 next time we go back to the ACRS, but I would say in the 22 next month or so. 23 MR. CALLAN: On the schedule chart it shows 24 revised template, revised criteria the end of November. 25 MR. BARRETT: Let me go to the schedule, which is ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 46 1 slide 24. I want to point out a couple of the major 2 features of the schedule. 3 The January senior management meeting cycle starts 4 later this month. We will be providing both the performance 5 charts and the economic trend plots, as I mentioned, for 6 this cycle. 7 In the next cycle, which begins in March of 1998, 8 we will also be putting the template into a trial 9 application using the PIMs data developed during the next 10 six months. 11 Starting in March of next year we plan to have a 12 public comment period. Leading up to that public comment 13 period we will be coming back to the Commission with a 14 Commission paper and a briefing to provide what we have at 15 that time in preparation for the public comment period. 16 Finally, in the late summer of 1998, after 17 completion of the public comment period and after completion 18 of all the trial applications, we intend to come back to the 19 Commission with recommendations on how to proceed from 20 there. By that time we will know more about the integrated 21 review of assessment processes. 22 CHAIRMAN JACKSON: Commissioner Diaz and then 23 Commissioner McGaffigan. 24 COMMISSIONER DIAZ: Looking at the schedule, I 25 just wanted to understand this Commission briefing February ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 47 1 of 1998 prior to the public comment. The process will 2 essentially be finalized by then so that it will be clear 3 that public comments in that process would just be kind of 4 fine tuning prior to the decision. In other words, you 5 intend to have a significant fraction of the process well 6 defined by February of 1998. 7 MR. BARRETT: That's right. We intend to have the 8 whole thing laid out in at least enough detail that we could 9 get significant public comment and industry feedback. 10 MR. MARTIN: Recognizing that by that time we will 11 not yet have had a trial with the template, because the raw 12 data won't be available until the March time frame. So 13 having gone through the complete process with what we think 14 is close to the final really won't be until a June or July 15 time frame. 16 COMMISSIONER DIAZ: The high level decision-making 17 on the processes will have been made. I don't know what I 18 mean by high level. The overriding major considerations. 19 COMMISSIONER McGAFFIGAN: The answer to 20 Commissioner Diaz' question actually makes me more 21 concerned. We had another briefing a few months on medical 22 where the question came up that by the time you put 23 something out for public comment all you really want is a 24 tweak or two and we're not really going to listen to public 25 comment. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 48 1 CHAIRMAN JACKSON: At the same time they need to 2 get started with having some flesh on the bones. 3 COMMISSIONER McGAFFIGAN: They have a lot flesh. 4 I suspect that at least some listeners to today's briefing 5 may go away with heart palpitations, or whatever. It 6 strikes me that if we are going to be an open agency that 7 during this period of the next couple of months, knowing 8 everything has to ultimately be decided, policy decisions by 9 the Commission, that there is no harm in having a lot of 10 dialogue with industry and the public, whoever, about 11 whether we are on the right track. GAO, for that matter. 12 And be relatively open as we search for solutions here. 13 Putting everything off until March when it's perfect or 14 perhaps locked in isn't a good idea. I throw that out. 15 CHAIRMAN JACKSON: I think the point is to reach 16 out and involve the various stakeholders and perhaps build 17 in more open processes. You are going to be briefing the 18 ACRS. Those are open meetings anyway. 19 MR. CALLAN: And we've gotten some valuable input 20 from attendees at the ACRS meetings. We got a good letter 21 from NEI. So we are getting feedback not only from the ACRS 22 but by attendees at those ACRS meetings. 23 COMMISSIONER McGAFFIGAN: I think I'm suggesting 24 in addition to ACRS that we go out and have meetings with 25 -- ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 49 1 CHAIRMAN JACKSON: That's what I'm saying. Build 2 in more structured public opportunity. 3 MR. BARRETT: Sooner rather than later. 4 Let me briefly conclude by reiterating that we 5 believe based on what we have seen so far -- 6 CHAIRMAN JACKSON: And remember, all your 7 stakeholders. 8 MR. BARRETT: We believe that the senior 9 management decisions will be best made with a combination of 10 both the template and the performance trends and that senior 11 management meeting decisions can be made by using numerical 12 criteria in conjunction with the trend plots, which we have 13 already shown give reasonable results but need further 14 refinement to be improved. 15 As I mentioned before, we are working on a 16 decision model and guidelines to be used with the template, 17 and we will report on that in the future. 18 Finally, with regard to economic indicators, we 19 see them as an early warning of potential performance 20 problems, not necessarily as a part of the formal decision 21 process itself, but more of an early warning of potential 22 performance problems. 23 CHAIRMAN JACKSON: Commissioner McGaffigan. 24 COMMISSIONER McGAFFIGAN: I think you are 25 splitting semantic hairs here in that last statement. My ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 50 1 fellow Commissioners are looking at me like I'm saying this 2 for all of them. When you use them in screening meetings, 3 even though perhaps the data is not going be fresh in 4 everybody's mind at the January meeting, they are used early 5 on in the decision process. Maybe you are saying they are 6 not used at the final decision phase, but they are used as 7 an early, gatekeeper phase, it sounds like. 8 CHAIRMAN JACKSON: What is the point in having 9 them to give you early warning if it doesn't at least inform 10 you as you go along in your decisions? 11 MR. BARRETT: I think it does inform you. The 12 point of the bullet is simply to say we don't intend to 13 write criteria that are built around economic indicators. 14 CHAIRMAN JACKSON: How are you going to use them, 15 then? 16 MR. BARRETT: As I said, we are going to provide 17 them to senior managers. I think they could easily be 18 provided to senior managers within the context of the senior 19 management meeting or outside the context of the senior 20 management meeting. They are meant to be information that 21 might provide an early warning. 22 CHAIRMAN JACKSON: I think you've got to come down 23 with a decision on it. You're either going to use it in the 24 senior management meeting process or you're not. No one has 25 to say that you have to use it in the same hard and fast way ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 51 1 that you might use some other indicators, but you need to 2 make a decision. If you mealy-mouth it, then it becomes the 3 stepchild -- you just decide what you are going to do. 4 MR. CALLAN: I don't think we can decide for a 5 while. We don't know enough about it. 6 CHAIRMAN JACKSON: That's fine. It's a process 7 that you are evolving to, but that's true of everything. 8 MR. CALLAN: I think we are all squeamish about 9 economic indicators. 10 CHAIRMAN JACKSON: That's clear, but that's what 11 I'm trying to tell you, that you are probably squeamish 12 about organizational effectiveness, too. 13 MR. CALLAN: Not too much. 14 CHAIRMAN JACKSON: It hasn't been developed to 15 this point, which suggests some lack of comfort in that 16 regard, and as you become more sophisticated, then you can 17 make a judgment as to what extent it really can validly be 18 used or not. You can't address it by backing away from it. 19 MR. CALLAN: Right. 20 CHAIRMAN JACKSON: You have to ensure that you do 21 that evaluation, however you come out. So you don't back 22 away from it. That's not the way you make the decisions. 23 You deal with it; you decide how good it can be in terms of 24 being used; and then you go on from there. But you can't 25 sort of say we're going to do it. It's like being a little ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 52 1 bit pregnant, or whatever. 2 MR. CALLAN: We didn't mean to come across that 3 way. We're going to give economic indicators a fair trial 4 and interact with the Commission and our stakeholders. 5 CHAIRMAN JACKSON: Absolutely. 6 MR. CALLAN: Right now that's the area that we are 7 probably, as I said, the most squeamish about. 8 CHAIRMAN JACKSON: Right. I just want to get out 9 on the table that I know you're squeamish, but you have to 10 go through a robust process and get to whatever the end 11 point is. 12 MR. CALLAN: I understand. Absolutely. 13 MR. MARTIN: Chairman, just another fact. A 14 couple of decisions have to be made. One is whether the 15 plant should be discussed. It may have a different role 16 there, forcing us to discuss them, and it may have a 17 different role in the decision process at the senior 18 management meeting. We'll look at both aspects. 19 CHAIRMAN JACKSON: I guess you are making kind of 20 an artificial distinction, which is what I think 21 Commissioner McGaffigan was getting at. When you use them 22 at some point in the process, you are using them in the 23 senior management meeting process. That is a separate 24 decision as to when you actually sit down in the senior 25 management meeting: Is it part of your go or no-go ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 53 1 decision-making? But the very fact that you use it 2 somewhere, at screening or whatever, you are using it in the 3 senior management meeting process. So let's not split these 4 kind of hairs artificially here. 5 MR. CALLAN: I understand. 6 MR. BARRETT: That concludes my remarks. 7 [Laughter.] 8 CHAIRMAN JACKSON: Thank you. 9 MR. CALLAN: Thank you, Rich. 10 Chairman, in the time remaining I understand that 11 we do have somewhat of a schedule this afternoon. So we 12 would like to proceed to continue with the short-term 13 actions. We have six areas we would like to cover as a 14 result of a direction that the staff has been given in 15 concert with short-term actions. I would like to ask Bill 16 Borchardt to cover those, and then we will quickly go into 17 the integrated review process. 18 MR. BORCHARDT: Slide 26, please. 19 It's the staff's intent to continue making 20 incremental improvements to the senior management meeting 21 process as opportunities arise. This slide shows a number 22 of the significant changes that have been made over the last 23 two years. 24 The first significant change was actually the 25 development of the management directive. Up until a couple ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 54 1 of years ago there was no real written procedural guidance 2 on conduct of the senior management meeting. We started 3 using this guidance in March of 1996 and it was eventually 4 published in a formal manner in June of 1997. 5 There have been a lot of changes to the screening 6 meetings. These changes include a far more active and 7 proactive involvement of the participants at the screening 8 meetings. The major participants now include the regional 9 administrator and the regional staff as appropriate, the 10 director of NRR and staff, AEOD, OI, and OE. All attend the 11 meeting and are active participants. In fact there is an 12 active solicitation of views at the screening meeting. 13 The threshold for discussion has changed slightly 14 or evolved slightly over the last several years. Now the 15 threshold for a plant moving forward to the senior 16 management meeting is, if one of those major participants 17 believes that the plant ought to be discussed, it moves 18 forward. It's not a vote; it's not a preponderance of votes 19 at the meeting. If one believes it ought to be discussed, 20 then that is the decision unless the ensuing discussion 21 makes that individual change their mind. 22 The discussion of plants includes all plants in 23 that region. The meeting takes place over a full day, and 24 the discussion is graded, depending upon the performance of 25 the plant. So the very best performing plants in the region ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 55 1 would get less discussion than the kind of plants that are 2 in the gray area you might describe, those that might be 3 worthy of discussion at the senior management meeting or 4 might not. 5 Rich has discussed that we are moving into 6 consideration of economic indicators, and the next meeting 7 will include use of the trend plots as well as economic 8 indicators. 9 The pro/con charts and the performance evaluation 10 template has served its purpose very well, I think, although 11 there are still some more improvements that need to be made. 12 But it has focused the discussion on objective information, 13 and it has provided a focal point for those discussions at 14 both the screening meeting and the senior management 15 meeting. 16 The active participation and documentation of 17 decisions started in real earnest at the January 1997 senior 18 management meeting. It includes an active facilitation by 19 the EDO and by the director of NRR and increased level of 20 interaction among all the participants and a focused 21 discussion through those pro/con charts and the site and 22 removal matrix for those plants that have been currently 23 categorized as category 2. 24 The bottom two bullets on this slide mention two 25 topics that the Commission has addressed recently. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 56 1 The plant issues matrix. We have recommended and 2 are moving forward with making that public beginning the 3 spring of next year. That will allow the regions to begin 4 developing the data, beginning next week actually, and so 5 we'll have six months at the time the next PPRs happen, and 6 then we will make that six months of information eventually 7 turn into at least a full year as the data is accumulated. 8 On the subject of trending letters, our 9 recommendation to the Commission is that we do not change 10 the policy at this time, although we recognize that the 11 trending letters and the PIM are both very important parts 12 of the integrated review that I am going to discuss next. 13 Slide 28, please. 14 What I would like to do next is provide a brief 15 overview of the integrated review that we are conducting on 16 the NRC's assessment processes. 17 Although this review is going to focus on the four 18 specific programs that are listed, I think we need to 19 constantly remind ourselves of the importance of the 20 inspection program and the basic inspection procedures and 21 inspection results that really form the factual base on 22 which all these other assessment processes operate. 23 The review effort grew out of a number of SRMs 24 that the Commission has provided to the staff recently. 25 Some were very specific in nature and some were rather wide ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 57 1 ranging. It was our suggestion that we conduct this 2 integrated review in order to give each of those SRMs the 3 appropriate consideration that they deserve. 4 The staff is really very excited about this 5 effort. It's a valuable opportunity, I think. It's the 6 first time that we have taken advantage of the opportunity 7 to do an integrated review of all these processes. I'm 8 going to very briefly go through these four processes. 9 They were all started at discrete points in time 10 for unique purposes. What has never been done is to look at 11 the cumulative effect both on the staff and on the industry 12 and on the public as an information source. So this is 13 really an opportunity to make life more effective and more 14 efficient for everyone involved. 15 Starting off with the SALP, it was implemented in 16 1980 following the TMI event. There have been numerous 17 changes over the last 18 years. At one point there were 17 18 SALP functional areas. The reports ranged 40 to 50 pages in 19 length. Now we have the four SALP areas that you are 20 familiar with: operations, engineering, maintenance, and 21 plant support. The interval now is normally 18 to 24 22 months, depending on plant performance. 23 The SALP has always served -- the major goals have 24 always included the allocation of inspection resources and a 25 communication tool with the licensees and the public. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 58 1 The SALP is conducted primarily by the regional 2 management with participation from NRR project management. 3 The senior management meeting was first 4 implemented in the mid-1980s following the Davis-Besse 5 event. Rich Barrett has discussed a lot of the details of 6 programs that are under development. It's held every six 7 months. It allows the senior managers to focus on the 8 plants of most concern, and one of the major outcomes is the 9 identification of an agency plan to address those plants. 10 One important point is that it has always been a 11 supplement to the normal regulatory process. We have never 12 waited to make a safety decision for the senior management 13 meeting to occur. The regional administrators, the NRR, the 14 major program offices all have a role in day-to-day 15 oversight and regulatory responsibility which are not 16 delayed in any way and never have been by the senior 17 management meeting. 18 Slide 29, please. 19 The plant performance reviews are largely a 20 regional effort. There is some NRR participation. It was 21 initially implemented in October of 1990, and it provides 22 the regional inspection staff primarily an opportunity to 23 perform midcourse corrections based upon the six-month 24 review of plant performance. 25 It's the intent that if licensee performance ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 59 1 weakness is identified in any particular area that the 2 inspection plans for that region would be adjusted 3 accordingly; we'd have some focused inspections; and 4 resources could be allocated, moved from one plant to 5 another, depending upon the real time perceptions of 6 performance by the region. 7 Renewed interest in it came out of the South Texas 8 Lessons Learned Task Force. One of the major findings of 9 that was that the NRC had the inspection information; we had 10 the findings, but we never put it together in an adequate 11 form to allow us to make decisions and put all the pieces of 12 the puzzle together. 13 The plant issues matrix is a significantly newer 14 initiative. It was implemented across the regions in the 15 spring of 1996. It's a listing of both the positive and 16 negative findings and conclusions out of the inspection 17 report. 18 Again, everything in the PIM has to be on the 19 public record. It doesn't have to be in an inspection 20 report; it could be in a document from NRR to the licensee 21 regarding a license amendment or some other document. But 22 it has to be on the docket. And it lists both positive and 23 negative findings. So there is an attempt to have some 24 balance, although by the nature of our job we do a much 25 better job identifying the weaknesses. You should expect ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 60 1 that a plant issues matrix will have a majority of issues 2 that identify concerns and weaknesses. 3 The spring of next year the PIMs will be made 4 public. 5 Go to slide 31. 6 Both slides 30 and 31 run through some of the 7 preliminary strengths and weaknesses identified by the staff 8 and the regions. 9 The integrated review group will do a far more 10 thorough job of identifying both the strengths and the 11 weaknesses of these programs, but in general, we know there 12 are some problems that we want to address through this 13 integrated review. 14 There is an element of redundancy among the 15 programs that we would like to minimize. I don't think we 16 can reduce it entirely. There is going to need to be some 17 overlap, but we want to reduce redundancy as much as we can. 18 We certainly want to reduce the level of different criteria. 19 We have already mentioned today we have a new 20 template configured. We have four SALP functional areas. 21 There is a strong argument that says we ought to assess 22 plants using the same criteria. Those four SALP functional 23 areas were created for a reason over time and the template 24 was created for a different reason. Now is the time to 25 reconcile it and come up with a single approach and start ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 61 1 using that on a routine basis. 2 There is the potential for inconsistent 3 implementation among the various regional offices and the 4 major program offices, and it's highly resource intensive. 5 Each of these programs start off with an objective in mind 6 of how much effort it's going to take, and inevitably it 7 takes more, and it keeps growing and growing and growing. 8 So this is an opportunity for us to kind of rebaseline our 9 resource efforts. 10 Slide 32. 11 This slide lists some of the attributes that we 12 would like to maximize and others that we want to try to 13 minimize. It's likely that there is going to have to be a 14 balance and some tradeoff between them. We won't get 15 absolutes on any of these. The review team has a difficult 16 task in front of them to try to meet these objectives. 17 The team will also be developing, to the extent it 18 can, some quantitative criteria to measure improvements in 19 the processes, some goals that we can establish, especially 20 regarding how many resources it takes the NRC staff to 21 complete these programs. I think this is one of the 22 valuable comments we received from the ACRS recently. So we 23 have just begun to try to come up with some of those 24 criteria. 25 The attributes to maximize include trying to come ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 62 1 up with a single assessment process, or at least a continuum 2 of discrete processes that could be used that relate to each 3 other and eliminate some of the unnecessary overlap; and 4 early identification of declining performance. 5 We'd like to have a clear understanding of roles, 6 especially within the NRC staff. This effort is a very 7 natural follow-on to the job task analysis we recently 8 completely in the regions. We're in the early stages now of 9 reviewing the documented results of the job task analysis. 10 So this fits in very well with that effort. 11 Of course we want to maximize the open dialogue 12 and use these tools as an effective communication device 13 both with the industry and the public. 14 Attributes to minimize include lessening the 15 opportunity for inconsistent assessment criteria, 16 eliminating overlapping responsibilities, trying to ensure 17 we have more consistent implementation, and trying to 18 eliminate as much as we can opportunities to send 19 conflicting messages to licensees. 20 Slide 33, please. 21 There are a few what we are calling boundary 22 conditions for this review. I need to say, I think, largely 23 the group is starting with a blank sheet of paper. There 24 are very few restrictions that we are placing on them. So 25 we are not tying the process that will come out of the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 63 1 integrated review effort to any status quo program that we 2 have now. 3 We are saying, though, that the inspection program 4 and the enforcement policy are being assumed to be 5 fundamentally sound at this point. This doesn't mean at all 6 that we think it's adequate to fulfill what will come out of 7 the process. It's an obvious follow-on activity that we 8 will have to identify gaps in the current inspection 9 program. We already talked about one today. If we look at 10 management effectiveness in the new template, there is 11 nothing in the inspection program now that directly inspects 12 that activity. It's an inferred judgment under the current 13 process. 14 We are going to have to identify new procedures, 15 new guidance, and then train the staff. So this isn't going 16 to be a quick solution to these identified weaknesses or 17 gaps in the program. 18 CHAIRMAN JACKSON: But it won't go on for a 19 decade. 20 MR. BORCHARDT: No. I hope not. 21 We know that these groups of processes are going 22 to have to assess all plants. The topic of using it for 23 public interaction and the opportunities for licensees to 24 respond is kind of the element that is currently in the SALP 25 program. We want to retain some element of that. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 64 1 CHAIRMAN JACKSON: Commissioner McGaffigan. 2 COMMISSIONER McGAFFIGAN: Two questions, one on 3 the second point. You said obviously we are going to change 4 the inspection program to try to take into account 5 management effectiveness and you are already doing that as a 6 result of a short-term action. Do you envision the 7 integrated assessment making recommendations with regard to 8 the inspection and enforcement program? 9 You are not saying you are reviewing them, but you 10 are going to end up with something that may not match up. 11 So you are going to have to make some recommendations at 12 least preliminarily how they might have to adjust to 13 whatever you are proposing. 14 MR. BORCHARDT: I would be personally satisfied if 15 they identify the gaps, where we need to do more work. I 16 think it might be too much to ask of them to come up with 17 recommended fixes given the time schedule that we are trying 18 to do this on and the number of resources we are applying to 19 it. It would be a quick turnaround activity for the 20 Inspection Program Branch and NRR to take those identified 21 gaps and then come up with a program to fill those. 22 COMMISSIONER McGAFFIGAN: My second question is on 23 the third tick, the performance of all plants categorized. 24 Is "categorized" a synonym for "scored"? Are there 25 conceivably two categories, the watch list and everyone ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 65 1 else? 2 The reason I raise that question is people come in 3 and tell me that being SALP-1 is a great motivator for the 4 folks at their plants. I went through this conversation the 5 other day with a licensee. When I was at Harvard and they 6 implemented pass-fail my senior year, we treated pass-fail 7 courses very differently from the courses where we got A's 8 and B's and C's, although I guess nowadays everybody gets 9 A's. 10 [Laughter.] 11 COMMISSIONER McGAFFIGAN: I actually think there 12 is a benefit to the old grading system where you are a tough 13 grader and you give out A's, B's and C's. 14 Is categorization a synonym for scored? 15 MR. CALLAN: Let me answer that. We have had a 16 lot of discussions on that subject. The first observation 17 I'll make is that the resources go up exponentially with the 18 way you parse scoring. If you have four categories and you 19 want to have a credible scoring system, it takes more than 20 four times as many resources to do that probably than it 21 would be to have one score, or to break performance down 22 into quartiles, top quartile, second quartile, third 23 quartile, fourth quartile. 24 I think what we can say at this point is there 25 will be some kind of ranking. How we score it is another ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 66 1 matter. We rank plants now as part of the screening 2 meeting; regions rank the plants. That's how you get the 3 best performers that you spend less time on. 4 That's a tough question, and it's linked, as I 5 said, directly to resources. 6 MR. COLLINS: Commissioner, there are a lot of 7 ways to look at evaluation of plants. Even with the 8 periodic reviews that are done we de facto rank plants by 9 assigning inspection resources. When the PIM becomes public 10 and our responses to the PIM, which is a resource-loaded 11 letter to the licensees, become more routine, there will be 12 a rack-up of the effort of plants. 13 I think part of coming to a conclusion of how do 14 we rank or grade plants becomes more of what do we intend 15 for the process to achieve beyond the allocation of 16 resources and the communication of performance to the 17 licensees and what's the most effective way to do that. 18 I understand some licensees come in and are 19 motivated by category 1's. I, quite frankly, haven't heard 20 that. I've heard the other side of the argument. 21 COMMISSIONER McGAFFIGAN: Maybe they only talk to 22 me, but it is more than a handful who seem to be motivated 23 to get to straight SALP-1. They believe that's a very 24 useful tool for motivating their workforce. 25 CHAIRMAN JACKSON: I guess I believe that we've ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 67 1 asked you to do it, so now we ought to let you do it and 2 come back to us with what you come back with. 3 MR. BORCHARDT: Slide 34, please. 4 The process is really rather simple. NRR has the 5 project lead. There is going to be a series of meetings. 6 Right now it's envisioned to be four. It may end up having 7 to be a few more than that. 8 The participants right now are shown on backup 9 slide number 6. They include participants from each of the 10 regions and some of the other program offices. 11 Even if program offices are not particularly 12 listed on that slide, there will be a series of less formal 13 meetings held here in headquarters with the other offices so 14 that they are all apprised of the ongoing results of each of 15 the steps and we can receive their input. 16 The schedule runs over the next 18 months or so. 17 The development will largely be, at least the initial 18 development of staff options, over the next months. May of 19 next year is a time period for public and industry comments. 20 The activities from around June, about the summer 21 of next year, until the end of the year are largely being 22 allocated for the development of new procedures, management 23 directives and training for the staff. 24 The eventual outcome is as significant as it could 25 be. It's going to be a significant mind-set change for the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 68 1 staff and the industry, but for the staff to implement, a 2 staff that is now very comfortable, I think, to a large 3 degree with it. Although highly labor intensive, they have 4 a lot of practice with it and they are comfortable with it. 5 So if we change it, it's going to be a significant training 6 evolution to get everybody adapted into the new program. 7 That completes my presentation. 8 CHAIRMAN JACKSON: Thank you. 9 Commissioner Dicus. 10 COMMISSIONER DICUS: I realize we are running over 11 and there are some other things that have got to be done. 12 Back to the senior management meeting. You 13 mentioned the ACRS had quite a few concerns that they have 14 brought to your attention, and you indicated that you would 15 probably be looking at these and bringing them into play at 16 some point and evaluating them. One of them in particular 17 was the idea that maybe your overall approach, which was 18 kind of from the bottom up, should be from the top down. 19 Would you care to comment on that? 20 MR. BARRETT: Yes. We actually have given a fair 21 bit of thought to a top-down approach although we haven't 22 emphasized it in our information. 23 We have been working with the Office of Research, 24 for instance, to develop a decision model. We feel that the 25 decision model is really the way in which you take what it ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 69 1 is right now, a list of categories, and integrate them into 2 what we believe is in fact a comprehensive structure for 3 performance assessment. 4 Backup slide number 4, for instance, gives you an 5 example of some of the thinking we've done along that line 6 in terms of trying to take these categories and show the 7 relationships among them and also show how they relate to 8 risk. In this particular slide what we are trying to do is 9 show how risk is a combination of three of these categories 10 and how they are supported by two of the other ones, namely, 11 engineering and organizational effectiveness, and then 12 problem identification and resolution as a feedback 13 mechanism. 14 So we have done some thinking about this and we 15 are committed to coming up with a model that is truly an 16 integrated model in response the ACRS as well as our own 17 motivation. 18 MR. KNAPP: I might add that Research is getting 19 very much involved in that particular concern the ACRS 20 raised, and hopefully we are providing a fair amount of 21 support to AEOD and doing, I think, more or less what they 22 have said, looking at the decision and then backing into the 23 template so that we are coming at this from both ways. I 24 think this will work pretty well by the time we are all 25 done. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 70 1 CHAIRMAN JACKSON: Commissioner Diaz. 2 COMMISSIONER DIAZ: I have a couple of comments 3 mixed with questions. In the interest of time, rather than 4 answering the questions, I will ask the staff to consider 5 them when they are putting their things together. I think 6 it is obvious the preoccupation that the senior management 7 process has created inside and outside this organization. 8 I think when we get to February 1998 there are a 9 series of questions that are important that we answer for 10 ourselves and for the outside. Maybe a basic question is, 11 what is the senior management process? 12 We need to be able to come and eventually define 13 it in a manner that is understandable to us and to our 14 stakeholders and to the Congress of the United States and to 15 everybody that really deals with it, including the press. 16 It is obvious that the senior management meeting 17 is no longer a meeting of senior managers; it has become a 18 much more elaborate process. The amount of time and 19 resources that it uses and this effort show its importance 20 in our organization. In 97-04, I think I recommended that 21 we change the name. I can't even remember. National 22 evaluation of licensee performance, which nobody liked; or 23 national assessment of plant performance. 24 CHAIRMAN JACKSON: He's trying to get them in 25 again. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 71 1 COMMISSIONER DIAZ: I wouldn't do that. 2 [Laughter.] 3 COMMISSIONER DIAZ: National assessment of plant 4 performance. Two people liked that. 5 [Laughter.] 6 COMMISSIONER DIAZ: In this regard, in answering 7 that question, I think there are two fundamental questions 8 in what is the senior management meeting or the national 9 assessment of plant performance. 10 [Laughter.] 11 COMMISSIONER DIAZ: And that is, what does this 12 process provide in terms of health and safety? What does 13 the actual process provide for this agency in terms of 14 health and safety? 15 When we answer that, then we need to ask ourselves 16 and reply, what does the watch list provide in terms of 17 health and safety? 18 I think those are very important questions that 19 really have been brought out even as early as yesterday by 20 Senator Biden's concern, which we need to reply to, and 21 those concerns need to be addressed and they need to be 22 addressed earlier rather than later, because I believe we 23 might have to be accountable to Congress in early 1998. So 24 February 1998 becomes an important date for the Commission 25 to have the appropriate information. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 72 1 Obviously when the process was created we did not 2 intend it to generate this much attention, but it has and it 3 is there. 4 As you further define these processes, I recommend 5 that we be able to establish clearly and for some reasonable 6 period of time, to us and to everybody, this basic question: 7 Is the senior management meeting, or the national assessment 8 of plant performance, a dominant inspection, assessment, 9 enforcement, and regulatory process? 10 If it is, let it so be known, and let the 11 Commission decide on whether that is what we want to do. If 12 it's not and your recommendation is that it not be, let it 13 so be known so that it occupies its proper place in our 14 regulatory infrastructure. 15 I believe that you have gone quite forward. I 16 think that the process is now converging, and I want to 17 thank you for your efforts in this regard. 18 CHAIRMAN JACKSON: Thank you. 19 I would like to thank the staff for a very 20 informative and very interactive briefing. These processes 21 play a vital role in helping to develop and to provide an 22 agency perspective on plant performance. 23 You've already heard we want you to continue along 24 the line you have been moving. I think there needs to be 25 additional focus on the issue of the decision-making process ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 73 1 at the senior management meeting. I think to some extent it 2 was finessed in this discussion. 3 It's my understanding that information that will 4 be available at the upcoming senior management meeting, and 5 I think you've indicated it, includes the template, the 6 performance trends, and the economic data plots. I think 7 you need to try to give the Commission -- no? 8 MR. MARTIN: Not the template. 9 CHAIRMAN JACKSON: Not the template. Okay. 10 You should provide to the Commission prior to the 11 next senior management meeting a little more information, as 12 much as you have developed, on how you intend to use the 13 information to reach decisions, what is the actual 14 decision-making process, and that you consider the 15 information and the suggestions provided in the recent ACRS 16 letter regarding the template and the senior management 17 meeting process. 18 There are some issues having to do with being able 19 to have the Commission have more information perhaps in a 20 graphical form that provides information about both false 21 alarm as well as detection processes as a function of the 22 observation periods. There was a discussion about how long 23 your data goes over. It's really kind of casting the 24 information in a somewhat different format. 25 As Commissioner Diaz' comments illustrated, I ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 74 1 think the important point is that we have clarity on what we 2 are trying to accomplish with the various processes. The 3 senior management meeting, I don't just focus on that, but 4 it's kind of a culmination point that everyone seems to 5 focus on. We have an overall set of processes, and I was 6 happy to hear Mr. Borchardt say how excited the staff was at 7 this opportunity to reassess. 8 The real point is, given our health and safety 9 mission, you should ask yourselves the following question. 10 I think this gets at what the Commissioner says. Do the 11 outputs give us the outcomes that we desire from the point 12 of view of our health and safety mission? And is it 13 consistent with our agency goals? And does it allow us to 14 have a measurement of the health and safety value added? 15 If you can do that and keep that in mind as you go 16 through this, then I think we will be in good shape at the 17 end. 18 I think Commissioner Diaz wants to make one last 19 comment. 20 COMMISSIONER DIAZ: Madam Chairman, I'm sorry, but 21 you said something that triggered my mind and I think it's 22 an important issue. As we define what these processes are, 23 I think it should be very, very clear to the stakeholders, 24 to the public, to the Congress that the senior management 25 meeting, however we cast its importance, is just one of the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 . 75 1 processes. 2 CHAIRMAN JACKSON: Right. 3 COMMISSIONER DIAZ: It doesn't take away 4 importance or weight or value from the day-to-day 5 inspections and assessments that are done by our people on 6 the line every day, and the decisions of health and safety 7 will be done on a daily basis independent of whether the 8 senior management process takes place, because I think that 9 is an overriding consideration. 10 CHAIRMAN JACKSON: It's something that people 11 misunderstand, but I think showing the connectivity from the 12 beginning to the end of everything we do helps to remove the 13 excessive focus on any given part of the process. 14 Unless there are further comments, we are 15 adjourned. Thank you. 16 [Whereupon at 3:16 p.m. the meeting was 17 concluded.] 18 19 20 21 22 23 24 25 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
Privacy Policy |
Site Disclaimer |