1
1 UNITED STATES OF AMERICA
2 NUCLEAR REGULATORY COMMISSION
3 - - -
4 BRIEFING ON
5 SENIOR MANAGEMENT ASSESSMENT PROCESS
6 FOR OPERATING REACTORS
7 - - -
8 PUBLIC MEETING
9 Nuclear Regulatory Commission
10 One White Flint North
11 Rockville, Maryland
12 Friday, September 19, 1997
13 The Commission met in open session, pursuant to
14 notice, at 1:30 p.m., Shirley A. Jackson, Chairman,
15 presiding.
16 COMMISSIONERS PRESENT:
17 SHIRLEY A. JACKSON, Chairman of the Commission
18 GRETA J . DICUS, Commissioner
19 NILS J. DIAZ, Commissioner
20 EDWARD McGAFFIGAN, JR., Commissioner
21 STAFF PRESENT:
22 JOHN C. HOYLE, Secretary of the Commission
23 STEPHEN BURNS, Deputy General Counsel
24
25
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 2
1 PRESENTERS:
2 JOSEPH CALLAN, EDO
3 SAM COLLINS, Director, NRR
4 R. WILLIAM BORCHARDT, Chief, Inspection Program
5 Branch, NRR
6 MALCOLM KNAPP, Acting Director, RES
7 THOMAS MARTIN, Director, AEOD
8 RICHARD BARRETT, Deputy Director, Incident
9 Response Division, AEOD
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 3
1 P R O C E E D I N G S
2 [1:30 p.m.]
3 CHAIRMAN JACKSON: Good afternoon, ladies and
4 gentlemen. I'm pleased to welcome members of the NRC staff
5 to brief the Commission on their activities vis-a-vis
6 improvements to the senior management meeting process, and
7 in particular the staff's plans for and the results of an
8 integrated review of the NRC assessment process for
9 operating commercial nuclear reactors.
10 As we all know, the senior management meeting
11 process is intended to facilitate the early identification
12 of plants which require increased regulatory attention.
13 The Commission previously has indicated its belief
14 that there is room for improvement in the senior management
15 meeting decision-making process. These improvements relate
16 to making the process more scrutable and using objective
17 data with well defined decision criteria. The objective
18 should be to obtain a clear, coherent picture of performance
19 at operating reactor facilities.
20 The staff will describe its current activities to
21 support these objectives as well as the disposition of the
22 Arthur Andersen recommendations.
23 The staff also should describe any incremental
24 improvements to the process that have already been
25 accomplished.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 4
1 Staff assessment processes other than the senior
2 management meetings include the systematic assessment of
3 licensee performance (SALP), the plant performance reviews
4 (PPRs), and the plant issues matrix (PIM).
5 The staff will discuss the strengths and
6 weaknesses of each of the processes and its plans for
7 conducting an integrated review of the NRC assessment
8 process for operating commercial reactors.
9 I understand that copies of the slide presentation
10 are available at the entrances to the room, and unless my
11 colleagues have any comments they would like to make,
12 Mr. Callan, please proceed.
13 MR. CALLAN: Thank you, Chairman. I am not going
14 to try to repeat Hugh Thompson's virtuoso performance this
15 morning and personally walk you through all these slides. I
16 intend to turn the discussion over to Sam Collins.
17 I will say, though, as noted by the introduction,
18 two offices have involvement in this effort, the Office of
19 NRR and AEOD. NRR, of course, has the lead and hence I'm
20 going to ask Sam to lead the discussion.
21 MR. COLLINS: Good afternoon. As you will see
22 during the presentation today and hear from members of the
23 staff, the development of the information base for the
24 senior management meeting process and the integrated review
25 are currently being performed in parallel and somewhat
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 5
1 independently. There is broad office participation and
2 involvement in this activity. Along with the members at the
3 table here today, we have a representative of the regions,
4 Mr. Ellis Merschoff from Region IV, who is in attendance,
5 representing the region and has been involved in the
6 process, as I mentioned.
7 CHAIRMAN JACKSON: And you have Research
8 represented.
9 MR. COLLINS: Yes.
10 NRR is committed to coordinating these two efforts
11 so that a new assessment process can be developed and
12 implemented in a timely manner. We believe that the
13 evaluation tools that are being developed by AEOD for the
14 senior management meeting process are tools that can be
15 applied equally to any new assessment process.
16 It is important to realize that there is more to
17 these efforts than simply developing the new process. It
18 will also take a concerted effort and a significant effort
19 devoted to developing these new management directives both
20 procedurally and with staff training. That has been
21 outlined previously to the Commission.
22 These include involving and achieving full support
23 and cooperation of the stakeholders, including the industry,
24 to ensure that successful implementation on an improved
25 process is a success.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 6
1 At this time I would like to turn the briefing
2 over to Mr. Rich Barrett.
3 MR. BARRETT: Thank you, Sam.
4 If I could have slide 6, please.
5 Good afternoon, Chairman, Commissioners. Today's
6 briefing deals with the development of an objective set of
7 indicators and standards for senior management meeting
8 decisions. We are looking to produce early indications of
9 performance problems and to promote consistency in
10 decision-making and scrutability of the basis for those
11 decisions.
12 This is a program which the Commission first
13 requested in June of 1996 and which the Commission has
14 repeatedly endorsed in subsequent SRMs.
15 The work I will describe has principally involved
16 AEOD staff in both the Incident Response Division and the
17 Safety Programs Division.
18 Significant support for this effort has been
19 provided by the Office of Research, primarily from their
20 experts in risk assessment, human performance, and
21 organizational effectiveness.
22 AEOD has employed the Idaho National Engineering
23 and Environmental Laboratory for statistical support and
24 Arthur Andersen Consulting for independent assessment.
25 The work has been overseen by NRR, who will be our
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 7
1 principal customer, and has had the benefit of continuous
2 regional involvement and regional oversight.
3 Slide 8, please.
4 I want to begin by reviewing the information that
5 NRC has available for performance assessment and by showing
6 how it relates to the products that we are developing.
7 On the left of this slide is a list of our
8 principal information sources, including the inspection
9 program, licensee event reports, and several other areas
10 listed here. I should point out that there are many other
11 sources of information that I have not listed.
12 From these sources we derive two types of
13 objective data as shown in the middle column. First there
14 are indicators, which are quantitative measures of
15 performance, such as the number of safety system failures,
16 or quantitative measures of economic stress, such as the
17 cost per kilowatt hour of electricity generated.
18 The other type of objective information are what
19 we are calling issues. These are qualitative findings but
20 nonetheless objective.
21 An example of an issue that might appear in the
22 plant issues matrix of a region would be an observation on
23 the part of an inspector that a licensee failed to restore a
24 system to its proper configuration following maintenance.
25 The category called issues represents the majority
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 8
1 of the information we have available to us.
2 We are developing methods to structure all of this
3 information, the issues and the indicators, to make it
4 scrutable, and we are developing criteria and guidelines to
5 assist senior managers in making decisions which are
6 objective and consistent.
7 The three methods we are developing or the three
8 products we are developing are shown in the right-hand
9 column. They are the performance template, the performance
10 trend methodology, and the economic trend methodology.
11 CHAIRMAN JACKSON: Are licensee self-assessments
12 folded into this at all?
13 MR. COLLINS: Licensee self-assessments could be
14 folded into this. We haven't done anything explicit at this
15 point.
16 MR. CALLAN: A fundamental ground rule, Chairman,
17 is that the information be publicly available; it has to be
18 in the docket file. To the extent that licensee
19 self-assessment insights are captured, and there are various
20 ways of doing that, either through inspection reports, by
21 the licensee submitting it to us under a cover a letter,
22 which happens from time to time, to the extent that that
23 information gets into the docket, then it is eligible to be
24 used in this process.
25 CHAIRMAN JACKSON: Are the results or issues that
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 9
1 arise from those self-assessments ever captured in
2 inspection reports?
3 MR. CALLAN: They are. We have specific
4 inspection modules. I happen to have memorized one
5 inspection, 40501 -- don't ask me why I remember that --
6 which is specifically directed at NRC follow-up of licensee
7 self-assessments to independently validate and verify the
8 effectiveness of them. That particular inspection module
9 essentially requires that we provide the significant
10 insights of that self-assessment into the docket, if not the
11 actual self-assessment. That's a judgment that has to be
12 made.
13 I'm guessing here. I think probably more often
14 than not the path of least resistance is for the licensee to
15 submit the self-assessment to us.
16 MR. BURNS: Chairman Jackson, might I add
17 something here? An issue that has come up over the years,
18 and in fact I think was an issue with Northeast Utilities in
19 terms of some early assessments, was their status as public
20 documents. At times licensees may submit them to us with a
21 request for a treatment as proprietary information under
22 FOIA Exemption 4, and there is legitimate treatment. There
23 is a body of law that is developed basically out of the
24 medical arena and other arenas which permits that type of
25 proprietary treatment.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 10
1 I think what Joe is saying is important in terms
2 of the public nature, but there may be a tension in terms of
3 some licensees with respect to their willingness to allow or
4 wanting their self-assessments in the public domain. There
5 has been some dialogue even more recently on those types of
6 things as well.
7 MR. CALLAN: It is probably worth further
8 digression here. This is a very important issue. There is
9 another degree of tension, and that tension has to do with
10 the chilling effect that a requirement from us for a
11 licensee to place self-assessments in the docket, the impact
12 of that requirement on the licensee's willingness to bare
13 their soul, so to speak, and to provide brutal critical
14 self-assessments. We are always concerned about that
15 aspect.
16 CHAIRMAN JACKSON: What is the nexus between that
17 and the inspection module?
18 MR. CALLAN: The inspection module that I
19 mentioned is a module that was created about three years ago
20 that enables us, under very restricted conditions, to
21 embrace a licensee's self-assessment, to take credit for it,
22 so to speak, to leverage our resources. In order to do
23 that, we have specific requirements levied. We have to do
24 some independent verification, validation; portions of the
25 self-assessment, if not the entire self-assessment, should
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 11
1 be made publicly available. If certain conditions are met,
2 we would then use those insights, capture them as our own
3 and act on them.
4 CHAIRMAN JACKSON: So they actually would play
5 into helping to develop the performance issues part of the
6 objective?
7 MR. CALLAN: Absolutely. In fact, let me give you
8 one example of kind of an extreme case. At the Cooper
9 Nuclear Station we used in effect a licensee's
10 self-assessment that we validated. The validation team
11 leader is behind me, Ellis Merschoff. That self-assessment
12 was put on the docket. We used that, and it was very
13 insightful. We have a similar effort, as you know, ongoing
14 at the Clinton Station. That's an extreme case. There are
15 lesser cases than that.
16 MR. BARRETT: Let me move on to slide 9. I would
17 like to talk for a little while about the performance trend
18 methodology.
19 Our performance trend method has been developed in
20 response to specific Commission guidance which is listed in
21 detail in slides 9 and 10. For instance, we have gone back
22 and done a complete reevaluation of each candidate
23 indicator. We actually looked at over 50 candidate
24 indicators. We looked at them from the perspective of
25 objectivity, their ability to resolve plant performance
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 12
1 issues, face validity, and also statistical correlation with
2 past senior management meeting results.
3 We have evaluated different ways of combining
4 indicators to produce trend plots. We have settled on two
5 ways, which I will be showing you in just a little while.
6 We have examined the use of various time periods
7 over which to aggregate and integrate the data. We have
8 looked at different weighting schemes, and we have also
9 looked at the issue of using fixed standards for comparison
10 versus floating standards. Again, we'll discuss that in a
11 moment as well.
12 In response to the Commission's request, we are
13 having this methodology peer reviewed by the ACRS. We plan
14 to submit it for public comment, at which time I am sure we
15 will also get a fair bit of industry feedback, and we plan
16 to hold a public workshop on it. In fact actually all of
17 this is going to be applied to the template and the economic
18 indicators as well.
19 We have done a preliminary benchmark of our
20 preliminary methods here against past senior management
21 meeting results, and I will show you some of that.
22 Also we plan, in accordance with Commission
23 guidance, to do a trial test of this methodology in the
24 January 1998 and the June 1998 senior management meeting
25 cycles.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 13
1 CHAIRMAN JACKSON: I think Commissioner McGaffigan
2 has a question.
3 COMMISSIONER McGAFFIGAN: It may be more in the
4 form of a comment, because I've said it before at meetings
5 on this subject. The statistical correlation with past
6 senior management meeting results worries me a little bit
7 because there is a presumption that those results were
8 correct, that we got the right discussion plants, that we
9 got the right trending letters, that we got the right people
10 on the list, and obviously we have been criticized that we
11 haven't done that. So I'm not sure whether statistical
12 correlation with past senior management meeting results is a
13 good thing or a bad thing. You're going to have to convince
14 me.
15 MR. CALLAN: ACRS has made the same point,
16 Commissioner.
17 COMMISSIONER McGAFFIGAN: In fact I made the same
18 point in a letter in today's reader.
19 MR. BARRETT: I think the point is a very good
20 one. I can only say a couple of things regarding that.
21 First of all, the Arthur Andersen report did say that from
22 the perspective of the discussion plants that every
23 indication from their study was that we had done a good job
24 of identifying plants for discussion, whereas we may have
25 been slow in putting plants on the watch list, for instance.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 14
1 The correlations we did were against the discussion plants.
2 COMMISSIONER McGAFFIGAN: I don't have the report
3 fresh in mind, but one of the examples -- they had sort of
4 three charts in their report, as I recall, and one of them
5 was a plant that we had on the watch list and we took it
6 off, and then its performance deteriorated worse than it
7 ever had been by their indicators, and we never discussed it
8 again.
9 The trouble with making a general statement is,
10 yes, maybe in general we got the right discussion plants,
11 but one of the three cases that they chose to highlight was
12 one where, whichever plant it was, we did not discuss it at
13 a time when their model would have called for at least
14 discussion if not getting it back on the list. That's my
15 recollection of one of the three examples they highlighted.
16 MR. BARRETT: That's correct. One of the things
17 that we did to try to test this question of whether the
18 discussion plant list was a good list -- Again, there is
19 always a certain amount of circularity here, because there
20 is no ground truth, there are no tablets in stone that say
21 which plants are good and which plants are not. We took
22 some of our indicators and we did what we call a clustering
23 analysis. We took indicators that we felt have face
24 validity, that is to say, indicators that we thought of and
25 by themselves were good indicators of performance.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 15
1 We went through a statistical process that I don't
2 fully understand and I wouldn't even attempt to describe
3 today to see if those of and by themselves in combinations
4 would nominate certain plants for discussion without any
5 reference to past discussion lists.
6 It turned out that the ones that they nominated in
7 these combinations correlated pretty well with past
8 decisions. Again, it is still circular and I don't think we
9 ever get the ground truth.
10 COMMISSIONER McGAFFIGAN: One question rather than
11 a comment. There are some proprietary indicators out there
12 and they are used when people come in to talk to
13 Commissioners. One of them is the WANO overall performance
14 indicator. I understand it's proprietary, but I understand
15 we also have access to it through the arrangements we have
16 with INPO. Have we looked at the proprietary indicators and
17 said, wow, one of these is so good in predicting -- I'm
18 sorry if I preempted your question.
19 COMMISSIONER DICUS: That's all right. Go ahead.
20 COMMISSIONER McGAFFIGAN: Have you looked at
21 propriety indicators, in particular the WANO overall
22 performance indicators?
23 MR. BARRETT: I don't know the answer to that
24 question.
25 CHAIRMAN JACKSON: Or looked at the correlation of
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 16
1 the Arthur Andersen prediction to what the WANO indicators
2 might say?
3 MR. CALLAN: Let me weigh in here. We met a
4 couple weeks ago. As you know, the senior managers had
5 their annual public meeting with INPO. INPO made a strong
6 point that there are over 400 plants internationally that
7 use the WANO indicators, and all the plants in the U.S. are
8 included in that number. They wanted the NRC to take a hard
9 look at those indicators so that we don't create our own set
10 and cause ambiguity and confusion. This is a request that
11 they made a couple weeks ago.
12 We need to do that. We need to take a hard look
13 at the WANO indicators and make a deliberate decision yes or
14 no, but at least I think we are obligated to do that.
15 WANO, as you know, also comes up with a figure of
16 merit, which is a single percent number. You probably
17 heard, Commissioner, the same thing I've heard from time to
18 time, which is that some utility executives believe that to
19 be the best indicator out.
20 COMMISSIONER McGAFFIGAN: Yes, I heard that
21 yesterday, which is why I asked the question.
22 MR. CALLAN: So I think we are obligated to take a
23 look at that very close.
24 CHAIRMAN JACKSON: Commissioner Dicus, did you
25 have an additional question?
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 17
1 COMMISSIONER DICUS: No. That was my question.
2 MR. BARRETT: We have looked at the individual
3 indicators, I think the complete set of individual WANO
4 indicators in one form or another.
5 Let me move on to page 11.
6 I just want to point out that at this point, based
7 on the work that we have done so far, the eight indicators
8 that you see here are the ones that we are concentrating
9 with the model and with the graphs that I am going to be
10 showing you in just a moment.
11 These are ones that passed the test that I
12 mentioned earlier, the test of face validity, the
13 correlation test. These are indicators that discriminate
14 well between discussion and non-discussion plants.
15 Also, these are indicators that have relatively
16 less subjectivity involved with them; relatively less
17 assessment is involved with these compared with some of the
18 other indicators that were included in the earlier Arthur
19 Andersen work.
20 CHAIRMAN JACKSON: Did you have criteria? What
21 you just described were the criteria you used to cull these
22 out of a larger set?
23 MR. BARRETT: Those and others, yes.
24 CHAIRMAN JACKSON: Thank you.
25 MR. BARRETT: If I could move to slide 12.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 18
1 In the next few slides I would like to just show
2 you a sample of the type of information that we will be
3 providing as a test case of this methodology for the January
4 senior management meeting cycle, which begins later this
5 month with the PPRs.
6 First, what we will be showing is just basically
7 an overview for each of the regions of the information in
8 each plant.
9 In Region D you can readily see which plants the
10 agency might want to look more closely at. Plant 103 is
11 certainly one of them.
12 CHAIRMAN JACKSON: What is the y axis?
13 MR. BARRETT: That axis is the number of hits.
14 This particular model is a threshold type of model and it's
15 a model similar to the Arthur Andersen model in which you
16 look at indicators one at a time and then count up the
17 number of hits.
18 COMMISSIONER DIAZ: Thresholds are wonderful
19 things. They are also very dangerous.
20 MR. BARRETT: They are, yes.
21 COMMISSIONER DIAZ: What is the meaning of the
22 threshold? Does it have special significance?
23 MR. BARRETT: This particular threshold is simply
24 three hits.
25 CHAIRMAN JACKSON: Does that mean that a plant is
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 19
1 a potential discussion plant?
2 MR. BARRETT: That's right. That's a candidate
3 threshold for discussion. That would mean that for three of
4 the eight indicators in this model this plant had a hit.
5 CHAIRMAN JACKSON: In this kind of trend model is
6 more recent data weighted differently than data earlier in
7 the assessment here?
8 MR. BARRETT: Not in this particular model. This
9 particular model carries six quarters of data equally
10 weighted.
11 CHAIRMAN JACKSON: What does regional average
12 mean?
13 MR. BARRETT: This would be for all of the plants
14 in this region there was an average of approximately 1.5
15 hits per plant.
16 COMMISSIONER McGAFFIGAN: In the Arthur Andersen
17 methodology, to get a hit you had to be twice as bad in the
18 indicator as the industry average. Is that carried over
19 here, or do you have multiple thresholds? You have a
20 threshold to get a hit and you have a threshold to be
21 considered for discussion?
22 MR. BARRETT: That's correct. In this particular
23 model it was one standard deviation from the mean of that
24 indicator averaged over six quarters.
25 COMMISSIONER McGAFFIGAN: That raises the question
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 20
1 as to whether you have the threshold at the right level. In
2 order to get hits they have to be worse than industry
3 average, and then you are looking for deviations from
4 industry average. You basically chose three times the
5 industry average or twice the regional average. If you are
6 going to try to get uniformity across the industry, it
7 presumably is the industry average that matters. Why did
8 you choose three as opposed to two, which would force
9 several of the plants, or at least plants 94 and 95, into
10 discussion space?
11 MR. BARRETT: At this point we haven't settled on
12 a threshold. I will show you some results based on this
13 threshold of three. We have a separate set of results based
14 on a threshold of two, and those are not too much different.
15 There are several issues we still need to examine.
16 One of them is whether we should be comparing to an industry
17 average or whether we should be setting a fixed standard.
18 These are still open questions.
19 CHAIRMAN JACKSON: Please go ahead.
20 COMMISSIONER DIAZ: What kind of statistical
21 distribution do you assume to come up with one standard
22 deviation from the mean? Obviously this is not a normal
23 population. There are very few numbers of plants.
24 MR. BARRETT: There was no need to assume a
25 distribution because we could take all the plants and just
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 21
1 calculate the standard deviation.
2 COMMISSIONER DIAZ: I see. Okay.
3 CHAIRMAN JACKSON: These hits are relative to the
4 performance parameters you had shown on page 11?
5 MR. BARRETT: That's correct.
6 CHAIRMAN JACKSON: Okay.
7 MR. BARRETT: Let me move on to slide 13, which
8 shows the performance trend for Plant 103. This is an
9 actual plant, by the way. There is no Region D. That's a
10 conglomeration of real plants but not in any particular
11 region.
12 This again, as I said, is the threshold model.
13 You can see for Plant 103 there is an increasing trend
14 forward in the indicators. That would obviously lead you to
15 ask, what is it that is driving this trend? So what we will
16 be providing in addition to the overall trend will be the
17 trends for the individual indicators, all eight of them.
18 In this particular case I am showing that forced
19 outage rate and safety system failures are certainly
20 contributing to this trend. That would lead you again to
21 further ask, well, what were those safety system failures,
22 how serious were they? We will be also supplying textual
23 information to back this up as to what is driving these
24 indicators.
25 CHAIRMAN JACKSON: Is there any double counting if
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 22
1 you look at forced outage rate and safety system failures?
2 MR. BARRETT: There could be double counting, yes.
3 In fact there probably will be double counting, because some
4 of these indicators relate to cause codes, and those cause
5 codes would relate perhaps to the same safety system
6 failures. That's why the textual information becomes
7 important so you get behind the numbers and understand what
8 is driving this.
9 If we could move on. Just briefly looking at page
10 14, as I mentioned, we do have two models. The second model
11 is a model that is based on a regression fit using similar
12 indicators. This is a regression fit to past discussion
13 plants. It gives a different picture, but still it gives a
14 picture for Plant 103 for what would appear to be degrading
15 performance.
16 COMMISSIONER DIAZ: In here you did weight them
17 more heavily as a function of time, right? The last few
18 periods were weighted heavier than the earlier ones when you
19 did the multiple step regression? If you look at the curve,
20 it would seem to me like weighting is directly proportional
21 to time or maybe even the square of time.
22 MR. BARRETT: We could, but this particular model,
23 the coefficients in the regression analysis were not given
24 any preference to more recent versus --
25 COMMISSIONER DIAZ: Look at the data and you will
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 23
1 see that it appears -- I don't know -- that the latest point
2 is heavier weighting than the earlier ones. Just by looking
3 at it, but I'm not sure. It does look like it comes up
4 earlier and it rises very, very steep, which might indicate
5 that it's weighted heavily towards the end.
6 MR. BARRETT: I understand your point. I don't
7 believe that's the case, but I could double check that.
8 CHAIRMAN JACKSON: The real question is this sharp
9 crossover at the 942 point.
10 What kind of regression model is this?
11 MR. BARRETT: I can give you the name of it.
12 CHAIRMAN JACKSON: Tell me the name.
13 MR. BARRETT: Logistic regression.
14 CHAIRMAN JACKSON: Okay.
15 MR. BARRETT: Let me move on to slide 15.
16 I'd like to ask you to look at the slide on the
17 monitor, because it has been changed from the slide that we
18 supplied to you earlier. The reason is we felt that we
19 needed to add additional information to this slide to
20 clarify it. The slide that we provided you could easily be
21 misinterpreted, we felt.
22 We have compared both of these models with the
23 results of past discussion lists. What we found is, first
24 of all, that there was what we call an 87 percent agreement.
25 That is to say, there were 79 plants out of 109 which both
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 24
1 the senior managers and the model thought should be
2 non-discussion plants. There were 15 plants which both the
3 senior managers and the model felt should be discussion
4 plants.
5 However, there were four non-discussion plants
6 that the model identified, and there were ten discussion
7 plants per the senior managers that were missed by this
8 model. In other words, this model identified 15 of the 25
9 discussion plants in this particular time period and
10 identified four plants that were not on the discussion list.
11 CHAIRMAN JACKSON: When you say model, you mean
12 the trend model or the regression model?
13 MR. BARRETT: Actually both models gave similar
14 results.
15 We will continue to refine this model, and we
16 think that we can improve the model. It remains to be seen
17 how much we can improve the model.
18 CHAIRMAN JACKSON: What do you think is the more
19 critical error, the ones that were identified by the model
20 that were not discussed, or the other way, the ones that
21 were discussed but not identified by the model?
22 MR. BARRETT: I don't know if I could make a
23 distinction between the two. I've actually gone and looked
24 at a couple of these plants, and we had a couple of
25 discussion plants for which the indicators just don't
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 25
1 indicate anything, and yet we had serious concerns about
2 those plants.
3 I believe you have to look behind each of the two
4 types of errors.
5 MR. CALLAN: I have an opinion on that. I think
6 Bill Borchardt's presentation gets into this, but the role
7 of the indicators is largely to provide a forcing function
8 for the discussion, to force the senior managers to have to
9 face facts, so to speak, and not to rationalize away
10 problems.
11 In that context, I guess you could say a false
12 positive, in other words, the ten discussion plants that
13 were missed by the model, would be a bigger problem. This
14 approach could accommodate false negatives. In other words,
15 the four that perhaps shouldn't be discussion plants that
16 become discussion plants because of the data scatter. The
17 process can accommodate that better than it can accommodate
18 missing ten plants that should be discussed.
19 CHAIRMAN JACKSON: Commissioner McGaffigan.
20 COMMISSIONER McGAFFIGAN: Did you do the obvious
21 check of looking at a lower threshold which would kick in in
22 your model plants 94 and 95? That obviously gives you more
23 plants for discussion. Were those the "right plants," the
24 plants that had been identified for discussion just by
25 varying the threshold from three to two?
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 26
1 MR. BARRETT: We've lowered the threshold. We did
2 at least one such sensitivity analysis. What it did, of
3 course, was identify more plants. It did a better job of
4 identifying plants that were discussed. It also produced
5 more false positives.
6 COMMISSIONER McGAFFIGAN: False positives by your
7 definition because they weren't discussed.
8 MR. BARRETT: Because they weren't discussed.
9 Again, always keeping in mind that we don't have ground
10 truth here. But I will tell you that the two plants that I
11 looked at that I felt were quite important were two plants
12 which we not only discussed but took further action. They
13 were not identified by the model. Those two plants remained
14 outside that universe. So there may well be a residual
15 class of plants that indicators don't pick up.
16 COMMISSIONER DICUS: I noticed on these two models
17 there are three -- I think three -- indicators that are
18 common to both models. Obviously they are going to impact
19 quite a bit. Could you give me a little bit of information
20 on why you did that?
21 MR. BARRETT: We were not trying to keep the
22 indicators separate. In fact, if anything, it might be an
23 ideal case if they were the same but applied in different
24 ways. In the case of the threshold model the emphasis is
25 more on trying to find indicators that you feel each in its
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 27
1 own way has face validity and indicates performance in a way
2 that you understand, whereas in the case of the regression
3 model, of course, you allow the regression to decide which
4 combination of indicators gives you the best results.
5 CHAIRMAN JACKSON: How sensitive are these results
6 to length of observation period?
7 MR. BARRETT: I couldn't tell you numerically. As
8 we got the observation period out towards six quarters, what
9 we found was that we were able to average out some of the
10 fluctuations that you get as a result of shutdowns and
11 things like that. But I don't have numbers as to how
12 sensitive we are.
13 I'd like to move and quickly talk about the
14 economic indicators. In the interest of time, I'd rather
15 not spend too much time on them, but I'd simply like to say
16 that we have performed a similar process with economic
17 indicators to the ones I just described with the performance
18 indicators.
19 We started with a large number of these
20 indicators, some of which were site-specific, some of which
21 were corporate, and we evaluated each one of them using
22 correlations with past senior management meeting
23 discussions.
24 We developed a trending methodology based on the
25 combination of the ones that gave the best results. I don't
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 28
1 know if anybody will be surprised by this, but we found that
2 the site-specific indicators showed a much better
3 correlation with past results than the corporate indicators.
4 We intend to supply economic trend plots similar
5 to the performance plots I just showed both as aggregates
6 and as individual plots of individual indicators for this
7 senior management meeting cycle along with explanatory
8 information so that everyone understands what these economic
9 indicators mean. This will be in mid-October of this year
10 for the January cycle.
11 CHAIRMAN JACKSON: To what extent were the
12 economic indicators leading indicators?
13 MR. BARRETT: The economic indicators were
14 leading. The correlations tended to show that where there
15 was a correlation there was a leading correlation. The
16 analyses were done by using delays in the -- again, if you
17 would like to know more about this, I might ask someone to
18 come to the podium, but they were shown to be somewhat
19 leading.
20 CHAIRMAN JACKSON: Commissioner McGaffigan.
21 COMMISSIONER McGAFFIGAN: Early on in your
22 presentation either you or Sam Collins said that at some
23 point you're going to get this ACRS comment and public
24 comment on all this. How is that going to work? Is that
25 going to work in time? If somebody looks at this from
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 29
1 outside and says you're all wrong here, is that going to be
2 in time to course adjust?
3 MR. BARRETT: The answer to that is yes. We will
4 always be in a position to course adjust. After we have
5 been through the public comment period and after we have
6 been through the trial tests of all these methods, we would
7 plan to come to the Commission -- our current schedule would
8 say the end of next summer -- with recommendations of how to
9 proceed with this information.
10 The ACRS peer review has already started. We met
11 with the ACRS in March of this year and gave them an
12 overview of the program. We met with them earlier this
13 month and showed them a much more detailed version of what
14 you are seeing today. We've gotten a great deal of feedback
15 from them and it's very useful feedback, and we will be
16 factoring it in.
17 We plan to go back to the ACRS in February. We
18 are scheduled to do that. We have an invitation from them
19 to come back at intermediate points to try out some of our
20 thinking as we go along.
21 So we believe that the ACRS comments can be
22 factored in as we go.
23 CHAIRMAN JACKSON: You mentioned what products you
24 are going to have for the January senior management meeting
25 cycle, but will any of this economic data be actually used
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 30
1 in the senior management meeting process?
2 MR. CALLAN: That's an NRR question.
3 MR. BORCHARDT: The intent right now is that we
4 use that information at the screening meeting primarily. If
5 it's found to be exceptionally relevant, it would move
6 forward to the actual senior management meeting, but its
7 primary purpose will be restricted to the screening meeting.
8 CHAIRMAN JACKSON: So it's at that point in the
9 process?
10 MR. BORCHARDT: Yes.
11 MR. CALLAN: Let me throw out a cautionary note on
12 the economic indicators. Even more so than the case of the
13 other indicators, which I think Commissioner McGaffigan
14 questioned because of the circular nature of the validation
15 process, I think the economic indicators are even more
16 subject to that kind of error. We are correlating these
17 indicators with plant performance in a period when they were
18 under economic regulation. So we are correlating the
19 indicators today with plant performance trends against that
20 context.
21 I don't think we can take much comfort from the
22 fact that we are seeing correlation or that we are even
23 seeing that they are leading indicators. I don't think we
24 know enough about the economic deregulation environment to
25 make too many flat statements on that.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 31
1 CHAIRMAN JACKSON: To draw too many conclusions.
2 MR. CALLAN: Right.
3 CHAIRMAN JACKSON: Commissioner Diaz.
4 COMMISSIONER DIAZ: Mr. Callan just answered my
5 question.
6 MR. BARRETT: I'd like to move on to slide 18 and
7 discuss the template.
8 CHAIRMAN JACKSON: Let me ask you a quick
9 question. The data to support the indicators that you have
10 on page 17, the site model indicators, is that publicly
11 available data?
12 MR. BARRETT: Yes, it is. It's information that
13 is available to us either from reports that are required to
14 be sent to the SEC or else monthly operating reports that we
15 get.
16 CHAIRMAN JACKSON: Okay.
17 MR. BARRETT: With regard to the template, the
18 Commission directed the staff to consider use of a plant
19 performance template and suggested that the template be used
20 as a way of making a connection between performance
21 information and the ensuing decisions.
22 You also directed us to show how the template and
23 the trend plots will be used in tandem in the decision
24 process and how the template can include both quantitative
25 as well as qualitative information.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 32
1 Finally, you directed us to more precisely and
2 objectively determine the specific criteria and thresholds
3 for NRC action levels.
4 If I could have slide 20, please.
5 Slides 20 and 21 present the current version of
6 our template. It's a version that was frozen as of July
7 31st, although this is still a work in progress. It
8 embodies a balanced and structured presentation of the key
9 elements that constitute plant performance.
10 I don't plan to go through this template in detail
11 because of the time constraints, but I would like to just
12 describe it in some outline.
13 The major categories in this template mirror the
14 categories in the template of Management Directive 8.14. We
15 decided to adopt those major categories after some
16 deliberation because we felt that it was a good set of
17 categories. We also felt that it was a set of categories
18 that were risk-informed.
19 To develop the subcategories we went back to the
20 record of the past senior management meetings to look at the
21 areas which past senior managers thought were important to
22 performance.
23 We went back to the briefing books for the senior
24 management meetings, to the minutes of the senior management
25 meetings, and to the transcripts of their briefings to the
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 33
1 Commission, and we built a database of approximately 1,700
2 specific statements that were made about plants.
3 From those we identified common characteristics
4 that were discussed. We aggregated those down to a couple
5 of dozen categories and then we looked to see if they fit
6 naturally into the major categories of Management Directive
7 8.14. And they did, with one exception.
8 We found that about a third of the judgments that
9 were stated in past senior management meetings related to
10 organizational effectiveness or management, and there was no
11 such category in the management directive template. So we
12 added a major category which we call organizational
13 effectiveness.
14 We took this template and we had it reviewed by
15 NRR and Research. The Office of Research suggested
16 significant changes to it, especially in the areas of human
17 performance and organizational effectiveness, and we
18 implemented those proposed changes.
19 We continue to refine this template, but in the
20 meantime we are using this version of the template for two
21 purposes. One is to define the information which will be
22 the input to this template, and secondly, to define the
23 criteria and decision model that will be based on this
24 template.
25 CHAIRMAN JACKSON: Right. That's what I was going
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 34
1 to point out, that what you have here is almost like a
2 listing of topics, a topical listing.
3 MR. BARRETT: That's correct.
4 CHAIRMAN JACKSON: The real question is what is
5 your hierarchy for decision-making and what are the criteria
6 you are going to use and what are the objective standards
7 that are applied.
8 MR. BARRETT: The ACRS talked about a decision
9 model.
10 CHAIRMAN JACKSON: How far are you away from
11 defining a decision model?
12 MR. BARRETT: The definition of the decision model
13 and the criteria is in its early stages. It's difficult to
14 say.
15 COMMISSIONER DICUS: It's written in pencil on a
16 piece of paper.
17 CHAIRMAN JACKSON: You decided to start it this
18 afternoon.
19 [Laughter.]
20 MR. BARRETT: No, we didn't start this afternoon.
21 We are a lot farther along than that, I'd say. In fact, I'd
22 like to discuss some of the early thinking as we get to that
23 point in a couple of slides.
24 CHAIRMAN JACKSON: Commissioner Diaz.
25 COMMISSIONER DIAZ: I was going to make a comment
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 35
1 and a question on this organizational effectiveness. I have
2 absolutely no problem with any of the categories except the
3 issue of culture. It's kind of a sensitive issue to me.
4 Making a small point in here in a relaxed afternoon, I used
5 to be in a place where people used to talk about having to
6 judge people by their revolutionary conscience.
7 [Laughter.]
8 COMMISSIONER DIAZ: I really don't want to get to
9 that point in our deliberations, and I'm sure that you are
10 very, very sensitive to that. When we assess culture we've
11 got to be very careful that we don't infringe on the freedom
12 of our institutions.
13 CHAIRMAN JACKSON: That's why what your criteria
14 are and what your decision model is is very important in
15 order to guard against that.
16 COMMISSIONER DIAZ: By the way, I flunked the
17 test.
18 [Laughter.]
19 MR. BARRETT: If I could move on to slide 22.
20 I want to talk a little bit about the information
21 that will populate this template. The template for a
22 specific plant will contain the issues that result from the
23 regional PIMs and other sources, as well as the indicators
24 that are appropriate to the various categories and
25 subcategories. We feel that these indicators can be
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 36
1 associated with the various categories.
2 We have actually done a trial application of this
3 approach using the PIMs data from a specific region, and we
4 found that the inspection data fit the template quite well,
5 again with one exception, and that is organizational
6 effectiveness. It's not surprising that we didn't find very
7 many findings related to management and organizational
8 effectiveness because inspectors are not encouraged to look
9 into management issues in the inspection process.
10 So that leaves a question, and the question is,
11 how are we going to populate the organizational
12 effectiveness category? That's a question that we are
13 currently working with the Office of Research. They have
14 some methodologies that we are evaluating to use the issues
15 from the other categories to populate the organizational
16 effectiveness category.
17 In addition to categorizing the issues, what we
18 intend to do is also assign a qualitative risk significance
19 to these issues. That could be a high versus low.
20 Certainly not a quantitative estimate. High/medium/low or
21 high/low. We are working with the Office of Research also
22 on this, to develop simple guidance on how to categorize
23 issues with respect to risk.
24 We are beginning a pilot application of the
25 template. We intend to ask the regional offices beginning
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 37
1 this fall to code new inspection findings as they go into
2 the PIMs, to code them in accordance with this template,
3 excluding the organizational effectiveness category.
4 By doing this we expect that when the June 1998
5 senior management meeting cycle begins, which is March of
6 1998, we will have a database of approximately six months
7 worth of information about these plants, and that will allow
8 us to do a trial application of this template in that senior
9 management meeting cycle.
10 CHAIRMAN JACKSON: Let me reemphasize something.
11 The issue of the organizational effectiveness and really
12 having a data and decision criteria relative to it is very
13 important for the kind of reason that Commissioner Diaz
14 mentioned. As I recall reading the Arthur Andersen report,
15 essentially it indicated that by the time the decisions
16 propagated to the senior management meeting that they were
17 basically anecdotal, that there was a database and there
18 seemed to be more linkage between the inputs and judgments
19 made at earlier stages in the process and things seemed to
20 work okay until you got to the senior management meeting,
21 and then there were a set of decisions that were made that
22 seemed unlinked or disjointed from everything that had gone
23 on before.
24 So it is very important that you develop the
25 criteria and what data needs to feed into that and then to
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 38
1 show what your decision model is that actually then gets
2 promulgated into the actual decisions in the senior
3 management. Otherwise you will always be accused of having
4 a process that is not scrutable. Either way. That you are
5 making arbitrary judgments about management that could be
6 negative without any supporting line and decision process
7 and data, or for those who would be the detractors of the
8 process, that you do it the other way, that you make
9 arbitrary decisions or you give credit for or you weigh
10 management behavior to the positive effect, again without
11 any real objective data.
12 So it is an important issue that I think you need
13 to give some heightened and accelerated attention to to the
14 extent that you can.
15 MR. BARRETT: We will.
16 MR. BORCHARDT: Chairman, if I may. I think the
17 problem is even more difficult than you describe. The
18 current configuration of the inspection program does not
19 right now support the data that Rich is alluding to that he
20 needs to set up that model. If the Commission decides that
21 we are going to go in that direction, there is a wide range
22 of inspection guidance and standards and training for the
23 inspection staff that needs to be completed, and that won't
24 be an easy task.
25 CHAIRMAN JACKSON: None of this is, though.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 39
1 Commissioner McGaffigan.
2 COMMISSIONER McGAFFIGAN: A related point is that
3 much of the information here is really available to the
4 licensee. The PIMs are or soon will be; LERs, obviously;
5 the economic indicator that the Chairman talked about back
6 on page 17. I assume if we are using publicly available
7 data that they can either replicate it or we could just hand
8 it to them. This is what the economic indicator that we are
9 using is. But this other stuff isn't, because it hasn't
10 been captured anywhere. I think that is where the greatest
11 chance of disconnect is.
12 CHAIRMAN JACKSON: Exactly. That's why you really
13 have to give that focused attention in this area, because
14 everybody needs to know what it is.
15 MR. CALLAN: That's one of the basic ground rules.
16 We don't use information that is not publicly available.
17 CHAIRMAN JACKSON: You do make decisions where you
18 make implicit management judgments.
19 MR. CALLAN: Derivative conclusions, right.
20 CHAIRMAN JACKSON: Without a clear decision-making
21 path for how you got there.
22 MR. CALLAN: That's right.
23 CHAIRMAN JACKSON: You can't unequivocally say
24 that you don't use it, because you do use it.
25 MR. CALLAN: Right.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 40
1 CHAIRMAN JACKSON: The real issue is clarifying it
2 and pulling it out and supporting it.
3 COMMISSIONER McGAFFIGAN: The one place where it
4 struck me that it's not publicly available is allegations.
5 How do you use allegations?
6 MR. CALLAN: The data that we use is the
7 information that we provide the licensees and I presume goes
8 in the docket.
9 MR. COLLINS: Yes, which are numbers in comparison
10 without specifics to the issues.
11 MR. CALLAN: But by category.
12 COMMISSIONER McGAFFIGAN: So the only way we use
13 allegations is, are there a lot of allegations at this
14 plant?
15 MR. CALLAN: And how many of them are harassment,
16 intimidation allegations, how many are technical
17 allegations, that sort of thing.
18 MR. MARTIN: We also focus on substantiated
19 allegations, which would have been communicated.
20 CHAIRMAN JACKSON: Right. So it's not just
21 counting the allegations.
22 Commissioner Diaz.
23 COMMISSIONER DIAZ: On slide 22, I imagine that
24 consistent with Commission guidance the assigning of
25 risk-significant is taking an appropriate priority in the
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 41
1 process for any kind of categorization that is made of
2 events or issues.
3 MR. BARRETT: We are giving a lot of attention to
4 making this whole process consistent with the agency's
5 initiative on risk-informed regulation.
6 MR. CALLAN: Let me clarify a point. We weigh
7 substantiated allegations heavily, obviously, but the
8 information we provide licensees includes all the
9 allegations we receive, and we tell them how many are
10 substantiated in various categories. So we give them a
11 pretty good set of data. We've been doing that for a couple
12 of years.
13 MR. COLLINS: We have been doing it for a few
14 years on request and we are doing it routinely now.
15 CHAIRMAN JACKSON: But making sure that you
16 maintain the confidentiality of the clients, et cetera,
17 right?
18 MR. MARTIN: In terms of data, your
19 characterization is absolutely correct, but if we
20 substantiate an allegation and then subsequently take
21 enforcement action, it's very clear they have the facts.
22 They may not know that it came about as an allegation, but
23 they know the fact that will appear in the PIM because it's
24 an enforcement item.
25 COMMISSIONER DIAZ: On slide 22, periodic
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 42
1 headquarters audit of implementation, what is that? Is that
2 a new interface you are creating now to audit the
3 implementation of the program?
4 MR. BARRETT: If we provide guidance to the
5 regions, for instance, on how to do the risk significance,
6 we would want to periodically look at samples of it just to
7 make sure that everyone understands the guidance. Perhaps
8 the word "audit" is a bit strong.
9 COMMISSIONER DIAZ: Does it refer to risk
10 significance in itself or to the entire process?
11 MR. BARRETT: The entire process. We want to make
12 sure that everyone understands what the template categories
13 mean and are implementing them uniformly as well as the risk
14 significance.
15 COMMISSIONER DIAZ: Because the senior management
16 provides themselves an evaluation of the entire process, I
17 was concerned that we might not be reevaluating the
18 reevaluation.
19 CHAIRMAN JACKSON: Let me make sure I understand
20 something here about the comment that you kind of made as a
21 sidebar comment, about what you provide to licensees
22 vis-a-vis allegations. There is a sensitivity issue having
23 to do with not revealing people's names. This is in terms
24 of protection of allegers. You don't mean that you just
25 give all information.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 43
1 MR. CALLAN: No. All we do is give them numbers.
2 We'll tell a licensee that in the 12-month period ending the
3 first of September the NRC received 20 allegations of which
4 5 were H&I.
5 CHAIRMAN JACKSON: I just wanted clarity for the
6 record.
7 COMMISSIONER McGAFFIGAN: I'm sorry to beat this
8 dead horse to some degree, but on allegations, isn't there a
9 chance that there will be time lag? I'm not saying that
10 this is bad. I think it's just a fact of life. You could
11 be in a senior management meeting in January and have a
12 bunch of substantiated allegations that since they require
13 enforcement action you have not shared with the licensee
14 because there is a time lag in enforcement. So it might
15 weigh in your decision as to whether the plant deserves to
16 be discussed.
17 MR. CALLAN: There are two facets to allegation
18 data. One facet gives you a window into the organizational
19 climate of the plant. To that extent, whether they are
20 substantiated or not is almost not real important. If you
21 get a lot of allegations of which a lot are H&I, then that
22 deserves close NRC scrutiny: is there a pathology there at
23 that site involving the management climate?
24 The other facet, of course, is what you are
25 getting to, which is the substance, the technical substance,
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 44
1 and is it enforceable or not. That second facet we handle
2 more routinely through the process, but when we call upon
3 the allegation coordinator who attends the senior management
4 meeting, we are really looking for the first set of
5 insights: what does the allegation data tell us about the
6 health of that organization? That's the insight we are
7 looking for.
8 MR. BARRETT: Let me just briefly on slide 23
9 discuss the question of criteria. As I said, the decision
10 criteria development is in an early stage. We believe that
11 both the trend plots and the templates are important to the
12 decision process. The trends are, of course, amenable to
13 strict thresholds and numerical criteria.
14 Interpretation of the template will require more
15 qualitative criteria and will entail some judgment. Some of
16 the factors that we think are important in these criteria,
17 first of all, would be the number and the risk significance
18 of the issues in a particular category or subcategory.
19 Also the significance of issues as they relate to
20 programmatic problems. So if you have a category that has a
21 lot of issues, no particular issue might be risk
22 significant, but as an aggregate they may point to a
23 programmatic problem. That's a precursor of
24 risk-significant activities.
25 Also the relative importance of various
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 45
1 categories. We know that not all six of these categories
2 would be equally important in a decision regarding the watch
3 list. We need to think about which ones are most important.
4 Finally, relationships among the categories. It
5 could very well be that poor performance in a particular
6 category might be mitigated by good performance in another
7 category, or a combination of two categories that have poor
8 performance might have more significance than some other
9 combination.
10 We have to look at this systemically; we have to
11 develop a decision model and develop qualitative criteria
12 around that model so that we can make this a consistent and
13 scrutable process.
14 We are working with the Office of Research and
15 with Arthur Andersen Consulting on this model.
16 CHAIRMAN JACKSON: When do you anticipate that the
17 guidelines will be available?
18 MR. BARRETT: I think that we would have a draft
19 set of guidelines available in the next couple of months.
20 We would certainly want to have them available before the
21 next time we go back to the ACRS, but I would say in the
22 next month or so.
23 MR. CALLAN: On the schedule chart it shows
24 revised template, revised criteria the end of November.
25 MR. BARRETT: Let me go to the schedule, which is
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 46
1 slide 24. I want to point out a couple of the major
2 features of the schedule.
3 The January senior management meeting cycle starts
4 later this month. We will be providing both the performance
5 charts and the economic trend plots, as I mentioned, for
6 this cycle.
7 In the next cycle, which begins in March of 1998,
8 we will also be putting the template into a trial
9 application using the PIMs data developed during the next
10 six months.
11 Starting in March of next year we plan to have a
12 public comment period. Leading up to that public comment
13 period we will be coming back to the Commission with a
14 Commission paper and a briefing to provide what we have at
15 that time in preparation for the public comment period.
16 Finally, in the late summer of 1998, after
17 completion of the public comment period and after completion
18 of all the trial applications, we intend to come back to the
19 Commission with recommendations on how to proceed from
20 there. By that time we will know more about the integrated
21 review of assessment processes.
22 CHAIRMAN JACKSON: Commissioner Diaz and then
23 Commissioner McGaffigan.
24 COMMISSIONER DIAZ: Looking at the schedule, I
25 just wanted to understand this Commission briefing February
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 47
1 of 1998 prior to the public comment. The process will
2 essentially be finalized by then so that it will be clear
3 that public comments in that process would just be kind of
4 fine tuning prior to the decision. In other words, you
5 intend to have a significant fraction of the process well
6 defined by February of 1998.
7 MR. BARRETT: That's right. We intend to have the
8 whole thing laid out in at least enough detail that we could
9 get significant public comment and industry feedback.
10 MR. MARTIN: Recognizing that by that time we will
11 not yet have had a trial with the template, because the raw
12 data won't be available until the March time frame. So
13 having gone through the complete process with what we think
14 is close to the final really won't be until a June or July
15 time frame.
16 COMMISSIONER DIAZ: The high level decision-making
17 on the processes will have been made. I don't know what I
18 mean by high level. The overriding major considerations.
19 COMMISSIONER McGAFFIGAN: The answer to
20 Commissioner Diaz' question actually makes me more
21 concerned. We had another briefing a few months on medical
22 where the question came up that by the time you put
23 something out for public comment all you really want is a
24 tweak or two and we're not really going to listen to public
25 comment.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 48
1 CHAIRMAN JACKSON: At the same time they need to
2 get started with having some flesh on the bones.
3 COMMISSIONER McGAFFIGAN: They have a lot flesh.
4 I suspect that at least some listeners to today's briefing
5 may go away with heart palpitations, or whatever. It
6 strikes me that if we are going to be an open agency that
7 during this period of the next couple of months, knowing
8 everything has to ultimately be decided, policy decisions by
9 the Commission, that there is no harm in having a lot of
10 dialogue with industry and the public, whoever, about
11 whether we are on the right track. GAO, for that matter.
12 And be relatively open as we search for solutions here.
13 Putting everything off until March when it's perfect or
14 perhaps locked in isn't a good idea. I throw that out.
15 CHAIRMAN JACKSON: I think the point is to reach
16 out and involve the various stakeholders and perhaps build
17 in more open processes. You are going to be briefing the
18 ACRS. Those are open meetings anyway.
19 MR. CALLAN: And we've gotten some valuable input
20 from attendees at the ACRS meetings. We got a good letter
21 from NEI. So we are getting feedback not only from the ACRS
22 but by attendees at those ACRS meetings.
23 COMMISSIONER McGAFFIGAN: I think I'm suggesting
24 in addition to ACRS that we go out and have meetings with
25 --
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 49
1 CHAIRMAN JACKSON: That's what I'm saying. Build
2 in more structured public opportunity.
3 MR. BARRETT: Sooner rather than later.
4 Let me briefly conclude by reiterating that we
5 believe based on what we have seen so far --
6 CHAIRMAN JACKSON: And remember, all your
7 stakeholders.
8 MR. BARRETT: We believe that the senior
9 management decisions will be best made with a combination of
10 both the template and the performance trends and that senior
11 management meeting decisions can be made by using numerical
12 criteria in conjunction with the trend plots, which we have
13 already shown give reasonable results but need further
14 refinement to be improved.
15 As I mentioned before, we are working on a
16 decision model and guidelines to be used with the template,
17 and we will report on that in the future.
18 Finally, with regard to economic indicators, we
19 see them as an early warning of potential performance
20 problems, not necessarily as a part of the formal decision
21 process itself, but more of an early warning of potential
22 performance problems.
23 CHAIRMAN JACKSON: Commissioner McGaffigan.
24 COMMISSIONER McGAFFIGAN: I think you are
25 splitting semantic hairs here in that last statement. My
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 50
1 fellow Commissioners are looking at me like I'm saying this
2 for all of them. When you use them in screening meetings,
3 even though perhaps the data is not going be fresh in
4 everybody's mind at the January meeting, they are used early
5 on in the decision process. Maybe you are saying they are
6 not used at the final decision phase, but they are used as
7 an early, gatekeeper phase, it sounds like.
8 CHAIRMAN JACKSON: What is the point in having
9 them to give you early warning if it doesn't at least inform
10 you as you go along in your decisions?
11 MR. BARRETT: I think it does inform you. The
12 point of the bullet is simply to say we don't intend to
13 write criteria that are built around economic indicators.
14 CHAIRMAN JACKSON: How are you going to use them,
15 then?
16 MR. BARRETT: As I said, we are going to provide
17 them to senior managers. I think they could easily be
18 provided to senior managers within the context of the senior
19 management meeting or outside the context of the senior
20 management meeting. They are meant to be information that
21 might provide an early warning.
22 CHAIRMAN JACKSON: I think you've got to come down
23 with a decision on it. You're either going to use it in the
24 senior management meeting process or you're not. No one has
25 to say that you have to use it in the same hard and fast way
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 51
1 that you might use some other indicators, but you need to
2 make a decision. If you mealy-mouth it, then it becomes the
3 stepchild -- you just decide what you are going to do.
4 MR. CALLAN: I don't think we can decide for a
5 while. We don't know enough about it.
6 CHAIRMAN JACKSON: That's fine. It's a process
7 that you are evolving to, but that's true of everything.
8 MR. CALLAN: I think we are all squeamish about
9 economic indicators.
10 CHAIRMAN JACKSON: That's clear, but that's what
11 I'm trying to tell you, that you are probably squeamish
12 about organizational effectiveness, too.
13 MR. CALLAN: Not too much.
14 CHAIRMAN JACKSON: It hasn't been developed to
15 this point, which suggests some lack of comfort in that
16 regard, and as you become more sophisticated, then you can
17 make a judgment as to what extent it really can validly be
18 used or not. You can't address it by backing away from it.
19 MR. CALLAN: Right.
20 CHAIRMAN JACKSON: You have to ensure that you do
21 that evaluation, however you come out. So you don't back
22 away from it. That's not the way you make the decisions.
23 You deal with it; you decide how good it can be in terms of
24 being used; and then you go on from there. But you can't
25 sort of say we're going to do it. It's like being a little
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 52
1 bit pregnant, or whatever.
2 MR. CALLAN: We didn't mean to come across that
3 way. We're going to give economic indicators a fair trial
4 and interact with the Commission and our stakeholders.
5 CHAIRMAN JACKSON: Absolutely.
6 MR. CALLAN: Right now that's the area that we are
7 probably, as I said, the most squeamish about.
8 CHAIRMAN JACKSON: Right. I just want to get out
9 on the table that I know you're squeamish, but you have to
10 go through a robust process and get to whatever the end
11 point is.
12 MR. CALLAN: I understand. Absolutely.
13 MR. MARTIN: Chairman, just another fact. A
14 couple of decisions have to be made. One is whether the
15 plant should be discussed. It may have a different role
16 there, forcing us to discuss them, and it may have a
17 different role in the decision process at the senior
18 management meeting. We'll look at both aspects.
19 CHAIRMAN JACKSON: I guess you are making kind of
20 an artificial distinction, which is what I think
21 Commissioner McGaffigan was getting at. When you use them
22 at some point in the process, you are using them in the
23 senior management meeting process. That is a separate
24 decision as to when you actually sit down in the senior
25 management meeting: Is it part of your go or no-go
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 53
1 decision-making? But the very fact that you use it
2 somewhere, at screening or whatever, you are using it in the
3 senior management meeting process. So let's not split these
4 kind of hairs artificially here.
5 MR. CALLAN: I understand.
6 MR. BARRETT: That concludes my remarks.
7 [Laughter.]
8 CHAIRMAN JACKSON: Thank you.
9 MR. CALLAN: Thank you, Rich.
10 Chairman, in the time remaining I understand that
11 we do have somewhat of a schedule this afternoon. So we
12 would like to proceed to continue with the short-term
13 actions. We have six areas we would like to cover as a
14 result of a direction that the staff has been given in
15 concert with short-term actions. I would like to ask Bill
16 Borchardt to cover those, and then we will quickly go into
17 the integrated review process.
18 MR. BORCHARDT: Slide 26, please.
19 It's the staff's intent to continue making
20 incremental improvements to the senior management meeting
21 process as opportunities arise. This slide shows a number
22 of the significant changes that have been made over the last
23 two years.
24 The first significant change was actually the
25 development of the management directive. Up until a couple
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 54
1 of years ago there was no real written procedural guidance
2 on conduct of the senior management meeting. We started
3 using this guidance in March of 1996 and it was eventually
4 published in a formal manner in June of 1997.
5 There have been a lot of changes to the screening
6 meetings. These changes include a far more active and
7 proactive involvement of the participants at the screening
8 meetings. The major participants now include the regional
9 administrator and the regional staff as appropriate, the
10 director of NRR and staff, AEOD, OI, and OE. All attend the
11 meeting and are active participants. In fact there is an
12 active solicitation of views at the screening meeting.
13 The threshold for discussion has changed slightly
14 or evolved slightly over the last several years. Now the
15 threshold for a plant moving forward to the senior
16 management meeting is, if one of those major participants
17 believes that the plant ought to be discussed, it moves
18 forward. It's not a vote; it's not a preponderance of votes
19 at the meeting. If one believes it ought to be discussed,
20 then that is the decision unless the ensuing discussion
21 makes that individual change their mind.
22 The discussion of plants includes all plants in
23 that region. The meeting takes place over a full day, and
24 the discussion is graded, depending upon the performance of
25 the plant. So the very best performing plants in the region
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 55
1 would get less discussion than the kind of plants that are
2 in the gray area you might describe, those that might be
3 worthy of discussion at the senior management meeting or
4 might not.
5 Rich has discussed that we are moving into
6 consideration of economic indicators, and the next meeting
7 will include use of the trend plots as well as economic
8 indicators.
9 The pro/con charts and the performance evaluation
10 template has served its purpose very well, I think, although
11 there are still some more improvements that need to be made.
12 But it has focused the discussion on objective information,
13 and it has provided a focal point for those discussions at
14 both the screening meeting and the senior management
15 meeting.
16 The active participation and documentation of
17 decisions started in real earnest at the January 1997 senior
18 management meeting. It includes an active facilitation by
19 the EDO and by the director of NRR and increased level of
20 interaction among all the participants and a focused
21 discussion through those pro/con charts and the site and
22 removal matrix for those plants that have been currently
23 categorized as category 2.
24 The bottom two bullets on this slide mention two
25 topics that the Commission has addressed recently.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 56
1 The plant issues matrix. We have recommended and
2 are moving forward with making that public beginning the
3 spring of next year. That will allow the regions to begin
4 developing the data, beginning next week actually, and so
5 we'll have six months at the time the next PPRs happen, and
6 then we will make that six months of information eventually
7 turn into at least a full year as the data is accumulated.
8 On the subject of trending letters, our
9 recommendation to the Commission is that we do not change
10 the policy at this time, although we recognize that the
11 trending letters and the PIM are both very important parts
12 of the integrated review that I am going to discuss next.
13 Slide 28, please.
14 What I would like to do next is provide a brief
15 overview of the integrated review that we are conducting on
16 the NRC's assessment processes.
17 Although this review is going to focus on the four
18 specific programs that are listed, I think we need to
19 constantly remind ourselves of the importance of the
20 inspection program and the basic inspection procedures and
21 inspection results that really form the factual base on
22 which all these other assessment processes operate.
23 The review effort grew out of a number of SRMs
24 that the Commission has provided to the staff recently.
25 Some were very specific in nature and some were rather wide
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 57
1 ranging. It was our suggestion that we conduct this
2 integrated review in order to give each of those SRMs the
3 appropriate consideration that they deserve.
4 The staff is really very excited about this
5 effort. It's a valuable opportunity, I think. It's the
6 first time that we have taken advantage of the opportunity
7 to do an integrated review of all these processes. I'm
8 going to very briefly go through these four processes.
9 They were all started at discrete points in time
10 for unique purposes. What has never been done is to look at
11 the cumulative effect both on the staff and on the industry
12 and on the public as an information source. So this is
13 really an opportunity to make life more effective and more
14 efficient for everyone involved.
15 Starting off with the SALP, it was implemented in
16 1980 following the TMI event. There have been numerous
17 changes over the last 18 years. At one point there were 17
18 SALP functional areas. The reports ranged 40 to 50 pages in
19 length. Now we have the four SALP areas that you are
20 familiar with: operations, engineering, maintenance, and
21 plant support. The interval now is normally 18 to 24
22 months, depending on plant performance.
23 The SALP has always served -- the major goals have
24 always included the allocation of inspection resources and a
25 communication tool with the licensees and the public.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 58
1 The SALP is conducted primarily by the regional
2 management with participation from NRR project management.
3 The senior management meeting was first
4 implemented in the mid-1980s following the Davis-Besse
5 event. Rich Barrett has discussed a lot of the details of
6 programs that are under development. It's held every six
7 months. It allows the senior managers to focus on the
8 plants of most concern, and one of the major outcomes is the
9 identification of an agency plan to address those plants.
10 One important point is that it has always been a
11 supplement to the normal regulatory process. We have never
12 waited to make a safety decision for the senior management
13 meeting to occur. The regional administrators, the NRR, the
14 major program offices all have a role in day-to-day
15 oversight and regulatory responsibility which are not
16 delayed in any way and never have been by the senior
17 management meeting.
18 Slide 29, please.
19 The plant performance reviews are largely a
20 regional effort. There is some NRR participation. It was
21 initially implemented in October of 1990, and it provides
22 the regional inspection staff primarily an opportunity to
23 perform midcourse corrections based upon the six-month
24 review of plant performance.
25 It's the intent that if licensee performance
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 59
1 weakness is identified in any particular area that the
2 inspection plans for that region would be adjusted
3 accordingly; we'd have some focused inspections; and
4 resources could be allocated, moved from one plant to
5 another, depending upon the real time perceptions of
6 performance by the region.
7 Renewed interest in it came out of the South Texas
8 Lessons Learned Task Force. One of the major findings of
9 that was that the NRC had the inspection information; we had
10 the findings, but we never put it together in an adequate
11 form to allow us to make decisions and put all the pieces of
12 the puzzle together.
13 The plant issues matrix is a significantly newer
14 initiative. It was implemented across the regions in the
15 spring of 1996. It's a listing of both the positive and
16 negative findings and conclusions out of the inspection
17 report.
18 Again, everything in the PIM has to be on the
19 public record. It doesn't have to be in an inspection
20 report; it could be in a document from NRR to the licensee
21 regarding a license amendment or some other document. But
22 it has to be on the docket. And it lists both positive and
23 negative findings. So there is an attempt to have some
24 balance, although by the nature of our job we do a much
25 better job identifying the weaknesses. You should expect
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 60
1 that a plant issues matrix will have a majority of issues
2 that identify concerns and weaknesses.
3 The spring of next year the PIMs will be made
4 public.
5 Go to slide 31.
6 Both slides 30 and 31 run through some of the
7 preliminary strengths and weaknesses identified by the staff
8 and the regions.
9 The integrated review group will do a far more
10 thorough job of identifying both the strengths and the
11 weaknesses of these programs, but in general, we know there
12 are some problems that we want to address through this
13 integrated review.
14 There is an element of redundancy among the
15 programs that we would like to minimize. I don't think we
16 can reduce it entirely. There is going to need to be some
17 overlap, but we want to reduce redundancy as much as we can.
18 We certainly want to reduce the level of different criteria.
19 We have already mentioned today we have a new
20 template configured. We have four SALP functional areas.
21 There is a strong argument that says we ought to assess
22 plants using the same criteria. Those four SALP functional
23 areas were created for a reason over time and the template
24 was created for a different reason. Now is the time to
25 reconcile it and come up with a single approach and start
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 61
1 using that on a routine basis.
2 There is the potential for inconsistent
3 implementation among the various regional offices and the
4 major program offices, and it's highly resource intensive.
5 Each of these programs start off with an objective in mind
6 of how much effort it's going to take, and inevitably it
7 takes more, and it keeps growing and growing and growing.
8 So this is an opportunity for us to kind of rebaseline our
9 resource efforts.
10 Slide 32.
11 This slide lists some of the attributes that we
12 would like to maximize and others that we want to try to
13 minimize. It's likely that there is going to have to be a
14 balance and some tradeoff between them. We won't get
15 absolutes on any of these. The review team has a difficult
16 task in front of them to try to meet these objectives.
17 The team will also be developing, to the extent it
18 can, some quantitative criteria to measure improvements in
19 the processes, some goals that we can establish, especially
20 regarding how many resources it takes the NRC staff to
21 complete these programs. I think this is one of the
22 valuable comments we received from the ACRS recently. So we
23 have just begun to try to come up with some of those
24 criteria.
25 The attributes to maximize include trying to come
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 62
1 up with a single assessment process, or at least a continuum
2 of discrete processes that could be used that relate to each
3 other and eliminate some of the unnecessary overlap; and
4 early identification of declining performance.
5 We'd like to have a clear understanding of roles,
6 especially within the NRC staff. This effort is a very
7 natural follow-on to the job task analysis we recently
8 completely in the regions. We're in the early stages now of
9 reviewing the documented results of the job task analysis.
10 So this fits in very well with that effort.
11 Of course we want to maximize the open dialogue
12 and use these tools as an effective communication device
13 both with the industry and the public.
14 Attributes to minimize include lessening the
15 opportunity for inconsistent assessment criteria,
16 eliminating overlapping responsibilities, trying to ensure
17 we have more consistent implementation, and trying to
18 eliminate as much as we can opportunities to send
19 conflicting messages to licensees.
20 Slide 33, please.
21 There are a few what we are calling boundary
22 conditions for this review. I need to say, I think, largely
23 the group is starting with a blank sheet of paper. There
24 are very few restrictions that we are placing on them. So
25 we are not tying the process that will come out of the
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 63
1 integrated review effort to any status quo program that we
2 have now.
3 We are saying, though, that the inspection program
4 and the enforcement policy are being assumed to be
5 fundamentally sound at this point. This doesn't mean at all
6 that we think it's adequate to fulfill what will come out of
7 the process. It's an obvious follow-on activity that we
8 will have to identify gaps in the current inspection
9 program. We already talked about one today. If we look at
10 management effectiveness in the new template, there is
11 nothing in the inspection program now that directly inspects
12 that activity. It's an inferred judgment under the current
13 process.
14 We are going to have to identify new procedures,
15 new guidance, and then train the staff. So this isn't going
16 to be a quick solution to these identified weaknesses or
17 gaps in the program.
18 CHAIRMAN JACKSON: But it won't go on for a
19 decade.
20 MR. BORCHARDT: No. I hope not.
21 We know that these groups of processes are going
22 to have to assess all plants. The topic of using it for
23 public interaction and the opportunities for licensees to
24 respond is kind of the element that is currently in the SALP
25 program. We want to retain some element of that.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 64
1 CHAIRMAN JACKSON: Commissioner McGaffigan.
2 COMMISSIONER McGAFFIGAN: Two questions, one on
3 the second point. You said obviously we are going to change
4 the inspection program to try to take into account
5 management effectiveness and you are already doing that as a
6 result of a short-term action. Do you envision the
7 integrated assessment making recommendations with regard to
8 the inspection and enforcement program?
9 You are not saying you are reviewing them, but you
10 are going to end up with something that may not match up.
11 So you are going to have to make some recommendations at
12 least preliminarily how they might have to adjust to
13 whatever you are proposing.
14 MR. BORCHARDT: I would be personally satisfied if
15 they identify the gaps, where we need to do more work. I
16 think it might be too much to ask of them to come up with
17 recommended fixes given the time schedule that we are trying
18 to do this on and the number of resources we are applying to
19 it. It would be a quick turnaround activity for the
20 Inspection Program Branch and NRR to take those identified
21 gaps and then come up with a program to fill those.
22 COMMISSIONER McGAFFIGAN: My second question is on
23 the third tick, the performance of all plants categorized.
24 Is "categorized" a synonym for "scored"? Are there
25 conceivably two categories, the watch list and everyone
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 65
1 else?
2 The reason I raise that question is people come in
3 and tell me that being SALP-1 is a great motivator for the
4 folks at their plants. I went through this conversation the
5 other day with a licensee. When I was at Harvard and they
6 implemented pass-fail my senior year, we treated pass-fail
7 courses very differently from the courses where we got A's
8 and B's and C's, although I guess nowadays everybody gets
9 A's.
10 [Laughter.]
11 COMMISSIONER McGAFFIGAN: I actually think there
12 is a benefit to the old grading system where you are a tough
13 grader and you give out A's, B's and C's.
14 Is categorization a synonym for scored?
15 MR. CALLAN: Let me answer that. We have had a
16 lot of discussions on that subject. The first observation
17 I'll make is that the resources go up exponentially with the
18 way you parse scoring. If you have four categories and you
19 want to have a credible scoring system, it takes more than
20 four times as many resources to do that probably than it
21 would be to have one score, or to break performance down
22 into quartiles, top quartile, second quartile, third
23 quartile, fourth quartile.
24 I think what we can say at this point is there
25 will be some kind of ranking. How we score it is another
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 66
1 matter. We rank plants now as part of the screening
2 meeting; regions rank the plants. That's how you get the
3 best performers that you spend less time on.
4 That's a tough question, and it's linked, as I
5 said, directly to resources.
6 MR. COLLINS: Commissioner, there are a lot of
7 ways to look at evaluation of plants. Even with the
8 periodic reviews that are done we de facto rank plants by
9 assigning inspection resources. When the PIM becomes public
10 and our responses to the PIM, which is a resource-loaded
11 letter to the licensees, become more routine, there will be
12 a rack-up of the effort of plants.
13 I think part of coming to a conclusion of how do
14 we rank or grade plants becomes more of what do we intend
15 for the process to achieve beyond the allocation of
16 resources and the communication of performance to the
17 licensees and what's the most effective way to do that.
18 I understand some licensees come in and are
19 motivated by category 1's. I, quite frankly, haven't heard
20 that. I've heard the other side of the argument.
21 COMMISSIONER McGAFFIGAN: Maybe they only talk to
22 me, but it is more than a handful who seem to be motivated
23 to get to straight SALP-1. They believe that's a very
24 useful tool for motivating their workforce.
25 CHAIRMAN JACKSON: I guess I believe that we've
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 67
1 asked you to do it, so now we ought to let you do it and
2 come back to us with what you come back with.
3 MR. BORCHARDT: Slide 34, please.
4 The process is really rather simple. NRR has the
5 project lead. There is going to be a series of meetings.
6 Right now it's envisioned to be four. It may end up having
7 to be a few more than that.
8 The participants right now are shown on backup
9 slide number 6. They include participants from each of the
10 regions and some of the other program offices.
11 Even if program offices are not particularly
12 listed on that slide, there will be a series of less formal
13 meetings held here in headquarters with the other offices so
14 that they are all apprised of the ongoing results of each of
15 the steps and we can receive their input.
16 The schedule runs over the next 18 months or so.
17 The development will largely be, at least the initial
18 development of staff options, over the next months. May of
19 next year is a time period for public and industry comments.
20 The activities from around June, about the summer
21 of next year, until the end of the year are largely being
22 allocated for the development of new procedures, management
23 directives and training for the staff.
24 The eventual outcome is as significant as it could
25 be. It's going to be a significant mind-set change for the
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 68
1 staff and the industry, but for the staff to implement, a
2 staff that is now very comfortable, I think, to a large
3 degree with it. Although highly labor intensive, they have
4 a lot of practice with it and they are comfortable with it.
5 So if we change it, it's going to be a significant training
6 evolution to get everybody adapted into the new program.
7 That completes my presentation.
8 CHAIRMAN JACKSON: Thank you.
9 Commissioner Dicus.
10 COMMISSIONER DICUS: I realize we are running over
11 and there are some other things that have got to be done.
12 Back to the senior management meeting. You
13 mentioned the ACRS had quite a few concerns that they have
14 brought to your attention, and you indicated that you would
15 probably be looking at these and bringing them into play at
16 some point and evaluating them. One of them in particular
17 was the idea that maybe your overall approach, which was
18 kind of from the bottom up, should be from the top down.
19 Would you care to comment on that?
20 MR. BARRETT: Yes. We actually have given a fair
21 bit of thought to a top-down approach although we haven't
22 emphasized it in our information.
23 We have been working with the Office of Research,
24 for instance, to develop a decision model. We feel that the
25 decision model is really the way in which you take what it
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 69
1 is right now, a list of categories, and integrate them into
2 what we believe is in fact a comprehensive structure for
3 performance assessment.
4 Backup slide number 4, for instance, gives you an
5 example of some of the thinking we've done along that line
6 in terms of trying to take these categories and show the
7 relationships among them and also show how they relate to
8 risk. In this particular slide what we are trying to do is
9 show how risk is a combination of three of these categories
10 and how they are supported by two of the other ones, namely,
11 engineering and organizational effectiveness, and then
12 problem identification and resolution as a feedback
13 mechanism.
14 So we have done some thinking about this and we
15 are committed to coming up with a model that is truly an
16 integrated model in response the ACRS as well as our own
17 motivation.
18 MR. KNAPP: I might add that Research is getting
19 very much involved in that particular concern the ACRS
20 raised, and hopefully we are providing a fair amount of
21 support to AEOD and doing, I think, more or less what they
22 have said, looking at the decision and then backing into the
23 template so that we are coming at this from both ways. I
24 think this will work pretty well by the time we are all
25 done.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 70
1 CHAIRMAN JACKSON: Commissioner Diaz.
2 COMMISSIONER DIAZ: I have a couple of comments
3 mixed with questions. In the interest of time, rather than
4 answering the questions, I will ask the staff to consider
5 them when they are putting their things together. I think
6 it is obvious the preoccupation that the senior management
7 process has created inside and outside this organization.
8 I think when we get to February 1998 there are a
9 series of questions that are important that we answer for
10 ourselves and for the outside. Maybe a basic question is,
11 what is the senior management process?
12 We need to be able to come and eventually define
13 it in a manner that is understandable to us and to our
14 stakeholders and to the Congress of the United States and to
15 everybody that really deals with it, including the press.
16 It is obvious that the senior management meeting
17 is no longer a meeting of senior managers; it has become a
18 much more elaborate process. The amount of time and
19 resources that it uses and this effort show its importance
20 in our organization. In 97-04, I think I recommended that
21 we change the name. I can't even remember. National
22 evaluation of licensee performance, which nobody liked; or
23 national assessment of plant performance.
24 CHAIRMAN JACKSON: He's trying to get them in
25 again.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 71
1 COMMISSIONER DIAZ: I wouldn't do that.
2 [Laughter.]
3 COMMISSIONER DIAZ: National assessment of plant
4 performance. Two people liked that.
5 [Laughter.]
6 COMMISSIONER DIAZ: In this regard, in answering
7 that question, I think there are two fundamental questions
8 in what is the senior management meeting or the national
9 assessment of plant performance.
10 [Laughter.]
11 COMMISSIONER DIAZ: And that is, what does this
12 process provide in terms of health and safety? What does
13 the actual process provide for this agency in terms of
14 health and safety?
15 When we answer that, then we need to ask ourselves
16 and reply, what does the watch list provide in terms of
17 health and safety?
18 I think those are very important questions that
19 really have been brought out even as early as yesterday by
20 Senator Biden's concern, which we need to reply to, and
21 those concerns need to be addressed and they need to be
22 addressed earlier rather than later, because I believe we
23 might have to be accountable to Congress in early 1998. So
24 February 1998 becomes an important date for the Commission
25 to have the appropriate information.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 72
1 Obviously when the process was created we did not
2 intend it to generate this much attention, but it has and it
3 is there.
4 As you further define these processes, I recommend
5 that we be able to establish clearly and for some reasonable
6 period of time, to us and to everybody, this basic question:
7 Is the senior management meeting, or the national assessment
8 of plant performance, a dominant inspection, assessment,
9 enforcement, and regulatory process?
10 If it is, let it so be known, and let the
11 Commission decide on whether that is what we want to do. If
12 it's not and your recommendation is that it not be, let it
13 so be known so that it occupies its proper place in our
14 regulatory infrastructure.
15 I believe that you have gone quite forward. I
16 think that the process is now converging, and I want to
17 thank you for your efforts in this regard.
18 CHAIRMAN JACKSON: Thank you.
19 I would like to thank the staff for a very
20 informative and very interactive briefing. These processes
21 play a vital role in helping to develop and to provide an
22 agency perspective on plant performance.
23 You've already heard we want you to continue along
24 the line you have been moving. I think there needs to be
25 additional focus on the issue of the decision-making process
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 73
1 at the senior management meeting. I think to some extent it
2 was finessed in this discussion.
3 It's my understanding that information that will
4 be available at the upcoming senior management meeting, and
5 I think you've indicated it, includes the template, the
6 performance trends, and the economic data plots. I think
7 you need to try to give the Commission -- no?
8 MR. MARTIN: Not the template.
9 CHAIRMAN JACKSON: Not the template. Okay.
10 You should provide to the Commission prior to the
11 next senior management meeting a little more information, as
12 much as you have developed, on how you intend to use the
13 information to reach decisions, what is the actual
14 decision-making process, and that you consider the
15 information and the suggestions provided in the recent ACRS
16 letter regarding the template and the senior management
17 meeting process.
18 There are some issues having to do with being able
19 to have the Commission have more information perhaps in a
20 graphical form that provides information about both false
21 alarm as well as detection processes as a function of the
22 observation periods. There was a discussion about how long
23 your data goes over. It's really kind of casting the
24 information in a somewhat different format.
25 As Commissioner Diaz' comments illustrated, I
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 74
1 think the important point is that we have clarity on what we
2 are trying to accomplish with the various processes. The
3 senior management meeting, I don't just focus on that, but
4 it's kind of a culmination point that everyone seems to
5 focus on. We have an overall set of processes, and I was
6 happy to hear Mr. Borchardt say how excited the staff was at
7 this opportunity to reassess.
8 The real point is, given our health and safety
9 mission, you should ask yourselves the following question.
10 I think this gets at what the Commissioner says. Do the
11 outputs give us the outcomes that we desire from the point
12 of view of our health and safety mission? And is it
13 consistent with our agency goals? And does it allow us to
14 have a measurement of the health and safety value added?
15 If you can do that and keep that in mind as you go
16 through this, then I think we will be in good shape at the
17 end.
18 I think Commissioner Diaz wants to make one last
19 comment.
20 COMMISSIONER DIAZ: Madam Chairman, I'm sorry, but
21 you said something that triggered my mind and I think it's
22 an important issue. As we define what these processes are,
23 I think it should be very, very clear to the stakeholders,
24 to the public, to the Congress that the senior management
25 meeting, however we cast its importance, is just one of the
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
. 75
1 processes.
2 CHAIRMAN JACKSON: Right.
3 COMMISSIONER DIAZ: It doesn't take away
4 importance or weight or value from the day-to-day
5 inspections and assessments that are done by our people on
6 the line every day, and the decisions of health and safety
7 will be done on a daily basis independent of whether the
8 senior management process takes place, because I think that
9 is an overriding consideration.
10 CHAIRMAN JACKSON: It's something that people
11 misunderstand, but I think showing the connectivity from the
12 beginning to the end of everything we do helps to remove the
13 excessive focus on any given part of the process.
14 Unless there are further comments, we are
15 adjourned. Thank you.
16 [Whereupon at 3:16 p.m. the meeting was
17 concluded.]
18
19
20
21
22
23
24
25
ANN RILEY & ASSOCIATES, LTD.
Court Reporters
1250 I Street, N.W., Suite 300
Washington, D.C. 20005
(202) 842-0034
Home > Electronic Reading Room > Document Collections > Commission Documents > Commission Meetings, Slides, Transcripts, Meeting SRMs, and Full Written Explanation for Closed Meetings > 1997
>
09/19/1997