 |
Search Options | |||
| Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us |  |
|||
Home > Electronic Reading Room > Document Collections > Commission Documents > Commission Meetings, Slides, Transcripts, Meeting SRMs, and Full Written Explanation for Closed Meetings > 1997
>
02/18/1997
1
UNITED STATES OF AMERICA
NUCLEAR REGULATORY COMMISSION
***
BRIEFING ON ANALYSIS OF QUANTIFYING
PLANT WATCH LIST INDICATORS
***
PUBLIC MEETING
***
Nuclear Regulatory Commission
Commission Hearing Room
11555 Rockville Pike
Rockville, Maryland
Tuesday, February 18, 1997
The Commission met in open session, pursuant to
notice, at 2:39 p.m., the Honorable SHIRLEY A. JACKSON,
Chairman of the Commission, presiding.
COMMISSIONERS PRESENT:
SHIRLEY A. JACKSON, Chairman of the Commission
KENNETH C. ROGERS, Member of the Commission
GRETA J. DICUS, Member of the Commission
EDWARD McGAFFIGAN, JR., Member of the Commission
NILS J. DIAZ, Member of the Commission
. 2
STAFF AND PRESENTERS SEATED AT THE COMMISSION TABLE:
JOHN C. HOYLE, Secretary of the Commission
KAREN D. CYR, General Counsel
EDWARD JORDAN, Deputy Executive Director for
Regulatory Effectiveness, Program Oversight,
Investigations, & Enforcement
DR. DENWOOD ROSS, Acting Director, AEOD
RICHARD BARRETT, Deputy Director, Incident
Response Division, AEOD
IRA GOLDSTEIN, Arthur Andersen, Partner, Federal
Industry
KAREN VALENTINE, Arthur Anderson, Senior Manager,
Office of Government Services
LOUIS ALLENBACH, Senior Management Consultant
KATHRYN KELLY, Senior Consultant
AARON LIEBERMAN, Senior Consultant
. 3
P R O C E E D I N G S
[2:39 p.m.]
CHAIRMAN JACKSON: Good afternoon, ladies and
gentlemen.
I am pleased to welcome members of the NRC Staff
to brief the Commission on the Arthur Andersen Assessment of
the Senior Management Meeting Process and Information Base.
The assessment was performed to ascertain how the
Senior Managers can improve the timeliness and thoroughness
of its plant safety assessments. The Senior Management
meeting process is intended to facilitate the early
identification of plants which require increased regulatory
attention
The Commission has indicated previously its belief
that there is room for improvement in the Senior Management
Meeting decisionmaking process. These improvements relate to
making the process more scrutable, using objective data with
well-defined decisions criteria.
The objective ultimately should be to attain a
clear, coherent picture of performance at operating reactor
facilities.
I understand that copies of the slide presentation
are available at the entrances to the meeting room, so
unless my fellow Commissioners have any opening comments,
Mr. Jordan, please proceed.
. 4
MR. JORDAN: We changed on you from the last
meeting.
CHAIRMAN JACKSON: That's right.
MR. JORDAN: Thank you very much, Chairman
Jackson, Commissioners.
With me at the table are Dr. Denny Ross, Acting
Director of AEOD and Rich Barrett, Deputy Director, Division
of Incident Response, who provided direct management
oversight of this effort.
Seated behind us are some of the Arthur Andersen
personnel who conducted the assessment.
Ira Goldstein is the partner in charge of Arthur
Andersen's Federal Industry --
CHAIRMAN JACKSON: Raise your hand high. Thank
you.
MR. JORDAN: Thank you -- Federal industry work.
Karen Valentine is the Senior Manager of the
Office of Government Services.
Louis Allenbach is the Senior Management
Consultant.
Kathryn Kelly is a Senior Consultant.
Aaron Lieberman is a Senior Consultant.
They are available to respond to specific
questions about their work that NRC Staff are unable to
answer.
. 5
The Arthur Andersen study of the Senior Management
Meeting grew out of a discussion at the June 25th, 1996
periodic Commission meeting on operating reactors in fuel
cycle facilities.
At that meeting the Commission raised a number of
questions about improving the information base of the Senior
Management meeting in order to make the Senior Management
Meeting decisions more objective, consistent and timely.
Following the issuance of an SRM on June 28th,
1996, the responsibility for this assessment was assigned to
AEOD by the Executive Director for Operations. The AEOD
staff decided to conduct an independent assessment of the
Senior Management Meeting process using a contractor with
extensive experience in management consulting and
performance indicators.
Arthur Andersen Consulting was selected for this
responsibility, using a streamlined process to select from a
list of GSA approved contractors. For the four-month period
of the study the AEOD staff provided Arthur Andersen with
the information and access they needed in order to provide a
creditable assessment.
The NRC Senior Advisory Panel was created to
review and comment on the NRC Staff proposed statement of
work and to provide input at key milestones in the study.
The Advisory Panel consisted of myself, Jim Milholland, Dave
. 6
Morrison, Stu Ebneter, and Frank Miraglia.
The report you have received and will be briefed
on today represents the views of Arthur Andersen Consulting.
The NRC Staff has begun an aggressive effort to evaluate the
recommendations and develop implementation options. The NRC
Staff recommendations will be presented in a Commission
paper which we plan to forward in the end of March, this
year.
The briefing this afternoon is intended to review
the findings and recommendations of the Arthur Andersen
report without providing NRC Staff views, and that is
normally difficult but Rich, I will ask you to begin the
presentation, please.
CHAIRMAN JACKSON: Commissioner?
COMMISSIONER McGAFFIGAN: Just one comment,
because we won't come back to this.
I would like to commend you for the process that
you went through, the streamlined procurement process, I
think to get this study in this timeframe. I think that is,
whether it was AEOD or working I'm sure with Procurement
shop, the strategy of going to the GSA approved list,
getting a contract with the appropriate qualifications and
getting them on board rapidly, that's very refreshing
because it often times takes a lot longer to get this sort
of study.
. 7
MR. JORDAN: I intended to give Admin the credit
for assisting us in that effort. Thank you.
MR. BARRETT: Good afternoon, Chairman Jackson,
Commissioners.
If I could have Slide 2, please.
Our intention this afternoon is to simply go
through the content of the Arthur Andersen report including
the methodology that they used in preparing the report and
also to present their findings about the outcomes of past
Senior Management Meetings as well as their findings and
recommendations regarding the information that we have used
in the past and the information we might use in the future
for Senior Management Meeting decisions, and the process we
use for making these decisions.
As Mr. Jordan pointed out, we will briefly at the
end talk about the schedule for the Staff's evaluation of
the recommendations and for development of options for
implementation.
Slide 3, please.
I think Mr. Jordan has already pretty well gone
over the chronology of the study. I would like to point out
however one thing I think is of interest.
The original Staff requirements memorandum
concentrated on the development of indicators that could
form a more objective basis for Senior Management Meeting
. 8
decisions. After Arthur Andersen came on board and began to
review the written documentation from the Senior Management
Meeting they made the recommendation to us that we expand
the study so that we also look at the process itself because
their feeling was that a great deal of what was happening in
the Senior Management Meeting was related to the process we
used and that to have a full examination of a step toward a
more objective measures required us to look at the process.
The Staff evaluated that recommendation and
concurred with it, so the contract was modified at that
point and we went forward with the fuller scope of work.
If I could have Slide 4, please.
Arthur Andersen assigned nine professional to this
task. As Mr. Jordan mentioned, it was led by a partner of
the firm as well as two senior managers of Arthur Andersen.
In addition, they involved part-time two of their senior
staff with extensive experience in utility finances as well
as nuclear operations, some experience in nuclear
operations, and four very capable staff members who worked
primarily almost full-time throughout the course of the
study.
The methodology they used was quite thorough in my
opinion. They first of all did a very thorough review of
the written record of the senior management meeting from
1992 to 1996.
. 9
That included the briefing books, which are
supplied to the Senior Managers prior to the meeting, the
Minutes that are published after the meeting, and the
transcripts of the Commission briefings that are given after
each one of the meetings.
Based on their review of the written record they
developed an extensive database. This was a database of the
characteristics and measures that were most often cited as
being the basis for the decisions, the performance
characteristics and performance measures.
In fact, they ended up with a database of 1700
records, which provided a great deal of insight into the
bases that we have used in the past for these decisions.
Secondly, Arthur Andersen conducted over 30
interviews of three types -- interviews with NRC Senior
Managers who have past experience with the Senior Management
Meeting, both Headquarters Managers and Regional
Representatives from all of the regions; we interviewed
Resident Inspectors and their immediate supervisors in the
Regional office; and we interviewed five senior utility
executives at the Vice President, Nuclear level.
Now the purpose of these interviews was different
in each case. In the case of the interviews with the NRC
Senior Managers, what we were trying to get there was an
understanding of how the Senior Management Meeting process
. 10
works, because it was very important for Arthur Andersen to
understand that and of course they had no opportunity to
attend the meeting.
Also, to understand in the opinion of the people
who participated what were the most important factors in
shaping the decisions that have been made in past Senior
Management Meetings and also to understand what the roles of
the various participants in the meetings are and finally to
see if these Senior Managers had any suggestions for process
improvements or if they felt that there were any plants that
if they had a chance to go back and look again they might
have treated differently -- so that was the purpose of
interviewing the NRC Senior Managers.
The purpose of interviewing the Resident
Inspectors and their immediate supervisors was for Arthur
Andersen to get a sense of how information that is
fundamental to the Senior Management Meeting performance
assessment process, how it is first gathered and how it is
developed an analyzed as it moves up through the chain of
events and then becomes part of our assessment, performance
assessment processes, such as the SALP and the Senior
Management Meeting.
Finally, the purpose of interviewing the utility
executives was to get a sense of how much they use
performance indicators in evaluating their own plants and
. 11
how they make use of performance indicators.
Also, we wanted to get a sense of what their
understanding of the Senior Management Meeting process was
from an outsider's perspective, so we conducted over 30
interviews.
Third, the third aspect of their methodology was
to create what they call a performance trend model, and we
will actually show you an example of the performance trend
model later in this presentation and we'll discuss it in
great detail, but the purpose of the performance trend model
was to demonstrate how indicators could be used in making
decisions related to the Senior Management Meeting,
indicators that are already available to the NRC Staff and
are already developed in the processes that we have
ongoing -- and how criteria could be used in conjunction
with those indicators to inform the process of
decisionmaking.
COMMISSIONER McGAFFIGAN: Could I ask --
MR. BARRETT: Sure.
COMMISSIONER McGAFFIGAN: -- maybe it's
appropriate to wait until later, but the indicators that are
available to the Staff, are they also available to the
public?
If you went through our documents, could you make
one of the charts that you are going to show us later from
. 12
the publicly available information?
MR. BARRETT: At the moment, seven of the nine
indicators we use are routinely made available to the
public.
These are the NRC performance indicators.
The two other indicators, which were related to
our enforcement and to numbers of allegations are not, I
believe, routinely made available to the public although I
don't believe there is any problem with making them
available to the public.
CHAIRMAN JACKSON: But we do have those indicators
ourselves and we make use of them.
MR. BARRETT: We do, yes.
CHAIRMAN JACKSON: Or we at least trend them at
this stage.
MR. BARRETT: We trend seven of the nine and the
other two I believe are just used internally within the
offices that they are developed in.
CHAIRMAN JACKSON: I see, but the information base
for developing indicators with them exists.
MR. BARRETT: It exists, yes.
Okay. It was not a great deal of effort for
Arthur Andersen to develop these charts with the
information.
The fourth item they did was to create an
. 13
integrated performance model, and again we will look at the
integrated performance model, but the purpose of the
integrated performance model was to illustrate how different
types of information could be used at various stages in the
performance assessment process, and again we will discuss
that in some detail.
Finally, Arthur Andersen developed a process map,
and part of that was developed -- this is a process that
describes how the NRC gathers information of various types,
how we analyze it, and use it in various processes such as
enforcement, the SALP process, and other processes leading
up to the Senior Management Meeting.
We don't plan to go into detail today on that
process map, but it is available in the report in Appendices
3 and 4.
If I could have Slide 5, please.
Arthur Andersen drew some conclusions about the
past record of the Senior Management Meeting with regard to
identifying poorly performing plants and with regard to
taking formal action.
CHAIRMAN JACKSON: Let me ask this question. Did
the use of the Arthur Andersen performance trend charts
identify any plants with poor performance which had not been
identified for discussion or vice versa?
MR. BARRETT: If you looked at the Arthur Andersen
. 14
performance trend plots and the criteria that they developed
as a straw man criteria, there would be plants that would
come up that were not on the list and were not discussed.
CHAIRMAN JACKSON: Would those have been ones that
upon discussion with Senior Managers or utility execs might
be agreed that should be on the list but were not, or -- or
was there any agreement that any that had previously been
placed on the list should not have been?
MR. BARRETT: There were no cases where plants had
been placed on the list where there was agreement among
anyone interviewed that it should not have been placed on
the list.
There were cases of plants, there were two cases
of plants that have been on the list where you could not
have identified those performance problems purely on the
basis of indicators. You would have to have looked at other
information to identify those as problem plants.
With regard to whether there were plants that
should have been on the list according to the charts that
were not on the list in the past. Yes, there were. There
were some that based on these, on this particular chart with
these criteria, would have been identified.
I would say that, and Arthur Andersen would say
this, that these particular indicators and these particular
criteria are not meant to be the set of indicators and
. 15
criteria and their recommendation to the NRC is that they
use the insights from the study to go in and do a systematic
look at indicators and criteria to come up with the ones
that we feel are the true indicators of performance.
CHAIRMAN JACKSON: So when you come back with the
paper in March, you intend to have identified what those
indicators really should be?
MR. JORDAN: Yes, Step 1. We will never have the
final answer but we will have an improved list --
CHAIRMAN JACKSON: With improved criteria or
refined criteria?
MR. JORDAN: Yes.
CHAIRMAN JACKSON: Dr. Ross?
DR. ROSS: I was going to say we were cautioned,
and this is in the report, in Appendix 2, page 2 -- in fact,
they felt strongly enough about it that they put it in
italics. They said that "The stress of our
recommendations" -- of course, meaning the Arthur Andersen
recommendations -- "lies in the methodology, not in the
numbers reported in the methodology. The NRC should first
conduct a review of the selected performance indicators to
be used when analyzing performance trends and then turn its
attention to formalizing a methodology such as the one
proposed to categorize plants."
And I think that is what we need to do.
. 16
CHAIRMAN JACKSON: Okay. Commissioner McGaffigan,
did you have a comment?
COMMISSIONER McGAFFIGAN: I'll come back.
CHAIRMAN JACKSON: Okay.
MR. BARRETT: I will come back to that question,
your question, in a moment.
First of all, with regard to the outcomes, in
general the Arthur Andersen concluded that for plants that
had performance problems the NRC has identified them for
discussion and that that was a fairly favorable result.
In addition, they concluded that plants that had
been put on the Watch List in the past had been placed there
appropriately, that the NRC has not been in the habit of
over-reacting in terms of putting plants on the Watch List.
Arthur Andersen also concluded, however, that the
NRC, the Senior Management Meeting has sometimes been slow
in taking formal actions in terms of trending letters or
Watch List designation and that NRC outcomes, Senior
Management Meeting outcomes, appear to be inconsistent.
That is to say that plants with apparently similar
performance have had experienced different outcomes.
Now if I could get back perhaps to a more full
discussion in answer to your question, the basis for the
Arthur Andersen's conclusions was really the entire scope of
the information they looked at.
. 17
We have no base truth here. We really don't have
anything with any fundamental principle we can go back to
and say based on this fundamental principle we now know
which plants should have been on the list or should not have
been on the list, so we had to use the preponderance of
information that was available, and the information that was
available, first of all, was their review of the written
record.
When Arthur Andersen reviewed the written record
of the Senior Management Meeting, their staff formed certain
impressions about the severity and the duration of
performance problems and based on that they came to
preliminary views about which plants seemed to deserve to be
put on the Watch List or deserved to get trending letters.
The second source of information that was used
were the interviews.
In the interviews with our own Senior Managers,
many of them expressed in hindsight the views that certain
plants probably should have been treated differently, so
that was a second source of information.
Finally, the performance trend model was developed
and was run for 109 plants and there were many cases where
the results of the model did not comport with the results of
the Senior Management Meeting.
The conclusions that Arthur Andersen drew are
. 18
based on a confluence of those three sources of information
where consistency could be seen in all three sources of
information.
CHAIRMAN JACKSON: Since we are talking about the
Arthur Andersen assessment, and since we have the benefit of
having this team sit here, I am going to ask whoever is the
senior-most person on that team to offer to give us any
further illumination you might wish to provide.
MR. GOLDSTEIN: Madam Chairman, I'm Ira Goldstein.
I am the partner responsible for this engagement and,
indeed, for our government work.
I think the summary that the staff has given to
you is an accurate reflection of the work that we did and of
the conclusions that we drew. I would focus for one moment
on the general perception, as I think Mr. Barrett said, that
the correct set of indicators had been looked at, that a
great deal of information had always been collected and that
if there was one indication of change that we concluded
should be focused on, it was the extent to which discussions
occurred that led to watch list placements somewhat later
than our model would indicate could have been the case.
The other conclusion, if you will, that I would
focus on is the balance between the objective indicators and
subjective judgments. Our belief, as I think the staff and
the staff of the NRC has always believed that ultimately
. 19
judgment must prevail. What we found was, as we looked at
the outcomes of the senior management meetings, that some of
the performance indicators could be used in a more objective
way as indicators that could lead to what we would like to
call a presumptive judgment and that is that the model can
give you some indication that there might be a presumption
that a particular plant could be appropriate for the watch
list, subject to rebuttal in a discussion. Our
recommendation secondly focused on that.
Thirdly, we also provided some recommendations
relating to the breadth and depth of the discussion in the
meeting and that we felt and I think the staff has expressed
sympathy with this perception that expanded participation
and expanded independent debate within that meeting could
lead to a fuller discussion of those indicators.
So with those three points of focus, I certainly
believe that the recounting that you hear is an accurate
reflection of what we reported.
CHAIRMAN JACKSON: Thank you.
MR. BARRETT: If I could have slide six --
COMMISSIONER ROGERS: Before you leave this slide,
I wonder if you could just clarify what you mean by "most"?
I see the word "most" appearing here a couple of times and I
just want to get a little feeling, particularly about the
second bullet. Most NRC senior manager utility executives
. 20
agreed that plants on the watch list were appropriately
placed.
How large was the disagreement there?
MR. BARRETT: I wouldn't say there was
disagreement. There was really a question of those people
who addressed the question and those people who did not. I
don't recall and perhaps Arthur Andersen recalls, but I
don't recall anyone saying that, disagreeing with that
statement. It was just a question of which people addressed
it and which people did not.
COMMISSIONER ROGERS: All right.
If that's what you found. Is that what you did
find in your interviews, folks from Andersen?
MR. GOLDSTEIN: Yes.
COMMISSIONER ROGERS: All right.
MR. BARRETT: If I could have slide six now?
The next three slides relate to findings and
recommendations of the Arthur Andersen study with regard to
the information base, the information we use for the current
senior management meeting decisions. And Arthur Andersen
made some favorable conclusions about our information base
which I think are very heartening.
First of all, one of their first impressions was
that the NRC has a wealth of information available to us, a
wealth of information that is directly applicable to the
. 21
assessment of performance and directly applicable to safety
and they don't always find that when they go out to assess
organizations. So there were no significant gaps that we
need to go out and start new major programs to develop new
information.
They also concluded that the performance
characteristics that have been used in past senior
management meeting decisions are indeed related to safety
and are related to risk, so again a very positive, positive
finding.
Arthur Andersen did identify what they considered
were conditions, however, related to how information was
handled and how information is used. First of all, they
concluded that the NRC focuses on events, tends to focus on
events or major problems that occur at plants and then,
based on those events, take a retrospective look at the
plant, looking for the root causes and quite frequently
finding the root causes in problems with management
effectiveness and operational effectiveness.
And what Arthur Andersen basically is recommending
is if we continue to focus on events in this way, we are
going to be identifying performance problems later than we
could. If, on the other hand, we had an ongoing systematic
program for assessing management effectiveness and
operations effectiveness, that we would have a program that
. 22
identified performance problems earlier and would give
licensees more of an opportunity to turn these problems
around before they become significant to safety.
CHAIRMAN JACKSON: That actually raises an
interesting question in my mind. The question is, is that
to say that the NRC does not assess management and
operational effectiveness on an ongoing basis or that that
assessment occurs but in the senior management meeting
decisions it is not focused on? And those are separate
issues. So I don't know if you want to speak to it or the
Arthur Andersen rep wants to speak to it or both.
MR. BARRETT: I think Arthur Andersen would say
that the management and operations effectiveness are clearly
focused on in most of the major programs, especially the
inspection program at the NRC. For instance, operations
effectiveness is a key focus of the SALP process.
What they are saying, basically, is that we need
to have a more systematic and structured way of developing
management effectiveness and operations effectiveness
information in a way that better feeds the senior management
meeting process. So it's a question of how information is
handled and how it's used.
DR. ROSS: The retrospective might be the key word
in terms of what are leading versus lag, and more focus on
the second bullet might produce leading indicators, which is
. 23
the main lesson to extract from this.
CHAIRMAN JACKSON: Okay, but there is a separate
question that underlies this and that is the question of is
there anywhere in our plant assessment processes that we
focus on management and operational effectiveness as leading
indicators? That's the first question, that's part A.
And part B is, if the answer is, yes, are we
saying that it is not used as such in the senior management
meeting process? So that's question one. Or is it that we
don't assess it?
I mean, those are two separate questions. You
see, we do SALP, we do plant performance reviews, we do
this, we do that. And the question is, do we focus on
management and operational effectiveness at those levels but
on an ongoing basis but it doesn't propagate to the senior
management meeting. Or are we saying that we don't,
anywhere in our program, focus on an ongoing basis on
assessing management and operational effectiveness and those
are two separate kinds of things.
MR. JORDAN: Right. I think I can try to answer
that.
Certainly the discussions in the senior management
meeting talk both about management and the SALP process
provides data input evaluations on operational
effectiveness. So they are both present.
. 24
In terms of having the data assimilated in a way
that is more easily used by the senior managers, I believe
that's the focus. So there are assessments but the
structure and collection of the information is not conducive
to use and we do, in fact, extract much of our information
about management effectiveness from things that happen as
opposed to a more I would say overview of capabilities.
And that is sort of historic. In the past, when
we try to look at capabilities, the industry itself was
critical of the NRC going in as a paragon of management
skills and knowledge and not looking at performance because
it really is an idea of management performance. So the
staff has been cautious, I believe, in assessing management
in terms of their capabilities as opposed to their
performance.
CHAIRMAN JACKSON: Mr. Goldstein looks as though
he has an itch.
Mr. Goldstein, I think when you sit down, we would
like you to sit in a green chair after this.
MR. GOLDSTEIN: A green chair?
CHAIRMAN JACKSON: here at the table.
MR. GOLDSTEIN: At the table, okay.
My wife points out every morning when I pick my
ties that I am close to color blind so that as we wave over
the chair --
. 25
CHAIRMAN JACKSON: Well, it turns out that your
tie matches the chair.
[Laughter.]
MR. GOLDSTEIN: I will mention that when I get
home.
CHAIRMAN JACKSON: She set you up.
MR. GOLDSTEIN: She has done that before, Madam
Chairman.
Let me reinforce something that Mr. Jordan said
and maybe even extend it a little further. We have put into
the report what we call an integrated performance model that
speaks very directly to the issue you are raising and
indeed, as Mr. Jordan said, management effectiveness is
discussed and is looked at and there is a great deal of
discussion in the record of management effectiveness.
But as one looks at risk and resource allocation,
the closer you get to an actual performance failure, the
more difficult it is to do something constructive and the
more the risk goes up and the more resources it takes to fix
the problem.
We like to view the levels of indicators as four
groupings. Furthest from the event is economic stress. If
you could see that, that would give you some more distant
indication. Management effectiveness, perhaps next.
Operational effectiveness, getting closer. And then
. 26
ultimately, performance results.
I would respond to your question by saying the
discussions of management effectiveness appear to be
triggered in the senior management meeting by results
events, by performance events as opposed to being a leading
edge of that type of performance.
CHAIRMAN JACKSON: Okay, thank you.
Commissioner Diaz?
COMMISSIONER DIAZ: Would it be fair to say that
as important as propagating an assessment of operating
effectiveness would be to negatively bias events that might
not have significance as part of this operational
effectiveness rather than propagating the event, the event
in a continuously amplified basis. Would you think it's --
would you like me to restate that?
MR. GOLDSTEIN: I would appreciate it. I am
having trouble understanding the relationship you're
drawing.
COMMISSIONER DIAZ: Okay, there are two issues in
here. One is we take an event and that event might tend to
dominate the process and then the actual assessment of
operational effectiveness might not propagate and be
properly amplified through the system to give it its
importance.
My point is that, as we look at the indicators, it
. 27
might be as important to negatively bias an event, okay,
that might not have full safety significance and comparable
impact on operational effectiveness and it is to amplify
properly those components that do have operational
significance on safety.
MR. GOLDSTEIN: I agree. It could be. But as
Mr. Jordan pointed out, it is much more difficult to assess
in any objective way management effectiveness and so I would
be cautious about using it as an amplifier or as a reduction
because it's a much softer indicator.
COMMISSIONER DIAZ: But I didn't say management.
I took the word "management" out. I said operational
effectiveness and event-related response.
MR. GOLDSTEIN: Easier to deal with, no question.
COMMISSIONER DIAZ: Thank you.
CHAIRMAN JACKSON: Commissioner McGaffigan?
COMMISSIONER McGAFFIGAN: Madam Chairman, this
goes back to a question you asked earlier about were there
plants that the performance trend model would indicate
should have been discussed and weren't discussed and there
clearly were some. If you also apply the decision criteria
suggested, there are plants that should have been on lists
and weren't on lists.
The thing that seems to, you know, bearing in mind
that italicized wording in Appendix 2, but the difference
. 28
between some of those plants and I've asked staff about one,
they said, oh, yeah, that's one of our lower quartile
plants, they limp along but they didn't have an event.
And so they can look quite bad on the Arthur
Andersen performance indicators over a very extended period
of time, one case a decade, but not be on the list because
they don't have an event. They are adequate. They are
getting SALP 3's and occasional 2's but they aren't trending
downward. And that's -- one of the insights you get from
the Arthur Andersen report, I think, is the relative
importance of events in sort of focusing us and I don't
know. I mean, at a previous meeting, Commissioner Rogers,
we talked, you know, about adequate -- we were getting the
SALP, a 3 trending downward or trending upward, what is a 3,
a three is adequate.
But what is a watch plant list? A watch plant, a
plant deserving to be on the watch list is a -- I'm not sure
we yet have the right criteria for. But that isn't going to
be decided today. It's just that we get a lot of insight
from looking at the 108 plants, not all of which are in the
report, and seeing, you know, comparing those judgments.
CHAIRMAN JACKSON: Well, we seem to be able --
that everyone might know that there is a plant that is, as
you would say, limping along and it is as if, well, we can't
do anything unless it has an event and so we are event
. 29
triggered. And then there is the potential that if one is
event triggered, if we are event triggered, can overreact to
an event at the same time. And so it's an interesting
issue, so I am interested to see how you are going to
suggest you are going to deal with it.
But Mr. Barrett came prepared to give his
presentation so let's let him continue.
MR. BARRETT: Well, let me move on with some of
the other problems that were identified with the information
base.
Arthur Andersen made the finding that the
information for the assessment was inconsistent from plant
to plant and from region to region. And what they mean by
that is that in the past, in the written record, information
that appeared to be important for one plant was not
mentioned for other plants.
For instance, SALP. Sometimes SALP was very
important in the discussion for one plant, not very
important for another plant and, in other cases, the results
seemed to be even inconsistent with the SALP. Of course,
SALP is a lagging indicator but nevertheless there were
examples of that where information seemed to be used in an
inconsistent manner.
Arthur Andersen recommends reengineering the
information, the way in which we deal with information
. 30
again. And we will talk about their integrated model in a
while.
By the way, there has been some discussion
recently about improvements that were made in the most
recent senior management meeting in the way in which
information was organized and presented and so while I was
not at the meeting, that sounds like perhaps an improvement
in that respect.
CHAIRMAN JACKSON: So is the inconsistency the
inconsistency in the information or inconsistency in its use
and application?
MR. BARRETT: It's the information, in this
particular case, in what information is brought to the
table.
CHAIRMAN JACKSON: So that's in its use?
MR. BARRETT: It's use, yes.
CHAIRMAN JACKSON: All right.
MR. BARRETT: If I could have slide seven.
I can move through some of these others with
regard to information.
Arthur Andersen found that the decision process is
highly subjective and that there is -- the process minimally
values objective indicators. Now, when they refer to
subjective information, I think it is important to
understand what they mean. Information can be unquantified
. 31
or unquantifiable and still be objective. It can still be
observable, it can still be inspectable.
When they refer to subjective information, they
are referring to information that can be viewed quite
differently by two observers and the examples that they most
frequently cite are the fact that the written record from
1992 to 1996 frequently emphasizes the importance of
personnel changes and reorganizations that have been made
recently at a plant and improvement plans that have been
developed. Arthur Andersen considers these to be subjective
information, information that we really can't evaluate a
priori and that this information appears to keep -- to carry
very high weight in the senior management meeting process.
Conversely, with regard to performance indicators
which have been available to the NRC for quite some many
years, the indications that they have from the interviews
are that not very many, in fact very few if any of the
senior managers interviewed, identified the performance
indicators as primary decision criteria for the senior
management meeting decisions. And Arthur Andersen also
observed they actually attended all of the January 1997
screening meetings, and their observations were that while
the performance indicators were mentioned, they were not
focused on. So that the bottom line of all of this is that
objective indicators appeared to be minimally valued in past
. 32
senior management meeting discussions.
CHAIRMAN JACKSON: Excuse me, Mr. Barrett, I want
to ask Mr. Goldstein, what do you mean when you say that the
personnel changes, reorganization or improvement plans are
subjective as opposed to objective?
MR. BARRETT: Two observers can watch the change
in leadership. One can draw the conclusion that it will
focus the organization more directly in the correct
direction and another person can determine that it's a step
backwards because the new leader does not have experience in
nuclear safety. Valuing that change as positive or negative
will be a subjective assessment.
CHAIRMAN JACKSON: I see.
COMMISSIONER ROGERS: Don't you have a little
difficulty here when you are talking about assessing
management effectiveness and at the same time trying to find
objective measures to do that? The kinds of things that we
are touching upon here relate to judgment calls about
management decisionmaking and therefor potential
effectiveness and isn't this really an area where it is very
difficult to have it both ways, to get away from subjective
measures or subjective judgments and yet judge management
effectiveness at all levels?
Now, I mean, at a lower level it is easier to do
than at the higher level in the organization to judge
. 33
management effectiveness and it seems to me that that's a
very thorny area to get into. It is one that Mr. Jordan
touched on, why we haven't gone further in that direction in
the past. And certainly I would like to hear, you know, any
thoughts you may have sometime on that issue because it is
central to overall safety and yet it is the most difficult
one for us to deal with.
MR. GOLDSTEIN: I believe that Mr. Jordan will be
presenting the integrated performance model that we put in
the report as well as the proposal we made for use of harder
indicators in the meetings and, if I could, I think it would
be more effective for me to wait until after that and then
use those to answer your question.
COMMISSIONER ROGERS: Fine. Fine, very good.
CHAIRMAN JACKSON: Mr. Barrett?
MR. BARRETT: Yes. I would like to take a shot at
that question, though. There are things that happen every
day at nuclear power plants which are objective indications
of the effectiveness of the organization and perhaps
organizational effectiveness is a better term than
management effectiveness. I think part of the challenge as
we evaluate options for implementing the Arthur Andersen
recommendation will be to find objective ways of --
objective, observable, inspectable findings that indicate
how effective the organization is and the management, as
. 34
opposed to behaviors which, of course, are -- management
behaviors, which are subjective.
Let me move on. Another finding of the Arthur
Andersen assessment was that the mass of unprioritized
information inundates senior managers. Many of the managers
we interviewed cited the large volume of information in the
briefing books and also many of them talked about the
difficulty in assimilating the information as it's presented
by the regional administrator. The numerous examples that
are put on the table that, after a while, the listener
begins to lose context and so that the Arthur Andersen
recommendation is that we pay more attention to the
formatting of information and the volume of information that
is presented to senior managers so that they can get a
better context of what it all means.
Analyze the information and present it in such a
way that conclusions might be more evident. Have a
consistent structure and order of presentation of
information so that problems can be put in context and
plants can be compared with plants previously discussed.
And I should point out and Arthur Andersen points
this out that there already has been a lot of progress in
this area over the past several senior management meetings
with some of the information, management strategies such as
the plant issues matrix, which is good.
. 35
If I could have slide eight, please?
This is the final slide on information issues.
One of the issues that they noticed was that a great deal of
manual effort goes into assimilating performance information
here at the NRC. And without going into a lot of detail,
their recommendation is that we could have a process that
would be much more efficient and have a much better sharing
of information if we continue to improve information access
through automation. And the agency, as you know, has some
efforts in place to improve our availability of information,
making sure that information is available in standard
formats that is available electronically to everyone who
wants to use it for whatever purpose. So this is an area
that Arthur Andersen feels would really help us to be more
efficient and more effective in our assessments.
And, finally --
CHAIRMAN JACKSON: Mr. Gillespie, when we were
talking about the reactor oversight program last week,
talked about some activities having to do with automating
things along the line beginning with inspection and various
other inputs. These beginning efforts that you are talking
about, is that what you are speaking of?
MR. BARRETT: Yes, that would definitely be
apropos.
CHAIRMAN JACKSON: And then a question I have is
. 36
how proprietary is that system to just NRR's use as opposed
to in fact being accessible and/or compatible with other
systems?
MR. BARRETT: I am not in a position to answer
that question. I don't know enough about that system.
MR. JORDAN: It is an NRC system. It would be
available to the regions and other managers.
CHAIRMAN JACKSON: Will it be available to other
parts of the agency not in NRR?
MR. JORDAN: Yes.
CHAIRMAN JACKSON: I mean not in just the reactor
part of the business?
MR. JORDAN: Yes.
CHAIRMAN JACKSON: Okay.
MR. BARRETT: The final finding regarding
information base has to do with economic stress. I don't
think it is any secret to anyone that there is a concern
about economic stress due to deregulation of the industry
and Arthur Andersen has made the finding that the NRC needs
to keep an eye on this kind of stress because economic
stress can be a cause of performance problems.
On the other hand, they caution us that economic
stress cannot -- is not necessarily a predictor of problems.
Economic stress can be handled by some organizations, quite
nicely, in fact. In some cases, can actually lead to an
. 37
improvement in performance. So they are not recommending
that we use economic stress in the context of the senior
management meeting as an indicator that would be used in the
decisionmaking process. They are rather recommending that
we have a process and have a system available whereby we can
choose economic indicators, track those indicators and use
them as a way of nominating plants for perhaps a little
extra oversight that we can see, keep an eye on whether
economic stress as it is indicated does indeed have an
impact on performance as time goes by.
CHAIRMAN JACKSON: Well, isn't it also true that
excessive expenditures of money can also be an indicator of
organizational ineffectiveness. It doesn't necessarily
mean -- what you are really saying is that you can't track
dollar expenditures to organizational effectiveness.
MR. GOLDSTEIN: I agree. I think what we are
saying is variations too far away from the norm ought to
catch your attention but, as Mr. Barrett said, we would not
put them into a model as one of a quantity of more
formalized indicators but one ought to go find out why
that's happening and keep an eye on it is really what we are
trying to say.
MR. BARRETT: We are also saying it is not
necessarily the absolute value of an indicator. It is the
trend over some period of time that may be more important to
. 38
look at.
COMMISSIONER McGAFFIGAN: If they want to go back
to slide 17, they might want to flash up there briefly, that
shows what the economic indicators proposed by Arthur
Andersen are.
CHAIRMAN JACKSON: Why don't we come to that and I
will offer them an opportunity to speak to it. But I assume
that's why on the next -- on page 9 that economic stress is
an ellipse and not a rectangle; is that right?
MR. BARRETT: Well, I'll say yes.
But I will say there is no plan to go at any point
in the presentation to slide 17, so --
CHAIRMAN JACKSON: But I'm saying you do now.
MR. BARRETT: Okay.
COMMISSIONER ROGERS: I would like to come back to
it myself.
MR. BARRETT: Okay, fine.
COMMISSIONER DICUS: I can save my question until
we come back.
We have questions.
MR. BARRETT: All right. Let's go to -- I think
I've lost track of where I am.
CHAIRMAN JACKSON: Page 9.
MR. BARRETT: Slide 9, yes.
Slide 9 is a conceptual representation of the
. 39
approach that we have already talked about to a certain
extent here. It's an approach for using four levels of
information in a coordinated way for assessments and from
the right-hand side as I look at it to the left-hand side,
you're getting information that has a greater and greater
value in terms of getting more and more warning of impending
performance problems.
On the right-hand side, the bar there is called
results and what that really refers to is the occurrence of
significant events or other issues that might be viewed as
having a direct impact on safety. You can certainly catch
performance problems using this type of indicator but this
is going to catch performance problems at a point where they
are going to have a higher safety implication and it is
going to take more resources on the part of the utility to
reverse the trend. These kind of indicators typically are
the kinds that we have used in terms of significant events
or severe accident precursors, SCRAMs, safety system
failures. These are occurrences that actually have safety
significance.
If you are looking for a more timely, ongoing type
of assessment, Arthur Andersen would ask you to move to the
left one block to operations effectiveness and get an
ongoing systematic way of looking at operations
effectiveness in a way that can be presented to the senior
. 40
management meeting.
Operations effectiveness refers to sort of those
categories that we use in the SALP process, the operations
program, the maintenance program, engineering and the other
plant support programs. We already have a large program to
inspect in these areas. What Arthur Andersen is proposing
is that we need a systematic approach to assessing
performance in these areas.
If you want a still more timely systematic way of
looking at performance that will give you earlier warning,
earlier indication, management effectiveness or, as I would
prefer to call it, organizational effectiveness should be
looked at in a systematic way. These are issues such as the
ability of the licensee to do self-assessment, the ability
to identify problems and resolve those problems, the ability
to coordinate and control work, the quality of procedures
and procedural adherence and issues of this type that are
sometimes referred to as soft issues.
Again, we look at these but quite frequently it is
a retrospective look in the wake of an event. Arthur
Andersen would like us to look at it in an ongoing way and
in a consistent and systematic way.
And finally, on the far left, we have economic
stress which, as I said before, can cause performance
problems and may be an early indicator and certainly should
. 41
be watched by the NRC. But, again, as I said, it is not
recommended for use in the senior management meeting itself.
Why don't we pull up 17. Slide 17, which is a
backup slide.
While we are waiting for slide 17, I think an
important point to make here, and I think this is something
that the Arthur Andersen people make quite frequently, is
that we shouldn't be looking necessarily for the magic set
of indicators. There are any number of good indicators that
can tell us about performance degradation. It is important
that we look at a spectrum of indicators and understand that
we are looking at indicators that are somewhat independent
of each other, but there is no magic set that is going to
tell you the answer.
And the five that they have given us here are five
that they are proposing as being ones that certainly have
promise but, again, they are recommending that the NRC do a
systematic look and see which ones that we are interested
in.
The first one here is operating costs per kilowatt
hour. Apparently, this is a measure that is quite
frequently used by utilities for their own internal look at
the operating effectiveness of a nuclear unit or any unit
for that matter. And it is certainly an indication of the
competitiveness of a particular unit in a market, especially
. 42
a competitive market where price is important. And if a
plant is not competitive, that may well be an indication it
will be experiencing economic stress in the future.
Debt to equity ratio is more of a measure of the
overall health of the company, especially a publicly traded
company, obviously. As debt to equity ratio rises, that can
be a negative measure on the overall strength of the company
and, again, perhaps a leading indicator of stress coming
down the road.
The next two, operating cost trends and capital
spending trends, are much more directly related to the way
in which the plant is operated. Capital spending trend, of
course, the indication is from past experience that capital
spending is one of the first things that's sacrificed when a
plant, when a company is undergoing economic stress and,
according to Arthur Andersen, this is one that may be a good
indicator of more immediate economic stress that a plant is
experiencing because of economic stress at a higher level in
the corporation.
COMMISSIONER McGAFFIGAN: How do you treat steam
generator replacements within that because that is sort of a
big lump that pops up that isn't necessarily a good
indicator other than that they want to continue to operate
for a while or whatever.
MR. BARRETT: Yes, that's -- clearly, a lot of
. 43
these indicators are -- there are many, many things that
happen in the life of a nuclear power plant. Just an
outage, for instance, which has to be taken into account.
And any of the indicators, even in the ones we currently
use, and certainly big expenditures like that, we would have
to look at these things in a smart way.
CHAIRMAN JACKSON: Well, the signal may be in
trend since the steam generator replacement is a delta
function.
MR. BARRETT: It's a delta function and it might
actually lead to loss of capital spending elsewhere as they
try to squeeze that in or it might not.
With regard to operating costs, Arthur Andersen
said that we should simply look at the trend in operating
costs. Either an increase in operating costs or a decrease
in operating costs should be looked at because it may -- we
should try to understand the underlying reason for that
change.
And, finally, one that kind of surprised me but
maybe it shouldn't have, is the percent of utility
generating capacity from nuclear. According to the Arthur
Andersen report that it's the opinion of their experts that
they have consulted that stress is greater on a utility that
has a high percentage of nuclear units, whether those units
are performing poorly or performing well. Nevertheless,
. 44
there is more economic stress on a utility that has a high
percentage of nuclear units.
So that is the rationale in a nutshell as to the
five that are proposed here but, as I said before, Arthur
Andersen is urging us to take an independent look at all the
indicators including economic stress indicators.
CHAIRMAN JACKSON: You had a question that you
wanted to ask, Commissioner Rogers?
COMMISSIONER ROGERS: Just I do think -- I was
listening for it and I think I heard it. And that is that
really it's changes that you have to be watching that
trigger your attention and that if something is changing you
better understand why it's changing, could be going up or
down and either one could be good or bad, depending upon the
reason for that.
Operating costs per kilowatt hour, generally
speaking, low is good but if you just try and reduce your
costs to get that down and you're not looking at the best
way to do that but just in a shortcut way, that's bad. So,
you know, it seems to me that what you are telling us is
watch for changes and try to understand what they are and
then use that as a way of screening or calling attention to
plants that you might want to look at more closely, but not
by themselves are determinant of whether somebody will go on
a watch list or not.
. 45
CHAIRMAN JACKSON: Commissioner?
COMMISSIONER DICUS: I have a question.
If we were to incorporate this in some fashion
into the overall decisionmaking process, in your view do you
think that the NRC staff has the resources to do this? And
perhaps even on a couple of these, the expertise to be able
to effectively evaluate them?
MR. JORDAN: Maybe I could answer by saying that
these five that are listed are commercially available. The
staff would not have to do any collection of information.
They are part of the financial community.
In terms of NRR does have persons that are
involved in the review of the financial capability of
utilities, a limited number. The object here would not be
to affect the decision process but to, in engineering terms,
if there is stress there may be strain so if the presence of
the stress is causing safety strain then there would be
communication to the staff to be watchful for safety strain.
So it would be a sensitization and so it would be
one of the earliest measures that one could become concerned
about but not as a basis for decision.
CHAIRMAN JACKSON: It wouldn't be a decision
trigger but I like the word of sensitization.
Please.
MR. BARRETT: If I could have slide 10?
. 46
Arthur Andersen made some findings and
recommendations regarding the process that we use. The
first one is a very positive one, namely that they feel that
the process is logically sound. They did take a look at our
process from front to back, bottom to top, and they feel
that we are using -- we have good processes for gathering
information. It is a logical progression of analysis and we
have the right people involved in the senior management
meeting.
Among the negative findings Arthur Andersen made
are, first of all, they feel that -- they concluded that the
senior management meeting process is dominated by the
regional administrator and the basis for that, first of all,
is that much of the information is developed in the region.
Secondly that a lot of this conclusion came from the
interviews that were conducted. Clearly NRC managers in
general tend to defer to the regional administrator's
greater depth of first-hand knowledge about the plants and
certainly that -- there is a certain amount of
reasonableness to that for sure.
They found that in interviews at the meeting,
while it involves many people, in the past at least it has
tended to be dominated by the regional administrator, the
EDO and the director of NRR and, among the three of those,
the deference is to the regional administrator.
. 47
The regional administrator is the principal
presenter at the meeting and the observation of Arthur
Andersen regarding their experience with the screening
meetings in January 1997, which they attended, was that the
regional administrator tended to act as a gatekeeper for
other participants and other information.
So the process is dominated by the regional
administrator and the role of some of the other senior
managers is unclear. So the recommendation that they make
is that there be a better balance among the participants,
that the NRC should strive to elevate the importance of
independent sources of information such as AEOD's event
information and enforcement information from OE, information
about investigations and allegations, that we try to elevate
the importance of these independent sources of information
and also that we consider a consensus building process, some
sort of techniques for consensus building. One of the
things that they suggested was the possible use of a
facilitator for the meeting.
I should note that in the January 29 meeting,
there was a fair bit of discussion about more discussion
among the various participants, a greater amount of
participation in the January 1997 meeting than has been
experienced in the past.
Slide 11.
. 48
One of the most important findings of the Arthur
Andersen assessment is that we have no clear criteria for
various levels of formal actions and that they view that as
a very important thing. We will discuss in a little while
the issue of objective criteria.
They found that the presentation of information at
the meeting is not balanced in structure, again coming back
to some of the things we said before. The regional
administrator presents his list of problems and at the last
senior management meeting apparently also the list of
strengths for each plant and the weight of this information
dominates all subsequent discussion.
The finding is that there is not sufficient weight
given to events and other types of information and
indicators and they are recommending a more rigorous and
structured presentation. That objective information be put
on the table first in a scrutable and compelling format and
that it be used as a rebuttable presumption. That the
objective information presents a case for some action and
then the discussion can be either to reinforce that case or
to rebut it for the rest of the meeting.
COMMISSIONER DIAZ: Who actually makes the
rebuttal? Has that been considered?
MR. BARRETT: Anyone who is at the meeting who has
information that is relevant.
. 49
COMMISSIONER DIAZ: You are not planning on
separating teams?
MR. BARRETT: There was no specific mention of
teams, no.
CHAIRMAN JACKSON: If in fact you are talking
about having the objective information presented in a way
that it forms the basis of or generates a rebuttable
presumption, aren't you in some sense really getting at the
screening meetings themselves? Because how plants come
forward or that is a rebuttable presumption that a plant be
discussed for inclusion in the watch list has to flow from
somewhere, you know, in order for it to get put on the
table. And really it is at that screening meeting level
that a lot of the -- essentially the bias in the system
occurs, whether it is either to put a plant onto the table
for discussion coupled with the discussion itself in the
meeting but it sounds like what you are saying is the
discussion follows what essentially has flowed out of that
regional discussion. Or to not put a plant onto the table
for discussion.
MR. BARRETT: I think you are absolutely right.
There was no discussion of that in the Arthur Andersen
report but I think you're right. This recommendation does
push the process back into the screening meeting.
COMMISSIONER McGAFFIGAN: To some degree it is
. 50
because if you go to chart 15, which is another one of the
backup slides you're probably not planning to use, it really
goes to the Chairman's question in that the first two
bullets are the screening meetings. Select discussion plans
using trend charts and decision criteria for input using
evaluation sheets and trend charts. Those are the two
places where the rebuttable presumption using the decision
criteria and the trend charts get put together really by
staff long before the meeting.
Then you have the discussion. Then they suggest
places that they go away from the rebuttable presumptions,
the accepted rebuttals, that that also be documented to the
Commission. So I think that chart sort of answers the
report, has at least some glimpse of that.
MR. BARRETT: Yes, it does. You're right,
absolutely right.
CHAIRMAN JACKSON: It's really like a three-part
process. It's what comes up through a prior -- whatever
prior process there is, screening. Then there is the actual
process in the meeting and then there is the documentation
and public presentation of whatever the results are. So
there are those three distinct phases and pieces.
MR. BARRETT: Arthur Andersen also found that
stakeholders do not understand the process and the outcomes
of the senior management meeting, that our discussions with
. 51
utility executives, there was a fair bit of consensus that
they were not clear on what it takes to get on the problem
plant list or off the list and they are not clear about what
the process is by which we make the decision.
Arthur Andersen feels that we must do a better job
of communicating to the Commission, to the public and to the
industry and they are recommending that we more fully
document the public record at the senior management meeting.
They are recommending that we consider publishing
transcripts of the meeting or at least that we publish a
more complete and accurate set of minutes at the meeting, so
that there can be a better understanding of what we decided
and why.
COMMISSIONER DIAZ: You could probably add
stakeholders and one commissioner right here.
CHAIRMAN JACKSON: You're a stakeholder,
commissioner.
COMMISSIONER DIAZ: Oh.
CHAIRMAN JACKSON: We know the commissioner is a
special beast but we are all stakeholders.
[Laughter.]
MR. BARRETT: If I could go to slide 12?
I would like to talk a little bit about the trend
plots before we actually put one up there.
The Arthur Andersen trend plots basically show how
. 52
NRC information can be used, could be used, along with some
reasonable criteria to greatly inform the decisions of the
senior management meeting. The model tracks the performance
of a plant against nine indicators in this particular case,
although Arthur Andersen, as Mr. Ross earlier said, wants us
to go back and do a systematic look at which indicators we
want to use.
Takes those nine indicators, including the
nine -- including the seven performance indicators of the
NRC plus an indicator of civil penalties and an indicator of
the number of allegations that a plant has experienced.
When a plant exceeds twice the average value for
the industry in any given indicator, then that becomes a
hit, twice the average for the industry, that's a hit. And
if it -- and that only has to exist for one quarter.
Hits accumulate. They accumulate for four
quarters and there is a four-quarter running sum of hits
that a plant carries with it.
CHAIRMAN JACKSON: Is each quarter weighted the
same?
MR. BARRETT: Yes.
CHAIRMAN JACKSON: And that particular averaging,
was that rooted in anything in particular or was it
arbitrary?
MR. BARRETT: It was arbitrary.
. 53
MR. GOLDSTEIN: The concept of using a rolling
average --
CHAIRMAN JACKSON: No, I know that. The issue is
how much -- what do you roll over. You know --
MR. GOLDSTEIN: How many quarters?
CHAIRMAN JACKSON: Exactly.
MR. GOLDSTEIN: No. I think enough so that you
can pick up changes and drop them in a timely fashion. You
don't want it too long.
CHAIRMAN JACKSON: All right.
MR. BARRETT: One of the assertions of the Arthur
Andersen study is that performance does not change
precipitously at the plants. It takes time for a plant's
performance to degrade and it takes time for it to recover.
CHAIRMAN JACKSON: Right. No, I agree with all
that. Part of the reason I bring that up is commissioners
have raised the point in the past or questions relative to
SALP and how it ties into the senior management meeting
process and the SALP covers a certain period of time that is
on the order of 18 to 24 months and that is the reason why I
asked the particular question about the number of quarters
over which you do the rolling average.
MR. BARRETT: So at any given point on the graph
is the sum of the hits for four quarters and for any four
quarters, the maximum number of hits you could have is 36,
. 54
four times nine.
What I think is important about this particular
model is not necessarily the details of it but two things
really. First of all, it is predicated on the idea that if
a plant is experiencing true performance problems it is
going to show up not in one indicator but in a variety of
indicators so you should be looking at a number of hits and
that you should be looking at it over an extended period of
time, not just for one quarter.
COMMISSIONER McGAFFIGAN: Could I ask a question
about the comment that NEI made after our briefing on the
senior management meeting? The heart of their comment was
you could be on the watch list today and would have been a
top quartile plant a decade ago and sort of built into these
performance indicators, and maybe it's fair to ask Arthur
Andersen, if the trend overall in performance indicators is
an improvement, being twice as bad as the industry average
and therefore deserving a hit, it could be quite a bit
better today than it was a decade ago. And so if there is
continued improvement and I know in recent years there has
been a sort of leveling off in the performance indicators
but if you have a declining trend then you are potentially
holding people to a moving target.
Is that a fair criticism of your model or --
MR. GOLDSTEIN: I think as Mr. Barrett explains
. 55
further, I believe this is in his explanation, the action
that we would propose that you take would be related to not
solely whether you have an accumulated number of hits above
a certain amount but, more importantly, to the trend over
time. That you would -- that a few quarters of growth would
lead to a discussion. Reduction over time would lead to a
step to take you or a rebuttable presumption that you be
taken off the watch list, so that our focus is on the -- is
on the trend over time as an indicator of risk, even if your
number of hits is higher than the average. You still,
perhaps, should be moving down the level of risk that the
Commission uses.
MR. BARRETT: Let me add a word to that --
MR. JORDAN: I think the answer to your question
is, yes.
MR. BARRETT: One of the things we might consider
is actually fixing the criteria. Rather than comparing to
an industry average, compare to some fixed value and it
might be the industry average.
MR. JORDAN: But we are responding to this model
and this model would facilitate a rising standard and
compare plants against a rising standard. This is an
intriguing model but we are not trapped by it; I think it is
a useful concept.
COMMISSIONER McGAFFIGAN: But that is a possible
. 56
problem with this model?
MR. JORDAN: Yes, correct.
COMMISSIONER McGAFFIGAN: As long as performance
indicators continue to improve in the industry, you would be
continuing to -- you would be moving against a moving
target.
I don't know what numbers in 1987 would get you a
hit but it's probably now, it would put you in the lowest
quartile.
MR. BARRETT: I suspect that still one SCRAM would
get you a hit.
COMMISSIONER McGAFFIGAN: One SCRAM would get you
a hit today whereas in '87 it might --
MR. BARRETT: Yes, because the industry average
would be less than half of a SCRAM per quarter. And there
are a number of indicators where that would be the case.
So it is not a fatal flaw in the model but it is
something that you would need to fix if, you know, we went
forward.
COMMISSIONER ROGERS: It is a question of
establishing some calibration for it, which is what you have
suggested might be a way to do it, and some absolute number.
And the other one is, you know, the obvious
problem with it and, you know, it's a bad thing to be below
average. I mean, you just can't be below average.
. 57
[Laughter.]
COMMISSIONER McGAFFIGAN: They didn't say you just
had to go below average, you had to be twice as bad as the
average.
COMMISSIONER ROGERS: I know, but there is always
going to be somebody twice as bad.
CHAIRMAN JACKSON: Well, we are all scientists and
engineers here for the most part and we all know that what
you normalize to is always the critical thing.
MR. BARRETT: Okay, well, let's move on to the
next slide which is an actual -- which is a performance
trend plot for an actual plant that was graphed from 1987 to
1996.
The curve with the diamonds represents the four-
quarter sum of hits for the actual plant. The squares
represent the industry average number of hits which ranges
from about five to six if you look on the right-hand scale.
Just to help you understand this, you can see the
peak there of the diamonds is 16 hits in that particular
quarter of 1991. And again, the maximum number you could
possibly have would be 36 hits. So, for this particular
plant, plant A, it ran along at about the industry average
or better than the industry average until 1991 when it took
a turn for the worse, peaking at 16 hits in the fourth
quarter of 1991 and then moving along through 1995 at
. 58
roughly that level.
On the left-hand margin, you will see the action
levels from one to five where a five is equivalent to being
a category three plant shutdown requiring Commission action
to allow them to restart. Action step four would be a watch
list plant. Three would be a trending letter and two would
be a discussion plant and one would be a plant that should
be removed from the list.
The yellow bars represent the actual NRC actions
with respect to this plant. It was discussed several times
starting in 1991 and was placed on the watch list by action
of the senior management meeting in January of 1996.
The green -- they turned out blue there, don't
they? Well, anyway, they're green when you're up closer.
The green bars are the criteria or the actions that would be
indicated by the Arthur Andersen criteria. And they would
have said that this plant would get a trending letter in
1992 and then be placed on the watch list in 1993.
This is a plant that would illustrate a case where
Arthur Andersen would say the NRC was slow to take formal
action and this was a plant that many NRC managers during
the interviews said they believed in retrospect might have
gone on the list earlier.
CHAIRMAN JACKSON: What triggered the action in
the first quarter of '96?
. 59
MR. BARRETT: The action on the part of the NRC?
CHAIRMAN JACKSON: Right.
MR. BARRETT: I --
CHAIRMAN JACKSON: It's just the way it happened?
COMMISSIONER McGAFFIGAN: That might identify the
plant, which --
CHAIRMAN JACKSON: Never mind. We're not supposed
to be discussing these guys. That's right.
COMMISSIONER McGAFFIGAN: Could I ask, another
problem with performance indicators that comes up when you
look at some of the charts, and I go away from this plant
but if you are shut down, it's hard to get SCRAMs so you
eliminate one category of hits. Now, if you're shut down,
you also may be getting plenty of additional inspectors
finding problems which gives you hits. But how do
you -- have you thought through, and maybe this is a March
31 question, how you are going to deal with normalizing the
performance indicators to things like what -- whether the
plant is in a shutdown condition or not and that sort of
thing?
MR. JORDAN: Clearly, this scheme has limitations
with respect to plants that are not operating and so it
simply doesn't work right for that and so there are a number
of conditions that for the March presentation -- we have to
look at the independence of the indicators, relative
. 60
weighting that one applies, the plant condition, whether a
rising standard is embedded in it.
So there are a lot of parameters that we have to
consider when we come back to say, okay, here is closer to
the ideal. But I think the model that they provided is a
real thought provoker and has a lot of merit to it but we
have to look further.
CHAIRMAN JACKSON: What was the indicator you used
for allegations and for enforcement action? Just numbers?
MR. JORDAN: Yes.
MR. BARRETT: Just number of allegations.
I believe it was number of civil penalties.
COMMISSIONER DIAZ: Should this plant --
MR. BARRETT: Excuse me, it's dollars of civil
penalties.
CHAIRMAN JACKSON: Dollars of civil penalties.
COMMISSIONER DIAZ: Should this plant have
remained on the watch list if it was placed on the watch
list after it broke down?
MR. BARRETT: Yes. As you can see, the green bar
there would not have indicated that they met the criteria
for removal. The Arthur Andersen model also has criteria
for removal.
COMMISSIONER DIAZ: Right. Another quarter would
have done that at that performance?
. 61
MR. BARRETT: Possibly. I would say, yes, because
that would be three quarters consecutively below the
industry average.
COMMISSIONER DIAZ: So it would be four quarters
below the industry average.
CHAIRMAN JACKSON: Mr. Goldstein?
MR. GOLDSTEIN: I would like to avoid focusing too
much on this model and these indicators. The objective of
the engagement was to probe on the issue of objective versus
subjective decisions at the senior management meeting. This
is one model. There are many, many others that can be used
and also replicated by the industry each plant in its own
behalf to tell how it will fare under a set of objective
criteria.
A lot of indicators have been put on the table
here and these may be the right nine. I'm not sure that
they should all be weighted equally. Dollars are used for
the indication of enforcement action. Maybe it should be
number of enforcement.
The key point is that models can be created that
can track historically and that is a test that has to be
done, and for which sensitivity analyses have to be done and
the time frame we had in this engagement neither did we
conduct some of the usual validity checks that have to do
with the sensitivity of the model to things like changing
. 62
the number of quarters and so forth. The one thing I guess
I would urge is that the individual elements of whatever
model is picked not be tested against an ironclad standard
but be viewed as a starting point.
It will take years to refine the right model that
gives you both the right objective standard and some
flexibility but the term "continuous improvement" in my
business is one way we try to convey to clients that it is
better to start and even if you're refining as you go along
it, at least in this environment, can be an improvement.
MR. JORDAN: I'd make one comment. We have a
remarkable historical record that we can use to benchmark
against. The variables that occur in the plant in terms of
objective measures and how their performance of those plants
has actually changed over time so the validation,
subsequently, can be reasonably powerful.
CHAIRMAN JACKSON: Okay.
MR. BARRETT: If I could have slide 14, I would
like to wrap this up.
In summary, Arthur Andersen concluded that we have
a logical process but that there are findings and
recommendations regarding the information and the process
itself that can greatly improve the way in which we conduct
our assessments.
We do not intend to implement the findings until
. 63
we have developed a staff consensus on what the right
options are to go forward and until we have had policy
guidance from the Commission but we will be preparing a
Commission paper which we expect to forward on March 31 and
we will proceed following Commission guidance.
The Commission paper will deal with options for
the process changes that have been recommended by Arthur
Andersen and also options and plans for development of the
leading indicators and the integrated process, the
integrated information system that is proposed by Arthur
Andersen.
In the meantime, we would expect that there might
be incremental changes implemented at the June 1997 senior
management meeting, mostly those that might relate to
process changes. It is a much more difficult challenge to
address the types of issues that have to be gotten over in
order to develop the information changes and we would expect
that those would be implemented on a trial basis in January
of 1988. So that concludes my presentation. If you have
any further questions, I would be happy to try to answer
them.
CHAIRMAN JACKSON: Commissioner Rogers?
COMMISSIONER ROGERS: Well, I think the report is
an extremely interesting one and I think that a number of
suggestions that have come out of it have been really very
. 64
good.
It is a question of details on things like the
model and whatnot and I think that the disclaimers that have
been made have been appropriate, don't get too hung up on it
right now but it is a very interesting and possibly quite
powerful approach.
A couple of points about the report. One is I
think you did say, I don't remember the pages now but I know
I read it carefully at one time at any rate and noticed that
you were emphasizing the importance of risk. But I really
didn't see anything much about risk in the report and I
wondered what you had in mind there, whether you were
talking about really a kind of qualitative judgment of risk
or something more mathematically defined, such as we would
come up with with a probabilistic risk assessment. And so
what is your concept of how we ought to fold risk into this
process?
MR. GOLDSTEIN: Are you asking me?
COMMISSIONER ROGERS: Yes.
MR. GOLDSTEIN: We learned early in the engagement
that the NRC has and we reviewed them, quantitative
standards that you use for what would be acceptable events,
the kind of radiation problems that would occur immediately
proximate to the plant and further out and those members of
the team who are anchored in risk issues for the nuclear
. 65
industry rapidly translate that into performance integrity
and assuring that the integrity of the plant and the
protection against some major operational failure is their
translator into risk.
I could contrast it to FAA. We do a great deal of
work for FAA where, although certainly a serious crash is a
disaster, it is not of the same magnitude. And so the
concept of risk isn't defined as zero defects; in fact, FAA
has a specific policy about refining designs as a result of
recurring air failures.
Our industry people in working with us here seem
to be very comfortable that the operating concept of risk
that you use and that we therefore could use is a zero
defect avoiding of operating failures and that is the -- we
did not go past that line to challenge the quantitative
models that you use, translate that into probability of
failure.
COMMISSIONER ROGERS: Well, I'm nodding my head.
That just means I heard you; I don't necessarily agree with
that definition.
MR. BARRETT: I would like to add a few words on
that subject.
COMMISSIONER ROGERS: Yes.
MR. BARRETT: We did in the process of this study
inform Arthur Andersen on the NRC's model of risk in terms
. 66
of its quantitative model of risk being consequences times
frequency and the major factors that tend to drive risk,
which is initiating events, failure probability and
equipment failures and common cause failures.
And we developed a qualitative model that relates
those to the types of things that tend to be assessed in the
context of the senior management meeting and that writeup is
actually Appendix 1 of the report, which was developed by
the NRC staff and given to Arthur Andersen. But there was
no intention and there is no intention of trying to make a
quantitative assessment of risk based on performance.
In the future, we have under development risk-
based indicators which, as they become available, as the
information becomes available to develop those indicators, I
could see that we could move those indicators into the
model, either to supplement the indicators we currently are
using or perhaps even to replace indicators that we are
currently using. But, basically, the answer to your
question is it is a qualitative rather than quantitative
connection.
COMMISSIONER ROGERS: Just one more point, I
think, before I get out of here and let other people have
their say. I think this suggestion with respect to
consensus decisionmaking and the idea of a rebuttable
presumption on the part of -- as a starting point for an
. 67
analysis I think is extremely interesting and I wonder if,
you know, there could be some more specific mechanisms
discussed for doing that, not necessarily right here today.
But I think if this process is to be one that is clearly
defensible and transparent to the public, then I think we
have to be pretty clear on exactly how we are going to get
to an end point starting with a rebuttable presumption and a
consensus decisionmaking process, just exactly what that
means.
CHAIRMAN JACKSON: Mr. Jordan, you were going to
make a comment?
MR. JORDAN: Rich covered my comment extremely
well.
CHAIRMAN JACKSON: Commissioner Dicus?
COMMISSIONER DICUS: Nothing further, thank you.
CHAIRMAN JACKSON: Commissioner Diaz?
COMMISSIONER DIAZ: Let's see, I've got one, two,
three, four, five. I'm going to throw them all away and go
back to zero defect. I'm going to throw all my questions
away.
This zero defect of operational failure which you
said is the basis on which you developed your performance
indicators, could you explain what an operational failure
is? Is that a core meltdown or is that control rods falling
in or is that a leaking pump? What is an operational
. 68
failure?
MR. GOLDSTEIN: We didn't develop the performance
indicators. The indicators that are here are the
indicators -- the seven indicators that the staff already
uses and that we are putting in the model. Those are what
we did use.
COMMISSIONER DIAZ: Zero defect.
MR. BARRETT: Well, I don't know that. Nuclear
power plants, as you well know probably as well as I or
better, are very complex machines and they are designed to
be somewhat forgiving of failures here and there so with
redundancy and diversity so --
COMMISSIONER DIAZ: We don't base performance
indicators on zero defects, do we?
MR. JORDAN: No. In the context you asked it, I
believe, what the Arthur Andersen report was saying was that
the NRC is adverse to risk and I would say in terms of a
severe accident, it is unacceptable to have a severe
accident. So that would be the connotation that I would put
on their comment.
COMMISSIONER DIAZ: Okay. So a connotation is a
severe accident that has significant impact on the health
and safety of the public versus, you know, the plant
shutting down because he has a bad seal on a pump.
MR. JORDAN: Correct. Correct.
. 69
COMMISSIONER DIAZ: So there is a very important
difference in there. Okay, thank you.
CHAIRMAN JACKSON: Well, we, I mean, I would
imagine, be hard pressed to prevent, you know, a seal on a
reactor coolant pump from failing. The question is, do we
pick up things ahead of time to not get to the severe
accident scenario.
Commissioner McGaffigan?
COMMISSIONER McGAFFIGAN: Just one comment. I do
think this was a remarkable effort over the last six months
and commend Admin for working with you at the start, as I
said, and I think the result is one of the best pieces of
work if not the best piece of work I have seen in the six
months I have been here.
That said, I would like to ask a question and that
is while this has been going on the General Accounting
Office is looking at exactly this set of issues. Are we
sharing all of our analysis and everything with GAO? How
are we trying to deal with being open and candid with the
Congress via the GAO?
MR. JORDAN: Certainly, the information that has
been developed is being made available or has been made
available to the GAO. They are aware of the effort and have
interviewed or are beginning to interview our staff.
COMMISSIONER McGAFFIGAN: So they have a sense
. 70
that we are struggling with the exact same set of issues
that they have been tasked to look at?
MR. BARRETT: They have conducted a number of
interviews of not only the people who worked with me as NRC
staff on this but they have also interviewed a number of the
Arthur Andersen panelists on the study. We have provided
them an early copy of the report prior to public release.
We have tried to be as --
COMMISSIONER McGAFFIGAN: Do we have a sense of
the timing? Will they -- the March 31 meeting where you are
going to tell us at least your preliminary views as to how
to deal with the report and what we might be able to adopt,
is that compatible, will that be ahead of GAO or will they
run ahead of that? Will they be able to wait and see what
you are proposing to us?
MR. JORDAN: We don't know what their schedule is.
We will find out and communicate with GAO.
CHAIRMAN JACKSON: Yes, Mr. Goldstein?
MR. VALENTINE: Let me just answer that because I
met with them twice. One thing we did have the advantage of
is both Ira and I used to work at GAO so we sort of --
COMMISSIONER McGAFFIGAN: This looked like a GAO
report.
MR. VALENTINE: Well, I hope it didn't look
completely like a GAO report. But we met with them and I
. 71
think one thing about GAO that I have found since I came
over to Arthur Andersen, we generally do things a little
quicker than GAO, so they are not going to be ready by March
31 with a detailed report but they are sort of interested in
what's going on here. They are very aware of what is going
on and as much as they could be supportive, they were
supportive.
CHAIRMAN JACKSON: Thank you.
I would like to thank the staff as well as the
representatives from Arthur Andersen for a very informative
briefing. I think what we can say is that the Arthur
Andersen report indicates that there is a relationship
between existing NRC indicators and plant performance and I
believe the staff should continue to evaluate to what extent
the existing indicators can be used to characterize plant
performance and you have kind of spoken to it, Mr. Barrett,
yourself that if the current set of indicators are
inadequate in the sense that they are not fully risk
informed, then the assumption is that the staff is exploring
the development of new indicators and will phase them in as
appropriate.
We have already talked about using management or,
as you said, organizational effectiveness as well as risk-
based indicators and I think those are very important.
The thing that has kind of been woven through this
. 72
but it seems needs more direct focus is the issue of the
screening meetings which feed the senior management meetings
and having them be as objective as possible. And a question
I would like to leave you with is whether the performance
indicators are perhaps better used at that point in terms of
developing the rebuttable presumptions about the plants and
having the meetings themselves focus on the kinds of process
improvements that you mentioned. And there was a plant
performance template that had been developed or was being
developed for use in that meeting and it would be useful to
know what intent you intend to make of that.
Then speaking further about the senior management
meeting itself, the scrutability of the framework and the
process, the process and the framework for decisionmaking
should display the connection, I think, that exists between
the plant performance data and what the actual ensuing
decisions are. And, as I said, it seemed that you had moved
along the lines of developing a plant performance template
to help do that. And I think the Commission would be very
interested in your establishing a consistency and if the
consistency already exists then establishing the evidence of
it, of the consistency between the senior management meeting
decisions and decisions that are reached in our other
evaluative processes. And here we are talking about the
SALP process, the plant performance reviews and the
. 73
inspection reports.
We had a briefing last week on the reactor
oversight program. It spoke to that. We have had a
discussion here about the performance indicators and their
uses. And we are speaking to it but we have to see the
connection in actual fact and so I think that's very
important.
So unless there are any further comments by the
commissioners, we are adjourned.
[Whereupon, at 4:21 p.m., the briefing was
adjourned.]
|
Privacy Policy |
Site Disclaimer |