Search Options
Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us

POLICY ISSUE
INFORMATION

SECY-00-0217

November 3, 2000

• PURPOSE:
• SUMMARY:
• BACKGROUND:
• DISCUSSION:
• STAFF ACTIONS:
• RESOURCES:
• CONCLUSIONS:
• COORDINATION:

PURPOSE:

To inform the Commission of continuing issues in obtaining and reporting accurate performance plan nuclear materials events data in a timely manner, and to report the actions the staff is taking to address these issues.

SUMMARY:

In a recent update of the fiscal year (FY) 1999 performance data, and the preparation of similar data for FY 2000, the staff noted fluctuations in some of the metrics previously reported in NUREG-1100, Volume 16, "Budget Estimates and Performance Plan, Fiscal Year 2001," dated February 2000. This paper discusses the staff's analysis of the data differences, the actions the staff is taking to assure the quality of data reporting, and the staff's presentation of revised materials safety metrics, based on more reliable historical data.

BACKGROUND:

The staff provided summary performance assessments of voluntary event reporting by the Agreement States to the Commission on August 12, 1996, and June 18, 1997, along with an addendum dated June 24, 1997. At that time, the findings indicated that most states were voluntarily providing information on "significant" events, (that is, events reportable by a licensee within 24 hours), but information on less significant events with a 30-day notification requirement, and follow-up information on all events was incomplete, inconsistent, and untimely. The 1996 data included no reportable events for three states, and five states did not report at all. The 1997 data found that four states indicated they had no reportable events, and one state did not report. Additionally, as a result of a review of 30-day report information in the Nuclear Materials Events Database (NMED), the staff found three "significant" events that were not reported on a timely basis. Event reporting guidance states that "significant" events should be reported to NRC within 24 hours of notification from an Agreement State licensee. Based on the findings, in September 1997, the staff included a requirement for event reporting as a compatibility element to the proposed final Policy Statement on Adequacy and Compatibility of Agreement State Programs.

In SECY-97-054, dated March 3, 1997, the staff transmitted to the Commission its Final Recommendations on Policy Statement and Implementing Procedures for: "Statement of Principles and Policy for the Agreement State Program" and "Policy Statement on Adequacy and Compatibility of Agreement State Program." The policy statement designated the reporting of event information under Compatibility Category "C" (the essential objectives of which an Agreement State should adopt to avoid conflicts, duplications, or gaps in the regulation of agreement material on a nationwide basis and that, if not adopted, would result in an undesirable consequence).

In a Staff Requirements Memorandum (SRM) dated June 30, 1997, the Commission approved the policy statements and implementing procedures contained in SECY-97-054. The Commission directed that, "since the staff's proposal would make Agreement State reporting of events to NRC Nuclear Materials Event Database an item of compatibility, references in the text to voluntary reporting should be eliminated, and the staff should make it clear that reporting of materials events is mandatory." The final policy statements became effective on September 3, 1997. The Agreement States were provided six months to implement this new program element. Implementation of mandatory reporting became effective March 3, 1998.

Following Commission direction, the staff provided the Agreement States copies of the final policy statements and implementing procedures in an all Agreement State letter dated September 15, 1997, which highlighted mandatory reporting of material events and subsequent evaluation under the Integrated Materials Performance Evaluation Program (IMPEP).

The 1998 and 1999 data indicate an increase in the number of states providing event information and improvements in the technical quality of the information. Improvements have also occurred in the number of states reporting electronically to NMED from 20% in 1996, to 66% reporting electronically in 1999. Staff also contacted several states having limited or no event report data entered into NMED for 1998 and 1999. Two states indicated that they had no reportable events in 1999. One of four programs in one Agreement State stated that they currently only share information that they determine could affect other licensees and do not routinely provide all event information to NRC. Although current guidance requests that the event information be provided on a monthly basis, many states provide event information on a less frequent basis.

DISCUSSION:

The staff has observed data fluctuations that affect some of the new performance metrics developed as part of the U.S. Nuclear Regulatory Commission's (NRC's) triennial update to the Strategic Plan to measure NRC and Agreement State performance. Of the eight (8) safety-related performance measures identified in the FY 2000-2005 Strategic Plan under the Nuclear Material Safety performance goal for maintaining safety, the staff plans to revise two performance measures based on more complete historical data. These measures were also included in the FY 2001 Budget and Performance Plan. However, based on OMB guidance, since the changes are due to more recent and quantifiable baseline data, NRC can reflect these changes in its next performance plan (i.e., FY 2002 Budget and Performance Plan). These measures would then be reported in a FY 2001 Performance Report. These measures were not in the FY 2000 Performance Plan and therefore will not be reported in the FY 2000 Performance Report.

In the nuclear materials safety arena, those measures supporting safety reflected NRC's shifting emphasis from managing outputs, such as the number of licensing actions taken, or inspections conducted, to managing outcomes, such as the number of events resulting in over-exposures, releases, losses, accidental criticality, medical events, attempted cases of malevolent use of material, and the number of breakdowns in physical protection or material control and accounting systems. Many of these new measures and metrics were developed based on 1996-1999 data, some of which was voluntarily provided by the States and NRC for entry into NMED. Until last year's initiative, the purpose of the data collection was primarily to analyze and trend events to determine areas requiring greater regulatory attention. The data were not used at that time to evaluate individual or collective program performance.

In the materials arena, these measures have certain unique attributes that are important to consider:

The staff now knows that some of the historical performance data included in NUREG-1100, Volume 16, were incomplete. These data were based on the best information available to the staff as of October/November 1999, as reported in NMED by NRC Headquarters, the regions, and the Agreement States. They appeared on page 73 of NUREG-1100, Volume 16, published in February 2000, along with proposed metrics. The proposed metrics were based on several years of historical data, much of it reported voluntarily, set at a high confidence level (in most cases, three sigmas), to identify potentially significant precursors that would indicate if the level of safety represented by the historical data had changed, and if a reevaluation of NRC's regulatory activities were required. Attachment 1 provides a comparison of the reported data with the revised data, based on more recent information, and a brief explanation of why the data have fluctuated over time. Attachment 1 also shows a comparison of the metrics as they first appeared in NUREG-1100, Volume 16, and as they appear in the NRC Strategic Plan.

The more recent figures (i.e., Attachment 1's adjusted data column) include data from NMED queries conducted in October 2000. The NRC staff and the NMED contractor have analyzed present and past years' data to verify the accuracy of the numbers, and provided revised performance data based on this information.

The analysis showed the data differences were largely based on the following factors:

STAFF ACTIONS:

With respect to the first, fourth, and fifth factors, staff believes that the quality of the data reporting is improving, particularly after event data reporting became a matter of compatibility on September 3, 1997, with implementation required by March 1998. Since that time, the States have made a more concerted effort to upgrade their event reporting, especially with respect to events that occurred in FY 1998 and 1999. However, there is still work to do to further improve the consistency and timeliness of data reporting. Therefore, staff intends to accept only the data from FY 1998 and FY 1999, and revise its calculation of the baseline safety metrics in the Strategic Plan based only on the data from those years, adding ensuing years' data, after the staff is confident that the data are more complete. This should minimize the need to revise the metrics for any given year due to the late reporting of data.

The FY 2001 performance measures will be revised to include the updated metrics. The updated metrics will be included in the FY 2002 Performance Plan currently under review by the Commission. This change will also serve as a change to the strategic plan.

The importance of accurate, complete, and timely data reporting has been, and will continue to be, emphasized within NRC, and with the Agreement States. OMB Circular A-11, which provides guidance on implementing the Government Performance and Results Act, requires agencies to identify the means they will use to verify and validate measured performance values. Therefore, staff plans to take steps to ensure that the performance information is valid and to verify that the data provided are accurate and reliable as they are collected, recorded, and analyzed. Towards this end, the staff plans to continue to evaluate this issue, through assessment of NMED data, during weekly Generic Assessment Panel reviews of material event information, and through the materials oversight program, IMPEP, which provides an opportunity to assess performance and to reinforce the importance of this issue as each State and region is reviewed. The Office of Nuclear Material Safety and Safeguards (NMSS), and the Office of State and Tribal Programs (OSTP) have and will continue to take advantage of other forums to stress the significance of event reporting. NRC staff members have traveled throughout the country providing NMED training to the States, NRC regions, and Headquarters personnel, while at the same time speaking about the importance of the data. This issue has also been discussed in All Agreement States meetings, and in meetings with the Conference of Radiation Control Program Directors. The staff also intends to include an executive summary in upcoming NMED reports, that will include graphs or tables highlighting any trends associated with these events as they relate to the new performance plan goals. Based on recent data reports, it appears that these efforts are beginning to achieve success. OSTP intends to take an additional step, and is preparing an All Agreement State letter reaffirming NRC's expectations with respect to timely and complete event reporting.

In addition, an NRC/Agreement State Working Group has formed to prepare recommendations for making the materials event program more effective, efficient, and realistic. As part of its efforts, the Working Group will identify what event information is needed to support implementation of the Strategic Plan, and compare that information to the current reporting requirements. The purpose is to determine if NRC and the Agreement States are collecting the right safety information across the nation, and at the appropriate level of detail.

With respect to the second and third factors, the staff believes it is realistic to accept such modifications to the preliminary reports, as necessary steps to increase the accuracy of the data. This will require an appropriate interval from the end of the reporting period, until these analyses are complete. Any data that must be reported to external stakeholders before that time, as part of the annual performance plan or budget submissions, will need to be annotated to indicate that the data are preliminary.

RESOURCES:

The staff actions described above will require no NRC additional resources, beyond what is already provided in the FY 2001 and 2002 NMSS and OSTP budget allocations.

CONCLUSIONS:

The staff believes that the data collection and reporting issues, as described above, persist to a certain extent, although recent experience suggests that the situation is improving. In the interim, the staff has taken a number of actions, described above, to accelerate improvement in the process, and to revise the materials safety metrics based on the revised historical data.

COORDINATION:

NMSS has coordinated this paper with OSTP. The Office of the General Counsel has no legal objection. The Office of the Chief Financial Officer has reviewed this paper and has no objections.

ATTACHMENT

NUCLEAR MATERIALS SAFETY DATA COMPARISON AND REASON FOR ADJUSTMENTS

1. A small number of Agreement State licensees are also engaged in waste arena activities, but the vast majority are involved primarily in materials arena activities.

2. Sometimes licensees submit preliminary event reports that are entered into NMED. At a later date, based on additional licensee or regulator analysis, it is determined that the event did not actually need to be reported and, therefore, should not have been included.

3. This measure and others in this column are taken from NUREG-1100, Vol. 16. The first measure includes loss of control due to one of the following events involving a reportable quantity of AEA material: missing inventory items, stolen material (usually portable moisture density gauges), abandoned material, material inadvertently sent to commercial landfills or to metal scrap yards, and well-logging sources abandoned downhole. Missing inventory items comprise the largest group, and often include material subsequently found within the licensees' facilities. Most of the material lost in this category were low activity sources in a shielded configuration.

4. The rounding in this column was directed by the Commission as part of the comments to the draft Strategic Plan.

5. For this particular metric, data collection reporting remains more difficult. The events described in footnote 1 require subjective evaluation against NRC reporting criteria by the NMED contractor and by some Agreement States. Therefore, the staff proposal is to retain the original metric of 350, even though a metric based purely on the historical data would support a number between 240-300.

6. New overexposure measure is based on the updated FY 1998 and 1999 overexposure data.

7. New medical measure, which did not change from the prior measure, is based on updated FY 1998 and FY 1999 medical events data.

8. New release metric is based on the revised release data for FY 1998 and FY 1999.