Search Options
Index | Site Map | FAQ | Facility Info | Reading Rm | New | Help | Glossary | Contact Us

SECY-98-266

November 13, 1998

PURPOSE:

To obtain Commission approval to publish a final amendment to 10 CFR Part 55 allowing, rather than requiring, power reactor facility licensees to prepare the initial operator licensing examinations and to publish a related revision to the Enforcement Policy.

SUMMARY:

From October 1995 to April 1996, the NRC conducted a voluntary pilot program to evaluate the feasibility of transferring responsibility for preparing the initial operator licensing written examinations required by 10 CFR 55.41 and 55.43 and the operating tests required by 10 CFR 55.45 from the NRC to power reactor facility licensees. Based on the success of the pilot program, the Commission directed the staff to continue the revised examination process on a voluntary basis and to develop a plan to amend 10 CFR Part 55 so that all power reactor facility licensees would be required to prepare their own examinations. Upon evaluating the rulemaking plan, including the pros and cons associated with implementing the revised process on an industry wide basis, the Commission directed the staff to proceed with the rulemaking while continuing to monitor the results of the facility-prepared examinations.

Although the staff had considered the possibility of implementing the revised examination process by issuing a generic letter rather than amending 10 CFR Part 55, that alternative was dismissed because variations in the level of participation would have complicated resource planning over the long term and because the staff considered it more appropriate to impose generic requirements through rulemaking. The staff recognized that the proposed rulemaking would impose new requirements on facility licensees but took the position that the backfit rule, 10 CFR 50.109, did not apply. However, upon further review after issuing the proposed rulemaking, the staff has concluded that requiring all facility licensees to prepare their own examinations could be considered a backfit and that there is insufficient basis to document a finding that paragraphs (a)(2) and (a)(3) of 10 CFR 50.109 are not applicable and a backfit analysis is not required. Consequently, the staff has decided to implement the provisions of the proposed rule on a voluntary basis, which would not constitute a backfit. The merits of the revised examination development process support proceeding with a rule change that gives facility licensees the option to prepare their own examinations, rather than terminating the pilot program and resuming the NRC-prepared examination process on an industrywide basis. In its comments on the proposed rulemaking, the Nuclear Energy Institute (NEI) asserted that preparing quality examinations with higher cognitive level questions requires detailed plant knowledge that can best be provided by facility licensees. NEI and a number of facility managers have also commented that the shift to facility-prepared examinations has improved the examination process and that it should be continued. The final rule will allow, rather than require, power reactor facility licensees to prepare their own operator licensing written examinations and operating tests and to proctor and grade the written examinations. In lieu of performing these activities, facility licensees may request the NRC to complete them. In making this final rule change, the NRC will continue to administer (i.e., manage and oversee) the initial operator licensing examination process by: (1) developing the generic fundamentals examinations (which are also proctored by facility licensees); (2) reviewing and approving the facility-developed, site-specific written examinations and operating tests; and (3) independently conducting and grading both the dynamic simulator and walk-through portions of the operating test, which is considered the most performance-based aspect of the licensing process and permits the NRC to evaluate the operator and senior operator applicants' competence under normal and abnormal plant conditions.

BACKGROUND:

The NRC's initial operator licensing examination consists of the following parts: (1) a written generic fundamentals examination (covering reactor theory, thermodynamics, and components) that license applicants have to pass as a prerequisite for taking the site-specific examination; (2) a site-specific written examination covering plant systems, emergency and abnormal plant procedures, and plant-wide generic knowledge and abilities; and (3) a site-specific operating test consisting of three categories, including a crew-based, dynamic simulator performance demonstration, an individual, task-based walk-through covering control room and in-plant systems, and various plant administrative requirements.

On March 24, 1995, the staff informed the Commission of its intent to revise the operator licensing program to allow greater participation by facility licensees and to eliminate contractor assistance in this area (SECY-95-075, "Proposed Changes to the NRC Operator Licensing Program"). The staff's proposal was motivated by: (1) the general improvement in the performance level of power reactor facility licensees' training programs and the licensees' demonstrated ability to develop and administer licensed operator requalification examinations; (2) the NRC's continuing efforts to streamline the functions of the Federal government; and (3) the need to accommodate anticipated resource reductions. In a staff requirements memorandum (SRM) dated April 18, 1995, the Commission approved the staff's proposal to initiate a transition process to revise the operator licensing program and directed the staff to consider carefully the experience from pilot examinations before fully implementing the changes. On August 15, 1995, the NRC staff issued Generic Letter (GL) 95-06, "Changes in the Operator Licensing Program," outlining the revised process for developing examinations, and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.

Between October 1, 1995, and April 5, 1996, the NRC staff reviewed, approved, and administered 22 operator licensing examinations using the revised process. The staff documented the results of the pilot examinations in SECY-96-123, "Proposed Changes to the NRC Operator Licensing Program," and briefed the Commission on June 18, 1996. In an SRM dated July 23, 1996, the Commission directed the staff to prepare a rulemaking plan to justify the changes to 10 CFR Part 55 and to present additional information on a number of issues related to the revised examination process. The SRM also authorized the staff to continue the pilot program on a voluntary basis pending a final Commission decision on the revised examination process. On September 25, 1996, the NRC staff issued SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 to Change Licensed Operator Examination Requirements." Upon Commission approval of the rulemaking plan in an SRM dated December 17, 1996, the staff prepared the proposed rulemaking (SECY-97-079, dated April 8, 1997), which was approved with modifications in an SRM dated June 26, 1997, and published in the Federal Register on August 7, 1997 (62 FR 42426). When the comment period expired on October 21, 1997, 11 comment letters had been received; 2 more arrived after the expiration date. The 13 comment letters and NRC's responses are summarized in the Statements of Consideration of the final rule (Attachment 2).

DISCUSSION:

From the time the pilot program began in October 1995 through the end of June 1998, the NRC staff reviewed, approved, and administered a total of 102 examinations that were voluntarily developed by facility licensees in accordance with the NRC's examination criteria and guidance.

Facility licensees prepared the written examinations and the operating tests, proctored the written examinations, and graded the written examinations using the guidance provided by the NRC in GL 95-06 during the early stages of the pilot program, and subsequently in interim Revision 8 of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors." NRC examiners reviewed the examinations and tests to determine if they were consistent with NRC standards, directed facility licensees to make whatever changes were necessary to achieve NRC standards, and approved the examinations and tests before they were administered. NRC examiners independently administered all of the operating tests, reviewed the grading of the written examinations, and made the final licensing recommendations for approval by NRC management pursuant to 10 CFR 55.51.

Pilot Process Effectiveness

The NRC's review of the 102 examinations revealed that the quality of the licensee-proposed examinations varied widely; essentially all of the proposed examinations required some changes and many needed significant changes to achieve NRC standards for quality and level of difficulty. In some cases, the examinations were delayed for a period of time in order for the NRC staff to evaluate the examinations and review the facility licensee's changes. Feedback from the NRC examiners who were involved with the pilot examinations indicates that the facility-prepared examinations that were finally approved by the NRC were comparable, in terms of quality and level of difficulty, to examinations prepared by the NRC before and since beginning the pilot program. Although the staff expected that the quality of the proposed examinations would improve as the industry became familiar with NRC examination criteria and expectations, and gained experience in preparing the examinations, some of the proposed examinations continue to require more changes than anticipated.

The average passing rates for reactor operators (ROs) and senior reactor operators (SROs) on both the written and operating portions of the 102 facility-prepared examinations administered through the end of June 1998 were only slightly lower than the corresponding passing rates on the examinations administered during fiscal year (FY) 1995, the last year in which all of the examinations were prepared by the NRC or its contractors. However, the average passing rates were consistent with the range of passing rates for FYs 1990 through 1994. This supports the NRC staff's conclusion that the facility-prepared examinations, revised as directed by the NRC, are comparable to NRC-prepared examinations. The lower passing rate on the facility-prepared written examinations suggests that the level of difficulty of the examinations may have increased slightly under the revised examination process. By taking advantage of the facility licensees' more detailed plant knowledge, the NRC examiners have been able to focus their attention on the psychometric quality (e.g., the cognitive level at which the questions are written and the plausibility of the distractors - those answer choices that should appear reasonable but, nevertheless, are wrong) rather than the technical details of each question, thereby improving the discrimination validity, and possibly the level of difficulty, of the examination. Considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training effectiveness and variability in licensees' screening of applicants) that could be responsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant. The examinations are discriminating at a conservative and acceptable level (i.e., the average level of difficulty has not declined) and the safe operation of the facilities is not being compromised. (Attachment 1 contains a detailed summary and analysis of the examination results.)

Pilot Process Efficiency

With regard to the efficiency of the revised examination process, the experience to date suggests that the average industry cost will not be significantly different from what it would be if the NRC prepared the licensing examinations. Feedback from the industry reflects that the cost for some facility licensees was higher than it would have been for an NRC-prepared examination; other licensees, however, were able to prepare acceptable quality examinations at a lower cost than if the examinations had been prepared by the NRC. The industry generally attributed the higher cost to the revised examination and administrative criteria under the pilot examination process. Although the NRC staff acknowledges that the revised criteria contribute to the elevated cost, many of the variables that affect the final cost of the examination (e.g., its initial quality) will be under the control of facility licensees if they elect to prepare their own examinations. Those facility licensees that submit acceptable quality examinations are likely to save resources despite the additional criteria that the NRC considers necessary under the revised examination process. However, those facility licensees that submit examinations that require many changes because they do not meet NRC standards, may have increased costs. The higher cost of revising a low-quality examination should serve as an incentive for facility licensees to submit acceptable quality examinations. (Attachment 3 contains an analysis of the industry resource burden.)

Comments from the NRC chief examiners who worked on the pilot examinations indicate that the average amount of time spent reviewing and revising the facility-prepared examinations was generally consistent with the estimates developed before starting the pilot program. Although it took longer than expected to revise a number of the examinations to meet NRC standards for quality, the resource burden was generally offset by other examinations that required less effort to review and revise.

The increased efficiency of the revised examination process has enabled the NRC to eliminate the use of contractors in the operator licensing program and to administer the initial operator licensing and requalification inspection programs with the existing NRC staff. Before initiating the pilot examination and transition process at the beginning of FY 1996, the NRC spent approximately $3 million per year on contractor assistance for initial examinations and requalification inspections. During FY 1996, when approximately 40 percent of the examinations were voluntarily prepared by facility licensees, the NRC spent approximately $1.6 million for contractor assistance (to prepare and administer initial examinations and assist with requalification inspections). In FY 1997, facility participation increased to include approximately 75 percent of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million. The budgets for FYs 1998 - 2000 reflect the complete elimination of contractor support for the operator licensing program (with the exception of the generic fundamentals examination). (Attachments 1 and 3 contain additional analysis of the NRC resource burden.)

Backfit

As issued for comment, the proposed rulemaking would have required all power reactor facility licensees to prepare their written operator licensing examinations required by 10 CFR 55.41 and 55.43 and the operating tests required by 10 CFR 55.45. Although the proposed rulemaking did impose new requirements on facility licensees, the NRC had taken the position that the backfit rule, 10 CFR 50.109, did not apply because the shift in responsibility for preparing the examinations would not: (1) constitute a "modification of the procedures required to operate a facility" within the scope of the backfit rule; (2) affect the basic procedures for qualifying licensed operators; or (3) require facility licensees to alter their organizational structures.

In its review of the final rulemaking, the Committee To Review Generic Requirements (CRGR) stated that the staff had not demonstrated its basis to support that position. Therefore, the CRGR recommended that the provisions of the proposed rule be implemented on a voluntary basis, which would not constitute a backfit. Although the staff had considered and dismissed that alternative during the proposed rulemaking, it has since concluded that the benefits of the revised examination process (e.g., improved regulatory efficiency and greater licensee control over the examination costs) remain substantial even if every facility licensee is not required to prepare its own examinations. Rather than terminate the pilot program and resume the NRC-prepared examination process on an industrywide basis, the staff has decided to amend the final rule to give facility licensees the option to prepare their own examinations or to have them prepared by the NRC. In that regard, this action is consistent with a comment made by NEI in response to the proposed rulemaking that the industry would prefer to continue the voluntary process because it would allow flexibility for those power reactor facility licensees with small training staffs.

Resources

Under a voluntary process, the NRC staff's ability to meet the examination demands and schedule needs of facility licensees will depend on the number of licensees that elect to prepare their own examinations. For purposes of comparison, the average NRC-prepared examination requires approximately two-thirds of an FTE (full-time equivalent) to prepare, administer, and document, while the average facility-prepared examination requires about one-third of an FTE to review, administer, and document. Therefore, with the 17 regional FTEs that are currently budgeted for the initial examination program in FYs 1999 and 2000, the NRC staff would be able prepare about 25 to 30 examinations per year, while it would be able review about 50 to 55 facility-prepared examinations. During the two full years in which facility licensees have been allowed to prepare their examinations under the voluntary pilot program, the level of industry participation has ranged from about 75 percent (41/55) in FY 1997 to about 90 percent (50/56) in FY 1998. Therefore, if the current level of industry participation is maintained, the staff expects that it will be able to satisfy most facility licensees' examination demands and scheduling needs within the current budget.

To manage the workload, the NRC staff will resume issuing an annual administrative letter that will request facility licensees to report their anticipated examination needs for the next four fiscal years and their intentions with regard to preparing each examination. Moreover, the staff intends to work with NEI in an effort to prioritize and coordinate the examination schedule, address workload conflicts, and promote facility participation in the examination development process.

Examination Security

As discussed in SECY-96-206 and SECY-97-079, the NRC staff has focused additional attention on examination security. Therefore, staff has amended 10 CFR 55.49 in the final rule to ensure that applicants, licensees, and facility licensees understand the scope of the regulation with regard to the integrity of examinations and tests. The staff has also revised the final rule to require facility licensees that elect to prepare their own operator licensing examinations to establish, implement, and maintain procedures to control examination security and integrity. As a separate action, the staff has developed a proposed revision to the "General Statement of Policy and Procedures for NRC Enforcement Actions" (Enforcement Policy) to address violations relative to the requirements in 10 CFR 55.49 (refer to Attachment 4). The proposed revision includes examples of violations that may be used as guidance in determining the appropriate severity level for violations involving an examination compromise regardless whether the examination was prepared by a facility licensee or the NRC staff. In addition to considering enforcement action, the NRC will delay any examination that may have been compromised until the scope of the potential compromise is determined and measures can be taken to address the integrity and validity of the examination.

Finally, the staff has strengthened the examination security and integrity guidelines in final Revision 8 of NUREG-1021. Although the guidelines in NUREG-1021 are not regulatory requirements, facility licensees that elect to prepare their own examinations are encouraged to consult NUREG-1021 when developing their required examination security procedures. Once a facility licensee has established the required procedures, the NRC intends to monitor this area to ensure that the procedures are implemented and maintained.

Conclusion

The NRC staff has concluded that the revised examination process is both effective and efficient, even if it is implemented on a voluntary basis. The staff has reviewed the vulnerabilities discussed in SECY-96-206 (i.e., quality and consistency, independence and public perception, examination security, NRC resource, program stability, and examiner proficiency) and evaluated the measures that have been taken to mitigate the vulnerabilities. (Attachment 1 contains a summary analysis of each vulnerability and the potential effects on reactor safety.) The revised program gives the staff the ability to maintain acceptable standards of performance in each of these areas, and the pilot program has demonstrated that it is capable of doing this. The final rule will require facility licensees that elect to prepare their examinations to follow the same guidelines that the NRC staff will use to prepare the examinations at nonparticipating facilities (i.e., the revision of NUREG-1021 that was in effect six months before the date of the examinations). NRC license examiners will review each facility-prepared examination to ensure that it conforms to the guidelines in NUREG-1021 regarding content, format, quality, and levels of knowledge and difficulty. The NRC will direct facility licensees to revise the examinations, as necessary, and will approve every examination before it is given. The NRC will not approve those examinations that do not meet NRC standards and, to the extent that facility employees do not follow the examination criteria in NUREG-1021 or NRC-approved alternatives, will address deficiencies in the submitted examinations as licensee performance issues. Moreover, because the NRC has discontinued the use of contractors, NRC examiners will directly observe and evaluate the performance of every license applicant on both the dynamic simulator and walk-through portions of the operating test, which is considered the most performance-based aspect of the licensing process and permits the NRC to evaluate the applicants' competence under normal and abnormal plant conditions. The staff believes that this will improve the NRC's ability to focus on operator performance issues and enable NRC examiners to gain more experience in a shorter time.

The revised examination process is more consistent with the NRC's other oversight programs because it requires NRC examiners to review materials prepared by facility licensees and holds facility licensees responsible for the quality of those materials. The revised process has enabled NRC examiners to focus more on the psychometric quality of the examinations (e.g., the cognitive level at which the questions are written and the plausibility of the distractors or "wrong answer" choices) prepared by the facility licensees than on the technical accuracy of the examinations, which was their primary focus when the examinations were prepared by the NRC. This shift in the NRC examiners' focus, coupled with the facility licensees' technical expertise, has the potential to improve the overall quality of the facility-prepared licensing examinations.

COORDINATION:

The Office of the General Counsel (OGC) has no legal objection to this final rulemaking. The Committee to Review Generic Requirements (CRGR) has reviewed the final rulemaking and endorses implementing the revised examination process on a voluntary basis. The Office of the Chief Financial Officer has reviewed this Commission paper for resource impacts and has no objections. The Advisory Committee on Reactor Safeguards (ACRS) reviewed the version of this rulemaking that would have required all power reactor facility licensees to prepare their licensing examinations and recommended the following: (1) that the rulemaking should be issued for use by the industry, and (2) that the staff should analyze the results of the pilot examinations to ensure that the quality and level of difficulty of the examinations are consistent across the regions. The staff has completed the recommended analysis and discussed the results in Attachment 1. The ACRS considered the proposed final amendment and decided not to review it. The proposed changes do not alter the Committee's previous recommendations on this matter, and the Committee has no objection to issuing the final rule for industry use.

The Office of the Chief Information Officer has reviewed the final rule for information technology and information management implications and concurs in it. However, the added information collection requirement must be submitted to the Office of Management and Budget (OMB) for approval before the final rule can be published.

RECOMMENDATIONS:

That the Commission

1. Approve the notice of final rulemaking (Attachment 2) for publication in the Federal Register.

2. Approve the notice amending the "General Statement of Policy and Procedures for NRC Enforcement Actions" (Enforcement Policy) (Attachment 4) for publication in the Federal Register.

3. Approve the submittal of a clearance request to OMB for the additional information collection required by the amended final rule (i.e., for facility licensees to establish, implement, and maintain examination security procedures if they elect to prepare their own examinations).

4. Certify that this rule, if promulgated, would not have a significant economic impact on a substantial number of small entities to satisfy the requirements of the Regulatory Flexibility Act, 5 U.S.C. 605(b).

5. Determine that neither an environmental impact statement nor an environmental assessment need be prepared because this proposed rule is eligible for a categorical exclusion as defined in 10 CFR 51.22(c)(1).

6. Note that

1. The Chief Counsel for Advocacy of the Small Business Administration will be informed of the certification regarding economic impact on small entities and the reasons for it as required by the Regulatory Flexibility Act.

2. The information collection requirements in the proposed rule have been approved by OMB, and the additional information collection requirement in the final rule (i.e., 10 CFR 55.40(b)(2)) is subject to review and approval by OMB. The staff will not publish the final rule until the OMB clearance has been received. If OMB requires any substantive changes, the staff will obtain Commission approval before publishing the final rule.

3. A regulatory analysis has been prepared (Attachment 3).

4. This is not a "major" rule as defined in the Small Business Regulatory Enforcement Fairness Act of 1996, 5 U.S.C. 804(2).

5. The appropriate Congressional committees will be informed (Attachment 5).

6. A public announcement will be issued when the final rulemaking is filed with the Office of the Federal Register (Attachment 6).

7. Copies of the notice of final rulemaking and final Revision 8 of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors," will be distributed to all facility licensees.

ATTACHMENT 1

PILOT PROGRAM RESULTS, CHANGES, AND VULNERABILITIES

• Background
• Pilot Program Overview
• SECTION IPILOT PROGRAM RESULTS
• Examination Quality
• Pilot Process Efficiency
• SECTION IISTAFF RESPONSES TO COMMENTS ON INTERIM REVISION 8 OF NUREG-1021
• Public Comments
• Comments From NRC Staff
• SECTION IIIVULNERABILITY REVIEW
• Quality and Consistency of the Examinations
• Independence and Public Perception
• Examination Security
• NRC Resources
• Program Stability
• Examiner Proficiency
• Reactor Safety

Background

In a staff requirements memorandum (SRM) of April 18, 1995, the Commission approved the NRC staff's proposal to initiate a transition process to revise the operator licensing program to require power reactor licensees to prepare the initial licensing examinations for their own facilities and directed the NRC staff (hereafter staff) to carefully consider experience gained from a pilot examination program before fully implementing the changes. With the Commission's approval, the staff issued Generic Letter (GL) 95-06, "Changes in the Operator Licensing Program," on August 15, 1995, outlining the revised examination development process and soliciting volunteers to participate in a pilot examination program to evaluate and refine the methodology.

Between October 1, 1995, and April 5, 1996, the staff reviewed and approved 22 operator licensing examinations prepared by facility licensees within the framework of a pilot program. These examinations were prepared using the guidance in Revision 7 (Supplement 1) of NUREG-1021, "Operator Licensing Examiner Standards," and the additional guidance in GL 95-06. The results of the pilot examinations were discussed in SECY-96-123, "Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996, and lessons learned from those examinations were incorporated in interim Revision 8 of NUREG-1021, which was retitled "Operator Licensing Examination Standards for Power Reactors." Based on the results of the pilot examinations, the staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the initial licensing examinations and to proctor and grade the written portion of the examinations.

In an SRM of July 23, 1996, the Commission authorized the staff to continue the pilot examination process on a voluntary basis and directed the staff to develop a detailed rulemaking plan to justify the changes that might be necessary to 10 CFR Part 55. The Commission also directed the staff to address the pros, cons, and vulnerabilities associated with the revised examination process to facilitate a Commission decision on whether to implement the revised process on an industrywide basis. With the Commission's approval, the staff resumed the pilot program in August 1996 (after having completed the original 22 examinations in April 1996), and by the end of June 1998 had reviewed, approved, and administered 80 additional examinations developed by facility licensees in accordance with NRC guidance, thereby raising the total number of pilot-style examinations to 102. The results of those examinations are discussed in Section I below.

On September 25, 1996, the staff forwarded the requested rulemaking plan and a discussion of the pros, cons, and vulnerabilities of the pilot process (including a detailed discussion of the measures taken to mitigate the vulnerabilities) to the Commission in SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 to Change Licensed Operator Examination Requirements." In an SRM of December 17, 1996, the Commission directed the staff to implement interim Revision 8 of NUREG-1021 on a voluntary basis and to proceed with the proposed rulemaking that would require all power reactor facility licensees to prepare their own operator licensing examinations. The staff forwarded the proposed rule (SECY-97-079, "Proposed Rule - Initial Licensed Operator Examination Requirements") to the Commission on

April 8, 1997, and on June 26, 1997, the Commission approved its publication in the Federal Register for a 75-day comment period. The notice of proposed rulemaking (62 FR 42426) dated August 8, 1997, also invited public comments on interim Revision 8 of NUREG-1021. Public comments related to the regulation are addressed in the Federal Register notice (FRN) (Attachment 2), and significant comments regarding interim Revision 8 of NUREG-1021 are discussed in Section II below (the staff also addressed several editorial comments during the revision process).

The staff has reviewed the vulnerabilities discussed in SECY-96-206 (i.e., quality and consistency, independence and public perception, examination security, NRC resources, program stability, and examiner proficiency) in light of the additional experience with the revised examination process since that paper was issued on September 25, 1996. The status of each vulnerability, including an overview of the compensatory measures, and the potential effects that the revised process might have on reactor safety are summarized in Section III below.

Pilot Program Overview

From the time the pilot program began in October 1995 through the end of June 1998, the staff reviewed, approved, and administered a total of 102 examinations that were voluntarily developed by facility licensees under the pilot examination and transition program.

Facility licensees prepared the written examinations and the operating tests, proctored the written examinations, and graded the written examinations using the guidance provided by the NRC in GL 95-06 during the early stages of the pilot program, and subsequently in interim Revision 8 of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors." NRC examiners reviewed the examinations and tests to determine if they were consistent with NRC standards, directed facility licensees to make whatever changes were necessary to achieve NRC standards if the submitted examinations and tests were deficient, and approved the examinations and tests before they were administered. NRC examiners independently administered all of the operating tests, reviewed the written examination grading, and made the final licensing recommendations for approval by NRC management.

SECTION I
PILOT PROGRAM RESULTS

The SRM for SECY-96-206 directed the staff to: (1) continue monitoring the results of the pilot examinations for indications that the revised examination development process may not be appropriate; and (2) to present the current data to the Commission along with the proposed and final rulemaking packages. Accordingly, Table 1.1 summarizes the results of the 102 pilot examinations conducted between October 1995 (the beginning of the pilot program) and the end of June 1998. For purposes of comparison, Table 1.1 also includes the examination results for fiscal years (FYs) 1990 through 1995 (the last year in which the NRC prepared all of the examinations) and the examinations prepared by the NRC since the pilot program was initiated (i.e., for FYs 1996, 1997, and the first three quarters of 1998). In conducting its analysis, the staff has focused primarily on the annual examination passing rates because the small number of applicants on most examinations makes it difficult to draw programmatic conclusions from the individual examination results. As recommended by the Advisory Committee on Reactor Safeguards (ACRS), the staff has also analyzed and compared the examination results across the NRC regional offices to determine whether the quality and level of difficulty of the examinations are consistent.

The average passing rates for reactor operators (ROs) and senior reactor operators (SROs) on both the written and operating portions of the 102 facility-prepared examinations administered through the end of June 1998 were only slightly lower than the corresponding passing rates on the NRC-prepared examinations administered during FY 1995, the last year in which all of the examinations were prepared by the NRC or its contractors. However, as shown in Table 1.1, the average passing rates tend to fluctuate, and the passing rates on the facility-prepared examinations were generally consistent with the historical range. The fluctuations in the passing rates from year to year and between the examinations prepared by the NRC and facility licensees could be caused by a number of factors including variations in the average level of experience of the license applicants, changes in the quality of the training or the facility licensee's threshold for screening its applicants before they take the licensing examination, or variations in the average level of difficulty of the examinations. Even if it is assumed that all of the variation in the passing rates on the facility-prepared examinations has been caused by the changes in the examinations, the magnitude and direction of the variation supports the staff's conclusion that, on the average, the facility-prepared examinations, revised as directed by the NRC, are comparable to NRC-prepared examinations. The examinations are discriminating at a conservative and acceptable level (i.e., the average level of difficulty has not declined and any possible increase is not considered to be of practical significance), and the safe operation of the facilities is not being compromised.

If the lower passing rate on the facility-prepared written examinations is indeed an indication that the level of difficulty of the examinations has increased, it may be a reflection of the NRC examiners' concentration on the psychometric quality (e.g., the cognitive level at which the questions are written and the plausibility of the distractors - those answer choices that should appear reasonable but, nevertheless, are wrong) in addition to the technical quality of the examinations. Although the staff did not intend for the level of difficulty or the failure rate on the examinations to increase, the examiners' efforts to achieve NRC standards regarding the cognitive level of questions and to improve the plausibility of the distractors may have improved the discrimination validity of the examinations. Consequently, those applicants who may have passed an examination containing lower cognitive level questions on which some of the distractors could be eliminated as implausible are now having more difficulty selecting the correct answers. The NRC examiners' efforts to improve the plausibility of the distractors has reduced the applicants' chances of passing the examination by simply guessing the correct answers. However, considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training deficiencies or less stringent screening of applicants by facility licensees) that could be responsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant. (The level of difficulty of the written examinations is further discussed below in connection with the review of pilot examination quality.)

The RO passing rates on the facility-prepared operating tests during FYs 1996 and 1997 were consistent with the passing rates on the tests prepared by the NRC during those same years. The 98 percent RO operating test passing rate during FY 1995 was unusually high compared to prior years. Although the passing rate on the facility-prepared SRO operating tests during FY 1997 was slightly lower than the FY 1995 baseline, it was the same as the passing rate on the NRC-prepared operating tests during FY 1996.

The SRO passing rates on the NRC-prepared written examinations and operating tests during FY 1997 were lower than the FY 1995 baseline, the historical ranges, and the facility-prepared results during FY 1997. The low passing rates on the NRC-prepared examinations are in large part attributable to two examinations on which the applicants' performance was unusually poor for reasons that may include variations in the quality of the applicants' training and the difficulty of the examinations. This result illustrates that the applicants' performance on the examinations can vary from year to year regardless of whether the examinations are prepared by the NRC or by facility licensees.

When the pass rates for FYs 1996 and 1997 were compared across the four NRC regions, only one statistically significant difference was identified. In FY 1996, the operating test pass rates for RO and SRO applicants taking NRC-prepared tests in one region were significantly lower than in the other regions. This pattern did not repeat itself in FY 1997. Therefore, the staff does not believe that there is any basis for concern or corrective action. The staff will analyze the FY 1998 regional performance when all the examination results, including the outstanding appeals, are final.

Another indicator of the relative consistency in the level of difficulty of the NRC-prepared and facility-prepared examinations is the amount of variation in the applicants' average grades on the written portion of the licensing examinations. Table 1.2 summarizes the average RO and SRO grades for FY 1995 through the third quarter of FY 1998. The data are consistent with the passing rates summarized in Table 1.1 and support the staff's conclusion that the facility-prepared examinations are discriminating at a conservative and appropriate level. The staff expects that minimally qualified applicants will be able to achieve a score of at least 80 percent on an NRC written licensing examination.

When the average written examination grades for FYs 1996 and 1997 were compared across the NRC regions, again, only one statistically significant difference was identified. In FY 1997, the average written examination grade for SRO applicants in one region was approximately two percentage points lower than the national average. Although two-point regional variations in the average grades were not unusual, all but one were not statistically significant. The lower average grade in FY 1997 was also reflected in a slightly lower pass rate on the SRO written examination in that region; however, the regional variation in the SRO pass rates was not statistically significant. Therefore, the staff has not identified any patterns that would lead to a conclusion that the licensing examinations are significantly different nationwide. However, the staff will continue to monitor the examination results for regional differences and trends; the FY 1998 data will be analyzed when the results of all the examinations given during the year, including any outstanding appeals, are final.

Examination Quality

Although the overall examination results indicate that the facility-prepared examinations are effective at discriminating between those applicants who have mastered the job requirements and those who have not, the NRC reviews showed that essentially all of the facility-prepared examinations required some changes. According to questionnaires completed by the NRC chief examiners responsible for the pilot examinations, the average facility-prepared examination required approximately 10 to 20 changes, which is consistent with the number of changes often required on examinations prepared by NRC contract examiners. However, a number (about 40 percent in FY 1997) of the facility-prepared examinations required significant changes to achieve NRC standards for quality and level of difficulty, and, in some cases, the examinations were delayed in order for the staff to fully evaluate the examination and allow time for the facility personnel to change the examinations. In FY 1997, about 10 percent of the examinations had to be delayed - some for only a week and others for up to a month - until the changes could be made. The written examinations generally required the most changes. The majority of the changes were necessary to improve the plausibility of the distractors. Both the walk-through and dynamic simulator portions of the operating tests required changes to replace or enhance nondiscriminatory tasks, events, and questions.

The staff evaluated comments made by the NRC chief examiners and facility licensees involved with the examinations to gain an understanding of why so many of the examinations have required significant changes. Four possible explanations have been considered, but their order and level of significance is uncertain: (1) the examination criteria and guidance in interim Revision 8 of NUREG-1021 may have been unclear; (2) the facility personnel assigned to prepare the licensing examinations may not have had the expertise to construct examinations meeting NRC standards of quality; (3) the facility personnel assigned to prepare the examinations may have been given insufficient time or other resources to prepare an adequate examination; and (4) it is possible that the facility personnel assigned to prepare the examinations may not have followed the guidance in NUREG-1021 because of a bias toward writing examinations that were less difficult. Each of these possibilities is discussed below.

With regard to the examination criteria and guidance in interim Revision 8 of NUREG-1021, the NRC chief examiners who responded to surveys on their experiences during the pilot-style examinations reported that the instructions were clear. They recommended some clarifications in the guidance, but no major changes to the examination criteria that have a direct bearing on examination quality. As discussed in SECY-96-206, the staff solicited industry comments on a draft of Revision 8 of NUREG-1021 in February 1996, but none of the respondents (which included the Nuclear Energy Institute (NEI) and two facility licensees) questioned the clarity of the guidance. When SECY-97-079 solicited comments on interim Revision 8 of NUREG-1021, one facility licensee (of the five that responded) commented that the guidance on level of difficulty needs to be clarified to enable facility licensees to satisfy the NRC's expectations in this area and to minimize the NRC examiners' subjective judgment. (Both the NRC examiners' and the facility licensees' comments on interim Revision 8 of NUREG-1021 are summarized in Section II below.) Based on the information available, the staff has concluded that, in general, the guidance in interim Revision 8 of NUREG-1021 is sufficiently detailed to yield examinations of acceptable quality, difficulty, and consistency. NUREG-1021 discusses and illustrates the guidelines for constructing questions and the psychometric principles that justify most of the changes that were made in the facility-prepared examinations. To further ensure that facility licensees understand the guidelines in NUREG-1021, the staff has conducted or participated in a number of public meetings and workshops in an effort to communicate its expectations to the facility employees who will prepare the examinations. Additional NRC and industry workshops are being planned to address examination quality and solicit industry feedback on NUREG-1021. The staff will continue to accept and review questions and comments on the NUREG and expects that additional clarifications and changes will become necessary in response to unanticipated issues.

Although it was not reflected in their public comments, some facility managers have since commented during meetings with the staff (e.g., a January 27 - 29, 1998, Region II public workshop to review test-writing techniques and other examination issues) that the NRC has raised its standard with regard to level of difficulty (particularly that of the written examinations) since beginning the pilot program. Although the staff did not intend to raise its standards during the pilot program, it is possible that the NRC examiners' efforts to maintain the effectiveness of the operator licensing process have raised the discrimination validity and level of difficulty of the examinations.

The NRC has always expected its licensing examinations to discriminate between applicants who have and have not mastered the knowledge, skills, and abilities required to perform licensed RO and SRO duties. To accomplish that objective, the NRC established a number of criteria against which its examiners are expected to evaluate test items in order to achieve an acceptable examination in terms of quality, consistency, level of knowledge and level of difficulty. In addition to being technically accurate, the NRC expects written examination questions to be written at the highest level of knowledge appropriate for the topics selected for the examination and the distractors in multiple choice questions to be plausible (i.e., they should appear reasonable but, nevertheless, be wrong). When the NRC or a contractor prepared an examination, the technical accuracy of the questions generally took precedence over their psychometric quality. However, the staff believes that the overall quality of the questions has improved over time as a result of repetitive use and revision and would have continued to improve even if the NRC had retained responsibility for developing the licensing examinations. Now that facility licensees are developing the examinations, focusing on the technical accuracy of the questions, NRC examiners are able to devote more effort to ensuring that the distractors are plausible and the questions are written at the appropriate level of knowledge.

In the late 1980s, when the NRC shifted the format of the written examination from essay and short answer to multiple choice questions, it took time to develop the NRC's bank of questions, particularly those that tested at the higher cognitive levels because they are more challenging and costly to write. By the time the NRC initiated the pilot program in FY 1996, the average written examination (based on audits performed by the staff) contained approximately 50 percent comprehension or analysis questions. To ensure that the examinations would remain effective, the staff revised interim Revision 8 of NUREG-1021 to establish a 50 percent lower limit on the higher cognitive level questions. The staff has concluded that the lack of an upper limit may have contributed to the increased variability in the level of difficulty of the examinations. The staff has also noted that some examiners may have suggested that facility licensees submit more than the minimum percentage of comprehension-level questions, thinking that the examination authors are more likely to over-estimate than under-estimate the cognitive levels of the questions. As discussed in Section II below in connection with the resolution of public comments on interim Revision 8 of NUREG-1021, the staff has clarified final Revision 8 by specifying a range of higher cognitive level questions, instead of a lower limit.

With regard to the expertise of the facility personnel assigned to prepare the examinations (i.e., the second possible explanation for the number of examination changes), the preliminary and proposed regulatory analyses for this rulemaking expressed the staff's expectation that the training personnel at power reactor facilities already possess the basic expertise needed to develop test items for the initial licensing examinations. The staff believed that many of the problems with examination quality and level of difficulty could be attributed to the facility licensees' lack of familiarity with the NRC's requirements and expectations and that, with experience, the industry would gain proficiency in preparing the examinations. However, the overall quality of the examinations submitted to the NRC during the pilot program did not improve as expected over time. Although approximately half of the 17 facility licensees that had prepared more than one examination by the end of FY 1997 did maintain or improve the quality of their second or third examinations, the quality of the other facility licensees' second or third examinations was lower. However, the pilot examination program has demonstrated that the NRC's review and approval process is ensuring that the quality and validity of the examinations that are administered to the license applicants, and upon which the NRC's licensing decisions are based, meet NRC standards.

Although it is unclear to what extent the problems with examination quality and level of difficulty are determined by the qualifications and training of the examination writers, the staff has asked the industry to address this area. Some facility licensees and NRC examiners have asserted that the personnel restrictions in NUREG-1021 could impede training or diminish examination quality because facility licensees must choose where to assign their best-qualified personnel and whether to hire contractors to write the examinations. Therefore, the staff has relaxed the personnel restrictions in final Revision 8 of NUREG-1021 to allow facility licensees greater flexibility regarding which instructors can prepare the examinations and to make the restrictions more closely parallel what is done on requalification examinations (as explained in Sections II and III below in connection with the staff's response to the public comments on interim Revision 8 of NUREG-1021 and the review of concerns regarding independence and public perception).

With regard to the level of facility resources required for preparing a licensing examination (the third possible explanation for the number of examination changes noted above), NRC examiners have seen instances where the personnel on some facility training staffs have been challenged to prepare the examinations without additional assistance. Moreover, NEI publicly commented that the personnel restrictions in NUREG-1021 have made it more difficult for facility licensees with small training staffs to prepare the examinations. Therefore, some facility licensees may have difficulty allocating the resources necessary for their training staffs to prepare good quality licensing examinations, while continuing to train the license applicants. The fact that some facility licensees have had to hire contractors to prepare their licensing examinations may also have had an effect on the technical quality of their examinations and the efficiency with which they were prepared. (Issues regarding the facility resource burden are further discussed in the regulatory analysis (Attachment 3), and the NEI comment regarding personnel restrictions, including the staff's rationale for relaxing those restrictions, is discussed in Section II below.)

As noted above, the fourth possible explanation for the number of examination changes is that facility personnel assigned to prepare the examinations may not have followed the guidance in NUREG-1021 because they may have been biased toward writing examinations that were less difficult. Some NRC examiners have reported a reluctance on the part of facility personnel to make NRC-directed changes that might increase the level of difficulty of the examination and expressed concern (in their public comments on the rulemaking) that the NRC's review and approval process may not adequately compensate for the conflict of interest inherent in the revised examination method. Two NRC examiners commented that the revised process puts training managers in a no-win situation and that some managers will prepare the easiest possible examination to minimize the risk of their applicants failing the examination. (A training manager made such a comment to an NRC examiner, and the region addressed the issue in a management meeting with the facility licensee.) The staff agrees that some facility personnel could prepare an examination that will maximize their applicants' chances for success despite the criteria and guidance in NUREG-1021 and their own expertise in testing and measurement techniques. The issues of independence and conflict of interest, as well as the controls that the staff has implemented to address these issues, are discussed in Section III below in connection with the review of the vulnerabilities associated with the revised examination process and in the FRN in connection with the resolution of public comments on the rule.

In summary, the staff has concluded that, despite its concerns regarding the quality of the facility-prepared examinations as submitted to the NRC, the revised examination process offers an effective alternative to examinations prepared by the NRC or an NRC contractor. The pilot program has demonstrated that NRC examiners and their supervisors can and will ensure that facility licensees revise the examinations as necessary to attain acceptable levels of quality and difficulty. Moreover, the staff will not approve those examinations that do not meet NRC standards and, when appropriate, address significant deficiencies in the submitted examinations as licensee performance issues in the examination reports.

Pilot Process Efficiency

With regard to the efficiency of the revised examination process, experience with the pilot examinations to date suggests that the average cost to facility licensees will not differ significantly from the cost of the NRC or its contractors preparing the licensing examinations. Comments from the industry reflect that it cost some facility licensees more to prepare their own examinations than they would have paid the NRC to prepare them; other licensees, however, were able to prepare good quality examinations at a lower cost than if the examinations were prepared by the NRC.

The industry attributed the higher cost to the revised examination and administrative criteria (e.g., documenting the source of test questions) under the pilot examination process. Although the staff acknowledges that the revised criteria contribute somewhat to the increased cost, facility licensees that elect to prepare their own examinations will be in a position to control many of the variables that affect the quality and, consequently, the final cost of an examination. For example, facility licensees can manage the size and quality of their examination banks and, to a large extent, the training and experience of the people they assign to write their licensing examinations. The revised examination process puts facility licensees in control of developing the examinations and holds them responsible for their quality. If a facility licensee submits an acceptable quality examination, it is likely to save resources despite the additional administrative requirements; however, if the facility licensee submits an examination that requires many changes, it is likely to cost more than if the NRC had prepared the examination.

Comments from the NRC chief examiners who worked on the pilot examinations indicate that the average amount of time NRC staff spent reviewing the facility-prepared examinations and directing changes was generally consistent with the estimates developed before starting the pilot program. Although about 20 percent of the examinations reviewed in FY 1997 required significantly more effort than expected to achieve the NRC's standards, the burden was generally offset by other examinations that required less effort to review and revise.

The increased efficiency of the revised examination process has enabled the NRC to eliminate the use of contractors in the operator licensing program and conduct the initial operator licensing and requalification inspection programs with the existing NRC staff. Before initiating the pilot examination and transition process at the beginning of FY 1996, the NRC spent approximately $3 million per year on contractor assistance for initial examinations and requalification inspections. During FY 1996, when approximately 40 percent of the examinations were prepared by facility licensees, the NRC spent approximately $1.6 million for contractor assistance (to prepare and administer initial examinations and assist with requalification inspections). In FY 1997, facility participation increased to include approximately 75 percent of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million. The FY 1998 and FY 1999 budgets reflect the complete elimination of contractor support for the operator licensing program (with the exception of the generic fundamentals examination).

SECTION II
STAFF RESPONSES TO COMMENTS ON INTERIM REVISION 8 OF NUREG-1021

Public Comments

As noted above, since beginning the pilot program in October 1995, the NRC staff has given the industry and public two opportunities to formally comment on the guidance in Revision 8 of NUREG-1021. The staff has also participated in a number of public meetings and workshops that have given the industry an alternative means to provide feedback or to raise questions regarding the guidance in NUREG-1021. The staff has considered all of the comments and questions and made changes or clarifications, as appropriate. The staff will continue to accept and review questions and comments as they arise and expects that additional clarifications and changes will become necessary in response to unanticipated issues.

Of the 13 individuals and organizations that responded to the proposed rule published in the Federal Register on August 7, 1997 (62 FR 42426), 7 also commented on interim Revision 8 of NUREG-1021. The staff considered all of the comments and revised interim Revision 8 of NUREG-1021 as appropriate; the significant comments and the staff's responses are summarized below. A copy of final Revision 8 of NUREG-1021 will be sent to each facility licensee upon Commission approval of the final rule. Copies of NUREG-1021, Revision 8, will also be available in the NRC Public Document Room (PDR), 2120 L Street, NW (Lower Level), Washington, DC, and on the internet at http://www.nrc.gov.

Comment: Seven respondents recommended changes to the restrictions that interim Revision 8 of NUREG-1021 places on facility personnel who can participate in developing the licensing examinations. The respondents from the nuclear industry generally believe that the restrictions place an undue burden on facility licensees with no apparent benefit and that the personnel who played an active role in training the license applicants should not be restricted from preparing the licensing examinations. The industry strongly endorsed adopting the NRC's requalification examination personnel restrictions so that any instructor could prepare the initial licensing examination as long as he or she refrained from teaching the applicants after starting work on their licensing examination. Some of the industry respondents asserted that the guidance regarding supervisory involvement in the examination development process needs to be clarified. One NRC license examiner recommended that the restrictions be clarified to prevent the same person from preparing both the audit examination that the facility licensee uses to confirm the applicants' readiness and the NRC licensing examination.

Response: The staff acknowledges that the personnel restrictions may have increased the burden for some facility licensees with smaller training staffs, and that the quality of the training could suffer if personnel are diverted from training assignments to prepare the examinations. Therefore, the NRC has revised the personnel restrictions in final Revision 8 of NUREG-1021 in a manner that the staff believes will reduce the burden on facility licensees without substantially increasing the potential for conflicts of interest that could bias the examination process. Specifically, final Revision 8 will allow facility instructors to prepare the written examination questions without regard to the amount of time they spent training the license applicants (as is the case on requalification examinations), but it will still preclude instructors from preparing questions related to those topics on which they provided instruction (which remains somewhat more restrictive than the requalification criteria) and from instructing the applicants once they begin working on the licensing examination (as was the case in interim Revision 8 and on requalification examinations). As a result of this change, a facility licensee that has two instructors who equally shared the responsibility for training the license applicants, can use both instructors to prepare examination questions related to the topics they did not teach, whereas, they previously would both have been restricted from any examination development. Facility licensees may still propose for staff consideration, on a case-by-case basis, other approaches to address the NRC's concern regarding conflict-of-interest. The vulnerability discussion regarding independence and public perception (in Section III) provides additional justification for this change and an overview of the controls that the staff has implemented to address this issue.

With regard to facility management involvement in the examination development and review process, the staff has revised the guidance in final Revision 8 of NUREG-1021 so that it does not unnecessarily restrict management activities or authority.

Additionally, the NRC has decided to implement the revised examination process on a voluntary basis rather than require all facility licensees to prepare their licensing examinations. This change, which was prompted by the staff's conclusion that the proposed rule may have entailed a backfit that could not be justified in accordance with 10 CFR 50.109, will minimize the effect of the personnel restrictions by affording facility licensees with small training staffs the option of having the NRC prepare their licensing examinations. This change is also consistent the NEI's comments on the proposed rulemaking that the industry would prefer to continue the voluntary process because it would allow flexibility for a few facility licensees with small training staffs.

Comment: One facility licensee commented that the guidance on level of difficulty for both the written examination and the operating tests needs to be clarified so that licensees can better satisfy the NRC's expectations on their first try. According to the licensee, the current guidance relies too heavily on the subjective judgment of the chief examiner, which fosters inconsistency.

Response: The staff agrees that assessing the level of difficulty of test items does involve some level of subjectivity. Writers of facility examinations and NRC examiners who are trained in the subject matter, measurement principles, and psychometrics, and who have general knowledge of operator and trainee performance on similar test items, can make informed judgments on level of difficulty based on the guidance in NUREG-1021.

Interim Revision 8 of NUREG-1021 discusses the concepts of discrimination validity and examination difficulty and includes a number of criteria directed at limiting the amount of variation in the difficulty of both the written examinations and the operating tests. For example, NUREG-1021, Appendix D, "Simulator Testing Guidelines," discusses the quantitative and qualitative attributes of simulator scenarios, and Section ES-301, "Preparing Initial Operating Tests," of NUREG-1021 provides a specific target range for each of the quantitative attributes applicable to the initial licensing examination. Similarly, NUREG-1021, Appendix B, "Written Examination Guidelines," discusses the principles of level of knowledge and level of difficulty, and ES-401, "Preparing Initial Site-Specific Written Examinations," specifies an upper limit on the number of questions based on fundamental knowledge or memory. However, it has come to the staff's attention that ES-401 does not specify an upper limit on the number of comprehension-level or analysis-level questions and that this may have contributed to variations in the level of difficulty of some examinations. Therefore, the staff has revised the criteria in interim Revision 8 of NUREG-1021 to specify ranges for memory and higher cognitive-level questions.

A number of other controls have been added to help minimize variation in the level of examination difficulty. Since beginning the pilot program, the NRC staff has participated in a number of meetings and workshops designed to help facility licensees better understand the NRC's standards (as stated in NUREG-1021) and expectations with regard to level of difficulty; additional workshops will be conducted. In addition, an NRC supervisor is required to review and approve every examination before it is administered, regardless of whether the examination was prepared by the NRC or a facility licensee; facility-prepared examinations are reviewed with particular emphasis on the changes recommended by the chief examiner. Moreover, the Office of Nuclear Reactor Regulation (NRR) periodically conducts examination reviews, which include an assessment of their level of difficulty, to ensure consistent regional implementation of the NRC's standards and expectations.

Comment: An NRC license examiner recommended that any question taken directly from a bank of questions that is available to the applicants for study should be considered a memory-level question regardless of the cognitive level at which it is written.

Response: Although this is a valid concern, making such a change would likely increase the burden on facility licensees (because of the increased challenge of writing additional questions at the higher cognitive levels) and increase the level of difficulty of the examinations. Therefore, the staff has made no changes to interim Revision 8 of NUREG-1021 in response to this comment.

Comment: An NRC license examiner recommended that NUREG-1021 should contain a detailed list of minimum physical security precautions and that facility licensees should be required to list the steps they have taken to ensure examination security, including measures to control any contractors who prepare the examinations.

Response: NUREG-1021 is a guidance document intended to help NRC examiners and facility licensees gain a better understanding of the procedures for, and NRC management's expectations regarding, preparation of the initial operator licensing and requalification examinations. The staff has clarified the examination security expectations in final Revision 8 of NUREG-1021, including the control of licensee contractors. In a separate action, the staff issued an Information Notice (IN) to advise power reactor facility licensees of the NRC's perspective and expectations regarding the integrity of examinations developed by the facility licensees' employees and representatives.

Furthermore, based on several security incidents that occurred since beginning the pilot examination program, the staff has concluded that applicants, licensees (operators), and facility licensees may not understand what is meant by the term "compromise," as used in 10 CFR 55.49. Therefore, the staff has concluded that it would be beneficial to clarify 10 CFR 55.49 in the final rule so that facility licensees understand their responsibility for maintaining control over the examination process. Moreover, the staff has revised the final rulemaking to require facility licensees that elect to prepare their licensing examinations to establish, implement, and maintain procedures to control examination security and integrity.

Comment: Two facility licensees suggested that applicants should be allowed to perform some or all of the five required reactivity manipulations on the simulator rather than on the actual reactor.

Response: The requirement for five significant control manipulations is addressed in 10 CFR 55.31(a)(5), and such a change is beyond the scope of this rulemaking. Therefore, the staff will address this comment separately.

Comment: An NRC license examiner recommended deleting the provision in interim Revision 8 of NUREG-1021 that allows up to 30 percent of the job performance measures (JPMs) on the walk-through portion of an operating test to be repeated on walk-through tests given on a later day.

Response: The issue of repeating JPMS during successive walk-through operating tests was discussed in SECY-96-206, and the 30 percent limit was included in interim Revision 8 of NUREG-1021 because no limit had existed previously when the NRC was preparing the examinations. The staff has revised its position on this issue in favor of the approach recommended by the examiner because it will improve examination security and be consistent with the policy on repeating simulator scenarios. The staff believes that the impact of the change will be minimal.

Comment: One nuclear industry employee commented that requiring facility licensees to prepare and submit an outline of the written examination and operating test in advance of the actual examination is an unnecessary administrative burden. He suggested that the due date for the examination itself be advanced and the requirement for a separate outline be eliminated.

Response: Preparing the outline, particularly for the written portion of the examination, in accordance with the guidance NUREG-1021 ensures that the required knowledge and abilities are systematically sampled and evaluated on the licensing examination. Submitting the outline to the NRC for review and approval before preparing the examination itself, enables the NRC to comment and the facility licensee to make the required changes before significant licensee resources are expended in developing the final examination materials.

In response to this comment and to comments (regarding the submittal of examination materials) from NRC examiners involved with the pilot examinations, the staff is changing the guidance in interim Revision 8 of NUREG-1021 by advancing the due dates for the examination outline and the proposed examination in order to provide additional time for facility licensees to resolve any concerns the NRC has with those submittals.

Comment: An NRC license examiner recommended that NUREG-1021 should contain guidance to ensure that the written examination outlines are systematically developed. He argued that this will minimize the possibility that facility licensees will bias the selection of topics toward those that were emphasized during the applicants' license training program. (Another NRC examiner submitted a differing professional view (DPV) regarding the potential for bias in the selection of topics for the examination and recommended that the NRC should continue to prepare the examination outlines.)

Response: The staff agrees that the specific written examination sampling guidance in interim Revision 8 of NUREG-1021 does not ensure that the written examination is free of biases that could affect its validity. Therefore, the staff has revised final Revision 8 of NUREG-1021 to reduce the risk of bias by ensuring that facility licensees prepare the examination outlines in a systematic manner. (The panel that reviewed the DPV concluded that this was an acceptable resolution of the issue.)

Comment: NEI and a nuclear industry employee recommended that facility licensees be allowed to use their site-specific task lists instead of NUREG-1122, "Knowledge and Abilities [K/A] Catalog for Nuclear Power Plant Operators: Pressurized Water Reactors," or NUREG-1123, "Knowledge and Abilities Catalog for Nuclear Power Plant Operators: Boiling Water Reactors," to establish the content validity of the examinations.

Response: The staff believes that allowing substitution of the NRC's K/A catalogs (which were derived from an industry job-task analysis and evaluated by a panel of subject matter experts representing the nuclear industry and the NRC) would be inappropriate and inconsistent with the regulations. Although the Part 55 rulemaking of 1987 generally endorsed the "systems approach to training," Sections 55.41(a), 55.43(a), and 55.45(a) all state that the knowledge, skills, and abilities selected for evaluation on a written examination and an operating test will be identified, in part (emphasis added), from learning objectives derived from a systematic analysis of licensed RO and SRO duties performed by each facility licensee. While NUREG-1262, "Answers to Questions at Public Meetings Regarding Implementation of Title 10, Code of Federal Regulations, Part 55 on Operators' Licenses," confirmed the NRC's intent that the training program's learning objectives would become the major source of the licensing examination, it also cautioned that the NRC would not be limited to those learning objectives. Moreover, since 1986, NUREG-1021 has encouraged examiners to use the facility licensees' learning objectives as a source of topics for the written examination, and it has always required examiners to reference the facility licensees' training materials when developing a licensing examination.

The systematic sampling procedures for preparing the written and walk-through examination outlines per interim Revision 8 of NUREG-1021 are designed around the structure of the NRC's K/A catalogs and may not be compatible with the facility-specific task lists. As discussed in SECY-96-206, interim Revision 8 of NUREG-1021 contains provisions for facility licensees to add, substitute, or delete specific knowledge and ability requirements on a case-by-case basis. Allowing facility licensees to substitute their entire site-specific task lists for the NRC's K/A catalogs could decrease the level of examination consistency, increase the potential for biasing the sample of material covered on the licensing examinations, and increase the level of effort for NRC examiners to review the proposed outlines to ensure adequate sampling per Part 55. The staff has concluded that the current approach of requiring facility licensees to explain deviations from the NRC's K/A catalogs is conservative, consistent, and efficient. Therefore, no changes were made to Revision 8 of NUREG-1021 in response to this comment.

Comment: NEI recommended that the time limit for taking the written examination should be increased from 4 to 5 hours because of the increased percentage of higher cognitive-level questions on the examinations.

Response: Feedback regarding the pilot examinations has confirmed that a number of applicants could not complete the examination in the 4 hours currently specified in NUREG-1021. The staff believes that increasing the time limit to 5 hours may be necessary because of the greater focus on improving the plausibility of the test question distractors (i.e., the wrong-answer choices) and the required range of higher cognitive-level questions.

The staff believes that the time limit can be increased to 5 hours without affecting the discrimination validity of the examinations. The nature of the NRC licensing examination is such that allowing sufficient time to demonstrate knowledge is of primary concern.

Comment: One facility licensee and a facility employee suggested that the simulator operating test administration procedure in NUREG-1021 should allow other operating assistants, such as a shift technical advisor (STA) and shift manager, to help the applicants during critical evolutions on the simulator when required by the plant's technical specifications and procedures.

Response: Because the operating tests are administered one-on-one (i.e., with one NRC examiner assigned to each applicant on the crew), the number of applicants on the crew can normally not exceed the number of NRC examiners dispatched to the site. Interim Revision 8 of NUREG-1021 already makes allowances for the operating crews to be supplemented with additional surrogate personnel up to the minimum RO staffing level required by the facility licensee's technical specifications, but it does not address the use of STAs or additional shift managers (i.e., senior operators). Therefore, in final Revision 8 of NUREG-1021, the staff will allow the use of STAs in accordance with the facility licensee's normal staffing practice for that position. However, the staff does not intend to permit more than one person to fill a senior operator position during the simulator test because the principal duties of the shift manager position (i.e., assuming the role of the emergency director, performing emergency classifications, and making protective action recommendations) are normally a part of the operating test for senior operator applicants.

Comment: An NRC license examiner suggested that the requalification examination procedures in NUREG-1021 should include limits on test item duplication between successive examinations.

Response: The staff's expectations regarding this issue were previously communicated to the NRC regional offices, and the staff agrees that the same information should be incorporated in final Revision 8 of NUREG-1021. Specifically, final Revision 8 will limit test item duplication between successive NRC requalification examinations to no more than 50 percent.

Comments From NRC Staff

In addition to the public comments discussed above, the staff considered a number of comments, recommendations, and questions from NRC examiners, supervisors, and managers involved in the operator licensing program. Although most of the issues related specifically to the pilot process, some applied to other aspects of the operator licensing program. A number of the issues were previously addressed in response to the public comments; the remaining significant comments and responses are summarized below. Several additional minor changes are summarized, as appropriate, in the Executive Summary section of final Revision 8 of NUREG-1021.

Comment: A senior manager suggested that NUREG-1021 should provide facility management with the opportunity to appeal to NRC management if the facility has significant concerns regarding the staff's application of the examination criteria in NUREG-1021.

Response: The staff agrees that such a provision is appropriate and consistent with the NRC's policy of minimizing regulatory impact when appropriate. ES-201, "Initial Operator Licensing Examination Process," of final Revision 8 of NUREG-1021 has been edited to incorporate such a practice.

Comment: A senior manager suggested that the staff should reconsider the need to review every question on written examinations prepared by facility licensees. Those questions that have been validated by successful use on a previous licensing examination at that facility should require no additional review by the NRC.

Response: The staff agrees that the benefit derived by conducting a detailed review of previously validated written examination questions does not justify the additional cost of conducting the review. Therefore, the staff has changed the guidance in final Revision 8 of NUREG-1021 to require only a general reading of written questions that were successfully used on an NRC licensing examination conducted since the beginning of FY 1996. In an effort to improve efficiency, the staff has also revised the written examination review process to allow the regions to return an examination to the facility licensee for revision depending on the outcome of a detailed review of a sample of questions.

Comment: A regional supervisor suggested that ES-301, "Preparing Initial Operating Tests," of NUREG-1021 should better define the NRC's expectations regarding the permissible number of closed-reference questions on the walk-through portion of the operating test.

Response: The staff agrees that the guidance in interim Revision 8 of NUREG-1021 lacks specificity and may contribute to variations in the quality and level of difficulty of the operating tests. Final Revision 8 of NUREG-1021 has been clarified to address the issue.

SECTION III
VULNERABILITY REVIEW

At the Commission's direction, SECY-96-206, the rulemaking plan for amendments to the examination requirements in 10 CFR Part 55, discussed the pros, cons, and vulnerabilities of proceeding with the revised examination process on an industrywide basis. The SRM that later approved the rulemaking plan directed the staff to monitor the pilot program results and inform the Commission if the staff gained any insights that suggest that the revised examination process should not be pursued. In response to this directive, the staff has reviewed the vulnerabilities (i.e., quality and consistency, independence and public perception, examination security, NRC resources, program stability, and examiner proficiency) that were discussed in the rulemaking plan in light of the staff's experience with the pilot process since SECY-96-206 was issued on September 25, 1996. The staff has also reviewed the potential effects on reactor safety.

For ease of review, the staff has categorized and addressed the public comments on the proposed rulemaking with respect to the same vulnerabilities. Appropriate cross-references to the FRN for the final rule are provided to minimize the need for repetition.

Quality and Consistency of the Examinations

The staff's expectations regarding the quality and level of difficulty of the proposed examinations have not yet been achieved. As discussed in Section I above, the staff has considered a number of possible reasons for the problems that have been observed, but no definitive explanations have been identified. Public comments related to this vulnerability are addressed in the associated FRN (Attachment 2).

To maintain acceptable uniformity of examination content, format, level of difficulty, and quality, the staff expects facility licensees that elect to prepare the written examinations and operating tests will do so in accordance with the procedures and guidelines in NUREG-1021. A number of administrative criteria and controls have been incorporated in Revision 8 of NUREG-1021 to implement the new examination development process. Those revisions include: limits on the number of test items (e.g., written questions and JPMs) that can be taken directly from the facility licensees' examination banks, the number of test items that can be developed by modifying existing bank items, and the number of newly-developed test items; limits on the repetition of test items from previous quizzes and examinations given at the facility; guidelines regarding the cognitive level of written examination questions; and improved quality assurance checklists.

The significant changes required on many of the proposed examinations suggests that the guidance in interim Revision 8 of NUREG-1021 may not have been understood. However, as noted in Section I above, the staff believes that the guidance in interim Revision 8 of NUREG-1021 was sufficiently clear to enable facility licensees to prepare examinations of acceptable quality and consistency, and, therefore, has made only minor revisions to NUREG-1021 since issuing SECY-96-206. The staff has conducted and participated in a number of public meetings and workshops in an effort to communicate its expectations to the facility employees who will prepare the examinations and to obtain feedback regarding the guidance in NUREG-1021. Additional NRC and industry workshops are being planned to address continued concerns regarding the quality of the submitted examinations. Moreover, as noted in Section I, the staff has asked the industry to address the issue of examination quality.

The revised examination process is dependent upon the ability and judgment of the NRC chief examiners to identify weak test items and to communicate the problems to the facility representatives so that the required corrections are understood and implemented. Although a number of the pilot examinations submitted by facility licensees have not met NRC standards for quality, level of knowledge, or level of difficulty, the NRC chief examiners have directed the facility licensees to revise the examinations so that they would discriminate at the appropriate level. The staff acknowledges that some of the facility-prepared examinations will be more difficult than others (yet within an acceptable range of variation) and that some of the variation can likely be traced to individual differences among the NRC examiners who review and approve the examinations. That is also the case when the NRC (or its contractors) prepares the examinations. However, the staff believes that the variation in the level of difficulty can be maintained within acceptable limits and would be even greater if NRC examiners did not take action to ensure that the submitted examinations are consistent with NRC standards.

To ensure consistent implementation of the guidance in NUREG-1021, an NRC supervisor is required to review and approve every examination before it is administered. Supervisors are also required to periodically evaluate every certified examiner during the conduct of an operating test to ensure that the on-site examination activities are being performed as intended. Furthermore, the NRR staff periodically reviews the licensing examinations given at selected facilities and the implementation of the operator licensing function in each region to verify the quality of the examination process and to ensure the consistent implementation of the NRC's standards and expectations. The NRR reviews include an assessment of the level of knowledge and level of difficulty of the written examinations and operating tests; each review is documented in a report to the responsible regional Division Director.

NRR has instructed regional managers and examiners not to approve or administer any examination that does not adhere to current NRC standards for content, format, quality, level of knowledge, and level of difficulty. If a facility licensee does not prepare appropriate licensing examinations, the NRC, which has the responsibility to ensure that public health and safety are maintained, will develop the examinations.

Independence and Public Perception

The staff implemented a number of measures during the pilot examination process to ensure that an adequate level of independence would be maintained and to minimize the potential for bias during examination development. GL 95-06, which outlined the pilot examination process in August 1995, restricted all facility employees who played a "substantial" role in training the license applicants from participating in the examination development. Questions regarding the meaning of "substantial" prompted the staff to clarify the personnel restrictions in interim Revision 8 of NUREG-1021, which incorporated lessons learned during the first 22 pilot examinations and was issued for use in February 1997. Interim Revision 8 prohibited facility instructors from participating in the examination development process if they had spent more than a specified amount of time training the applicants who would be taking the NRC examination and prohibited any instructor from preparing written examination questions related to those topics in which the instructor had provided training. The personnel restrictions in the GL and interim Revision 8 supplemented a related provision, which, for purposes of maintaining examination security, barred anyone who had knowledge of the contents of the NRC examination from further training-related activities involving the license applicants. The latter provision had been adopted from the requalification examination program but did not address the separate but equally important perception that the people who were responsible for training the applicants, if allowed to prepare the examination, could bias its content or level of difficulty.

The nuclear power industry has argued that the applicants' instructors should be permitted to prepare the licensing examinations, just as the instructors who perform the requalification training are permitted to prepare the requalification examinations required by 10 CFR 55.59(a). As noted in the public comments summarized in Section II above and in the associated FRN (Attachment 2), the industry has indicated that the additional personnel restrictions implemented during the pilot examination process have unnecessarily increased the facility licensees' administrative burden in preparing the examinations and forced facility licensees to divert their best-qualified instructors from their primary duty of training the license applicants or to use less-qualified instructors to prepare the licensing examination, with a possible decrease in quality.

The staff has evaluated the industry's concerns in light of the other public comments related to this vulnerability and the other changes that are being made in final Revision 8 of NUREG-1021. Therefore, as noted in Section II above in response to the public comments, the staff has removed the restriction that prohibits instructors from preparing any written examination questions based solely on the amount of time they spent training the license applicants. This change should provide licensees with increased flexibility in managing their resources and possibly reduce their costs without adversely affecting the independence of the examination process. For example, if a facility licensee has two instructors who equally shared the responsibility for training the license applicants, they will now be permitted to prepare the examination questions related to the topics they did not teach, whereas, they previously would both have been restricted from any examination development. As discussed in Section II, final Revision 8 will also include explicit guidance for systematically sampling the important, safety-related K/As from NUREG-1122 or NUREG-1123 when preparing the written examination outline. This change should reduce the chances that the written examination might be biased in favor of those topics that were emphasized during the training program and away from other important topics that received less attention. The operating test outline will continue to be independently developed by an individual who was not involved in the applicants' license training. These measures should provide sufficient assurance that the examinations are prepared in an impartial manner.

The staff maintains, for the reasons that follow, that the remaining personnel restrictions are needed to limit the potential for bias in the examination development process.

• The personnel restrictions were an important component of the pilot examination process. The conclusions of the acceptability of the pilot process, such as the comparability of examination passing rates and test scores, are linked to the independence of the test developer. To eliminate controls on who can write examination material at this point in the transition process could undermine the staff's conclusions regarding the effectiveness of the pilot process.

• The personnel restrictions maintain greater consistency with other professional licensing and certification examinations (e.g., medical, legal, aircraft, and engineering), which are prepared by an independent body.

• The initial licensing examinations are considered "NRC" examinations upon approval, even if they were prepared by the facility licensee, whereas, the requalification examinations required by 10 CFR 55.59(a) are classified as "facility" examinations. Unlike the requalification examinations, the NRC's initial licensing examinations are not a part of the facility licensee's training program. The initial examination is the cornerstone of the NRC's licensing decision.

• On June 18, 1996, the staff briefed the Commission on the results of the pilot program, noted its concerns regarding the potential for facility employees to bias the examinations, and discussed the personnel restrictions that the staff had implemented to mitigate those concerns. The Commission acknowledged the vulnerability in this area, including the concern that instructors could teach to the test.

• Eliminating the personnel restrictions could affect the public perception regarding the revised process and indicate that the NRC is not concerned about the potential for examination bias.

With regard to the issue of public perception, The Washington Post reported, shortly after the staff disclosed its intent to revise the examination process in March 1995, that consumer advocates were concerned that allowing facility licensees to prepare the examinations might endanger reactor safety because the examinations might be less rigorous than those prepared by the NRC. However, in the three years since the original proposal, the NRC has afforded the public a number of opportunities to comment on the revised examination process, but none have been received. For example, Generic Letter 95-06 and its supplement announced and later summarized the results of the pilot examination program; comments were invited, but none were received. When a Federal Register notice (61 FR 6869) dated February 22, 1996, announced the availability of draft Revision 8 of NUREG-1021 and solicited comments, only NEI and two facility licensees responded. And, when the proposed rule was published in the Federal Register (62 FR 42426) on August 7, 1997, the 13 respondents included NEI, five nuclear utilities and one employee, four NRC or contract examiners, one non-power reactor facility licensee, and the State of Illinois, but no consumer advocates or members of the general public.

Concerns regarding the potential for conflict of interest have also prompted the NRC to review the clarity of 10 CFR 55.49. The regulation encompasses not only activities like cheating and lapses in security but also activities that compromise the integrity or validity of the examination (e.g., noncompliance with the criteria designed to limit the potential for bias in the selection of topics to be evaluated on the written examination). Therefore, the NRC has concluded that it would be beneficial to amend 10 CFR 55.49 in the final rule to ensure that applicants, licensees, and facility licensees, understand the scope of the regulation. If the NRC determines that a facility licensee has biased the scope, content, or level of difficulty of an examination to enhance the chances that its applicants would pass the examination, the NRC will utilize its enforcement authority including, as warranted, civil penalties, orders against the individuals involved, and, charging the individuals involved with deliberate misconduct pursuant to 10 CFR 50.5.

In summary, the pressures to conduct the examinations on schedule and the reduced involvement of the NRC in the examination development process could lead to examinations that reflect the biases of the facility licensee and that discriminate at a lower level than is currently the norm. However, a number of process requirements are in place to reduce the biases: the revised examination process includes restrictions on personnel, the systematic selection of topics to be tested on the written examination, independent NRC review, NRC-directed revisions, NRC approval of the examinations, and independent NRC administration of the operating tests that should provide sufficient safeguards to reduce the possibility of biased licensing decisions. The pilot examination results, to date, suggest that these measures, in combination with the NRC examiners' increased focus on the psychometric quality of the written questions, will be effective at maintaining the discrimination validity of the facility-prepared licensing examinations.

A summary of the public comments related to this vulnerability and the staff's responses are included in the associated FRN (Attachment 2).

Examination Security

As noted in SECY-96-206, the staff expects that the majority of facility licensees will continue to maintain proper examination security, regardless of whether they elect to prepare their own examinations or to utilize examinations prepared by the NRC. Nevertheless, the revised examination development process may provide greater opportunities for examination compromise (as that term is clarified in an amendment to 10 CFR 55.49) than the traditional process in which fewer facility employees had knowledge of and access to the examination for a shorter period of time. Since beginning the pilot examination program, there have been a number of instances in which facility licensees have: (1) failed to exercise positive control of the examinations by not locking the room or container in which the examinations were being prepared or stored; (2) failed to control the activities of personnel involved in preparing or reviewing the examinations, as required by the signed security agreements, or to restrict unauthorized personnel from gaining access to the examinations; and (3) lost control of the examinations while printing or making copies.

NRC examiners are required by Revision 8 of NUREG-1021 to be attentive to examination security measures, to review the NRC's security expectations with the facility licensee at the time the examination arrangements are confirmed, and to report any security concerns to NRC management. If the NRC determines during its preparation that the integity of an examination may have been compromised, it will not administer the examination until the scope of the potential compromise is determined and measures can be taken to address the integrity and validity of the examination. If the compromise is discovered after the examination has been administered, the NRC will not complete the licensing action for the affected applicants until the staff can make a determination regarding the validity of the examination. If the compromise is not discovered until after the licensing action is complete, the NRC will reevaluate the licensing decision pursuant to 10 CFR 55.61(b)(2) if it determines that the original licensing decision was based on an invalid examination. As a separate measure, the NRC will take enforcement action, when appropriate, and consult with the Office of Investigations in cases involving the possibility of personal wrongdoing.

The staff has initiated a number of actions since the issuance of SECY-96-206 to increase the NRC examiners' and the facility licensees' sensitivity to examination security issues and to mitigate the vulnerability in this area. Those actions include: (1) the development of a revision to NUREG-1600, Revision 1, "General Statement of Policy and Procedure for NRC Enforcement Actions," to address enforcement action against the license applicants, Part 55 licensees, and Part 50 facility licensees that are subject to the requirements in 10 CFR 55.49; (2) a change in the final rule, based on comments on the proposed rule, to clarify 10 CFR 55.49 and to require facility licensees that elect to prepare their own licensing examinations to establish, implement, and maintain procedures to control examination security and integrity; (3) further clarification and elaboration on the guidance in NUREG-1021 about examination security; and (4) the issuance of an IN to advise power reactor facility licensees of the NRC's perspective and expectations regarding the security of examinations developed by the facility licensees' employees and representatives.

NRC Resources

When the staff developed the pilot examination process, it estimated that it would take approximately 370 examiner-hours to review and otherwise prepare for, administer, grade, and document an average facility-prepared operator licensing examination. Although a number of the proposed examinations required more NRC resources than expected to achieve NRC's standards, the burden was generally offset by other examinations that required less effort to review and revise.

The increased efficiency of the revised examination process has enabled the NRC to eliminate the use of contractors in the operator licensing program and conduct the initial operator licensing and requalification inspection programs with the existing NRC staff. Before initiating the pilot examination and transition program at the beginning of FY 1996, the NRC spent approximately $3 million per year for contractor assistance to prepare and administer initial examinations and assist with requalification inspections. During FY 1996, when approximately 40 percent (27/68) of the examinations were voluntarily prepared by facility licensees, the NRC spent approximately $1.6 million for contractor assistance in those areas. In FY 1997, facility participation increased to include approximately 75 percent (41/55) of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million (approximately 3 full-time equivalents (FTEs)). During FY 1997, the NRC expended approximately 16.8 staff FTEs to conduct all the site-specific initial operator licensing examinations, plus 3.8 staff FTEs for generic operator licensing activities. The licensed operator requalification inspection program required approximately 4.5 additional staff FTEs (plus a minimal level of contractor support), bringing the total resources used for the operator licensing program in FY 1997 to approximately 28.1 FTEs. This actual resource burden is generally consistent with the estimates prepared before beginning the pilot examination program and supports the staff's conclusion that the revised examination process is an efficient alternative to the traditional NRC-prepared examinations.

Experience during the pilot program has shown that the quality of the facility-prepared examinations can vary widely, making it difficult to predict the amount of time necessary for the NRC to review and revise any particular examination in order to meet NRC standards. Although the staff had expected the quality of the examinations to improve over time, some of the proposed examinations continue to require more changes than anticipated. The added examiner workload could increase the possibility of lower examination quality, raise the NRC's cost, and affect the staff's ability to satisfy the facility licensees' needs for licensing examinations. As noted in SECY-97-079, NRR has issued a memorandum to the regional administrators emphasizing the importance of: (1) assigning adequate resources to carry out the operator licensing task; (2) completing a thorough review of every facility-prepared examination; and (3) not administering any examination that does not meet NRC standards for quality and level of difficulty. Furthermore, as noted in Section I above, the staff has asked the industry to address the issue of examination quality, which should reduce the need for changes and the cost of preparing the examinations.

As noted in the discussion of alternative approaches in the proposed rulemaking, implementing the revised examination process on an elective basis will complicate the task of resource management because the number of licensees that elect to prepare their own examinations could fluctuate substantially from year to year. For purposes of comparison, the average NRC-prepared examination requires approximately two-thirds of an FTE to prepare, administer, and document, while the average facility-prepared examination requires about one-third of an FTE to review, administer, and document. Therefore, with the 17 regional FTEs that are budgeted for the initial examination program in FYs 1999 and 2000, the NRC staff will be able to prepare about 25 to 30 examinations per year, while it would be able to review about 50 to 55 facility-prepared examinations. During the two full years in which facility licensees have been allowed to prepare their examinations under the voluntary pilot program, the level of industry participation has ranged from about 75 percent (41/55) in FY 1997 to about 90 percent (50/56) in FY 1998. Therefore, if the current level of industry participation is maintained, the staff expects that it will be able to satisfy most facility licensees' examination demands and scheduling needs.

To manage the workload, the NRC staff will resume issuing an annual administrative letter that will request facility licensees to report their anticipated examination needs for the next four fiscal years and their intentions with regard to preparing each examination. Moreover, the staff intends to work with NEI in an effort to coordinate the examination schedule and address workload conflicts.

The resource burden on facility licensees is addressed in the associated FRN (Attachment 2) and regulatory analysis (Attachment 3).

Program Stability

As noted in SECY-96-206, when it started the pilot program, the staff intended that the format, content, and level of difficulty of the facility-prepared examinations would remain consistent with previous NRC-prepared examinations so that the transition from NRC-prepared to facility-prepared examinations would be imperceptible to the license applicants. Although the staff has made a number of changes in the examination criteria and guidelines in Revision 8 of NUREG-1021, it has made every effort to maintain consistency. Of the three parameters, level of difficulty is the most subjective and most complex to define and control. However, based on the overall examination results (i.e., the slight decline in the passing percentage and average written examination grades discussed in Section I above) since starting the transition process, it appears that the level of difficulty of the facility-prepared examinations, revised as directed by the NRC, is consistent with those prepared by the NRC. Considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training program quality and screening of applicants by facility licensees) that could be responsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant.

Examiner Proficiency

As noted in the responses to the public comments summarized in the FRN (Attachment 2), the staff believes that by participating in the revised examination process, NRC examiners will maintain their proficiency. An NRC examiner will review and approve every facility-prepared examination before it is administered to ensure that it conforms to the guidelines in NUREG-1021 regarding content, format, quality, and levels of knowledge and difficulty. NRC examiners will also continue to independently administer and grade both the dynamic simulator and the plant walk-through portions of the operating tests. Because the use of contractors has been discontinued and NRC examiners will be administering all of the operating tests, the revised process will enable the examiners to accrue more experience in a shorter period of time and to maintain their proficiency. The smaller pool of all NRC examiners may also improve the consistency of the operating test evaluations and the licensing decisions.

The Commission directed the staff in the SRM of December 17, 1996, to ensure that examiners maintain proficiency by writing at least one initial operator licensing examination per calendar year in each NRC region. Because the final rulemaking will allow power reactor facility licensees to have their examinations prepared by the NRC, the staff does not anticipate any difficulty meeting this directive.

New NRC license examiners will still be required to complete a standardized training program, including the development of a complete written examination and operating test, as part of their certification process. Once certified, examiners are required to maintain their qualification by periodically conducting examinations and attending refresher training. NRR has revised the refresher training syllabus to focus more attention on reviewing, critiquing, and approving written examination questions. Furthermore, as directed by the SRM of December 17, 1996, the staff intends to conduct examiner training conferences at intervals not to exceed 24 months; the last conference was convened in October 1997.

Routine management oversight activities performed by the regions and NRR should provide additional assurance that examiner proficiency is being maintained. Regional supervisors are required to review and approve the examination changes recommended by their chief examiners and they periodically evaluate every certified examiner during the conduct of an operating test. Moreover, the NRR staff periodically reviews the licensing examinations given at selected facilities and the implementation of the operator licensing function in each region to verify the quality of the examination process and to ensure the consistent implementation of the NRC's standards and expectations.

Reactor Safety

As noted above and in the responses to the public comments summarized in the FRN (Attachment 2), the staff believes that the revised examination process will improve the staff's ability to focus on operator performance because NRC examiners will be administering all of the operating tests. The staff acknowledges that the total number of facility operating procedures reviewed in the process of examination development may decrease under the revised method. However, NRC examiners are still expected to review and identify discrepancies in the procedures that will be exercised during the walk-through JPMs and the simulator scenarios. Because only staff examiners will be administering the operating tests and reviewing all of the applicable procedures, the revised process will enable the examiners to accrue more experience in a shorter period of time.

The staff also acknowledges that the revised examination process will place an extra burden on the training staffs of those facility licensees that elect to prepare their own examinations and may divert resources from their primary training activities so that they can prepare the licensing examinations. The NRC's decision to continue the revised examination process on an optional basis will afford those facility licensees with small training staffs greater flexibility in managing their resources. If a facility licensee that elects to prepare its own examinations places insufficient resources on either training or testing, the quality of its proposed licensing examinations or the passing rate on those examinations would most likely suffer. Although many of the facility-prepared examinations have required numerous revisions to achieve NRC quality standards, the examination results, to date, are consistent with the results on previous NRC-prepared examinations, suggesting that the quality of the facility licensees' training programs has not been affected.

The NRC will continue to set the standard for operator performance by ensuring that facility licensees that elect to prepare their own examinations maintain appropriate standards for examination quality and level of difficulty. The staff will remain actively involved in the operator licensing process by reviewing and approving every written examination and operating test before it is administered. The NRC will not approve and administer unacceptable examinations.

The overall results of the facility-prepared examinations support the staff's conclusion that the licensing decisions that have been made on the basis of those examinations are as valid and conservative as those made using the traditional examination process. The fact that facility licensees will have the option of preparing the examinations with NRC review and approval, should have no negative effect on the safe operation of the plants.

ATTACHMENT 2

[7590-01-P]

NUCLEAR REGULATORY COMMISSION
10 CFR Part 55
RIN 3150-AF62

Initial Licensed Operator Examination Requirements

AGENCY: Nuclear Regulatory Commission.

ACTION: Final rule.

SUMMARY: The Nuclear Regulatory Commission (NRC) is amending its regulations to allow nuclear power facility licensees to prepare, proctor, and grade the required written examinations and to prepare the required operating tests that the NRC uses to evaluate the competence of individuals applying for operator licenses at those plants. The amendment requires facility licensees that elect to prepare the examinations to prepare the examinations in accordance with NRC operator licensing examination standards for power reactors; establish, implement, and maintain procedures to control examination security and integrity; submit, upon approval by an authorized representative of the facility licensee, each examination and test to the NRC for review and approval; and proctor and grade the written examinations upon NRC approval. In making this final rule change, the NRC will continue to administer (i.e., manage and oversee) the initial operator licensing examination process by: (1) developing the generic fundamentals examinations (which are also proctored by facility licensees); (2) reviewing and approving the facility-developed, site-specific written examinations and operating tests; and (3) independently conducting and grading both the dynamic simulator and walk-through portions of the operating test, which is considered the most performance-based aspect of the licensing process and permits the NRC to evaluate the operator and senior operator applicants' competence under normal and abnormal plant conditions. The amendment preserves the NRC's authority to prepare the examinations and tests in lieu of licensees and to exercise its discretion and reject a power reactor facility licensee's determination to prepare, proctor, and grade the written examinations and prepare the operating tests. The Commission is concerned with examination integrity; therefore, the amendment will also revise the regulations to ensure that applicants, licensees, and facility licensees understand the scope of the regulation.

EFFECTIVE DATE: This final rule is effective on (insert the date 180 days from date of publication in the Federal Register).

FOR FURTHER INFORMATION CONTACT: Siegfried Guenther, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, telephone: 301-415-1056; e-mail: sxg@nrc.gov.

SUPPLEMENTARY INFORMATION:

• Background
• Discussion
• Summary of Public Comments
• Other Comments
• Availability of Guidance Document for Preparing Operator Licensing Examinations
• Final Rule
• Environmental Impact: Categorical Exclusion
• Paperwork Reduction Act Statement
• Public Protection Notification
• Regulatory Analysis
• Regulatory Flexibility Certification
• Backfit Analysis
• Enforcement Policy
• List of Subjects in 10 CFR Part 55
• PART 55 - OPERATORS' LICENSES
• �55.8 Information collection requirements; OMB approval.
• �55.40 Implementation.
• �55.49 Integrity of examinations and tests.

Background

Section 107 of the Atomic Energy Act (AEA) of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for an operator's license, to prescribe uniform conditions for licensing these individuals, and to issue licenses as appropriate. Pursuant to the AEA, 10 CFR Part 55 requires an applicant for an operator license to pass an examination that satisfies the basic content requirements specified in the regulation. The licensing examination consists of the following parts: (1) a written generic fundamentals examination (covering reactor theory, thermodynamics, and components) that license applicants have to pass as a prerequisite for taking the site-specific examination; (2) a site-specific written examination covering plant systems, emergency and abnormal plant procedures, and plant-wide generic knowledge and abilities; and (3) a site-specific operating test consisting of three categories, including a crew-based, dynamic simulator performance demonstration, an individual, task-based walk-through covering control room and in-plant systems, and various plant administrative requirements. Although neither the AEA nor Part 55 specifies who must prepare, proctor, or grade these examinations, the NRC has traditionally performed those tasks itself or through its contract examiners. The NRC and its contract examiners have used the guidance in NUREG-1021, "Operator Licensing Examination Standards for Power Reactors," once titled "Operator Licensing Examiner Standards," to prepare the initial operator licensing examinations. This document has been revised as experience has been acquired in preparing the examinations. The current version is designated Revision 8.⁽¹⁾

In accordance with 10 CFR 170.12 (i), the NRC's staff and contractual costs are recovered from facility licensees that receive examination services. In Fiscal Year (FY) 1995, the NRC spent approximately $3 million on contractor support for the preparation and administration of the initial operator licensing examinations and for support of requalification program inspections. On March 24, 1995, in SECY-95-075, "Proposed Changes to the NRC Operator Licensing Program," the staff advised the Commission of its intent to eliminate the use of contractors by allowing facility licensees to prepare the examinations. The NRC staff's proposal was motivated by the general improvement in the performance level of power reactor facility licensees' training programs, the NRC's continuing efforts to streamline the functions of the Federal government, and the need to accommodate anticipated resource reductions.

On April 18, 1995, the Commission approved the NRC staff's proposal to initiate a transition process to revise the operator licensing program and directed the NRC staff to consider carefully the experience from pilot examinations before fully implementing the changes. On August 15, 1995, the NRC issued Generic Letter (GL) 95-06, "Changes in the Operator Licensing Program,"¹ outlining the revised examination development process and soliciting volunteers to participate in pilot examinations to evaluate and refine the methodology.

Between October 1, 1995, and April 5, 1996, the NRC reviewed and approved 22 operator licensing examinations, including both the written examinations and the operating tests, prepared by facility licensees as part of a pilot program. These examinations were prepared using the guidance in Revision 7 (Supplement 1) of NUREG-1021¹ and the additional guidance in GL 95-06.

The results of the pilot examinations were discussed in SECY-96-123, "Proposed Changes to the NRC Operator Licensing Program," dated June 10, 1996. Based on the results of the pilot program, the NRC staff recommended that the Commission approve the implementation of the new examination process on a voluntary basis until rulemaking could be completed to require all power reactor facility licensees to prepare the entire initial operator licensing examination and to proctor and grade the written portion of the examination. On July 23, 1996, the Commission authorized the staff to continue the pilot examination process on a voluntary basis and directed the staff to develop a rulemaking plan to justify the changes that would be necessary to 10 CFR Part 55. The Commission also directed the staff to address a number of additional items (e.g., pros, cons, and vulnerabilities) regarding the revised examination process to facilitate a Commission decision on whether to implement the revised process on an industrywide basis.

With Commission approval, the NRC staff resumed conducting pilot-style examinations on August 19, 1996, and by the end of June 1998 had reviewed, approved, and administered 80 additional examinations that were developed by facility licensees. This raised the total number of examinations completed using the pilot process to 102.

On September 25, 1996, the NRC staff forwarded the rulemaking plan and a response to the additional items to the Commission in SECY-96-206, "Rulemaking Plan for Amendments to 10 CFR Part 55 to Change Licensed Operator Examination Requirements." SECY-96-206 identified a number of areas (i.e., quality and consistency, independence and public perception, examination security, NRC resources, program stability, and examiner proficiency) in which the NRC could be more vulnerable under the revised examination process and described the measures that the NRC has taken to manage the vulnerabilities. On December 17, 1996, the Commission directed the staff to proceed with the proposed rulemaking. The NRC staff forwarded the proposed rule (SECY-97-079, "Proposed Rule - Initial Licensed Operator Examination Requirements") to the Commission on April 8, 1997, and on June 26, 1997, the Commission approved publication of the proposed rule for a 75-day comment period. The proposed rule was published in the Federal Register (62 FR 42426) on August 7, 1997.

After the public comment period expired on October 21, 1997, 11 comment letters were received. Two additional comment letters arrived after the expiration date but were also considered in the development of the final rule.

As written, the proposed rule would have required all power reactor facility licensees to prepare their operator licensing examinations and to proctor and grade the written portion of those examinations. Although the proposed rule would have imposed new requirements on facility licensees, the NRC took the position that the backfit rule, 10 CFR 50.109, did not apply because the shift in responsibility for preparing the examinations would not: (1) constitute a "modification of the procedures required to operate a facility" within the scope of the backfit rule; (2) affect the basic procedures for qualifying licensed operators; or (3) require facility licensees to alter their organizational structures. However, based upon further review after issuing the proposed rule, the NRC has concluded that there is insufficient basis to support the original position. Therefore, the NRC has decided to revise the final rule so power reactor facility licensees may elect to prepare their written examinations and operating tests (and proctor and grade the written examinations) in accordance with NUREG-1021, or to have the NRC prepare the examinations, thereby making a backfit analysis unnecessary.

Discussion

The pilot examinations demonstrated that the revised process, under which facility licensees prepare the written examinations and operating tests, is generally effective and efficient. From the time the pilot program began in October 1995 through the end of June 1998, the NRC staff reviewed, approved, and administered a total of 102 examinations that were voluntarily developed by facility licensees under the pilot examination and transition program.

Facility licensees prepared the written examinations and the operating tests, proctored the written examinations, and graded the written examinations using the guidance provided by the NRC in GL 95-06 during the early stages of the pilot program, and subsequently in interim Revision 8 of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors." NRC examiners thoroughly reviewed the examinations and tests to determine if they were consistent with NRC standards, directed facility licensees to make whatever changes were necessary to achieve NRC standards if the submitted examinations and tests were deficient, and approved the examinations and tests before they were administered. NRC examiners independently administered all of the operating tests, reviewed the written examination grading, and made the final licensing recommendations for approval by NRC management.

Comments from the NRC chief examiners who evaluated the pilot examinations indicate that the quality and level of difficulty of the licensee-prepared examinations (when modified as directed by the NRC) were generally comparable to the examinations prepared by the NRC (i.e., by the staff or NRC contractors). The passing rate on the 102 pilot-style examinations administered through the end of June 1998 was only slightly lower than the passing rate on the power reactor licensing examinations administered during FY 1995, the last year in which all examinations were prepared by the NRC. However, considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training program quality and screening of applicants by facility licensees) that could be responsible for some or all of the observed difference, the Commission has concluded that the observed change in the passing rates is not significant. The average grades on the facility-prepared, NRC-approved written examinations were also comparable if slightly lower than the grades on examinations prepared by the NRC during FY 1995. These data support the conclusion that the facility-prepared examinations are discriminating at a conservative and acceptable level and that the revised examination process is effective. Therefore, the fact that some facility licensees will be preparing the examinations with NRC review and approval, should have no negative effect on the safe operation of the plants.

Although the NRC-approved examinations were comparable to NRC-prepared examinations, essentially all of the examinations prepared by facility licensees required some changes subsequent to NRC review, and many of the examinations required significant rework. The NRC had originally believed that, with training and experience, the industry would quickly gain proficiency in preparing the examinations, but the overall quality of the examinations submitted to the NRC during the pilot program did not improve as expected over time. Although approximately half of the 17 facility licensees that had prepared more than one examination by the end of FY 1997 did maintain or improve the quality of their second or third examination submittals, the quality of the other facility licensees' second or third examinations was lower. Consequently, the NRC has asked the industry to address the issue of examination quality and determine the need for additional training on examination development. The NRC will continue to: (1) direct facility licensees that prepare their examinations to revise the examinations as necessary to achieve an acceptable level of quality and discrimination; (2) withhold approval of those examinations that do not meet NRC standards; (3) oversee the regional implementation of the operator licensing process to ensure consistency; (4) address significant deficiencies in the submitted examinations as licensee performance issues in the examination reports, as appropriate; (5) conduct or participate in workshops, as necessary, to ensure that facility licensees understand the NRC's examination criteria; and (6) prepare the licensing examinations for those facility licensees that elect not to prepare their own examinations.

With regard to the efficiency of the revised examination process, the experience to date supports the conclusion that the average industry cost will not differ significantly from the cost of NRC-prepared examinations. Comments from the industry reflect that the cost for some facility licensees to prepare the examination was higher than it would have been for an NRC-prepared examination; however, other licensees prepared good quality examinations at lower cost than the NRC. The industry generally attributed the higher cost to the revised examination and administrative criteria under the pilot examination process. Although the NRC acknowledges that the revised criteria contribute somewhat to the elevated cost, many of the variables that affect the quality and, consequently, the cost of the examination will be under the facility licensees' control and can present an opportunity for cost savings. For example, facility licensees that elect to prepare the examinations will be able to manage the size and quality of their examination banks and the training and experience of the personnel they select to write their licensing examinations. The revised examination process allows facility licensees to control the development of the examinations and holds them responsible for their quality. If a facility licensee submits an acceptable quality examination, it is likely to save resources despite the additional administrative criteria; however, if the facility licensee submits an examination that requires many changes, it will likely cost more than if the NRC had prepared the examination.

Comments from the NRC chief examiners who worked on the pilot examinations indicate that the average amount of time spent reviewing and revising the facility-prepared examinations was generally consistent with the estimates developed before starting the pilot program. Although a number (approximately 20 percent in FY 1997) of the examinations required significantly more NRC effort than originally anticipated to bring them up to the NRC's standards, the resource burden was generally offset by other examinations that required less effort to review and revise. The increased efficiency of the revised examination process has enabled the NRC to eliminate the use of contractors in the operator licensing program and conduct the initial operator licensing and requalification inspection programs with the existing NRC staff. Before initiating the pilot examination and transition process at the beginning of FY 1996, the NRC spent approximately $3 million per year on contractor assistance for initial examinations and requalification inspections. In FY 1997, when facility licensees prepared approximately 75 percent of the examinations, the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million. The FY 1998 and FY 1999 budgets reflect the complete elimination of contractor support for the operator licensing program (with the exception of the generic fundamentals examination). Future resource requirements for the operator licensing program will, in large part, be driven by changes in the level of facility participation in the voluntary examination development process.

In order to maintain the integrity of the operator licensing written examinations required by 10 CFR 55.41 and 55.43 and the operating tests required by 10 CFR 55.45, the Commission has amended the final rule by adding a requirement for those power reactor facility licensees that elect to prepare, proctor, and grade the written examinations and prepare the operating tests, to establish, implement, and maintain procedures that control the security and integrity of those examinations and tests. The Commission's regulations in 10 CFR 55.49 already prohibit applicants, licensees (operators), and facility licensees from engaging in any activity that compromises the integrity of any examination or test required by 10 CFR 55. However, based on the number of examination security incidents that have occurred since the pilot examination program began, the Commission has concluded that applicants, licensees, and facility licensees may not be aware that the requirements of 10 CFR 55.49 cover more than just those activities directly involving the physical administration of an examination or test. In that regard, the Commission considers the integrity of an examination or test to be compromised if any activity occurs that could affect the equitable and consistent administration of the examination or test, regardless of whether the activity takes place before, during, or after the administration of the examination or test. Therefore, in addition to requiring certain facility licensees to establish, implement, and maintain procedures that control the security and integrity of the examinations and tests, the Commission is also amending 10 CFR 55.49 to clarify the scope of that regulation.

Revision 8 of NUREG-1021 identifies a number of examination security and integrity guidelines (e.g., physical security precautions, including the use of simulators and the mailing of examination materials) that the affected facility licensees (i.e., those that elect to prepare their own written examinations and operating tests) should consider when establishing their procedures. Although the security and integrity guidelines in NUREG-1021 are not regulatory requirements, once a facility licensee has established its required procedures, the Commission intends to monitor this area to ensure that the procedures are implemented and maintained.

Consistent with the examination security and integrity guidelines in NUREG-1021, facility employees with specific knowledge of any NRC examination before it is given should not communicate the examination contents to unauthorized individuals and should not participate in any further instruction of the students scheduled to take the examination. Before they are given access to the examination, facility employees are expected to sign a statement acknowledging their understanding of the restrictions. When the examinations are complete, the same employees are expected to sign a post-examination statement certifying that they have not knowingly compromised the examination.

NRC examiners are expected to be attentive to the facility licensee's examination security measures, to review the security expectations with the facility licensee at the time the examination arrangements are confirmed, and to report any security concerns to NRC management. If the NRC determines during its preparation that an examination may have been compromised, it will not administer the examination until the scope of the potential compromise is determined and measures can be taken to address the integrity and validity of the examination. Pursuant to 10 CFR 55.51, the NRC must make a determination before issuing a license that the test or examination is valid, meeting the requirements of the AEA and the Commission's regulations. If the compromise is discovered after the examination has been administered, the NRC will not complete the licensing action for the affected applicants until the NRC staff can make a determination regarding the validity of the examination. If the compromise is not discovered until after the licensing action is complete, the NRC will reevaluate the licensing decision. If the NRC determines that the original licensing decision was based on an invalid examination, it will take appropriate action pursuant to 10 CFR 55.61(b)(2).

As a separate action, the Commission is modifying its "General Statement of Policy and Procedures for NRC Enforcement Actions" (Enforcement Policy) to provide examples of violations that may be used as guidance in determining the appropriate severity level for violations involving the compromise of an examination or test. The NRC staff will evaluate all potential compromises of an examination or test required by 10 CFR 55 to determine whether a violation of 10 CFR 55.49 has occurred. A compromise that is not detected before the test or examination is administered would be considered a significant regulatory concern and categorized at least at Severity Level III. However, depending on the circumstances as explained in the Enforcement Policy, the severity level may be increased or decreased. The NRC intends to utilize its enforcement authority including, as warranted, civil penalties and orders against individuals and facility licensees who: (1) compromise the integrity of an examination in violation of 10 CFR 55.49; (2) commit deliberate misconduct in violation of 10 CFR 50.5; or (3) provide incomplete or inaccurate information to the NRC in violation of 10 CFR 50.9. In addition, cases involving willful violations may be referred to the Department of Justice for criminal prosecution.

The Commission has reviewed the vulnerabilities and costs associated with the revised examination process and considered the measures that the NRC staff has taken to mitigate the vulnerabilities. With regard to examination quality and level of difficulty, the Commission acknowledges that the effectiveness of the revised examination process is contingent on the NRC staff's review of the facility-proposed examinations to ensure that NRC standards are achieved. The Commission has concluded, based on the results of the pilot examination program, that the controls implemented by the NRC staff will provide reasonable assurance that the examinations that are administered to the license applicants will provide a valid and consistent basis upon which to make the licensing decisions regardless of whether the examinations were prepared by the facility licensee or the NRC. The Commission also realizes that the frequency of examination security incidents and the risk of undetected compromises may increase for those examinations that are prepared by facility licensees. However, the Commission is confident that the measures discussed above will sufficiently control the vulnerability in this area.

The Commission is aware that the original expectation that facility licensees would eventually realize cost savings under the revised process as they gain proficiency in preparing the examinations has not yet been realized. However, the Commission has concluded that neither the increased vulnerabilities nor the absence of clear industry cost benefit provides sufficient basis for discontinuing the revised examination process. The Commission also finds that the revised examination process is more consistent with the NRC's other oversight programs because it requires NRC examiners to review materials prepared by facility licensees. The revised process enables NRC examiners to focus more on the psychometric quality of examinations (e.g., the cognitive level at which the questions are written and the plausibility of the distractors or wrong answer choices) prepared by the facility licensees than on the technical accuracy of the examinations, which was their primary focus when the examinations were prepared by NRC contractors. This shift in the NRC examiners' focus, coupled with the facility licensees' technical expertise, has the potential to improve the overall quality of the facility-prepared licensing examinations.

In the proposed rule, the NRC took the position that the backfit rule (10 CFR 50.109) did not apply to this rulemaking. However, in its review of the final rule, the Committee To Review Generic Requirements (CRGR) opined that it was inclined to view the rule as a backfit and recommended that the provisions of the proposed rule be implemented on a voluntary basis, which would not constitute a backfit. Although the NRC had considered and dismissed that alternative during the proposed rulemaking because of concerns regarding resource planning, it has since concluded that the benefits of the revised examination process (e.g., improved regulatory efficiency and greater licensee control over the examination costs) remain substantial even if every facility licensee is not required to prepare its own examinations. Rather than terminate the pilot program and resume the NRC-prepared examination process on an industrywide basis, the NRC has decided to amend the final rule to give facility licensees the option to prepare their own examinations or to have them prepared by the NRC.

Summary of Public Comments

The 75-day public comment period began when the notice of proposed rulemaking was published in the Federal Register (62 FR 42426) on August 7, 1997, and closed on October 21, 1997. The notice (FRN) requested public comment on the proposed rule, on the implementation guidance in interim Revision 8 of NUREG-1021, and on the following two questions:

1. Are there portions of the operator exams that are common to all licensees, and would, therefore, be more efficiently developed by the NRC?

2. Is the conclusion in the regulatory analysis correct that it would be less costly for each licensee to prepare its own initial operator examinations to be reviewed, revised, and administered by the NRC, than to have one NRC contractor prepare these exams for all licensed operators with the costs to be reimbursed by licensee fees?

The NRC received 13 comment letters on the proposed rule; two of the letters arrived after the comment period closed, but they were considered nonetheless. The respondents included three NRC examiners, one contract examiner, five nuclear utilities and one utility employee, one nonpower reactor facility licensee, the State of Illinois, and the Nuclear Energy Institute (NEI), which submitted its comments on behalf of the nuclear power industry. Copies of the public comments are available in the NRC Public Document Room, 2120 L Street, NW (Lower Level), Washington, DC, and on the internet at "http://ruleforum.llnl.gov/cgi-bin/ rulemake?source=OE_ PRULE".

Seven of the respondents (three NRC examiners, one contract examiner, one utility employee, one nonpower facility licensee, and the State of Illinois) recommended that the rule change be disapproved. Five of the industry respondents (NEI and four utilities) supported the rule change; however, one utility endorsed NEI's comments but stated that it did not agree with the proposed rule in its present form. NEI and two of the utilities stated that they would rather continue with a voluntary program because it would allow greater flexibility for those facility licensees with small training staffs. However, they would support mandatory participation with the rule change rather than return to the previous process under which NRC contractors wrote most of the examinations.

Those comments related to the two specific questions raised in the proposed rule, and those that have a direct bearing on the rule are discussed below. The comments are categorized as they relate to reactor safety and the vulnerabilities discussed in SECY-96-206 (i.e., quality and consistency, independence and public perception, security, NRC resources, and examiner proficiency). NRC received no comments related to program stability.

One NRC examiner, NEI, four of the utilities, and the utility employee also provided specific comments and recommendations regarding the implementation guidance in interim Revision 8 of NUREG-1021. Those comments are addressed in Attachment 1 of the Commission (SECY) paper associated with this rulemaking. A copy of the SECY is available in the NRC Public Document Room, on the internet at http://www.nrc.gov, or from Siegfried Guenther, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, at 301-415-1056 or e-mail at sxg@nrc.gov.

Comment: With regard to the first specific question included in the proposed rulemaking, 2 of the 13 respondents (NEI and one utility) stated that all of the common material is already included in the generic fundamentals examination (GFE) and that the remaining elements are best covered as part of the site-specific examination.

Response: It appears that the current allocation of topics between the GFE and site-specific written examinations is generally perceived to be an efficient method of covering the topics required by 10 CFR 55.41 and 55.43. Therefore, the Commission finds no basis for changing the process to have the NRC separately develop portions of the initial examination that would be common to all facilities.

Comment: Seven of the 13 respondents (NEI, two utilities, a utility employee, and three examiners) directly or indirectly addressed the second specific question in their letters. NEI and one utility stated that the revised examination criteria in interim Revision 8 of NUREG-1021 have increased the level of effort and will result in higher licensing fees regardless of who prepares the examinations. However, NEI and another utility agreed that comparing the cost of facility-prepared examinations to those prepared by the NRC is difficult, but they concluded that it should be less costly for facility licensees to prepare the examinations than to have the NRC prepare them under the same criteria.

NEI also stated that the relative cost of the two examination processes should not be the only factor in deciding whether to proceed with the rulemaking that would have required all power reactor facility licensees to prepare their licensing examinations. NEI indicated that preparing higher cognitive level questions requires detailed plant knowledge, better provided by facility licensees, and that the revised process (which has eliminated the use of NRC contractors to administer the operating tests) will allow NRC staff to evaluate each applicant without relying on third-party observers.

Two NRC examiners, one contract examiner, and a utility employee asserted that the facility licensees' cost has increased under the revised examination process. They cited various reasons for the increased cost, including training personnel to write the examinations and then restricting them from training the applicants, and upgrading equipment to maintain examination security. The NRC examiners based their comments on feedback from facility training personnel; one examiner indicated that it took facility licensees an average of 700 hours to prepare each examination. The utility employee stated that the rule change will simply transfer the cost of contractors from the NRC to the utilities.

Response: The NRC acknowledges that the revised administrative criteria in particular (e.g., the restrictions on facility training personnel and the need to document the source of the test items) have probably caused the cost of preparing the examinations to be somewhat higher than it would have been if facility licensees had been allowed to prepare the examinations using the same criteria that applied to the NRC and its contractors before starting the pilot program. However, when the NRC first developed the revised examination process, with its additional administrative criteria, the NRC still believed that the cost for facility licensees to prepare the examinations would be offset by the reduction in the licensing fees and that a cost savings could be realized as facility licensees gained experience with the process. Many of the facility licensees that participated in the pilot program demonstrated that it is possible to prepare an acceptable quality examination at the same or lower cost than the NRC or its contractors could prepare a comparable examination. The fact that a number of facility licensees did not prepare acceptable examinations may be as much an indication of the licensees' inefficiency and inexperience as it is a symptom of deficiencies in the examination criteria. Those facility licensees that did not initially submit acceptable examinations, eventually paid more in fees because of the additional effort required for the NRC to review, and the licensees' staffs to rewrite, the examinations. Finally, it is possible that the magnitude of the increase in effort and cost may be perceived to be higher than it actually is because the industry had originally expected to save money if the NRC would have allowed facility licensees to prepare the examinations using the version of NUREG-1021 that was in effect before beginning the pilot program.

With regard to the additional security costs cited by the examiners, the Commission has stressed the importance of maintaining examination security, but the NRC has not required facility licensees to invest in additional physical security systems. However, the frequency of security incidents since beginning the pilot examination program has prompted the NRC to: (1) clarify the intent of 10 CFR 55.49 in the final rule; (2) amend the final examination rule to require facility licensees that elect to prepare their examinations to establish, implement, and maintain procedures to control examination security and integrity; and (3) include additional security guidance in the final version of Revision 8 of NUREG-1021. These actions will help ensure, among other things, that facility licensees understand their responsibility for maintaining control over the examination process.

The pilot examinations demonstrated that some of the people assigned by facility licensees to develop the examinations did not have sufficient expertise required to prepare good quality examination materials consistent with NRC standards. As noted earlier, the NRC has asked the industry to address the issue of examination quality and the need for additional training on examination development. The NRC acknowledges that the restrictions on the use of instructors to prepare the licensing examinations may be partially responsible for limiting the availability of qualified examination preparers. Therefore, the staff has revised the personnel restrictions in the final version of Revision 8 of NUREG-1021 in an effort to reduce the burden on facility licensees. Specifically, instructors will be allowed to prepare the written examination questions without regard to the amount of time they spent training the license applicants; however, the instructors will still be precluded from preparing questions related to those topics on which they provided instruction and from instructing the applicants once they begin working on the licensing examination (as was the case in interim Revision 8). This change should provide licensees that elect to prepare their examinations with increased flexibility in managing their resources and possibly reduce their costs without adversely affecting the independence of the examination process.

The NRC has revised the regulatory analysis in response to the public comments and lessons learned from the pilot program. The NRC has also reevaluated the additional administrative criteria in interim Revision 8 of NUREG-1021 and considers them reasonable and essential to mitigate the vulnerabilities (e.g., quality, security, and conflict of interest) of the new examination process and to facilitate the NRC staff's review of the proposed examinations. These criteria are retained in the final version of Revision 8 of NUREG-1021.

The issue of cost has lost much of its importance because the NRC has decided to continue the revised examination process on a voluntary basis rather than require each power reactor facility licensee to prepare the examinations. It will be up to each facility licensee to compare the cost of preparing its own examinations in accordance with the criteria in the effective revision of NUREG-1021 with the cost of having the NRC staff prepare the examinations and then make a decision based on its available resources (and other considerations).

Comment: Two NRC examiners with pilot-examination experience asserted that the quality of the simulator and walk-through tests has decreased significantly and that, in most cases, the quality and difficulty of the submitted examinations have been below NRC standards. All four examiners who submitted comments cited various reasons why the quality and difficulty of the facility-prepared examinations might be lower than examinations prepared by the NRC or its contract examiners, including: (1) the facility licensees' tendency to narrow the scope of the operating test to those procedures that the facility believes are important (and emphasized in the training program); and (2) the belief that most facility training personnel do not have the expertise to develop valid test items. Two NRC examiners asserted that the quality of the examinations has not improved during the pilot program and is not likely to improve because there is nothing to prevent licensees from using different people to develop successive examinations. A utility employee asserted that the utilities' limited contact with the process by preparing an examination once every 18 to 24 months will not foster consistency or develop skilled examination writers.

Two NRC examiners asserted that the elimination of NRC contract examiners who participated in examinations across the four NRC regions will be detrimental to examination consistency. One NRC examiner asserted that the guidance in interim Revision 8 of NUREG-1021 is not sufficiently prescriptive to ensure nationwide consistency in the level of knowledge tested and the level of difficulty of the examinations and that several specific changes should be included in NUREG-1021 to address his concerns.

The State of Illinois asserted that the quality and consistency of the written examination questions can be maintained because the NRC can change and approve the questions before they are used. However, the State also recommended that the NRC should compile the examination questions and proctor the examinations (refer to the conflict-of-interest discussion below).

According to NEI, the recent facility-prepared examinations were of higher quality than the examinations prepared by the NRC before the pilot program started. Many of the NRC-prepared examinations had to be revised in response to the facility licensees' technical reviews.

Response: Essentially all of the facility-prepared examinations required some changes and many required significant changes to make them conform to the NRC's standards for quality and level of difficulty. According to the questionnaires completed by the NRC chief examiners responsible for the pilot examinations, the average facility-prepared written examination required approximately 10 to 20 changes, which is consistent with the number of changes often required on examinations prepared by NRC contract examiners. Most NRC chief examiners judged the final examinations (with the NRC's changes incorporated) to be comparable to recent NRC-prepared examinations in terms of quality and level of difficulty. Moreover, the fact that the passing rate on the facility-prepared examinations is generally consistent with the historical passing rate on examinations prepared by the NRC suggests that the NRC-approved examinations have discriminated at an acceptable level and that they have provided an adequate basis for licensing the applicants at those facilities.

Although the NRC expected that the proposed examination quality would improve as facility licensees gained experience and familiarity with the NRC's requirements and expectations, the overall quality of examinations submitted to the NRC during the transition process did not improve appreciably over time. Although approximately half of the 17 facility licensees that had prepared more than one examination by the end of FY 1997 did maintain or improve the quality of their second or third examination submittals, the quality of the other facility licensees' second or third examinations was lower. Although it is unclear to what extent the problems with proposed examination quality and difficulty have been caused by a lack of sufficient expertise on the part of the examination writers, the NRC has asked the industry to address this issue. Furthermore, the NRC staff has conducted and participated in a number of public meetings and workshops in an effort to communicate its expectations to the facility employees who will be preparing the examinations. Additional NRC and industry workshops will be conducted to address examination quality and solicit industry feedback.

In SECY-96-206, the NRC staff discussed the issues of examination quality and consistency and how they might be affected when a large number of facility employees assume the role that had been filled by a smaller number of experienced NRC and contract examiners. The NRC staff's comprehensive examination reviews versus the examination criteria in NUREG-1021, in combination with supervisory reviews and the examination oversight activities conducted by the Office of Nuclear Reactor Regulation, should mitigate the vulnerability in this area. Moreover, the industry and staff initiatives to improve the expertise of the examination writers should eventually enhance the quality and consistency of the facility-prepared examinations.

Comment: All four examiners who submitted comments, a nonpower reactor facility licensee, and the State of Illinois asserted that allowing the facility licensees to prepare the operator licensing examinations decreases the level of independence and creates a conflict of interest for facility personnel having responsibility for training and licensing the operators. Their letters maintained that the new process makes it possible for the utilities to "teach the examination," to test applicants only on what was taught, or to avoid testing in areas with known difficulties. One NRC examiner noted that the new process places training managers in a no-win situation because if applicants fail the examination, the managers look like poor trainers, and if the examination is too easy, the NRC gives them a bad report. He and another NRC examiner asserted, based on their experience during the pilot examinations, that some facility personnel openly admitted that they would develop the easiest possible examination to ensure that all their applicants would pass.

One NRC examiner noted that the NRC review and approval process cannot adequately compensate for the conflict-of-interest problems inherent in the revised examination process and recommended a change to interim Revision 8 of NUREG-1021 that would limit the licensees' latitude in selecting topics for the examination outline. The State of Illinois suggested that the NRC should compile the questions and proctor the examination to maintain more of the checks and balances that existed under the old process.

The nonpower reactor facility licensee noted that most professional licensing examinations are developed by independent agencies, and that this fosters a sense of professionalism in the license applicants.

Response: The NRC agrees that the revised examination process decreases the level of independence in the licensing process and may create a potential conflict of interest for facility personnel involved in preparing the examination. As discussed in SECY-96-206, the NRC included a number of measures in Revision 8 of NUREG-1021, including restrictions on which facility employees can develop the examination and a performance-based review of every examination by an NRC examiner, to ensure that an adequate level of independence is maintained and to minimize the potential for bias in the examination development process. Although the NRC will no longer prohibit instructors from participating in the examination development based solely on the amount of time they spent training the license applicants (as discussed above in response to comments concerning the industry burden under the revised examination process), this change is not expected to substantially increase the potential for bias in the operator licensing process because instructors will still be prohibited from writing questions related to the topics in which they provided instruction. Moreover, as a separate measure, the NRC has amended the final version of Revision 8 of NUREG-1021 to include an expectation that facility licensees will use an objective, systematic process for preparing the written examination outline. This process enhancement should further limit the potential for bias in the selection of topics to be evaluated on the written examination.

The Commission has concluded that the amended personnel restrictions, in combination with the systematic selection of topics for the written examination and the continued independent selection of topics for the operating test, will establish a sufficient level of independence to address the vulnerability associated with the conflict of interest inherent in the revised examination process. However, if the NRC determines that a facility licensee has intentionally biased the scope, content, or level of difficulty of an examination (i.e., compromised its integrity contrary to 10 CFR 55.49) to enhance the chances that its applicants would pass the examination, the NRC will utilize its enforcement authority including, as warranted, civil penalties, orders against the individuals involved, and, charging the individuals involved with deliberate misconduct pursuant to 10 CFR 50.5.

Concerns regarding the potential for conflict of interest and the frequency of security incidents since beginning the pilot examination program have prompted the NRC to review the clarity of 10 CFR 55.49. The regulation encompasses not only activities like cheating and lapses in security but also activities that compromise the integrity or validity of the examination itself (e.g., noncompliance with the criteria designed to limit the potential for bias in the selection of topics to be evaluated on the written examination). Therefore, the NRC has concluded that it would be beneficial to amend 10 CFR 55.49 to clarify its intent and to amend the examination rule to require power reactor facility licensees that elect to prepare their licensing examinations to establish procedures to control examination security and integrity.

Comment: Three NRC examiners and the State of Illinois asserted that the revised examination process increases the threat to examination security. One examiner noted that the examination is onsite for a longer period of time, thereby proportionally increasing the risk of being compromised. Another examiner cited the fact that a number of examination reports have documented problems with security.

Response: As discussed in SECY-96-206 and SECY-97-079, the Commission is aware of the vulnerability in this area because several security incidents have occurred since beginning the pilot examination program. Therefore, based on the comments received and the experience with security incidents, the NRC has: (1) clarified 10 CFR 55.49 in the final rule to ensure that applicants, licensees, and facility licensees understand the scope and intent of the regulation; (2) amended the final examination rule to require facility licensees that elect to prepare their licensing examinations to establish, implement, and maintain procedures to control examination security and integrity; (3) strengthened the discussion of examination security in the final version of Revision 8 of NUREG-1021; and (4) modified NUREG-1600, "General Statement of Policy and Procedures for NRC Enforcement Actions," to address enforcement action against parties subject to the requirements in 10 CFR 55.49. NRC examiners are expected to review the NRC's physical security guidelines and the facility licensee's specific plans for ensuring examination security at the time the examination arrangements are confirmed with the designated facility contact. Furthermore, the NRC has issued an Information Notice to advise power reactor facility licensees of the NRC's perspective and expectations regarding the integrity of examinations developed by the facility licensees' employees and representatives, and it has asked NEI to take the initiative in developing a model for securing examinations.

As a separate action, the NRC will not administer any examination that may have been compromised until the scope of the potential compromise is determined and measures can be taken to address the integrity and validity of the examination. If the compromise is discovered after the examination has been administered, the NRC will not complete the licensing action for the affected applicants until the staff can make a determination regarding the impact that the compromise has had on the examination process. If the compromise is not discovered until after the licensing action is complete, the NRC will reevaluate the licensing decision pursuant to 10 CFR 55.61(b)(2) if it determines that the original licensing decision was based on an invalid examination.

Comment: One NRC examiner disagreed with the conclusion in the proposed rulemaking that the facility-prepared examination process is an efficient use of NRC resources when compared to the NRC-prepared or contractor-prepared examinations. He noted that, in most cases, the quality and difficulty of the proposed examinations have been below NRC standards (as discussed above) and that it has taken a significant effort on the part of the NRC chief examiner to achieve an acceptable product.

An NRC contract examiner asserted that NRC cost-saving is a poor reason for changing the rule, since the utilities pay for the examinations anyway. He noted that the pilot examination process has led to a loss of certified examiners and contends that those NRC examiners who are left will become more dissatisfied with their jobs and will leave because they will be required to travel more to compensate for the loss of contractors.

Response: The NRC acknowledges that many of the facility-prepared examinations (about 20 percent in FY 1997) required significantly more NRC examiner time than desired or planned in order to achieve NRC quality standards. However, questionnaires filled out by NRC chief examiners for the pilot examinations indicate that the average amount of time spent on reviewing and upgrading the examinations is generally consistent with the estimates developed before starting the pilot program (i.e., approximately 170 examiner-hours). As noted in SECY-97-079, the NRC has issued a memorandum to its regional administrators emphasizing the importance of: (1) assigning adequate resources to carry out the operator licensing task; (2) completing a review of every facility-prepared examination; and (3) not administering any examination that fails to meet NRC standards for quality and level of difficulty. Furthermore, all the time that NRC examiners spend reviewing an examination and modifying it so that it meets NRC standards is ultimately billed to the facility licensee.

The Commission acknowledges that facility licensees bear the cost of preparing the licensing examinations whether or not the NRC performs this function. However, this rule will give facility licensees more control over the cost of licensing operators at their facility, and the pilot examination program has demonstrated that some facility licensees will save resources if they elect to prepare their own licensing examinations.

The NRC's budget cuts have necessitated agencywide downsizing, which can be expected to increase the burden of travel for many NRC employees, not just the operator licensing examiners. The number of NRC full-time equivalent (FTE) license examiners has remained essentially constant throughout the pilot program and, aside from normal attrition and staff turnover, the loss of certified examiners has been limited to NRC contractors.

Comment: Two NRC examiners expressed concern that examiner proficiency will decrease as a result of implementing the revised examination process. One of the examiners stated that examination reviewers will not maintain the same base of knowledge as examination writers maintained and that they will lose their familiarity with plant operating procedures.

Response: The Commission has concluded that the revised examination process affords sufficient NRC staff involvement that NRC examiners will maintain an acceptable level of proficiency. An NRC examiner will review and approve every facility-prepared examination before it is administered to ensure that it conforms to the criteria specified in NUREG-1021 for content, format, quality, and level of knowledge and difficulty. NRC examiners will also continue to independently administer and grade both the dynamic simulator and the plant walk-through portions of the operating tests. Because NRC examiners will be administering all of the operating tests, the Commission believes that the revised process will enable the examiners to accrue more experience in a shorter period of time and to maintain their proficiency. New NRC license examiners will still be required to complete a standardized training program, including the development of a complete written examination and operating test, as part of their qualification process. Moreover, the NRC will ensure that the in-house capability to prepare the examinations is maintained by: (1) requiring a regional supervisor to review and approve every examination and the Office of Nuclear Reactor Regulation to conduct periodic examination reviews; (2) conducting examiner refresher training; and (3) convening an operator licensing examiners' training conference at intervals not to exceed 24 months. Although experience during the voluntary pilot program and informal feedback from the industry suggests that facility licensees are likely to request the NRC to prepare a sufficient number of examinations to maintain the proficiency of its examiners, each region will be required to write at least one initial operator licensing examination per calendar year.

Comment: A utility employee asserted that the revised examination process will not enhance the competency of the operators or reactor safety because the facilities' training resources will be diverted from their primary purpose (i.e., training the applicants) as much as six months before the examination date. Three NRC examiners also took issue with the conclusion in the proposed rulemaking that the NRC staff's focus on operator performance and its core of experience will improve under the pilot examination process because contractors will no longer be used to administer the operating tests. Two of the examiners asserted that the reduction in the amount of procedural research by examiners will result in the identification and correction of fewer procedural problems. Two of the examiners also stated that the contract examiners help maintain examination consistency across the NRC regions and that their contribution to the operator licensing program goes beyond simple task completion.

Response: The Commission expects that those training departments that cannot readily and safely absorb the examination development work will use the funds that they were previously paying to the NRC through the fee recovery program to secure the additional personnel to do the extra work or request the NRC to prepare the examinations. If a facility licensee decides to prepare the examination and, as a result, places insufficient resources on either training or testing, the quality of its proposed licensing examinations or the passing rate on those examinations would most likely suffer. Although many of the facility-prepared examinations have required significant changes to achieve NRC quality standards, the examination results, to date, are generally consistent with the results on previous NRC-prepared examinations, suggesting that the quality of the facility licensees' training programs has not been affected. Therefore, the fact that facility licensees will have the option of preparing the examinations is not expected to have a negative effect on reactor safety.

The NRC acknowledges that the contract examiners identified procedural and training problems in addition to their primary responsibility for preparing and administering the licensing examinations, and that they helped maintain examination consistency by working on examinations in each of the NRC's regions. As noted in connection with the discussion of examination quality, the Commission realizes that the revised examination process increases the possibility of inconsistency, but it believes that the examination criteria in the final version of Revision 8 of NUREG-1021, in combination with the NRC's examination oversight programs, will minimize these inconsistencies so that they remain within acceptable limits.

When the NRC initiated the pilot program, its goal was to eliminate the need for NRC contract examiners without compromising the existing levels of reactor safety. Because NRC examiners will be administering all of the operating tests, the revised process will enable the NRC examiners to accrue more experience in a shorter period of time and may improve the consistency of the operating test evaluations and the licensing decisions. Although the total number of procedures reviewed in the process of developing examinations may be fewer under the revised method, NRC examiners will still be expected to review and identify discrepancies in the procedures that will be exercised during the walk-through portion of the operating test and during the simulator scenarios.

Other Comments

Since beginning the pilot examination program, the NRC has sought to obtain up-to-date insights regarding the effectiveness of the revised examination process based on the staff's growing body of experience in reviewing the facility-prepared examinations. Many of the staff comments received have paralleled the public comments and require no further attention in this notice. However, one recommendation to amend the wording of the proposed regulation is considered worthy of discussion and incorporation. Specifically, it was recommended that the rule should indicate that a key manager would be responsible for submitting the examination because that individual would be in a position to ensure that the facility licensee's operations and training departments apply sufficient resources to prepare a quality examination. The NRC finds that the recommendation is consistent with normal NRC practice and the analogous regulatory requirement in �55.31(a)(3), which requires "...an authorized representative of the facility licensee by which the applicant will be employed..." to submit a written request that examinations be administered to the applicant. Therefore, the wording of the final examination rule has been amended to require an authorized representative of the facility licensee to approve the written examinations and operating tests before they are submitted to the NRC for review and approval.

Availability of Guidance Document for Preparing Operator Licensing Examinations

As a consequence of preparing and administering the initial operator licensing examinations over a number of years, the NRC has developed a substantial body of guidance to aid its examiners. That guidance has been published in various versions of NUREG-1021, the latest version of which (final Revision 8) incorporates lessons learned since interim Revision 8 was published in February 1997, as well as refinements prompted by the comments submitted in response to the FRN of August 7, 1997 (62 FR 42426), which solicited public comments in conjunction with the proposed rulemaking. A copy of the final version of Revision 8 of NUREG-1021 will be mailed to each facility licensee; in accordance with NRC practice, revisions of NUREG-1021 are announced in the Federal Register when they are issued and become effective six months after the date of issuance. Copies may be inspected and/or copied for a fee at the NRC's Public Document Room, 2120 L Street NW (Lower Level), Washington, DC. Final Revision 8 of NUREG-1021 is also electronically available for downloading from the internet at "http://www.nrc.gov."

The NRC will prepare, administer, and grade initial operator licensing examinations when requested by facility licensees and at least four times a year to maintain the proficiency of its examiners. NRC examiners will use the criteria in the effective version of NUREG-1021 to evaluate whether an applicant meets the Commission's regulations. In this regard, NUREG-1021 is comparable to the Standard Review Plan (SRP), which establishes the criteria that the NRC uses to evaluate Part 50 license applications. Licensees that elect to prepare their own licensing examinations will also be required to use the guidance in the effective version of NUREG-1021. As provided in NUREG-1021, licensees may identify differences from the NUREG-1021 examination criteria and evaluate how the proposed alternatives provide an acceptable method of complying with the Commission's regulations. The NRC staff will review any proposed alternatives and make a decision regarding their acceptability. The NRC will not approve any alternative that would compromise its statutory responsibility of prescribing uniform conditions for the operator licensing examinations.

Final Rule

This regulation adds a new section, �55.40, "Implementation," to Subpart E of 10 CFR Part 55. Paragraph (a) of �55.40 states the NRC's intent to use the criteria in the version of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors," in effect six months before the examination date when preparing and evaluating the written examinations required by ��55.41 and 55.43, and the operating tests required by �55.45. The NRC uses the criteria in NUREG-1021 to evaluate whether an applicant meets the Commission's regulations. In this regard, NUREG-1021 is comparable to the Standard Review Plan, which establishes the criteria that the NRC uses to evaluate Part 50 license applications. Pursuant to Section 107 of the AEA of 1954, as amended, the NRC must prescribe uniform conditions for licensing individuals applying for operator licenses.

Based on the success of the pilot examination program, paragraph (b) of �55.40 allows power reactor facility licensees to prepare, proctor, and grade the written examinations required by ��55.41 and 55.43 and to prepare the operating tests required by �55.45, subject to the following conditions:

(1) To ensure uniformity pursuant to the AEA, the facility licensee shall prepare the examinations and tests in accordance with NUREG-1021;

(2) To minimize the possibility that the required written examinations and operating tests might be compromised, the facility licensee shall establish, implement, and maintain procedures to control the security and integrity of the examinations and tests;

(3) To ensure that the facility licensee's operations and training departments apply sufficient resources to prepare a quality examination, an authorized representative of the facility licensee shall approve the examinations before they are submitted to the NRC for review and approval; and

(4) To ensure that NRC standards for quality are maintained, the facility licensee must receive Commission approval of its proposed written examinations and operating tests before they are given.

These requirements are contained in ��55.40(b)(1), (2), (3), and (4) respectively.

As provided in NUREG-1021, licensees may identify differences from the NUREG-1021 examination criteria and evaluate how the proposed alternatives provide an acceptable method of compliance with NRC regulations. The NRC staff will review any proposed alternatives and make a decision regarding their acceptability. However, the NRC will not approve any alternative that would compromise its statutory responsibility of prescribing uniform conditions for the operator licensing examinations. The NRC staff will review the facility-prepared written examinations and operating tests against the criteria in NUREG-1021 and direct whatever changes are necessary to ensure that adequate levels of quality, difficulty, and consistency are maintained. After the NRC staff reviews and approves a written examination, the facility licensee will proctor and grade the examination consistent with the guidance in NUREG-1021. The NRC staff will continue to independently administer and grade the operating tests, review and approve the written examination results, and make the final licensing decisions. The facility licensee will not conduct parallel operator evaluations during the dynamic simulator or the walk-through tests.

Pursuant to the requirements in �55.40(c), the NRC staff will prepare the licensing examinations and tests upon written request by a power reactor facility licensee in accordance with �55.31(a)(3). In addition, the NRC may exercise its discretion to reject a power reactor facility licensee's determination to prepare the required written examinations and operating tests, and to proctor and grade the written examinations. The NRC will then prepare, proctor, and grade the written examinations and prepare the operating tests for the facility licensee. This provision of the regulation allows the NRC to maintain its proficiency and to perform these activities if the NRC questions a licensee's ability to prepare an acceptable examination.

Paragraph (d) of �55.40 reasserts that the NRC will continue to prepare and administer the written examinations and operating tests for non-power reactor facility licensees. The NRC has taken this position because the non-power reactor community does not have an accreditation process for training and qualification or the resources to prepare the examinations.

This regulation also amends �55.49 because the NRC has determined, after the proposed rule was published, that applicants, licensees, and facility licensees may be interpreting �55.49 too narrowly by limiting it to actual cases of cheating. The amendment clarifies that the regulation pertains to all activities that could affect the equitable and consistent administration of the examination, including activities before, during, and after the examination is administered.

Environmental Impact: Categorical Exclusion

The NRC has determined that this rule is the type of action described as a categorical exclusion in 10 CFR 51.22(c)(1). Therefore, neither an environmental impact statement nor an environmental assessment has been prepared for this regulation.

Paperwork Reduction Act Statement

This final rule amends information collection requirements that are subject to the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq). These were approved by the Office of Management and Budget (OMB), approval number 3150-0101. The additional public reporting burden for this collection of information is estimated to average 500 hours per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information (i.e., preparing the examinations). The additional, one-time burden for power reactor facility licensees that elect to prepare their licensing examinations to establish procedures to prevent the examinations from being compromised is not expected to exceed 100 hours per facility; and the burden of maintaining those procedures is estimated at approximately 10 hours per facility per year. Send comments on any aspect of this collection of information, including suggestions for reducing the burden, to the Information and Records Management Branch (T-6F33), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by internet electronic mail to bjs1@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0101), Office of Management and Budget, Washington, DC 20503.

Public Protection Notification

The NRC may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

Regulatory Analysis

The Commission has prepared a regulatory analysis on this regulation. The analysis examines the costs and benefits of the alternatives considered by the Commission. The regulatory analysis is available for inspection in the NRC Public Document Room, 2120 L Street NW (Lower Level), Washington, DC. Single copies of the analysis may be obtained from Siegfried Guenther, Office of Nuclear Reactor Regulation, U.S. Nuclear Regulatory Commission, at 301-415-1056 or by e-mail at sxg@nrc.gov.

Regulatory Flexibility Certification

In accordance with the Regulatory Flexibility Act of 1980, (5 U.S.C. 605(b)), the Commission certifies that this rule does not have a significant economic impact on a substantial number of small entities. This rule affects only the licensing and operation of nuclear power plants. The companies that own these plants do not fall within the scope of the definition of "small entities" described in the Regulatory Flexibility Act or the Small Business Size Standards stated in regulations issued by the Small Business Administration at 13 CFR Part 121.

Backfit Analysis

In the proposed rule, the NRC took the position that the backfit rule (10 CFR 50.109) did not apply because the proposed shift in responsibility for preparing the examinations: would not constitute a "modification of the procedures required to operate a facility" within the scope of the backfit rule; would not have affected the basic procedures for qualifying licensed operators; and would not have required facility licensees to alter their organizational structures. However, upon further review, the NRC has concluded that there is insufficient basis to support the original position. Therefore, the NRC has decided to revise the final rule so that power reactor facility licensees may elect to prepare their written examinations and operating tests (and proctor and grade the written examinations) in accordance with NUREG-1021 or to have the NRC prepare the examinations. Eliminating the requirement for all facility licensees to prepare their examinations and tests obviates the need for a backfit analysis.

Enforcement Policy

In conjunction with this final rule, the Commission is separately publishing modifications to NUREG-1600, "General Statement of Policy and Procedure for NRC Enforcement Actions," to address enforcement action against parties subject to the requirements in 10 CFR 55.49 (i.e., Part 55 license applicants/licensees and Part 50 licensees).

List of Subjects in 10 CFR Part 55

Criminal penalties, Manpower training programs, Nuclear power plants and reactors, Reporting and recordkeeping requirements.

For the reasons given in the preamble and under the authority of the Atomic Energy Act of 1954, as amended; the Energy Reorganization Act of 1974, as amended; and 5 U.S.C. 552 and 553; the NRC adopts the following amendments to 10 CFR Part 55.

PART 55 - OPERATORS' LICENSES

1.The authority citation for Part 55 continues to read as follows:

AUTHORITY: Secs. 107, 161, 182, 68 Stat. 939, 948, 953 , as amended, sec. 234, 83 Stat. 444, as amended (42 U.S.C. 2137, 2201, 2232, 2282); secs. 201, as amended, 202, 88 Stat. 1242, as amended, 1244 (42 U.S.C. 5841, 5842).

Sections 55.41, 55.43, 55.45, and 55.59 also issued under sec. 306, Pub. L. 97-425, 96 Stat. 2262 (42 U.S.C. 10226). Section 55.61 also issued under secs. 186, 187, 68 Stat. 955 (42 U.S.C. 2236, 2237).

2. In �55.8, paragraph (c)(4) is revised to read as follows:

�55.8 Information collection requirements; OMB approval.

*****

(4) In ��55.40, 55.41, 55.43, 55.45, and 55.59, clearance is approved under control number 3150-0101.

*****

3. A new �55.40 is added to read as follows:

�55.40 Implementation.

(a) The Commission shall use the criteria in NUREG-1021, "Operator Licensing Examination Standards for Power Reactors,"⁽²⁾ in effect six months before the examination date to prepare the written examinations required by ��55.41 and 55.43 and the operating tests required by �55.45. The Commission shall also use the criteria in NUREG-1021 to evaluate the written examinations and operating tests prepared by power reactor facility licensees pursuant to paragraph (b) of this section.

(b) Power reactor facility licensees may prepare, proctor, and grade the written examinations required by ��55.41 and 55.43 and may prepare the operating tests required by �55.45, subject to the following conditions:

(1) Power reactor facility licensees shall prepare the required examinations and tests in accordance with the criteria in NUREG-1021 as described in paragraph (a) of this section;

(2) Pursuant to �55.49, power reactor facility licensees shall establish, implement, and maintain procedures to control examination security and integrity;

(3) An authorized representative of the power reactor facility licensee shall approve the required examinations and tests before they are submitted to the Commission for review and approval; and

(4) Power reactor facility licensees must receive Commission approval of their proposed written examinations and operating tests.

(c) In lieu of paragraph (b) of this section and upon written request from a power reactor facility licensee pursuant to �55.31(a)(3), the Commission shall, for that facility licensee, prepare, proctor, and grade, the written examinations required by ��55.41 and 55.43 and the operating tests required by �55.45. In addition, the Commission may exercise its discretion and reject a power reactor facility licensee's determination to elect paragraph (b) of this section, in which case the Commission shall prepare, proctor, and grade the required written examinations and operating tests for that facility licensee.

(d) The Commission shall prepare, proctor, and grade the written examinations required by ��55.41 and 55.43 and the operating tests required by �55.45 for non-power reactor facility licensees.

4. Section 55.49 is revised to read as follows:

�55.49 Integrity of examinations and tests.

Applicants, licensees, and facility licensees shall not engage in any activity that compromises the integrity of any application, test, or examination required by this part. The integrity of a test or examination is considered compromised if any activity, regardless of intent, affected, or, but for detection, would have affected the equitable and consistent administration of the test or examination. This includes activities related to the preparation and certification of license applications and all activities related to the preparation, administration, and grading of the tests and examinations required by this part.

Dated at Rockville, Maryland, this ____ day of ______________, 1998.

ATTACHMENT 3

REGULATORY ANALYSIS FOR RULEMAKING ON REQUIREMENTS
FOR INITIAL LICENSED OPERATOR EXAMINATIONS

• 1. Statement of Problem and Objective
• 2. Background
• 3. Identification and Analysis of Three Alternative Approaches
• 3.1 Alternative 1 - Take No Action
• 3.2 Alternative 2 - Provide Regulatory Guidance
• 3.3 Alternative 3 - Amend 10 CFR Part 55
• 4. Regulatory Impact - Qualitative Costs and Benefits
• Facility Licensees
• Operator License Applicants
• NRC Staff
• 5. Decision Rationale
• 6. Implementation Schedule

1. Statement of Problem and Objective

Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing these individuals, and to issue licenses as appropriate. To implement this statutory mandate, operator license applicants are required by 10 CFR Part 55, "Operators' Licenses," to pass a written examination and an operating test. The written examination and operating test must satisfy the basic content requirements that are specified in the regulation. Although the NRC has traditionally prepared, administered, and graded the examinations and tests itself or through its contract examiners, neither the AEA nor Part 55 specifies who must perform those tasks. That fact, in conjunction with the general improvement in the performance level of power reactor facility licensees' training programs, prompted the NRC staff to develop an alternative approach under which nuclear power plant licensees would prepare the examinations and submit them to the NRC for review and approval. This approach has been tested and assessed through a voluntary pilot program and has been deemed by the NRC staff to be feasible. The monitoring and assessment of this voluntary pilot program has demonstrated that examinations prepared by facility licensees and modified as directed by the NRC are comparable in terms of their quality and discrimination validity to those prepared by the NRC and its contract examiners under the traditional process; therefore, the safe operation of the nuclear power plants is not compromised by using the alternative approach.

Based on the success of the pilot program, the NRC is amending 10 CFR Part 55 so that nuclear power plant licensees may elect whether to prepare these examinations or have them prepared by the NRC. The NRC is also publishing a final version of Revision 8 of NUREG-1021, "Operator Licensing Examination Standards for Power Reactors," for use by the NRC and facility licensees in preparing, administering, and grading the examinations and tests under the revised process.

This change in policy is part of the NRC's continuing effort to streamline the functions of the Federal government consistent with the Administration's initiatives and to accommodate NRC resource reductions. Pursuant to the provisions of the AEA, the NRC will ensure that the quality of the operator licensing examinations and the effectiveness of the operator licensing program are maintained. These changes are not intended to affect the format, content, scope, quality and level of difficulty of the examinations, thereby minimizing the impact of the rule change on the operator license applicants.

2. Background

10 CFR 55.31(a)(3) requires the applicant for an operator's license to submit a written request from an authorized representative of the facility licensee that the written examination and the operating test be administered to the applicant. Furthermore, 10 CFR 55.33(a)(2) states that the Commission will approve an initial application for a license if it finds that the applicant has passed the requisite written examination and operating test in accordance with ��55.41 and 55.45 for reactor operators, or ��55.43 and 55.45 for senior operators. These written examinations and operating tests determine whether the applicant for an operator's license has learned to operate a facility competently and safely, and additionally, in the case of a senior operator, whether the applicant has learned to direct the licensed activities of licensed operators competently and safely.

As stated above, the NRC or its contract examiners have traditionally prepared, administered, and graded the written examinations and operating tests. In accordance with 10 CFR 170.12 (i), the NRC staff and contractual costs are recovered from the facility licensees that receive examination services. Before beginning the pilot program, the NRC spent approximately $3 million per year to retain contractor support for the operator licensing programs (including initial licensing examinations and requalification inspections). However, the increased efficiency of the revised examination process has enabled the NRC to eliminate the use of contractors in the operator licensing program and conduct the initial operator licensing and requalification inspection programs with the existing NRC staff. During fiscal year (FY) 1996, when approximately 40 percent of the examinations were voluntarily prepared by facility licensees, the NRC spent approximately $1.6 million for contractor assistance (to prepare and administer initial examinations and assist with requalification inspections). In FY 1997, facility participation increased to include approximately 75 percent of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million. The FY 1998 and FY 1999 budgets reflect the complete elimination of contractor support for the operator licensing program (with the exception of the generic fundamentals examination).

Based on the success of the pilot program, the NRC originally proposed to amend 10 CFR Part 55 to require all power reactor facility licensees to prepare their own operator licensing examinations and to proctor and grade the written portion of those examinations. The staff took the position in the proposed rulemaking that the backfit rule, 10 CFR 50.109, did not apply because the proposed shift in responsibility for preparing the examinations would not constitute a "modification of the procedures required to operate a facility" within the scope of the backfit rule. Moreover, the rule would not have affected the basic procedures for qualifying licensed operators, nor would it have required facility licensees to alter their organizational structures. However, based upon further review since issuing the proposed rulemaking, the NRC has concluded that there is insufficient basis to support the original position. Therefore, the NRC has decided to revise the final rulemaking so that power reactor facility licensees may elect to prepare their written examinations and operating tests (and proctor and grade the written examinations) in accordance with NUREG-1021 or to have the NRC prepare the examinations. Eliminating the requirement for all facility licensees to prepare their examinations and tests has obviated the need for a backfit analysis.

3. Identification and Analysis of Three Alternative Approaches

3.1 Alternative 1 - Take No Action

As discussed above, the budget for FYs 1998 and 1999 already reflects the elimination of contract support as a result of facility licensees' voluntary participation in the revised examination process. Therefore, if 10 CFR Part 55 is not amended (and the NRC terminates the voluntary pilot program), it will require the NRC to take other actions to enable NRC examiners to satisfy power reactor facility licensees' requirements for licensed operators and to conduct the requalification inspection program. Such actions might include the following: (1) changing the format or length of the examinations to reduce the NRC examiners' workload; (2) reconsidering the possibility of recentralizing the operator licensing function to improve efficiency and effectiveness; and (3) increasing the direct examiner resources budgeted for the operator licensing program.

3.2 Alternative 2 - Provide Regulatory Guidance

Before initiating the proposed rulemaking, the NRC staff had considered the possibility of implementing the revised examination process by issuing a supplement to Generic Letter 95-06, "Changes in the Operator Licensing Program," which on August 15, 1995, had outlined the revised examination process and solicited volunteers to participate in the pilot program. This alternative was rejected in favor of amending 10 CFR Part 55 to require all power reactors to prepare their own licensing examinations because the NRC staff considered implementation of the new process on a voluntary basis less desirable over the long term. Allowing rather than requiring facility licensees to prepare the initial operator licensing examinations could affect the NRC staff's ability to meet the examination demands and scheduling needs of facility licensees, depending on the demand for examinations and the percentage of licensees that elect to prepare their own examinations. The resource problem is further compounded by the elimination of contractor support for the operator licensing program and by other unanticipated demands on the examiner work force (e.g., examination appeals).

In the proposed rulemaking, the NRC took the position that the backfit rule did not apply to this rulemaking. However, it has concluded upon further review since issuing the proposed rule, that there is insufficient basis to support that position. Consequently, the NRC has reconsidered its options and decided to implement the revised examination process on an elective basis. The merits of the revised process (e.g., improved regulatory efficiency and greater licensee control over the examination costs) justify proceeding with a rule change that gives facility licensees the option to prepare the examinations, rather than terminating the pilot program and resuming the NRC-prepared examination process on an industrywide basis.

3.3 Alternative 3 - Amend 10 CFR Part 55

This alternative, as implemented by the proposed rulemaking, would have required every power reactor facility licensee to prepare the initial operator licensing examinations and to proctor and grade the written portion of the examination.

However, as noted in the discussion of Alternative 2 above, the NRC staff has concluded, since publishing the proposed rule, that there is insufficient basis to support the position that the backfit rule does not apply to this rulemaking. Therefore, the NRC has decided to withdraw from this alternative and revise the final rulemaking so that power reactor facility licensees may elect to prepare their written examinations and operating tests (and proctor and grade the written examinations) in accordance with NUREG-1021 or to have the NRC prepare the examinations.

4. Regulatory Impact - Qualitative Costs and Benefits

Facility Licensees

Before beginning the pilot examination program, the NRC depended on NRC employees and contractors to prepare and administer the initial operator licensing examinations required by 10 CFR Part 55. NRC contractors also assisted in the inspection of requalification programs administered by facility licensees. In accordance with 10 CFR 170.12 (i), the cost of NRC time spent and any related contractual costs were (and are) billed directly to the facility licensees that receive(d) the examination services.

Under the revised examination process as implemented by this final rulemaking, each power reactor facility licensee will be able to elect whether to prepare the site-specific initial operator licensing examinations at its facilities or have the NRC prepare those examinations. Facility licensees that elect to prepare their own examinations will be required to prepare the proposed examinations (including the written examination, the walk-through, and the dynamic simulator tests) based on the guidance in NUREG-1021 and to submit the examinations to the NRC for review and approval. Facility licensees will have the option to hire a contractor to prepare the license examination (as the NRC often did before starting the pilot process).

As provided in NUREG-1021, licensees may identify differences from the NUREG-1021 examination criteria and evaluate how the proposed alternatives provide an acceptable method of complying with the Commission's regulations. The NRC staff will review any proposed alternatives and make a decision regarding their acceptability. The NRC will not approve any alternative that would compromise its statutory responsibility of prescribing uniform conditions for the operator licensing examinations.

This rule change gives facility licensees that elect to prepare their own examinations more control over the cost of their examination services because it puts them in a position to manage the quality of the product that is submitted to the NRC. The higher the quality of the examination that the facility licensee submits, the lower the resulting NRC charges. If the NRC or one of its contractors writes an examination, the facility licensee is responsible for the entire cost of preparing the examination.

The NRC will review the examinations prepared by the facility licensees consistent with the guidance in NUREG-1021 and will direct the facility training staffs to make whatever changes are necessary to achieve a product that meets the NRC's standards for quality and level of difficulty. Each participating facility licensee will be billed only for the time that the NRC staff spends to review the examination prepared by the facility licensee, to direct the revisions that are necessary for the examination to meet NRC standards, and to administer and grade the operating tests.

After the NRC reviews and approves a facility-prepared examination, the facility licensee will be required to proctor and grade the written portion according to the guidance in NUREG-1021. The NRC staff will continue to administer and grade the operating tests, review and approve the written examination results recommended by the facility licensee, and make the final licensing decisions.

Based on lessons learned and feedback from the pilot examinations completed at the time, the proposed regulatory analysis predicted that the average time spent by a facility licensee to prepare the written examination and operating tests would be approximately 600 to 800 staff hours. Because a portion of that time (about 200 hours) would be spent reviewing and assisting with the administration of NRC-developed examinations under the traditional examination process, 200 hours should be subtracted from the total. The resulting average burden of approximately 400 to 600 staff hours was somewhat higher than the 400 hours that the NRC staff or its contract examiners typically take to prepare an examination. The extra burden was generally attributable to the facility licensees' unfamiliarity with specific NRC examination expectations and to the additional administrative tasks, such as documenting the source of the examination questions, required to maintain examination integrity. The NRC staff believed that the facility training staffs already had the basic knowledge, skills, and abilities necessary to evaluate operator performance and develop test items for the initial licensing examination and expected that the burden would diminish as facility training staffs gained familiarity and experience with the NRC's specific examination requirements and expectations. Moreover, the NRC staff expected that the facility employees' more detailed knowledge of their facility and easier access to the reference materials required to prepare the examinations would eventually enable them to prepare acceptable quality examinations in less time than the NRC or a contractor could.

In an effort to quantify the potential cost savings, the proposed regulatory analysis assumed that facility licensees would eventually prepare the examinations in the same amount of time as the NRC allotted its contract examiners to perform the task (i.e., approximately 400 hours), and that the time would be equally distributed between contractors and the licensee's own in-house staff at a cost of $120 per hour and $60 per hour, respectively. This translated into an industrywide burden of $2.16 million, assuming 60 site-specific licensing examinations per year with all facility licensees preparing their own examinations in accordance with the proposed rulemaking, and was roughly the same as the cost would be if the NRC were to prepare the examinations relying equally on in-house and contractor staff efforts. The NRC staff had no basis for estimating the cost savings that might be gained if facility licensees availed themselves of the aforementioned efficiencies, but a 10 percent reduction in the burden would result in a small industrywide savings of about $220,000 per year. As an elective program, the yearly industry cost savings will be even more difficult to predict because of variations in the total demand for licensing examinations, the percentage of those examinations that are prepared by facility licensees and the NRC, and the efficiency with which each participating facility licensee is able to prepare its examinations.

The proposed rulemaking that was published in the Federal Register (62 FR 42426) on August 7, 1997, specifically requested public comments on the NRC staff's conclusion in the proposed regulatory analysis that it would eventually be less costly for each facility licensee to prepare its own initial operator examinations rather than to have an NRC contractor prepare the examinations for all licensed operators with the costs reimbursed by licensee fees. The proposed rulemaking also invited public comments on interim Revision 8 of NUREG-1021, which defines the NRC's examination criteria and expectations. Seven respondents directly or indirectly addressed the resource question in their comment letters, and seven respondents submitted specific comments and recommendations related to NUREG-1021.

The Nuclear Energy Institute (NEI) and one utility stated that the revised examination criteria in interim Revision 8 of NUREG-1021 have increased the level of effort and will result in higher licensing fees regardless of who prepares the examinations. However, NEI and another utility agreed that comparing the cost of utility-prepared examinations to those prepared by the NRC is difficult. They concluded that it should be less costly for utilities to prepare the examinations than to have the NRC prepare them, provided the preparers are using exactly the same criteria. NEI also stated that the relative cost of the two examination processes should not be the only factor in deciding whether to proceed with the rulemaking. NEI indicated that preparing higher-cognitive level questions requires detailed plant knowledge better known to licensees and that the revised process will allow NRC staff to concentrate on directly evaluating each applicant without relying on third-party observers.

Two NRC examiners, one contract examiner, and a utility employee asserted that the facility licensees' cost has increased under the revised examination process. They cited various reasons for the increased cost, including training personnel to write the examinations and then restricting them from training the applicants and upgrading equipment to maintain examination security. The NRC examiners based their opinions on informal feedback from facility training personnel. One examiner indicated that it was taking facility licensees an average of 700 hours to prepare each examination. The utility employee stated that the rule will simply transfer the cost of contractors from the NRC to the utilities.

The NRC staff acknowledges that the revised examination and administrative criteria in interim Revision 8 of NUREG-1021 have probably caused the cost of the examinations to be somewhat higher than it would have been if facility licensees had been allowed to prepare the examinations using the same criteria that applied to the NRC and its contractors before starting the pilot program. The staff has reevaluated the revised criteria in light of the public comments and lessons learned during the pilot examination program and concluded that the criteria remain necessary to mitigate the vulnerabilities (e.g., quality, security, and conflict of interest) inherent in the revised examination process and to facilitate the NRC staff's review of the proposed examinations. The staff has incorporated several minor changes (e.g., allowing instructors to prepare written examination questions on subjects they did not teach, without regard to the amount of time they spent training the applicants) and clarifications in the final version of Revision 8 of NUREG-1021. However, the changes are not expected to significantly affect the cost of preparing an examination.

Despite the revised examination and administrative criteria, many of the facility licensees that participated in the pilot program demonstrated that it is possible to prepare an acceptable quality examination at the same or lower cost than the NRC or its contractors could prepare a comparable examination. However, approximately 40 percent of proposed pilot examinations have not met NRC standards for quality and have required significant changes. The fact that the lower quality examinations take longer to review and require significant rework by the facility licensees has driven up their total cost and caused the staff to question the validity of its original expectation that the industry may realize additional cost reductions as it gains experience with the NRC's examination requirements.

As noted earlier, the proposed regulatory analysis assumed that the training personnel at power reactor facilities already have the basic expertise necessary to develop test items for the initial licensing examination. During the second half of the 1980s, the industry increased its emphasis in the training area, and all power reactor licensees established formal training programs that were based on a systems approach to training (SAT) and were accredited by the National Academy for Nuclear Training. Moreover, pursuant to 10 CFR 50.120 and 55.4, SAT-based training programs must evaluate the trainees' mastery of training objectives, and NRC inspections of licensee requalification programs for licensed operators confirmed that training staffs generally possessed the skills needed to evaluate the trainees' knowledge.

However, the public comments on the proposed rule, comments from NRC examiners involved with the pilot examinations, and the number of significant changes required on many of the pilot examinations have raised concerns about the capability of some of the facilities to write the licensing examinations. The NRC staff realizes that the restrictions on the use of instructors to prepare the licensing examinations may have prompted some facility licensees having smaller training staffs to use less-qualified personnel to prepare the examinations. And although the NRC continues to believe that personnel restrictions are necessary to address the potential for conflicts of interest in the examination development process, it has reconsidered the scope of the restrictions in interim Revision 8 of NUREG-1021 in an effort to reduce the potential burden on facility licensees. Consequently, final Revision 8 will allow facility instructors to prepare the written examination questions without regard to the amount of time they spent training the license applicants. However, final Revision 8 will still preclude instructors from preparing questions related to those topics on which they provided instruction and from instructing the applicants once they begin working on the licensing examination (as was the case in interim Revision 8).

Additionally, the fact that the NRC has decided to implement the revised examination process on an elective basis rather than require all facility licensees to prepare their licensing examinations will minimize the effect of the personnel restrictions by affording facility licensees with small training staffs the option of having the NRC prepare their licensing examinations. This change is also consistent with NEI's comment on the proposed rulemaking that the industry would prefer to continue the voluntary process because it would allow flexibility for a few facility licensees with small training staffs.

It is unclear to what extent the examination quality has been affected by the training and qualification of the examination writers, but the NRC staff has asked the industry to address this issue and to determine the need for additional standards for training on testing and measurement principles and examination development techniques. The cost of this initiative is uncertain at this time, but the benefits would be realized throughout the facility licensees' training programs and not just in the initial operator licensing program.

With regard to the additional security costs cited in the public comments, it is important to note that although the NRC has stressed the seriousness of maintaining examination security, the staff has not required that facility licensees invest in additional physical security systems. The NRC's expectations regarding the facility licensees' responsibility for maintaining examination security were outlined in draft Revision 8 of NUREG-1021, which was issued for public comment in February 1996, and subsequently clarified in interim Revision 8 of NUREG-1021, which was issued for use in February 1997. The security incidents that occurred during the pilot program and the public comments on the proposed rule prompted the NRC to include additional security guidelines in final Revision 8 of NUREG-1021; however, the guidelines are instructional in nature and do not require any security system improvements. The security incidents that occurred during the pilot program have also prompted the NRC to conclude that applicants, licensees, and facility licensees may not understand the provisions of �55.49, "Integrity of Examinations and Tests," pertain to more than just those activities involving the physical administration of an examination. Therefore, the NRC has determined that it would be beneficial to amend �55.49 in the final rule to clarify the intent of the regulation. Moreover, the NRC has revised the final examination rulemaking to state clearly that facility licensees that elect to prepare their own licensing examinations are required to establish, implement, and maintain procedures to control examination security and integrity. The NRC staff believes that most facility licensees have already established examination security policies and practices, so the additional cost of documenting those policies and practices in formal procedures should not be significant. Establishing formal security procedures should raise facility employees' awareness of this important issue and could avert future security incidents that might require examinations (or portions of the examinations) to be delayed and replaced at significant expense to the affected facility licensees.

As noted in the proposed regulatory analysis, the revised examination process will eliminate the need for facility licensees to duplicate and ship multiple sets of reference materials because NRC contract examiners (who would have required a separate copy of the reference material) will not be used to prepare the examinations. Feedback from the industry in response to the NRC staff's solicitation of public comments on the draft revision of NUREG-1021 (refer to 61 FR 6869, of February 22, 1996) indicated that facility licensees had been spending an additional 80 to 160 hours to prepare and ship the reference materials under the existing examination process. Under the revised process, facility licensees will generally submit only materials that are needed for the NRC chief examiner to verify the accuracy of the examination questions. This is a reduction of resources, but it has not been quantified in this analysis.

In summary, before beginning the pilot program, the NRC relied on its own staff and contractors to write the site-specific operator licensing examinations, and the cost was billed to the facility licensees in accordance with 10 CFR 170.12 (i). To prepare and administer such examinations, the NRC staff and contractors had to learn the details of the operation of each specific plant. To prepare the examinations, the examining staff had to, in effect, duplicate the technical expertise already present at each site. This rule will reduce this inefficiency by allowing each power reactor facility licensee to elect whether to prepare its own licensing examinations or have them prepared by the NRC. Efficiency should be gained if the NRC focuses its efforts on maintaining the appropriate scope, depth, and quality of the examinations, leaving the preparation of the detailed examination materials to the facility licensees.

The NRC expects that the initial period of inefficiency will continue until facility licensees learn the process for preparing the examinations, expand and improve their examination question banks, and upgrade the capabilities of their training staffs, as necessary. Experience during the pilot examination program has demonstrated that the quality of the examinations and the efficiency of the process will be slow to improve. It may take considerable time and effort to develop a bank of written examination questions appropriate for use on the initial licensing examination, but such a bank should improve the quality of future examinations and should be more efficient. The NRC still expects that, with experience and training, facility licensees will be able to prepare acceptable quality examinations that require less NRC review resources and provide larger potential savings to the licensees. Those facility licensees that submit acceptable quality examinations are likely to save resources despite the additional administrative criteria that the NRC considers necessary under the revised examination process. However, those facility licensees that submit examinations that require many changes because they do not meet NRC quality standards, will likely end up with a more costly examination than if it had been prepared by the NRC.

Implementing the revised examination process on an elective basis has obviated the need for a backfit analysis because it will allow each facility licensee to assess the costs and benefits of preparing its own examinations in accordance with the criteria in NUREG-1021 with the costs and benefits of having the NRC staff prepare the examinations (as has been the traditional practice). The yearly industry cost savings will depend upon the total demand for licensing examinations, the percentage of those examinations that are prepared by facility licensees and the NRC, and the efficiency with which each participating facility licensee is able to prepare its examinations.

Operator License Applicants

To the extent possible, the format, content, quality, and level of difficulty of the examinations should remain unchanged, thereby minimizing the impact of the rule change on the operator license applicants. NRC examiners will continue to review, to direct revisions where necessary, and approve every facility-prepared written examination and operating test before it is administered. The NRC acknowledges that some of the facility-prepared examinations will be more difficult than others and that some of the variation can likely be traced to individual differences among the NRC examiners who review and approve the examinations. This was and is also the case when the NRC prepares the examinations. However, the NRC believes that the variation in the level of difficulty can be maintained within acceptable limits and would be even greater if NRC examiners do not take action to ensure that the submitted examinations are consistent with NRC standards.

The average reactor operator (RO) and senior operator (SRO) passing rates on both the written and operating portions of the 102 facility-prepared examinations administered through the end of June 1998 were only slightly lower than the corresponding passing rates on the NRC-prepared examinations administered during FY 1995, the last year in which all of the examinations were prepared by the NRC or its contractors. However, the average passing rates tend to fluctuate, and the pilot examination results were comparable to the range of passing rates for FYs 1990 through 1994. The average passing rates for ROs on the facility-prepared operating tests during FYs 1996 and 1997 were consistent with the passing rates on the tests prepared by the NRC during those same years. Although the passing rate on the facility-prepared SRO operating tests during FY 1997 was slightly lower than in FY 1995, it was the same as the passing rate on the NRC-prepared operating tests during FY 1996.

The NRC staff believes that the lower passing rate on the facility-prepared written examinations for reactor operators and senior operators may be a reflection of the NRC examiners' increased concentration on the psychometric quality (e.g., the cognitive level at which the questions are written and the plausibility of the distractors or wrong answer choices) in addition to the technical quality of the examinations. Although the NRC did not intend for the level of difficulty or the failure rate of the examinations to increase, it is possible that the NRC examiners' efforts to achieve NRC standards regarding the cognitive level of the questions and the plausibility of the distractors have affected the discrimination validity of the examinations. Consequently, those marginal applicants who may have passed an examination on which two of the distractors could often be eliminated as implausible are now having more difficulty. However, considering the historical fluctuation in the average examination passing rates and the other factors (e.g., training program quality and screening of applicants by facility licensees) that could be responsible for some or all of the observed decline, the staff has concluded that any increase in the level of difficulty is not significant.

If the NRC prepares the examination in lieu of the facility licensee, the NRC examiners will use the same procedures and guidance (i.e., NUREG-1021) that the facility licensees would use to prepare the examinations.

NRC Staff

When the staff developed the pilot examination process, it estimated that it would take approximately 370 examiner-hours to review and otherwise prepare for, administer, grade, and document an average facility-prepared operator licensing examination. Although a number of the proposed examinations required more NRC resources than expected to achieve NRC's standards, the burden was generally offset by other examinations that required less effort to review and revise.

The increased efficiency of the revised examination process has enabled the NRC to eliminate the use of contractors in the operator licensing program and conduct the initial operator licensing and requalification inspection programs with the existing NRC staff. Before initiating the pilot examination and transition program at the beginning of FY 1996, the NRC spent approximately $3 million per year for contractor assistance to prepare and administer initial examinations and assist with requalification inspections. During FY 1996, when approximately 40 percent (27/68) of the examinations were voluntarily prepared by facility licensees, the NRC spent approximately $1.6 million for contractor assistance in those areas. In FY 1997, facility participation increased to include approximately 75 percent (41/55) of the examinations, and the NRC's spending on contractor assistance for the licensing examinations and requalification inspections decreased to approximately $0.5 million (approximately 3 full-time equivalents (FTEs)). During FY 1997, the NRC expended approximately 16.8 staff FTEs to conduct all the site-specific initial operator licensing examinations, plus 3.8 staff FTEs for generic operator licensing activities. The licensed operator requalification inspection program required approximately 4.5 additional staff FTEs (plus a minimal level of contractor support), bringing the total resources used for the operator licensing program in FY 1997 to approximately 28.1 FTEs. This actual resource burden is generally consistent with the estimates prepared before beginning the pilot examination program and supports the staff's conclusion that the revised examination process is an efficient alternative to the traditional NRC-prepared examinations.

Experience during the pilot program has shown that the quality of the facility-prepared examinations can vary widely, making it difficult to predict the amount of time necessary for the NRC to review and revise any particular examination in order to meet NRC standards. Although the staff had expected the quality of the examinations to improve over time, some of the proposed examinations continue to require more changes than anticipated. The added examiner workload could increase the possibility of lower examination quality, raise the NRC's cost, and affect the staff's ability to satisfy the facility licensees' needs for licensing examinations. As noted in SECY-97-079, NRR has issued a memorandum to the regional administrators emphasizing the importance of: (1) assigning adequate resources to carry out the operator licensing task; (2) completing a thorough review of every facility-prepared examination; and (3) not administering any examination that does not meet NRC standards for quality and level of difficulty.

As noted in the discussion of alternative approaches, implementing the revised examination process on an elective basis will complicate the task of resource management and may affect the NRC staff's ability to meet the examination demands and scheduling needs of facility licensees because the number of licensees that elect to prepare their own examinations could fluctuate from year to year. For purposes of comparison, the average NRC-prepared examination requires approximately two-thirds of an FTE to prepare, administer, and document, while the average facility-prepared examination requires about one-third of an FTE to review, administer, and document. Therefore, with the 17 regional FTEs that are budgeted for the initial examination program in FYs 1999 and 2000, the NRC staff will be able to prepare about 25 to 30 examinations per year, while it would be able to review about 50 to 55 facility-prepared examinations. During the two full years in which facility licensees have been allowed to prepare their examinations under the voluntary pilot program, the level of industry participation has ranged from about 75 percent (41/55) in FY 1997 to about 90 percent (50/56) in FY 1998. Therefore, if the current level of industry participation is maintained, the staff expects that it will be able to satisfy most facility licensees' examination demands and scheduling needs.

To manage the workload, the NRC staff will resume issuing an annual administrative letter that will request facility licensees to report their anticipated examination needs for the next four fiscal years and their intentions with regard to preparing each examination. Moreover, the staff intends to work with NEI in an effort to coordinate the examination schedule and address workload conflicts. If the level of industry participation declines, other actions will be needed to enable NRC examiners to satisfy power reactor facility licensees' requirements for licensed operators and to conduct the requalification inspection program. Such actions might include the following: (1) changing the format or length of the examinations to reduce the NRC examiners' workload; (2) reconsidering the possibility of recentralizing the operator licensing function to improve efficiency and effectiveness; and (3) assigning and training additional examiners to accommodate the increased workload.

5. Decision Rationale

The amendments to 10 CFR Part 55 will allow power reactor facility licensees to elect whether to prepare the entire initial operator licensing examinations and to proctor and grade the written portion of the examinations or to have the NRC perform those tasks. The qualitative assessment of costs and benefits discussed above, leads the NRC to conclude that the overall impact of the rulemaking will not significantly increase licensee costs and may eventually result in a savings to licensees that elect to prepare their own examinations as they become more familiar with the NRC examination guidelines. Any improvements in efficiency will likely be due, in part, to the facility employees' better understanding of the plant design and operating characteristics and their ready access to the reference materials required to prepare and validate the examinations. The apparent need of some facilities to improve their examination development capabilities in order to achieve the expected standards for examination quality may delay the realization of cost savings for some facility licensees that elect to prepare their own examinations and may limit the overall industry cost benefit. Facility licensees that do not prepare examinations that meet the NRC's quality standards will likely derive no cost benefit from this action because of the additional effort required to review and rewrite the examinations. Moreover, facility licensees that elect not to prepare their own examinations will derive no cost benefit from this action.

The voluntary pilot program has demonstrated that the revised examination process is effective as well as efficient. The pilot examinations, revised as directed by the NRC, were comparable in terms of their quality to examinations prepared by the NRC and its contract examiners. The overall passing rate and average written examination grades on the 102 pilot examinations administered through the third quarter of FY 1998 were the same as, or slightly lower than, those on the power reactor licensing examinations during FY 1995, when all the examinations were prepared by the NRC or its contractors. Therefore, the safe operation of the facilities was in no way compromised.

The NRC staff believes that its ability to focus on operator performance and its examiner experience may improve as a result of this action because every applicant will be directly observed by an NRC employee. Before beginning the transition process, contract examiners administered about half of the operating tests and collected the observations that formed the basis for the NRC's licensing actions, but the increased efficiency of the revised process has enabled the NRC to eliminate the use of contractors as of FY 1998. Consequently, the revised process will enable the NRC examiners to gain more experience in a shorter period of time and may improve the consistency of the operating test evaluations and the licensing decisions. Although the total number of procedures reviewed in the process of examination development may decrease under the revised method, NRC examiners are still expected to review and identify discrepancies in the procedures that will be exercised during the plant walk-through test and the dynamic simulator scenarios.

Although the proposed rulemaking had endorsed Alternative 3 as the preferred method for implementing the revised examination process, the NRC has determined, based upon further review, that there is insufficient basis to support the earlier position that the backfit rule does not apply to this action. Consequently, the staff has reconsidered the alternatives and decided to implement the revised examination process on an elective basis (Alternative 2). The merits of the revised process justify proceeding with a rule change that gives facility licensees the option to prepare the examinations, rather than terminating the pilot program and resuming the NRC-prepared examination process on an industrywide basis. This action has the potential to save facility licensees money if they become proficient in preparing the examinations, has negligible impact upon operator license applicants, provides a substantial cost savings to the NRC, and has the potential for an improvement in the staff's examination experience and ability to focus on operator performance.

6. Implementation Schedule

The rule will become effective 180 days after the date it is published in the Federal Register. No effect on other schedules is anticipated.

ATTACHMENT 4

[7590-01-P]

NUCLEAR REGULATORY COMMISSION
[NUREG-1600, Rev. 1]
Revision of NRC Enforcement Policy

SUMMARY: The Nuclear Regulatory Commission (NRC) is amending its "General Statement of Policy and Procedure for NRC Enforcement Actions" (Enforcement Policy) to conform to the amendments to the regulations that govern operators' licenses published in this issue of the Federal Register as a separate action. Those amendments allow nuclear power facility licensees to prepare, proctor, and grade the written examinations and prepare the operating tests that the NRC uses to evaluate the competence of individuals applying for operator licenses at the facility licensees' plants. Moreover, the amendment requires facility licensees that elect to prepare their own examinations to establish, implement, and maintain procedures to control examination security and integrity, and it clarifies the regulations to ensure that applicants, licensees, and facility licensees understand what it means to compromise the integrity of a required test or examination. Therefore, the Enforcement Policy is being amended to add examples of violations that may be used as guidance in determining the appropriate severity level for violations involving the compromise of applications, tests, and examinations.

EFFECTIVE DATE: This action is effective [insert date of publication in the Federal Register], while comments are being received. Submit comments on or before [insert date 30 days from date of publication in the Federal Register].

ADDRESSES: Submit written comments to: David Meyer, Chief, Rules Review and Directives Branch, Office of Administration, Mail Stop: T6D59, U. S. Nuclear Regulatory Commission, Washington, DC 20555-0001. Hand deliver comments to: 11555 Rockville Pike, Rockville, Maryland, between 7: 45 am and 4: 15 pm, Federal workdays. Copies of comments received may be examined at the NRC Public Document Room, 2120 L Street, NW, (Lower Level), Washington, DC 20555-0001. Copies of NUREG-1600 may be purchased from the Superintendent of Documents, U.S. Government Printing Office, Mail Stop SSOP, Washington, DC 20402-9328. Copies are also available from the National Technical Information Service, 5285 Port Royal Road, Springfield, Virginia 22161. Copies are also available for inspection and copying for a fee in the NRC Public Document Room.

FOR FURTHER INFORMATION CONTACT: James Lieberman, Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; telephone: (301) 415-2741; e-mail: jxl@nrc.gov.

SUPPLEMENTARY INFORMATION:

• Paperwork Reduction Act
• Public Protection Notification
• Small Business Regulatory Enforcement Fairness Act
• APPENDIX B: SUPPLEMENTS - ENFORCEMENT EXAMPLES
• SUPPLEMENT I--REACTOR OPERATIONS

The Commission's "General Statement of Policy and Procedure for NRC Enforcement Actions" (Enforcement Policy) was first issued on September 4, 1980. Since that time, the Enforcement Policy has been revised on a number of occasions. On May 13, 1998 (63 FR 26630), the Enforcement Policy was revised in its entirety and was also published as NUREG-1600, Rev. 1. The Enforcement Policy primarily addresses violations by licensees and certain non-licensed persons, as discussed further in the Enforcement Policy in footnote 3 to Section I, "Introduction and Purpose," and in Section X, "Enforcement Action Against Non-Licensees."

By a separate action published in this issue of the Federal Register, the NRC is amending its regulations in 10 CFR Part 55 to allow nuclear power facility licensees to prepare, proctor, and grade the written examinations and prepare the operating tests that the NRC uses to evaluate the competence of individuals applying for operator licenses at the facility licensees' plants. Section 107 of the Atomic Energy Act (AEA) of 1954, as amended, requires the NRC to determine the qualifications of individuals applying for operator licenses, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Pursuant to the AEA, 10 CFR Part 55 requires applicants for operator licenses to pass an examination that satisfies the basic content requirements specified in the regulation. Because the NRC considers the integrity of the licensing tests and examinations to be essential to the safe operation of nuclear facilities, the NRC is also amending 10 CFR 55.49 to clarify that the integrity of a test or examination required by 10 CFR Part 55 is considered compromised if any activity, regardless of intent, affected, or but for detection, would have affected the equitable and consistent administration of the test or examination. Moreover, the NRC is amending 10 CFR Part 55 to require power reactor facility licensees that elect to prepare their own examinations to establish, implement, and maintain procedures to control examination security and integrity.

The NRC intends to use its enforcement authority to emphasize that a compromise of an application, test, or examination required by 10 CFR Part 55 cannot be accepted. Therefore, the NRC is amending the Enforcement Policy by adding examples of violations in Supplement I, "Reactor Operations," to provide guidance in determining the appropriate severity level for violations involving the compromise of an application, test, or examination used to evaluate the competence of individuals applying for operator licenses or to evaluate the continued competence of licensed operators. A compromise of an application, test, or examination that is not detected before the test or examination is administered is considered a significant regulatory concern and normally categorized at least at Severity Level III. Willful acts to compromise an application, test, or examination add to the significance. Therefore, a willful compromise of an application, test, or examination that was detected before the test or examination was administered would also be normally categorized at least at Severity Level III. A willful compromise that was not detected before the test or examination was administered would be normally categorized at a Severity Level I or II depending on the circumstances. A compromise that was not willful and was discovered before a test or examination was given would be normally categorized at Severity Level IV, as would other violations of 10 CFR 55.49 that are of more than minor concern, such as failures to establish, implement, and maintain procedures to control the security of the examination process or failures to take adequate corrective action in response to a previous compromise.

Notwithstanding this guidance, the NRC may choose to exercise discretion in accordance with Section IV, "Severity of Violations," and Section VII, "Exercise of Discretion," of the Enforcement Policy and either escalate or mitigate the severity level and enforcement sanction for a particular violation involving the compromise of an application, test, or examination required by 10 CFR Part 55, based on the circumstances of the case, to ensure that the resulting enforcement action appropriately reflects the level of NRC concern regarding the issue and conveys the appropriate message to the individuals (i.e., applicants, licensees, or other) or facility licensees involved. Circumstances to be considered include factors such as the timing of the compromise, when the compromise was detected, the number of individuals involved, the scope of the compromise, the advantage received as a result of the compromise, the degree of willfulness, if any, programmatic aspects, and the facility licensee's response to the compromise. In addition, depending on the circumstances of the case, the severity level of the violation may differ between the individual and the facility licensee. The NRC intends to utilize its enforcement authority, as warranted, and issue notices of violation, civil penalties, and orders to individuals and facility licensees who (1) compromise an application, test, or examination in violation of 10 CFR 55.49, (2) commit deliberate misconduct in violation of 10 CFR 50.5, or (3) provide incomplete or inaccurate information to the NRC in violation of 10 CFR 50.9. In addition, cases involving willful violations may be referred to the Department of Justice for criminal prosecution.

Paperwork Reduction Act

This policy statement does not contain a new or amended information collection requirement subject to the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq.). Existing requirements were approved by the Office of Management and Budget, approval number 3150-0136. The approved information collection requirements contained in this policy statement appear in Section VII.C.

Public Protection Notification

The NRC may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

Small Business Regulatory Enforcement Fairness Act

In accordance with the Small Business Regulatory Enforcement Fairness Act of 1996, the NRC has determined that this action is not a major rule and has verified this determination with the Office of Information and Regulatory Affairs of OMB.

Accordingly, Supplement I--Reactor Operations of Appendix B of the NRC Enforcement Policy is revised to read as follows:

APPENDIX B: SUPPLEMENTS - ENFORCEMENT EXAMPLES

* * * * *

SUPPLEMENT I--REACTOR OPERATIONS

This supplement provides examples of violations in each of the four severity levels as guidance in determining the appropriate severity level for violations in the area of reactor operations.

C. Severity Level III - Violations involving for example:

* * * * *

5. A compromise of an application, test, or examination required by 10 CFR Part 55 that is not detected before the test or examination is administered, or a willful compromise that is detected before the test or examination is administered.

D. Severity Level IV - Violations involving for example:

* * * * *

2. A compromise of an application, test, or examination required by 10 CFR Part 55 that is discovered before the test or examination is administered, or other failures to meet 10 CFR 55.49 that are of more than minor concern.

Dated at Rockville, MD, this _______ day of ______________ , 1998.

ATTACHMENT 5

The Honorable Dan Schaefer, Chairman
Subcommittee on Energy and Power
Committee on Commerce
United States House of Representatives
Washington, DC 20515

Dear Mr. Chairman:

The NRC has sent the enclosed amendments to the Commission's rules in 10 CFR Part 55 to the Office of the Federal Register for publication. This rulemaking will allow facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor and grade the written portion of the examination. Facility licensees that elect to prepare their own examinations will be required to submit the examinations to the NRC for review and approval and establish procedures to control examination security and integrity. This rulemaking requires the NRC to prepare those examinations and tests not prepared by facility licensees and preserves the NRC's authority to prepare the examinations and tests, as necessary.

Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55, "Operators' Licenses," to pass an examination satisfying the basic content requirements that are also specified in the regulation. Although the AEA is not specific as to who will prepare and administer the examination, the NRC has traditionally performed these functions through its staff or contract examiners. The NRC is increasing power reactor facility licensees' role in the examination development process in recognition of the improvements in industry training programs, to make the operator licensing program more efficient, and to realize budgetary savings for the Federal Government.

The NRC will continue to (1) prepare and administer the generic fundamentals examination to operator license applicants, (2) review every facility-prepared examination, direct changes as required to achieve NRC standards, and approve the examination before it is given, (3) administer and grade the operating test portion of the examination, including a control room simulator examination and a one-on-one demonstration of specific operating tasks, and (4) prepare and administer the licensing examinations as requested by facility licensees and at least once per year in each of the NRC's four regions.

Enclosure: Federal Register Notice

cc: Representative Ralph Hall

Dear Mr. Chairman:

The NRC has sent the enclosed amendments to the Commission's rules in 10 CFR Part 55 to the Office of the Federal Register for publication. This rulemaking will allow facility licensees to prepare the entire initial examination for reactor operators and senior reactor operators and to proctor and grade the written portion of the examination. Facility licensees that elect to prepare their own examinations will be required to submit the examinations to the NRC for review and approval and establish procedures to control examination security and integrity. This rulemaking requires the NRC to prepare those examinations and tests not prepared by facility licensees and preserves the NRC's authority to prepare the examinations and tests, as necessary.

Section 107 of the Atomic Energy Act of 1954 (AEA), as amended, requires the NRC to determine the qualifications of individuals applying for an operator license, to prescribe uniform conditions for licensing such individuals, and to issue licenses as appropriate. Operator license applicants are required by 10 CFR Part 55, "Operators' Licenses," to pass an examination satisfying the basic content requirements that are also specified in the regulation. Although the AEA is not specific as to who will prepare and administer the examination, the NRC has traditionally performed these functions through its staff or contract examiners. The NRC is increasing power reactor facility licensees' role in the examination development process in recognition of the improvements in industry training programs, to make the operator licensing program more efficient, and to realize budgetary savings for the Federal Government.

The NRC will continue to (1) prepare and administer the generic fundamentals examination to operator license applicants, (2) review every facility-prepared examination, direct changes as required to achieve NRC standards, and approve the examination before it is given, (3) administer and grade the operating test portion of the examination, including a control room simulator examination and a one-on-one demonstration of specific operating tasks, and (4) prepare and administer the licensing examinations as requested by facility licensees and at least once per year in each of the NRC's four regions.

Enclosure: Federal Register Notice

cc: Senator Bob Graham

ATTACHMENT 6

COMMISSION PUBLISHES FINAL RULE FOR EXPANDED UTILITY ROLE
IN TESTING REACTOR OPERATOR LICENSE CANDIDATES

The Nuclear Regulatory Commission has amended its regulations to provide an expanded role for its nuclear power plant licensees in certifying reactor operators.

The Commission took this action in recognition of improvements in industry training programs, to make the operator licensing program more efficient, and as a cost-cutting measure for the agency.

Applicants for a license to manipulate the controls of a nuclear power plant must pass a written test and demonstrate knowledge of control room operation fundamentals to be considered for the final, hands-on, site-specific license examination. Under its new rule, licensees operating nuclear power plants will be permitted to prepare, proctor, and grade their exams and prepare their operating tests - subject to NRC approval. The NRC will continue to administer and grade the hands-on portion of the test, including a control room simulator examination and a one-on-one demonstration of specific operating tasks.

The rule change follows a 1995 decision by the Commission to begin a pilot program in which nuclear power plant licensees would prepare the tests under NRC oversight.

All examinations and tests written by utilities and tailored to their specific plants will be subject to review, modification, and approval by NRC staff before the tests are given. If the submitted test fails to meet NRC's standards, the agency has the option of preparing its own exam in lieu of accepting or modifying one prepared by a utility.

The NRC staff will prepare the licensing examinations upon request by facility licensees and at least once annually in each of the agency's four regions to ensure that the NRC staff retains proficiency in examination writing and to serve as a quality check on the process.

Licensing of operators for research and test reactors will continue to be handled by the NRC and will be unaffected by the rule change. Utilities also will continue to prepare and administer requalification examinations to licensed operators as part of an NRC-approved training program.

Full details of the final rule are included in a notice published in an upcoming issue of the Federal Register.

###

1. Copies are available for inspection or copying for a fee from the NRC Public Document Room (PDR) at 2120 L Street NW, Washington, DC 20555-0001; the PDR's mailing address is Mail Stop LL-6; telephone is 202-634-3273; fax is 202-634-3343. Revision 8 of NUREG-1021 is also available for downloading from the internet at http://www.nrc.gov.

2. Copies of NUREGs may be purchased from the Superintendent of Documents, U.S. Government Printing Office, P.O. Box 38082, Washington, DC 20402-9328. Copies are also available from the National Technical Information Service, 5285 Port Royal Road, Springfield, VA 22161. A copy is available for inspection and/or copying in the NRC Public Document Room, 2120 L Street, NW (Lower Level), Washington, D.C.