[DNFSB LETTERHEAD]

 

June 18, 2004

 

The Honorable Linton Brooks

Administrator

National Nuclear Security Administration

U.S. Department of Energy

1000 Independence Avenue, SW

Washington, DC 20585-0701

 

Dear Ambassador Brooks:

 

The design, fabrication, procurement, inspection, and maintenance of special tooling have an important impact on the safety of nuclear explosive operations at the Pantex Plant.  The Department of Energy (DOE), and subsequently the National Nuclear Security Administration (NNSA) have expended significant resources at the Pantex Plant to develop seamless safety processes for various weapon programs that rely on specially designed tools to eliminate or minimize potential hazards to nuclear explosive operations.  During the past several years, there have been a number of occurrences related to this tooling.  These occurrences and subsequent tooling program reviews at the Pantex Plant conducted by DOE, the Pantex Plant contractor, and the Defense Nuclear Facilities Safety Board (Board), have resulted in the development of several corrective action plans.

 

The Board’s staff recently conducted another review of the tooling program at the Pantex Plant.  A report documenting the results of this review is enclosed.  The Board is concerned that there continue to be serious weaknesses in the tooling program.  The Pantex Site Office (PXSO) and BWXT Pantex (BWXT) have identified most of these weaknesses and have developed corrective action plans to address them.  However, some of these deficiencies are of a longstanding nature and previous corrective actions have proven ineffective in resolving the issues.  A significant lesson to be learned from the long history of tooling program reviews is the importance of sustaining and periodically evaluating the effectiveness of improvements being made.

 

A majority of the key corrective actions identified in the latest Pantex tooling improvement plan are scheduled to be completed in fiscal year 2004.  PXSO has acknowledged the need for NNSA to conduct a thorough and comprehensive review of the tooling program at Pantex and the Board agrees.

 

The Board believes that an effective quality assurance program, as required by Title 10,

U.S. Code of Federal Regulations, Part 830, Subpart A, is essential to safely design, fabricate, purchase, inspect, and maintain special tooling.  Therefore, pursuant to 42 U.S.C. § 2286b(d), the Board requests that NNSA report to the Board within 30 days when NNSA will conduct a comprehensive review of quality assurance as it affects the tooling program at Pantex, and the intended scope and schedule of the review.  Further, the Board requests that it be briefed on the results of the review.

 

Sincerely,

 

John T.  Conway

Chairman

 

c:   Mr. Daniel E. Glenn

Mr. Mark B. Whitaker, Jr.

 

Enclosure

---

 

DEFENSE NUCLEAR FACILITIES SAFETY BOARD

 

Staff Issue Report

 

June 1, 2004

 

MEMORANDUM FOR:      J. K. Fortenberry, Technical Director

 

COPIES:                                 Board Members

 

FROM:                                   D. Kupferer

 

SUBJECT:                             Tooling Program Review at Pantex

 

The staff of the Defense Nuclear Facilities Safety Board (Board) visited the Pantex Plant from March 30 to April 1, 2004, to review the site-wide tooling program.  Staff members T. Hunt, D. Kupferer, and J. Shackelford, as well as outside expert R. West, participated in this review.  The Board’s staff met with personnel from BWXT-Pantex (BWXT) and the National Nuclear Security Administration’s (NNSA) Pantex Site Office (PXSO) to discuss the following topics related to tooling:  design packages, project team interaction, calculations, fabrication, receiving and inspection, the special tooling program, recent tooling-related occurrences, and past site-wide tooling program studies.  This report documents the staff’s observations from this review.  In addition, the Board’s staff has continued to review tooling deficiencies reported at the Pantex Pant.  This report provides a brief summary of recent tooling related incidents through the end of May 2004.  BWXT is currently in the process of reorganizing the tooling program at Pantex and implementing a tooling improvement plan.

 

Summary.  The Board’s staff concluded that a number of weaknesses exist with respect to the tooling program at Pantex.  Some of these deficiencies are of a longstanding nature.  PXSO and BWXT re-identified some of these weaknesses in recent reviews and have planned or taken actions to address them.

 

It is of particular concern that multiple site-wide tooling program reviews during the past several years have identified similar weaknesses and corrective actions.  Many corrective actions have been ineffective or not fully implemented.

 

Another concern is that BWXT personnel could not show that the recently implemented functional testing of tooling ensures that all credited safety features are tested.  The staff also noted issues with receiving and inspection of new and modified tools, failure analyses of tooling designs, control of tooling changes, establishment of quality assurance requirements for procured tools, use of lessons learned, and performance of peer reviews.  The staff intends to follow closely the implementation of planned improvements to the Pantex tooling program.

 

Background.  During the past several years, there have been a number of tooling-related occurrences and subsequent site-wide tooling program reviews conducted by the Department of Energy (DOE)/NNSA, the site contractor, and the Board’s staff.  These include the following:

 

Tooling Task Force Review (1996)—Early in 1995, Mason & Hanger Corporation undertook various efforts to increase the level of formality associated with tooling design, maintenance, and configuration control.  A tooling task force developed 24 principal recommendations to improve the site-wide tooling program.

 

Board’s Staff Review—In May 2000, the Board’s staff conducted a review that resulted in a letter and two reports sent by the Board to DOE that discussed tooling design, manufacturing, procurement, and control at Pantex.  The staff made observations that failure modes analyses of complex tooling were not performed, periodic testing of tools with credited safety functions was not well-defined, and training of production technicians in the use of tooling and the reasoning behind specific design features was substandard.

 

Tooling Verification Action Plan—The Tooling Verification Action Plan (TVAP) was developed after production technicians discovered a misassembled tool in September 2003.  Corrective actions in the TVAP included verifying bay and cell tools against drawings; functionally testing cell and bay tools; reviewing tooling modifications, repairs, and fabrications; performing additional tool inspections in the tooling warehouse prior to issuance; and establishing a tooling tryout facility for functional testing.

 

Tooling Improvement Plan—BWXT recognized the continuing problems with the overall tooling program at Pantex and formed a team to evaluate the program and recommend improvements.  The resulting Tooling Improvement Plan (TIP) superceded the TVAP and was forwarded to PXSO on March 15, 2004.  The TIP contains more than 80 corrective actions to be completed during fiscal years 2004 and 2005.  At the time of the Board’s staff review, 25 of the corrective actions were reported as complete.  The planned/completed corrective actions include improving receiving and inspection (R&I); establishing a process for conducting tryouts of credited tooling; creating a tooling department responsible for tool design, fabrication, and tryout; establishing a tracking and trending program for tooling concerns; improving inventory accuracy for tooling; improving design documentation through implementation of a design requirements document; and improving ease of access to tooling documentation through a centralized computer database.

 

Quality Assurance Survey—A PXSO Quality Assurance Survey (QAS) of the BWXT tooling program was completed in March 2004.  The QAS resulted in 12 findings as well as 25 deficiencies and weaknesses.  The 12 findings include issues related to unauthorized screening of tooling deviations, inconsistent usage of categorical exclusions for deviations on tooling drawings, unauthorized modifications of tooling, improper inspection of tooling, examples of in-use tooling that did not meet design/drawing specifications, lack of a tracking/trending program, inventory control problems, and other documentation problems.

 

Recent Occurrences—There have been a number of tooling related incidents in the recent past:

 

• In August 2003, the radiation case of a unit was scored during a cutting and removal procedure.

 

• In September 2003, during disassembly operations, production technicians discovered that a rotating tool was misassembled such that the tool rotated in a direction opposite to the direction intended.

 

• In December 2003, a common hand tool that was not previously addressed in the hazard analysis was used during disassembly operations.

 

• In January 2004, during dismantlement operations, cracks in a high explosive charge, most likely caused by tooling loadings, were discovered.

 

• In January 2004, an extraction tool failed during disassembly operations.

 

• In January 2004, a potential failure mechanism for a lifting and rotating fixture was discovered.

 

• In January 2004, during readiness activities, misassembled tooling was discovered.

 

• In March 2004, a tool was inappropriately altered without authorization.

 

• In May 2004, on two occasions, production technicians discovered an expired preventive maintenance sticker on an enhanced transportation cart that contained a weapon.

 

• In May 2004, Pantex personnel discovered that the pressurization force associated with a piece of tooling was not addressed in the Weapon Specific Hazard Analysis.

 

• In May 2004, during bay operations the transfer of a unit from a work stand to a transportation cart was prevented because the two pieces of tooling could not be aligned.

 

• In May 2004, it was discovered that an in-service inspection requirement for detonator covers was not adequately captured in drawings or procedures, and was not being performed.

 

 

Tooling Program Observations.

 

Cross Walk of Credited Tooling Functions, Technical Safety Requirements, and Functional Testing of Tooling—BWXT has identified approximately 300 tool designs (almost 3000 copies) that have been credited with one or more safety functions in the various weapon program hazard analyses.  The tooling task force (in 1996) recommended the creation of a tooling tryout facility to verify the performance of safety functions following fabrication and maintenance.  However, this recommendation was not implemented.  A tooling tryout facility is now being implemented as a result of the more recent TIP.  The Board’s staff did not find a clear translation of all safety functions, which are credited in the safety basis, to the functional tests being performed.  The staff also found that copies of credited tooling that have not been through a formal and documented process intended to functionally test all of the safety features may have already been issued for use.

 

Tooling Design and Modification—The processes established for designing and modifying tools generally lack formality and appear difficult to implement.  BWXT allows tooling design engineers to authorize tooling modifications verbally.  It is the responsibility of the design engineer to issue a tooling design instruction (TDI) and update the controlled design drawing to incorporate the verbally directed modification as soon as possible.  The TDI is placed in the tooling work package at R&I to prevent a tool that does not meet the most current design requirements from being issued.  The Board’s staff noted several weaknesses in the flowdown of requirements and documentation related to TDIs.

 

In response to questions about the methodology used to ensure tools can perform their design function, BWXT personnel stated that the first copy of a tool is put through a rigorous series of tests to verify its ability to meet design requirements.  There was no directive that stated how this testing was to be conducted and no documentation was available to show it had been accomplished for any tools.  Although the Board’s letter dated May 23, 2000, noted the lack of a formal failure analysis process, no action has been taken to address this issue.

 

Project Team Interaction—When the need for a new tooling design is identified, a project team is assembled, consisting of members from tooling design engineering, process engineering, the authorization basis group, the design agencies, the training department, and the production technician core team.  However, the roles and responsibilities of each project team member are not clearly defined or documented.  In addition, there do not appear to be formalized procedures for communicating tooling information and data between the project team members.  For example, during the hazards identification process, weapon response requests are generated.  It is unclear what groups are responsible for determining the weapon response information that is needed, and approving that determination.  In January 2004, an extraction tool failed during disassembly operations.  The pull force applied to the component was greater than that allowed in the authorization basis.  It appears that a miscommunication occurred between design engineering, the authorization basis group, and the design agency.  It is unclear if proposed corrective actions will be effective in preventing similar miscommunications in the future.

 

Receiving and Inspection Procedures and Documentation—The Board’s staff identified deficiencies in the R&I process for special tooling.  The staff found a design drawing used for a recent R&I that contained a pen-and-ink change to a critical dimension and noted that there was no TDI to support the pen-and-ink change.  A TDI was present, but the R&I had been made to the existing drawing, contrary to the approved process.  Despite these deficiencies, the R&I form was signed as acceptable and peer reviewed by the responsible supervisor.  Further, the supervisor was unsure as to what specific criteria to apply in the conduct of a peer review.

 

Tooling Procurement—The majority of special tooling is fabricated by outside vendors.  Before a piece of equipment is procured at Pantex, it is designated with an Acquisition Level.  Acquisition Level 1 (AL-1) is procured to a high level of quality requirements, which are defined in the procurement manual.  Tooling is procured to Acquisition Level S (AL-S), which denotes procurement of a service.  Despite an extensive briefing in which BWXT personnel asserted that AL-S provides the same level of quality assurance as AL-1, it is still unclear to the Board’s staff that this is the case.  A Product Description Quality Requirements Document (PDQRD) is generated during the procurement process for all AL-1 items.  The PDQRD specifically defines the quality assurance requirements for that item.  Rather than a PDQRD, a technical data package (TDP) and statement of work (SOW) are generated for AL-S equipment, including tooling.  Discussions with cognizant BWXT personnel revealed confusion as to the content of these key tooling procurement documents.  Neither the TDP nor the SOW specifically define quality assurance requirements.  Instead, the quality assurance requirements for AL-S items are specified in a blanket contract with each individual vendor.  BWXT performs audits of vendors to review quality assurance practices at least once every three years.  BWXT was unable to supply documentation that showed that the combination of the TDP, SOW, and vendor contract provided the same degree of quality assurance as required for an AL-1 procurement.

 

Lessons Learned—The staff noted that BWXT does not have an effective system in place to ensure that the appropriate insights and lessons learned from tooling program deficiencies and occurrences are adequately incorporated into the overall tooling program.  In particular, a number of weaknesses in the R&I program had been identified previously and no effective feedback mechanism was in place to ensure that similar deficiencies would be prevented.  The production technicians and training specialists were not familiar with the details of the recent failure of an extraction tool.  The lessons learned documentation lacked detail, limiting the effectiveness of the feedback.  In another case, a tool was found to be misassembled because the drawing was difficult to interpret.  In this example, BWXT was unable to provide documented evidence of any actions taken to minimize the possibility of a recurrence.

 

The Peer Review Process—BWXT has implemented peer reviews to address some of the identified deficiencies in the special tooling program.  These are intended to provide separate reviews for certain important attributes of the program, including design calculations, analyses, testing, and inspection.  However, the critical elements, criteria, and standards to be used in the peer review process are not defined or documented. BWXT stated that this problem was recognized and action was being taken.