[DNFSB
LETTERHEAD]
June 18, 2004
The Honorable Linton Brooks
Administrator
National Nuclear Security
Administration
U.S. Department of Energy
1000 Independence Avenue, SW
Washington, DC 20585-0701
Dear Ambassador Brooks:
The design, fabrication, procurement,
inspection, and maintenance of special tooling have an important impact on the
safety of nuclear explosive operations at the Pantex Plant. The Department of Energy (DOE), and
subsequently the National Nuclear Security Administration (NNSA) have expended
significant resources at the Pantex Plant to develop seamless safety processes
for various weapon programs that rely on specially designed tools to eliminate
or minimize potential hazards to nuclear explosive operations. During the past several years, there have been
a number of occurrences related to this tooling. These occurrences and subsequent tooling
program reviews at the Pantex Plant conducted by DOE, the Pantex Plant
contractor, and the Defense Nuclear Facilities Safety Board (Board), have resulted
in the development of several corrective action plans.
The Board’s staff recently
conducted another review of the tooling program at the Pantex Plant. A report documenting the results of this
review is enclosed. The Board is concerned
that there continue to be serious weaknesses in the tooling program. The Pantex Site Office (PXSO) and BWXT Pantex
(BWXT) have identified most of these weaknesses and have developed corrective
action plans to address them. However,
some of these deficiencies are of a longstanding nature and previous corrective
actions have proven ineffective in resolving the issues. A significant lesson to be learned from the
long history of tooling program reviews is the importance of sustaining and
periodically evaluating the effectiveness of improvements being made.
A
majority of the key
corrective actions identified in the latest Pantex tooling improvement plan are
scheduled to be completed in fiscal year 2004. PXSO has acknowledged the need for NNSA to
conduct a thorough and comprehensive review of the tooling program at Pantex
and the Board agrees.
The Board believes that an
effective quality assurance program, as required by Title 10,
U.S. Code of Federal
Regulations, Part 830, Subpart A, is
essential to safely design, fabricate, purchase, inspect, and maintain special
tooling. Therefore, pursuant to 42 U.S.C.
§ 2286b(d),
the Board requests that NNSA report
to the Board within 30 days when NNSA will conduct a comprehensive
review of quality assurance as it affects the tooling program at Pantex, and
the intended scope and schedule of the review. Further, the Board requests that it be briefed
on the results of the review.
Sincerely,
John T. Conway
Chairman
c: Mr. Daniel E. Glenn
Mr.
Mark B. Whitaker, Jr.
Enclosure
DEFENSE
NUCLEAR FACILITIES SAFETY BOARD
Staff
Issue Report
June
1, 2004
MEMORANDUM FOR: J. K. Fortenberry, Technical Director
COPIES: Board Members
FROM: D. Kupferer
SUBJECT: Tooling Program Review at Pantex
The staff of the Defense Nuclear
Facilities Safety Board (Board) visited the Pantex Plant from March 30 to April
1, 2004, to review the site-wide tooling program. Staff members T. Hunt, D. Kupferer, and J.
Shackelford, as well as outside expert R. West, participated in this review. The Board’s staff met with personnel from
BWXT-Pantex (BWXT) and the National Nuclear Security Administration’s (NNSA)
Pantex Site Office (PXSO) to discuss the following topics related to tooling: design packages, project team interaction,
calculations, fabrication, receiving and inspection, the special tooling
program, recent tooling-related occurrences, and past site-wide tooling program
studies. This report documents the staff’s
observations from this review. In
addition, the Board’s staff has continued to review tooling deficiencies
reported at the Pantex Pant. This report
provides a brief summary of recent tooling related incidents through the end of
May 2004. BWXT is currently in the
process of reorganizing the tooling program at Pantex and implementing a
tooling improvement plan.
Summary. The
Board’s staff concluded that a number of weaknesses exist with respect to the
tooling program at Pantex. Some of these
deficiencies are of a longstanding nature.
PXSO and BWXT re-identified some of these weaknesses in recent reviews
and have planned or taken actions to address them.
It is of particular concern that
multiple site-wide tooling program reviews during the past several years have
identified similar weaknesses and corrective actions. Many corrective actions have been ineffective
or not fully implemented.
Another concern is that BWXT
personnel could not show that the recently implemented functional testing of
tooling ensures that all credited safety features are tested. The staff also noted issues with receiving and
inspection of new and modified tools, failure analyses of tooling designs,
control of tooling changes, establishment of quality assurance requirements for
procured tools, use of lessons learned, and performance of peer reviews. The staff intends to follow closely the
implementation of planned improvements to the Pantex tooling program.
Background. During
the past several years, there have been a number of tooling-related occurrences
and subsequent site-wide tooling program reviews conducted by the Department of
Energy (DOE)/NNSA, the site contractor, and the Board’s staff. These include the following:
Tooling
Task Force Review (1996)—Early
in 1995, Mason & Hanger Corporation undertook
various efforts to increase the level of formality associated with tooling
design, maintenance, and configuration control. A tooling task force developed 24 principal recommendations
to improve the site-wide tooling program.
Board’s
Staff Review—In
May 2000, the Board’s staff conducted a review that resulted in a letter and
two reports sent by the Board to DOE that discussed tooling design, manufacturing,
procurement, and control at Pantex. The
staff made observations that failure modes analyses of complex tooling were not
performed, periodic testing of tools with credited safety functions was not
well-defined, and training of production technicians in the use of tooling and
the reasoning behind specific design features was substandard.
Tooling
Verification Action Plan—The
Tooling Verification Action Plan (TVAP) was developed after production
technicians discovered a misassembled tool in September 2003. Corrective actions in the TVAP included
verifying bay and cell tools against drawings; functionally testing cell and
bay tools; reviewing tooling modifications, repairs, and fabrications; performing
additional tool inspections in the tooling warehouse prior to issuance; and establishing
a tooling tryout facility for functional testing.
Tooling
Improvement Plan—BWXT
recognized the continuing problems with the overall tooling program at Pantex
and formed a team to evaluate the program and recommend improvements. The resulting Tooling Improvement Plan (TIP)
superceded the TVAP and was forwarded to PXSO on March 15, 2004. The TIP contains more than 80 corrective
actions to be completed during fiscal years 2004 and 2005. At the time of the Board’s staff review, 25 of
the corrective actions were reported as complete. The planned/completed corrective actions
include improving receiving and inspection (R&I); establishing a process
for conducting tryouts of credited tooling; creating a tooling department
responsible for tool design, fabrication, and tryout; establishing a tracking
and trending program for tooling concerns; improving inventory accuracy for
tooling; improving design documentation through implementation of a design requirements
document; and improving ease of access to tooling documentation through a centralized
computer database.
Quality
Assurance Survey—A
PXSO Quality Assurance Survey (QAS) of the BWXT tooling program was completed
in March 2004. The QAS resulted in 12
findings as well as 25 deficiencies and weaknesses. The 12 findings include issues related to
unauthorized screening of tooling deviations, inconsistent usage of categorical
exclusions for deviations on tooling drawings, unauthorized modifications of
tooling, improper inspection of tooling, examples of in-use tooling that did
not meet design/drawing specifications, lack of a tracking/trending program,
inventory control problems, and other documentation problems.
Recent
Occurrences—There
have been a number of tooling related incidents in the recent past:
Tooling Program
Observations.
Cross Walk of Credited
Tooling Functions, Technical Safety Requirements, and Functional Testing of
Tooling—BWXT
has identified
approximately 300 tool designs (almost 3000 copies) that have been credited
with one or more safety functions in the various weapon program hazard analyses.
The tooling task force (in 1996)
recommended the creation of a tooling tryout facility to verify the performance
of safety functions following fabrication and maintenance. However, this recommendation was not implemented.
A tooling tryout facility is now
being implemented as a result of the more recent TIP. The Board’s staff did not find a clear translation
of all safety functions, which are credited in the safety basis, to the
functional tests being performed. The
staff also found that copies of credited tooling that have not been through a
formal and documented process intended to functionally test all of the safety
features may have already been issued for use.
Tooling
Design and Modification—The
processes established for designing and modifying tools generally lack
formality and appear difficult to implement. BWXT allows tooling design engineers to
authorize tooling modifications verbally. It is the responsibility of the design
engineer to issue a tooling design instruction (TDI) and update the controlled
design drawing to incorporate the verbally directed modification as soon as
possible. The TDI is placed in the
tooling work package at R&I to prevent a tool that does not meet the most
current design requirements from being issued. The Board’s staff noted several weaknesses in
the flowdown of requirements and documentation related to TDIs.
In response to questions about
the methodology used to ensure tools can perform their design function, BWXT
personnel stated that the first copy of a tool is put through a rigorous series
of tests to verify its ability to meet design requirements. There was no directive that stated how this
testing was to be conducted and no documentation was available to show it had
been accomplished for any tools. Although the Board’s letter dated May 23, 2000,
noted the lack of a formal failure analysis process, no action has been taken
to address this issue.
Project
Team Interaction—When
the need for a new tooling design is identified, a project team is assembled,
consisting of members from tooling design engineering, process engineering, the
authorization basis group, the design agencies, the training department, and
the production technician core team. However, the roles and responsibilities of
each project team member are not clearly defined or documented. In addition, there do not appear to be
formalized procedures for communicating tooling information and data between
the project team members. For example,
during the hazards identification process, weapon response requests are generated.
It is unclear what groups are
responsible for determining the weapon response information that is needed, and
approving that determination. In January
2004, an extraction tool failed during disassembly operations. The pull force applied to the component was
greater than that allowed in the authorization basis. It appears that a miscommunication occurred
between design engineering, the authorization basis group, and the design
agency. It is unclear if proposed corrective
actions will be effective in preventing similar miscommunications in the future.
Receiving
and Inspection Procedures and Documentation—The Board’s staff identified deficiencies in the R&I
process for special tooling. The staff
found a design drawing used for a recent R&I that contained a pen-and-ink
change to a critical dimension and noted that there was no TDI to support the
pen-and-ink change. A TDI was present,
but the R&I had been made to the existing drawing, contrary to the approved
process. Despite these deficiencies, the
R&I form was signed as acceptable and peer reviewed by the responsible
supervisor. Further, the supervisor was
unsure as to what specific criteria to apply in the conduct of a peer review.
Tooling
Procurement—The
majority of special tooling is fabricated by outside vendors. Before a piece of equipment is
procured at Pantex, it is designated with an Acquisition Level. Acquisition Level 1 (AL-1) is procured to a
high level of quality requirements, which are defined in the procurement manual.
Tooling is procured to Acquisition Level
S (AL-S),
which denotes procurement of a service. Despite an extensive briefing in which BWXT
personnel asserted that AL-S provides
the same level of quality assurance as AL-1, it is still unclear to the Board’s
staff that this is the case. A
Product Description
Quality Requirements Document (PDQRD) is generated during the procurement
process for all AL-1 items. The PDQRD
specifically defines the quality assurance requirements for that item. Rather than a PDQRD, a technical data package
(TDP) and statement of work (SOW) are generated for AL-S
equipment,
including tooling. Discussions with
cognizant BWXT personnel revealed confusion as to the content of these key
tooling procurement documents. Neither
the TDP nor the SOW specifically define quality assurance requirements. Instead, the quality assurance requirements
for AL-S items are specified in a blanket contract with each individual vendor.
BWXT performs audits of vendors to
review quality assurance practices at least once every three years. BWXT was unable to supply documentation that
showed that the combination of the TDP, SOW, and vendor contract provided the
same degree of quality assurance as required for an AL-1 procurement.
Lessons
Learned—The
staff noted that BWXT does not have an effective system in place to ensure that
the appropriate insights and lessons learned from tooling program deficiencies
and occurrences are adequately incorporated into the overall tooling program. In particular, a number of weaknesses in the
R&I program had been identified previously and no effective feedback
mechanism was in place to ensure that similar deficiencies would be prevented. The production technicians and training
specialists were not familiar with the details of the recent failure of an
extraction tool. The lessons learned
documentation lacked detail, limiting the effectiveness of the feedback. In another case, a tool was found to be
misassembled because the drawing was difficult to interpret. In this example, BWXT was unable to provide
documented evidence of any actions taken to minimize the possibility of a
recurrence.
The Peer
Review Process—BWXT
has implemented peer reviews to address some of the identified deficiencies in
the special tooling program. These are
intended to provide separate reviews for certain important attributes of the
program, including design calculations, analyses, testing, and inspection. However, the critical elements, criteria, and
standards to be used in the peer review process are not defined or documented.
BWXT stated that this problem was recognized and action was being taken.