[DNFSB
LETTERHEAD]
September 10, 2007
The Honorable Thomas P. D'Agostino
Administrator
National Nuclear Security Administration
Dear Mr. D'Agostino:
The Defense Nuclear Facilities
Safety Board (Board) has completed a review of nuclear criticality safety (NCS)
implementation, conduct of operations, and configuration management in the Plutonium Facility vault at Los Alamos National
Laboratory. The enclosed report prepared
by the Board's staff provides a detailed discussion of the results of this
review. This letter focuses on one of
the issues identified during the review that deals with the improper utilization
of the laboratory's Materials Accountability and Safeguards System (MASS) software
to ensure compliance with criticality safety limits.
The laboratory uses MASS to
track material movements and special nuclear material storage throughout the
Plutonium Facility, including movements into and out of the vault. MASS often provides the sole source of
information about a given item, such as mass and isotopic composition. Despite the laboratory's contention that MASS
is merely an operator aid and is not relied upon to determine the safety of
material movements, discussion with operators and review of procedures revealed
that MASS fulfills critical safety-related functions, such as determining
compliance with NCS limits. MASS was initially conceived as a security-related accountability
program and accordingly was not designed to perform a criticality
safety-related function. As a result,
its criticality safety aspects were not subjected to
the rigorous quality assurance requirements and configuration management that
its current safety-related usage would require.
MASS was developed by the
laboratory in the 1970s in a programming language that is difficult to maintain
and has not seen a significant software upgrade since its introduction. Consequently, the laboratory has
long-standing plans to transition MASS to a modern computing platform and
programming language. Even though the
need for a MASS update was identified more than 10
years ago, upgrade efforts have been unsuccessful. The laboratory recently revived the effort to
modernize and upgrade MASS, but progress is not yet evident. In addition, it is unclear whether the current
concept for a new MASS acknowledges its safety-related function and therefore
identifies the appropriate software quality assurance and configuration management
requirements.
Sincerely,
A.J. Eggenberger
Chairman
c: Mr. Glenn S.
Podonsky
Mr. Donald L. Winchell, Jr.
Mr. Mark B.
Whitaker, Jr.
Enclosure
DEFENSE NUCLEAR
FACILITIES SAFETY BOARD
Staff Issue Report
August 6, 2007
MEMORANDUM FOR: J. Kent Fortenberry
FROM: C. Goff and E. Elliott
SUBJECT: Materials Accountability and Safeguards System
Software at
This report documents a
review by the staff of the Defense Nuclear Facilities Safety Board (Board) of
nuclear criticality safety (NCS) implementation in the Plutonium Facility (PF-4) vault at Los Alamos National
Laboratory. The review included a site visit on June 5-7, 2007,
as well as a follow-up teleconference held on July 19, 2007. Members of the Board's staff B. Broderick, R. T. Davis, E. Elliott, C.
Goff, C. Keilers, M. Moury,
and J. Plaue participated in the review.
Materials
Accountability and Safeguards System Software. The laboratory tracks material movements and special
nuclear material (SNM) storage throughout PF-4 using the software program
referred to as the Materials Accountability and Safeguards System (MASS). MASS
often provides the sole source of information about a given item, including
mass, isotopic composition, material form, and storage location. MASS is also used to generate the material label
that is affixed to the outside of each SNM container. Before SNM is physically transferred to a new
location in the facility, the transfer must be completed
electronically in MASS by the operators. In some cases, the criticality limits for a
given location (e.g., glovebox or vault) are loaded
into MASS.
Reliance on MASS for Safety - According to laboratory management, MASS is regarded as an operator aid that provides information for
criticality control purposes, but is not relied upon to perform a criticality
safety function. This belief is
consistent with the description of MASS in the administrative procedure that
governs NCS implementation in PF-4 (TA55-AP-522) which states that MASS is
"a useful administrative tool to assist nuclear material handlers in
managing fissile material inventories within approved limits." The NCS group also informed the Board's staff
that operators use MASS only to obtain data;
compliance with NCS limits is to be determined by hand calculation while
referencing the approved values contained in the NCS posting and associated
Criticality Safety Limit Approval (CSLA).
The assertion that MASS is
only an operator aid is not consistent with the laboratory's implementing
procedures. In some situations, MASS automatically provides both NCS limit information and
performs the calculation to verify compliance with the limits. In TA55-AP- 522-the procedure that describes MASS
as an "administrative tool'-there are procedural steps detailing the
movement of items into a glovebox that state,
"Before physically moving the fissile material, perform the MASS
transaction to check that the amount and form of the material being
transferred is within the criticality safety limit for the
destination glovebox." Hand calculations are not
mentioned in the procedure or the associated training. Additionally, a Technical Area-55 Notice (TA55-Notice-011)
sent to all employees on October 3 1, 2006, instructs
operators to execute the electronic transfer in MASS before physically
moving the material into the location, with the express purpose of preventing
an overmass condition. The notice uses the following lessons-learned
statement to explain the new requirement: "The practice was to physically move items,
then enter the information into MASS. This means that computer material
accountability checks were not performed until the item was in the new
location. This permitted human error, which
can result in exceedance of criticality safety
limits.” In both the procedure and the
notice, the laboratory is instructing operators to use MASS to comply with NCS
limits.
After speaking with vault operators,
it was clear to the Board's staff that MASS was also being
relied upon in the vault for compliance with criticality safety limits. For example, some vault storage locations have
NCS limits that are based on the total mass and allow
multiple SNM containers. MASS keeps a
running total of the mass in each storage location, and will output a warning
message that informs the operators if an NCS limit will be
exceeded when they electronically process an SNM transfer. According to the vault operator procedure
(NMT4-WI- 101) if MASS generates a warning that a criticality limit will be
exceeded, the operators are instructed to consult the CSLA to check the NCS
limit, after MASS has alerted them. The operators also consult a handwritten Vault
Availability Book (the Blue Book), which is used primarily to ensure that the
item will physically fit into the storage location. The Blue Book also provides a form to indicate
the location's running mass total, which could serve as a second check
of the total mass; however, it does not contain the NCS limits. In this case, MASS is relied upon to prevent an "overmass"
condition, which means that MASS is performing a safety function exceeding that
of an "operator aid."
Safety Software Requirements-MASS was originally developed
in the 1970s as a security-related accountability program, and thus it lacks
the configuration management control and software quality assurance pedigree of
a safety-related system. Department of
Energy Order 414.1 C, Quality Assurance, requires that software performing safety functions meet certain
requirements, including appropriate configuration management of key data,
verification and validation of software operation, and documentation. Currently, MASS does not meet many of these
requirements. As an example, changes to
NCS limit data in MASS that should require configuration management control are handled informally via email and are not verified or audited
by the NCS group.
Planned Improvements-As mentioned previously, MASS was developed
more than 20 years ago, and was
written in a programming language that is difficult to maintain; it has not undergone
a software upgrade since its inception. The
original author of the code, now retired,
is the only person capable of editing the source code and is still relied upon to make improvements and correct
problems, which included three instances of system inoperability during the
past year.
Because of the age of the
code and the limited ability to update it, the laboratory began efforts to
transition MASS to a modern computing platform and programming language more than
10 years ago; however, these efforts were unsuccessful. A March 2006 report by the laboratory's Audits
and Assessments office found that the MASS upgrade project lacked "full management
support, project leadership, defined scope, prepared schedule and budget, and quality
assurance plan for a successful functional and timely implementation." This finding was repeated
from two previous assessments, one performed in July 2004 and the other in
March 2002.
Recently, the laboratory
revived the effort to modernize and upgrade MASS. In particular, laboratory management formally:
(1) consolidated responsibility for the system, which historically has been a
significant hindrance to progress and (2) committed to providing the necessary
resources for the upgrade. During the
teleconference with the Board's staff on
July 19, 2007, the laboratory further indicated that early planning had
begun for a renewed effort that is expected to be mature in early fall 2007. No project manager had been assigned to the project,
and substantive planning incorporating lessons learned from
previous upgrade attempts was not evident. It was unclear whether the concept for a new
MASS acknowledges its safety-related function and therefore identifies the
appropriate software quality assurance and configuration management
requirements. The effort to upgrade and
modernize MASS would appear to be an excellent opportunity to implement these
requirements formally. A representative
from the Los Alamos Site Office (LASO) expressed the expectation that the managerial
rigor of the MASS update effort would be commensurate with that of other major design
and construction projects, but did not explicitly acknowledge MASS as
safety-related software.
Observations
on Vault Operations. The
following observations made by the Board's staff have been
informally communicated to the laboratory for its use.
NCS Involvement in Operational Changes-The staff was briefed on findings from an internal
assessment by the PF-4 Deputy Facilities Operations Director. This
assessment identified that the NCS group was not always involved when an
operation underwent a change. This lack of NCS involvement in the change
process has resulted in ongoing operations that are not in
compliance with NCS postings. The
NCS group is required by ISD 130- 1 .O, Nuclear Criticality Safety Program Manual,
to evaluate new or revised operations involving significant quantities of
fissionable material. However, this
requirement is weakly implemented at the present time,
a situation that the improved Unreviewed Safety
Question process should rectify by requiring NCS concurrence on proposed
changes to fissile material operations. In the interim, the NCS group issued a letter stating
that NCS approval is needed for any changes to
fissionable material operations.
Adequacy of Vault Operator Staffing-Vault operations require two trained vault operators. Currently, there are only
two trained vault operators, with a third in training who is approximately 1
year away from full qualification. Because of the limited number of qualified personnel,
the vault is typically accessible to users only during the morning and must close if one of the two vault operators is absent from
work. Overall, the laboratory has not
developed adequate staffing plans to safely support the expected
increase in programmatic mission, as well as ongoing nuclear material
stabilization and repackaging activities, including actions to implement the Board's
Recommendations 94- 1, Improved Schedule
for Remediation in the Defense Nuclear Facilities Complex, and 2000- 1, Prioritization for Stabilizing Nuclear Materials.
Priority of NCS Reevaluation for the Vault - Based on earlier walkdowns
conducted as part of the NCS Program Improvement Plan (PIP), vault operations were categorized as a medium-risk activity, and NCS
evaluations governing the vault were deemed to have only "administrative
deficiencies." The Board's staff
examined the vault documentation, which consists of a number of convoluted NCS
evaluations that rely heavily on expert judgment. In some cases, the evaluations contradict one
another. In at least one case, controls
identified as necessary in the evaluations (i.e., use of neutron-absorbing
material) are not being implemented appropriately. These deficiencies in the NCS evaluations for
the vault are unlikely to be addressed in the near
term, because the NCS group is currently focused on evaluating and documenting
other operations deemed to be of higher risk during NCS PIP walkdowns.
Additionally, bringing the vault NCS evaluations into compliance with DOE
Standard 3007, Guidelines for Preparing
Criticality Safety Evaluations at Department of Energy Nonreactor Nuclear
Facilities, will require substantial staff resources. According to the current schedule, the vault
evaluations will not be rewritten until the third
quarter of fiscal year 2009. The Board's
staff believes the existing vault evaluations contain technical deficiencies
and that their rewriting should be accorded higher priority, especially given
the large number of vault movements and the fact that the vault is nearing
capacity.
Weaknesses in the Conduct of Operations - The staff observed a movement of SNM into the vault.
Overall, the vault staff and material
handlers appeared to be highly knowledgeable; however, the Board's staff did
observe weaknesses in the conduct of operations. For example, the vault operators informed the
staff that the local copy of the procedure that governs the use of MASS was 2
years out of date and did not accurately reflect some of their current
practices. While an updated procedure was available in the facility's document
control system, it was not being utilized. The Board's staff emphasized the need for the
correct version of the procedure to be present in the workspace. Subsequently, the local (and out-of-date) copy
of the procedure was replaced.
Status
of Federal NCS Oversight. Currently, LASO has one individual performing NCS oversight,
with augmentation and monitoring provided by personnel from the National Nuclear
Security Administration (NNSA)