This is the accessible text file for GAO report number GAO-09-804T
entitled 'Combating Nuclear Smuggling: Lessons Learned from DHS Testing
of Advanced Radiation Detection Portal Monitors' which was released on
June 25, 2009.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Testimony:
Before the Subcommittee on Investigations and Oversight, Committee on
Science and Technology, House of Representatives:
United States Government Accountability Office:
GAO:
Combating Nuclear Smuggling:
Lessons Learned from DHS Testing of Advanced Radiation Detection Portal
Monitors:
Statement of Gene Aloise, Director:
Natural Resources and Environment:
GAO-09-804T:
GAO Highlights:
Highlights of GAO-09-804T, a testimony before the Subcommittee on
Investigations and Oversight, Committee on Science and Technology,
House of Representatives.
Why GAO Did This Study:
The Department of Homeland Security’s (DHS) Domestic Nuclear Detection
Office (DNDO) is responsible for addressing the threat of nuclear
smuggling. Radiation detection portal monitors are key elements in the
nation’s defenses against such threats. DHS has sponsored testing to
develop new monitors, known as advanced spectroscopic portal (ASP)
monitors, to replace radiation detection equipment being used at ports
of entry. DNDO expects that ASPs may offer improvements over current-
generation portal monitors, particularly the potential to identify as
well as detect radioactive material and thereby to reduce both the risk
of missed threats and the rate of innocent alarms, which DNDO considers
to be key limitations of radiation detection equipment currently used
by Customs and Border Protection (CBP) at U.S. ports of entry. However,
ASPs cost significantly more than current generation portal monitors.
Due to concerns about ASPs’ cost and performance, Congress has required
that the Secretary of Homeland Security certify that ASPs provide a
significant increase in operational effectiveness before obligating
funds for full-scale ASP procurement.
This testimony addresses (1) GAO findings on DNDO’s latest round of ASP
testing, and (2) lessons from ASP testing that can be applied to other
DHS technology investments. These findings are based on GAO’s May 2009
report GAO-09-655 and other related reports.
What GAO Found:
GAO’s report on the latest round of ASP testing found that DHS
increased the rigor in comparison with previous tests and thereby added
credibility to the test results. However, GAO’s report also questioned
whether the benefits of the ASPs justify the high cost. In particular,
the DHS criteria for a significant increase in operational
effectiveness require only a marginal improvement in the detection of
certain weapons-usable nuclear materials, which DNDO considers a key
limitation of current-generation portal monitors. The marginal
improvement required of ASPs is particularly notable given that DNDO
has not completed efforts to fine-tune current-generation equipment to
provide greater sensitivity. Moreover, the preliminary test results
show that ASPs performed better than current-generation portal monitors
in detection of such materials concealed by light shielding
approximating the threat guidance for setting detection thresholds, but
that differences in sensitivity were less notable when shielding was
slightly below or above that level. Finally, DNDO has not yet updated
its cost-benefit analysis to take into account the results of the
latest round of ASP testing and does not plan to complete computer
simulations that could provide additional insight into ASP capabilities
and limitations prior to certification even though test delays have
allowed more time to conduct the simulations. DNDO officials believe
the other tests are sufficient for ASPs to demonstrate a significant
increase in operational effectiveness. GAO recommended that DHS assess
ASPs against the full potential of current-generation equipment and
revise the program schedule to allow time to conduct computer
simulations and to uncover and resolve problems with ASPs before full-
scale deployment. DHS agreed to a phased deployment that should allow
time to uncover ASP problems but disagreed with the other
recommendations, which GAO believes remain valid.
The challenges DNDO has faced in developing and testing ASPs illustrate
the importance of following best practices for investments in complex
homeland security acquisitions and for testing of new technologies. GAO
recently found that many major DHS investments, including DNDO’s ASP
program, had not met the department’s requirements for basic
acquisition documents necessary to inform the investment review
process, which has adopted many acquisition best practices. As a
result, DHS had not consistently provided the oversight needed to
identify and address cost, schedule, and performance problems in its
major investments. A primary lesson to be learned regarding testing is
that the push to replace existing equipment with the new portal
monitors led to an ASP testing program that until recently lacked the
necessary rigor. Even for the most recent round of testing, DNDO’s
schedule consistently underestimated the time required to conduct tests
and resolve problems uncovered during testing. In contrast, GAO has
previously found that testing programs designed to validate a product’s
performance against increasing standards for different stages in
product development are a best practice for acquisition strategies for
new technologies. Aspects that improved the latest round of ASP testing
could also, if properly implemented, provide rigor to DHS’s testing of
other advanced technologies.
View [hyperlink, http://www.gao.gov/products/GAO-09-804T] or key
components. For more information, contact Gene Aloise at (202) 512-3841
or aloisee@gao.gov.
[End of section]
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss GAO's work on the Department
of Homeland Security's (DHS) testing of advanced spectroscopic portal
(ASP) radiation detection monitors. As you are aware, the national
security mission of U.S. Customs and Border Protection (CBP), an agency
within DHS, includes screening for smuggled nuclear or radiological
material that could be used in a nuclear weapon or radiological
dispersal device (a "dirty bomb"). To screen cargo at ports of entry,
CBP conducts primary inspections with radiation detection equipment
called portal monitors--large stationary detectors through which cargo
containers and vehicles pass as they enter the United States. When
radiation is detected, CBP conducts secondary inspections using a
second portal monitor to confirm the original alarm and a handheld
radioactive isotope identification device to identify the radiation's
source and determine whether it constitutes a threat.
The polyvinyl toluene (PVT) portal monitors currently in use can detect
radiation but cannot identify the type of material causing an alarm. As
a result, the monitors' radiation alarms can be set off even by benign,
naturally occurring radioactive material. One way to reduce the rate of
such innocent alarms--and thereby minimize unnecessary secondary
inspections and enhance the flow of commerce--is to adjust the
operational thresholds (i.e., operate the PVTs at a reduced level of
sensitivity). However, reducing the sensitivity may make it more
difficult to detect certain nuclear materials.
To address the limitations of current-generation portal monitors, DHS's
Domestic Nuclear Detection Office (DNDO) in 2005 began to develop and
test ASPs, which are designed to both detect radiation and identify the
source.[Footnote 1] DNDO hopes to use the new portal monitors to
replace at least some PVTs currently used for primary screening, as
well as PVTs and handheld identification devices currently used for
secondary screening. However, we estimated in September 2008 that the
lifecycle cost of each standard cargo version of the ASP (including
deployment costs) to be about $822,000, compared with about $308,000
for the PVT standard cargo portal, and that the total program cost for
DNDO's latest plan for deploying radiation portal monitors--which
relies on a combination of ASPs and PVTs and does not deploy radiation
portal monitors at all border crossings--would be about $2 billion.
[Footnote 2]
Concerned about the performance and cost of the ASP monitors, Congress
required the Secretary of Homeland Security to certify that the
monitors will provide a "significant increase in operational
effectiveness" before DNDO obligates funds for full-scale ASP
procurement.[Footnote 3] The Secretary must submit separate
certifications for primary and secondary inspection. In response, CBP,
DNDO, and the DHS management directorate jointly issued criteria in
July 2008 for determining whether the new technology provides a
significant increase in operational effectiveness. The primary
screening criteria require that the new portal monitors detect
potential threats as well as or better than PVTs, show improved
performance in detection of highly enriched uranium (HEU), and reduce
innocent alarms. To meet the secondary screening criteria, the new
portal monitors must reduce the probability of misidentifying special
nuclear material (e.g., HEU and plutonium) and the average time to
conduct secondary screenings.
DNDO designed and coordinated a new series of tests, originally
scheduled to run from April 2008 through September 2008, to determine
whether the new portal monitors meet the certification criteria for
primary and secondary screening and are ready for deployment. Key
phases of this testing program include concurrent testing led by DNDO
of the new and current equipment's ability to detect and identify
threats and of ASPs' readiness to be integrated into operations for
both primary and secondary screening at ports of entry; field
validation led by CBP at four northern and southern border crossings
and two seaports; and an independent evaluation, led by the DHS Science
and Technology Directorate at one of the seaports, of the new portal
monitors' effectiveness and suitability.
Since 2006, we have issued six reports and four testimonies on
development of radiation detection portal monitors, including today's
testimony, and have made 19 recommendations for improving DNDO's
efforts to develop and test portal monitors. Our concerns have focused
on key areas in which DNDO's efforts have lacked the necessary rigor
given ASPs' high cost and the importance of the radiation portal
monitor project to our national security. These areas include objective
and realistic testing of ASPs' performance in comparison with the
performance of current-generation equipment; full disclosure and
reporting of the limitations of tests used to support a decision by the
Secretary of Homeland Security on ASP certification; development of a
cost estimate that considers the full costs of the plan for deploying
radiation detection portal monitors; and development of a cost-benefit
analysis based on ASPs' demonstrated performance and a complete
accounting of the portal monitor project's costs. (App. I presents a
summary of our key findings and recommendations related to ASPs.) As I
will discuss today, DNDO has improved the rigor of testing but has not
yet updated the cost-benefit analysis that is critical to a decision on
whether to replace radiation detection equipment already deployed at
ports of entry with the significantly more expensive ASPs.
Specifically, my testimony discusses (1) our key findings on the most
recent round of ASP testing and (2) lessons from ASP testing that can
be applied to other DHS technology investments. These findings are
based on our report released this week and other related GAO reports.
[Footnote 4] We conducted this performance audit work in June 2009 in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to produce a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our statement
today.
The Latest Round of Testing Highlights the Limitations of ASPs:
Our report on the latest round of ASP testing found that DHS increased
the rigor of ASP testing in comparison with previous tests and that a
particular area of improvement was in the performance testing at the
Nevada Test Site, where DNDO compared the capability of ASP and current-
generation equipment to detect and identify nuclear and radiological
materials. For example, unlike in prior tests, the plan for the 2008
performance test stipulated that there would be no system contractor
involvement in test execution. Such improvements addressed concerns we
previously raised about the potential for bias and provided credibility
to the results.
Nevertheless, based on the following factors, we continue to question
whether the benefits of the new portal monitors justify the high cost:
* The DHS criteria for a significant increase in operational
effectiveness. Our chief concern with the criteria is that they require
a marginal improvement over current-generation portal monitors in the
detection of certain weapons-usable nuclear materials when ASPs are
deployed for primary screening. DNDO considers detection of such
materials to be a key limitation of current-generation portal monitors.
We are particularly concerned about the marginal improvement required
of ASPs because the detection threshold for the current-generation
portal monitors does not specify a level of radiation shielding that
smugglers could realistically use. DOE and national laboratory
officials told us that DOE's threat guidance used to set the current
detection threshold is based not on an analysis of the capabilities of
potential smugglers to take effective shielding measures but rather on
the limited sensitivity of PVTs to detect anything more than certain
lightly shielded nuclear materials. DNDO officials acknowledge that
both the new and current-generation portal monitors are capable of
detecting certain nuclear materials only when unshielded or lightly
shielded. The marginal improvement in detection of such materials
required of ASPs is particularly notable given that DNDO has not
completed efforts to fine-tune PVTs' software and thereby improve
sensitivity to nuclear materials. DNDO officials expect they can
achieve small improvements in sensitivity, but DNDO has not yet funded
efforts to fine-tune PVTs' software. In contrast to the marginal
improvement required in detection of certain nuclear materials, the
primary screening requirement to reduce the rate of innocent alarms
could result in hundreds of fewer secondary screenings per day, thereby
reducing CBP's workload and delays to commerce. In addition, the
secondary screening criteria, which require ASPs to reduce the
probability of misidentifying special nuclear material by one-half,
address the inability of relatively small handheld devices to
consistently locate and identify potential threats in large cargo
containers.
* Preliminary results of performance testing and field validation. The
preliminary results presented to us by DNDO are mixed, particularly in
the capability of ASPs used for primary screening to detect certain
shielded nuclear materials. Preliminary results show that the new
portal monitors detected certain nuclear materials better than PVTs
when shielding approximated DOE threat guidance, which is based on
light shielding. In contrast, differences in system performance were
less notable when shielding was slightly increased or decreased: Both
the PVTs and ASPs were frequently able to detect certain nuclear
materials when shielding was below threat guidance, and both systems
had difficulty detecting such materials when shielding was somewhat
greater than threat guidance. With regard to secondary screening, ASPs
performed better than handheld devices in identification of threats
when masked by naturally occurring radioactive material. However,
differences in the ability to identify certain shielded nuclear
materials depended on the level of shielding, with increasing levels
appearing to reduce any ASP advantages over the handheld identification
devices. Other phases of testing uncovered multiple problems in meeting
requirements for successfully integrating the new technology into
operations at ports of entry. Of the two ASP vendors participating in
the 2008 round of testing, one has fallen behind due to severe problems
encountered during testing of ASPs' readiness to be integrated into
operations at ports of entry ("integration testing"); the problems may
require that the vendor redo previous test phases to be considered for
certification. The other vendor's system completed integration testing,
but CBP suspended field validation after 2 weeks because of serious
performance problems resulting in an overall increase in the number of
referrals for secondary screening compared with existing equipment.
* DNDO's plans for computer simulations. DNDO does not plan to complete
injection studies--computer simulations for testing the response of
ASPs and PVTs to simulated threat objects concealed in cargo
containers--prior to the Secretary of Homeland Security's decision on
certification even though delays to the ASP test schedule have allowed
more time to conduct the studies. According to DNDO officials,
injection studies address the inability of performance testing to
replicate the wide variety of cargo coming into the United States and
the inability to place special nuclear material and other threat
objects in cargo during field validation. DNDO had earlier indicated
that injection studies could provide information comparing the
performance of the two systems as part of the certification process for
both primary and secondary screening. However, DNDO subsequently
decided that performance testing would provide sufficient information
to support a decision on ASP certification. DNDO officials said they
would instead use injection studies to support effective deployment of
the new portal monitors.
* Lack of an updated cost-benefit analysis. DNDO has not yet updated
its cost-benefit analysis to take into account the results of the
latest round of ASP testing. An updated analysis that takes into
account the results from the latest round of testing, including
injection studies, might show that DNDO's plan to replace existing
equipment with ASPs is not justified, particularly given the marginal
improvement in detection of certain nuclear materials required of ASPs
and the potential to improve the current-generation portal monitors'
sensitivity to nuclear materials, most likely at a lower cost.
Our report recommended that the Secretary of Homeland Security direct
DNDO to (1) assess whether ASPs meet the criteria for a significant
increase in operational effectiveness based on a valid comparison with
PVTs' full performance potential and (2) revise the schedule for ASP
testing and certification to allow sufficient time for review and
analysis of results from the final phases of testing and completion of
all tests, including injection studies. We further recommended that, if
ASPs are certified, the Secretary direct DNDO to develop an initial
deployment plan that allows CBP to uncover and resolve any additional
problems not identified through testing before proceeding to full-scale
deployment. DHS agreed to a phased deployment that should allow time to
uncover ASP problems but disagreed with GAO's other recommendations,
which we continue to believe remain valid.
Procurement Decisions for New Technologies Require Rigorous Testing and
Thorough Analysis of Results:
The challenges DNDO has faced in developing and testing ASPs illustrate
the importance of following existing DHS policies as well as best
practices for investments in complex homeland security acquisitions and
for testing of new technologies. The DHS investment review process
calls for executive decision making at key points in an investment's
life cycle and includes many acquisition best practices that, if
applied consistently, could help increase the chances for successful
outcomes. However, we reported in November 2008 that, for the period
from fiscal year 2004 through the second quarter of fiscal year 2008,
DHS had not effectively implemented or adhered to its investment review
process due to a lack of senior management officials' involvement as
well as limited monitoring and resources.[Footnote 5] In particular, of
DHS's 48 major investments requiring milestone and annual reviews under
the department's investment review policy, 45 were not assessed in
accordance with this policy. In addition, many major investments,
including DNDO's ASP program, had not met the department's requirements
for basic acquisition documents necessary to inform the investment
review process. As a result, DHS had not consistently provided the
oversight needed to identify and address cost, schedule, and
performance problems in its major investments. Among other things, our
November 2008 report recommended that the Secretary of Homeland
Security direct component heads, such as the Director of DNDO, to
ensure that the components have established processes to manage major
investments consistent with departmental policies. DHS generally
concurred with our recommendations, and we noted that DHS had begun
several efforts to address shortcomings in the investment review
process identified in our report, including issuing an interim
directive requiring DHS components to align their internal policies and
procedures by the end of the third quarter of fiscal year 2009. In
January 2009, DHS issued a memorandum instructing component heads to
create acquisition executives in their organizations to be responsible
for management and oversight of component acquisition processes. If
fully implemented, these steps should help ensure that DHS components
have established processes to manage major investments.
Based on our body of work on ASP testing, one of the primary lessons to
be learned is to avoid the pitfalls in testing that stem from a rush to
procure new technologies. GAO has previously reported on the negative
consequences of pressures imposed by closely linking testing and
development programs with decisions to procure and deploy new
technologies, including the creation of incentives to postpone
difficult tests and limit open communication about test results.
[Footnote 6] We found that testing programs designed to validate a
product's performance against increasing standards for different stages
in product development are a best practice for acquisition strategies
for new technologies. In the case of ASPs, the push to replace existing
equipment with the new portal monitors led to a testing program that
until recently lacked the necessary rigor. Even for the most recent
round of testing, DNDO's schedule consistently underestimated the time
required to conduct tests, resolve problems uncovered during testing,
and complete key documents, including final test reports. In addition,
DNDO's schedule did not anticipate the time required to update its cost-
benefit analysis to take into account the latest test results. DNDO's
original working schedule anticipated completion of testing in mid-
September 2008 and the DHS Secretary's decision on ASP certification
between September and November 2008. However, testing is still not
completed, and DNDO took months longer than anticipated to complete the
final report on performance testing.
As previously mentioned, a number of aspects of the latest round of ASP
testing increased the rigor in comparison with earlier rounds and, if
properly implemented, could improve the rigor in DHS's testing of other
advanced technologies. Key aspects included the following:
* Criteria for ensuring test requirements are met. The test and
evaluation master plan established criteria requiring that the ASPs
meet certain requirements before starting or completing any test phase.
For example, the plan required that ASPs have no critical or severe
issues rendering them completely unusable or impairing their function.
The criteria provided a formal means to ensure that ASPs met certain
basic requirements prior to the start of each phase of testing. DNDO
and CBP adhered to the criteria even though doing so resulted in
integration testing taking longer than anticipated and delaying the
start of field validation.
* Participation of the technology end user. The participation of CBP
(the end user of the new portal monitors) provided an independent
check, within DHS, of DNDO's efforts to develop and test the new portal
monitors. For example, CBP added a final requirement to integration
testing before proceeding to field validation to demonstrate ASPs'
ability to operate for 40 hours without additional problems and thereby
provide for a productive field validation. In addition, the
participation of CBP officers in the 2008 round of performance testing
allowed DNDO to adhere more closely than in previous tests to CBP's
standard operating procedure for conducting a secondary inspection
using the handheld identification devices, thereby providing for an
objective test.
* Participation of an independent test authority. The DHS Science and
Technology Directorate, which is responsible for developing and
implementing the department's test and evaluation policies and
standards, will have the lead role in the final phase of ASP testing
and thereby provide an additional independent check on testing efforts.
The Science and Technology Directorate identified two critical
questions, related to ASPs' operational effectiveness (i.e., detection
and identification of threats) and suitability (e.g., reliability,
maintainability, and supportability), and drafted its own test plan to
address those questions.
Mr. Chairman, this completes my prepared statement. I would be happy to
respond to any questions that you or other Members of the Subcommittee
may have at this time.
GAO Contact and Staff Acknowledgments:
For further information about this testimony, please contact me at
(202) 512-3841 or aloisee@gao.gov. Ned Woodward (Assistant Director),
Joseph Cook, and Kevin Tarmann made key contributions to this
testimony. Dr. Timothy Persons (Chief Scientist), James Ashley, Steve
Caldwell, John Hutton, Omari Norman, Alison O'Neill, Amelia Shachoy,
and Rebecca Shea also made important contributions.
[End of section]
Appendix I: Key Findings and Recommendations from Related GAO Products
on Testing and Development of ASPs:
Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation
Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain.
[hyperlink, http://www.gao.gov/products/GAO-06-389]. Washington, D.C.:
March 22, 2006.
* Key findings. Prototypes of advanced spectroscopic portals (ASP) were
expected to be significantly more expensive than current-generation
portal monitors but had not been shown to be more effective. For
example, Domestic Nuclear Detection Office (DNDO) officials'
preliminary analysis of 10 ASPs tested at the Nevada Test Site found
that the new portal monitors outperformed current-generation equipment
in detecting numerous small, medium-size, and threatlike radioactive
objects and were able to identify and dismiss most naturally occurring
radioactive material. However, the detection capabilities of both types
of portal monitors converged as the amount of source material
decreased.
* Recommendations. We recommended that the Secretary of Homeland
Security work with the Director of DNDO to analyze the benefits and
costs of deploying ASPs before any of the new equipment is purchased to
determine whether any additional detection capability is worth the
additional cost. We also recommended that the total program cost
estimate for the radiation portal monitor project be revised after
completion of the cost-benefit analysis.
Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the
Purchase of New Radiation Detection Portal Monitors Was Not Based on
Available Performance Data and Did Not Fully Evaluate All the Monitors'
Costs and Benefits. [hyperlink,
http://www.gao.gov/products/GAO-07-133R]. Washington, D.C.: October 17,
2006.
Combating Nuclear Smuggling: DHS's Decision to Procure and Deploy the
Next Generation of Radiation Detection Equipment Is Not Supported by
Its Cost-Benefit Analysis. [hyperlink,
http://www.gao.gov/products/GAO-07-581T]. Washington, D.C.: March 14,
2007.
* Key findings. DNDO's cost-benefit analysis issued in response to our
March 2006 recommendation did not provide a sound analytical basis for
DNDO's decision to purchase and deploy ASPs. We identified a number of
problems with the analysis of both the performance of the new portal
monitors and the costs. With regard to performance, DNDO did not use
the results of its own tests and instead relied on assumptions of the
new technology's anticipated performance level. In addition, the
analysis focused on identifying highly enriched uranium (HEU) and did
not consider how well the new portal monitors can correctly detect or
identify other dangerous radiological or nuclear materials. With regard
to costs, DNDO did not follow the DHS guidelines for performing cost-
benefit analyses and used questionable assumptions about the
procurement costs of portal monitor technology.
* Recommendations. We recommended that DHS and DNDO conduct a new cost-
benefit analysis using sound analytical methods, including actual
performance data and a complete accounting of all major costs and
benefits as required by DHS guidelines, and that DNDO conduct realistic
testing for both ASPs and current-generation portal monitors.
Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the
National Laboratories' Test Results on Radiation Portal Monitors in
Support of DNDO's Testing and Development Program. [hyperlink,
http://www.gao.gov/products/GAO-07-347R]. Washington, D.C.: March 9,
2007.
* Key findings. DNDO had not collected a comprehensive inventory of
testing information on current-generation portal monitors. Such
information, if collected and used, could improve DNDO's understanding
of how well portal monitors detect different radiological and nuclear
materials under varying conditions. In turn, this understanding would
assist DNDO's future testing, development, deployment, and purchases of
portal monitors.
* Recommendations. We recommended that the Secretary of Homeland
Security, working with the Director of DNDO, collect reports concerning
all of the testing of current-generation portal monitors and review the
test reports in order to develop an information database on how the
portal monitors perform in both laboratory and field tests on a variety
of indicators, such as their ability to detect specific radiological
and nuclear materials.
Combating Nuclear Smuggling: Additional Actions Needed to Ensure
Adequate Testing of Next Generation Radiation Detection Equipment.
[hyperlink, http://www.gao.gov/products/GAO-07-1247T]. Washington,
D.C.: September 18, 2007.
* Key findings. We found that tests conducted by DNDO in early 2007
were not an objective and rigorous assessment of the ASPs'
capabilities. Specifically, we raised concerns about DNDO using biased
test methods that enhanced the apparent performance of ASPs; not
testing the limitations of ASPs' detection capabilities--for example,
by not using a sufficient amount of the type of materials that would
mask or hide dangerous sources and that ASPs would likely encounter at
ports of entry; and not using a critical Customs and Border Protection
(CBP) standard operating procedure that is fundamental to the
performance of handheld radiation detectors in the field.
* Recommendations. We recommended that the Secretary of Homeland
Security delay Secretarial certification and full-scale production
decisions on ASPs until all relevant tests and studies had been
completed and limitations to tests and studies had been identified and
addressed. We further recommended that DHS determine the need for
additional testing in cooperation with CBP and other stakeholders and,
if additional testing was needed, that the Secretary of DHS appoint an
independent group within DHS to conduct objective, comprehensive, and
transparent testing that realistically demonstrates the capabilities
and limitations of ASPs.
Combating Nuclear Smuggling: DHS's Program to Procure and Deploy
Advanced Radiation Detection Portal Monitors Is Likely to Exceed the
Department's Previous Cost Estimates. [hyperlink,
http://www.gao.gov/products/GAO-08-1108R]. Washington, D.C.: September
22, 2008.
* Key findings. Our independent cost estimate suggested that from 2007
through 2017 the total cost of DNDO's 2006 project execution plan (the
most recent official documentation of the program to equip U.S. ports
of entry with radiation detection equipment) would likely be about $3.1
billion but could range from $2.6 billion to $3.8 billion. In contrast,
we found that DNDO's cost estimate of $2.1 billion was unreliable
because it omitted major project costs, such as elements of the ASPs'
life cycle, and relied on a flawed methodology. DNDO officials told us
that the agency was no longer following the 2006 project execution plan
and that the scope of the agency's ASP deployment strategy had been
reduced to only the standard cargo portal monitor. Our analysis of
DNDO's summary information outlining its scaled-back plan indicated the
total cost to deploy standard cargo portals over the period 2008
through 2017 would be about $2 billion but could range from $1.7
billion to $2.3 billion. Agency officials acknowledged the program
requirements that would have been fulfilled by the discontinued ASPs
remained valid, including screening rail cars and airport cargo, but
the agency had no plans for how such screening would be accomplished.
* Recommendations. We recommended that the Secretary of Homeland
Security direct the Director of DNDO to work with CBP to update the
projection execution plan to guide the entire radiation detection
program at U.S. ports of entry, revise the estimate of the program's
cost and ensure that the estimate considers all of the costs associated
with its project execution plan, and communicate the revised estimate
to Congress so that it is fully apprised of the program's scope and
funding requirements.
Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and
Complete All Tests Prior to Making a Decision on Whether to Purchase
Advanced Portal Monitors. [hyperlink,
http://www.gao.gov/products/GAO-08-1178T]. Washington, D.C.: September
25, 2008.
* Key findings. In preliminary observations of the 2008 round of ASP
testing, we found that DNDO had made progress in addressing a number of
problems we identified in previous rounds of ASP testing. However, the
DHS criteria for significant increase in operational effectiveness
appeared to set a low bar for improvement--for example, by requiring
ASPs to perform at least as well as current-generation equipment when
nuclear material is present in cargo but not specifying an actual
improvement. In addition, the ASP certification schedule did not allow
for completion of computer simulations that could provide useful data
on ASP capabilities prior to the Secretary's decision on certification.
Finally, we questioned the replacement of current-generation equipment
with ASPs until DNDO demonstrates that any additional increase in
security would be worth the ASPs' much higher cost.[Footnote 7]
Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced
Portal Monitors Does Not Fully Disclose the Limitations of the Test
Results. [hyperlink, http://www.gao.gov/products/GAO-08-979].
Washington, D.C.: September 30, 2008.
* Key findings. DNDO's report on the second group of ASP tests in 2007
(the Phase 3 tests) did not appropriately state test limitations. As a
result, the report did not accurately depict the results and could
potentially be misleading. The purpose of the Phase 3 tests was to
conduct a limited number of test runs in order to identify areas in
which the ASP software needed improvement. While aspects of the Phase 3
report addressed this purpose, the preponderance of the report went
beyond the test's original purpose and made comparisons of the
performance of the ASPs with one another or with currently deployed
portal monitors. We found that it would not be appropriate to use the
Phase 3 test report in determining whether the ASPs represent a
significant improvement over currently deployed radiation equipment
because the limited number of test runs did not support many of the
comparisons of ASP performance made in the report.
* Recommendations. We recommended that the Secretary of DHS use the
results of the Phase 3 tests solely for the purposes for which they
were intended--to identify areas needing improvement--and not as a
justification for certifying whether the ASPs warrant full-scale
production. If the Secretary intends to consider the results of the
Phase 3 tests in making a certification decision regarding ASPs, we
further recommended that the Secretary direct the Director of DNDO to
revise and clarify the Phase 3 test report to more fully disclose and
articulate the limitations present in the Phase 3 tests and clearly
state which insights from the Phase 3 report are factored into any
decision regarding the certification that ASPs demonstrate a
significant increase in operational effectiveness. Finally, we
recommended that the Secretary direct the Director of DNDO to take
steps to ensure that any limitations associated with the 2008 round of
testing are properly disclosed when the results are reported.
Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation
Detection Portal Monitors, but Preliminary Results Show Limits of the
New Technology. [hyperlink, http://www.gao.gov/products/GAO-09-655].
Washington, D.C.: May 21, 2009.
* Key findings. We reported that the DHS criteria for a significant
increase in operational effectiveness require a large reduction in
innocent alarms but a marginal improvement in the detection of certain
weapons-usable nuclear materials. In addition, the criteria do not take
the current-generation portal monitors' full potential into account
because DNDO has not completed efforts to improve their performance.
With regard to ASP testing, we found that DHS increased the rigor in
comparison with previous tests, thereby adding credibility to the test
results, but that preliminary results were mixed. The results showed
that the new portal monitors performed better than current-generation
portal monitors in detection of certain nuclear materials concealed by
light shielding approximating the threat guidance for setting detection
thresholds, but differences in sensitivity were less notable when
shielding was slightly below or above that level. Testing also
uncovered multiple problems in ASPs meeting the requirements for
successful integration into operations at ports of entry. Finally, we
found that DNDO did not plan to complete computer simulations that
could provide additional insight into ASP capabilities and limitations
prior to certification even though delays to testing allowed more time
to conduct the simulations.
* Recommendations. We recommended that the Secretary of Homeland
Security direct the Director of DNDO to assess whether ASPs meet the
criteria for a significant increase in operational effectiveness based
on a valid comparison with current-generation portal monitors' full
performance potential and revise the schedule for ASP testing and
certification to allow sufficient time for review and analysis of
results from the final phases of testing and completion of all tests,
including computer simulations. If ASPs are certified, we further
recommended that the Secretary of Homeland Security direct the Director
of DNDO to develop an initial deployment plan that allows CBP to
uncover and resolve any additional problems not identified through
testing before proceeding to full-scale deployment.
[End of section]
Footnotes:
[1] DNDO was established within DHS in 2005; its mission includes
developing, testing, acquiring, and supporting the deployment of
radiation detection equipment at U.S. ports of entry. CBP began
deploying portal monitors in 2002, prior to DNDO's creation, under the
radiation portal monitor project.
[2] GAO, Combating Nuclear Smuggling: DHS's Program to Procure and
Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed
the Department's Previous Cost Estimates, [hyperlink,
http://www.gao.gov/products/GAO-08-1108R] (Washington, D.C.: Sept. 22,
2008).
[3] Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 121
Stat. 1844, 2069 (2007); Consolidated Security, Disaster Assistance,
and Continuing Appropriations Act, 2009, Pub. L. No. 110-329, 121 Stat.
3574, 3679 (2008).
[4] GAO, Combating Nuclear Smuggling: DHS Improved Testing of Advanced
Radiation Detection Portal Monitors, but Preliminary Results Show
Limits of the New Technology, [hyperlink,
http://www.gao.gov/products/GAO-09-655] (Washington, D.C.: May 21,
2009).
[5] GAO, Department of Homeland Security: Billions Invested in Major
Programs Lack Appropriate Oversight, [hyperlink,
http://www.gao.gov/products/GAO-09-29] (Washington, D.C.: Nov. 18,
2008).
[6] GAO, Best Practices: A More Constructive Test Approach Is Key to
Better Weapon System Outcomes, [hyperlink,
http://www.gao.gov/products/GAO/NSIAD-00-199] (Washington, D.C.: July
31, 2000).
[7] This testimony also summarized our September 2008 report on the
life cycle cost estimate to deploy ASPs [hyperlink,
http://www.gao.gov/products/GAO-08-1108R].
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Phone:
The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site,
[hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548: