Summary
The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are part of the U.S. defense against such threats. In 2007, Congress required that funds for new advanced spectroscopic portal (ASP) monitors could not be spent until the Secretary of DHS certified that these machines represented a significant increase in operational effectiveness over currently deployed portal monitors. In addition to other tests, DNDO conducted the Phase 3 tests on ASPs to identify areas in which the ASPs needed improvement. GAO was asked to assess (1) the degree to which the Phase 3 test report accurately depicts the test results and (2) the appropriateness of using the Phase 3 test results to determine whether ASPs represent a significant improvement over current radiation detection equipment. GAO also agreed to provide its observations on special tests conducted by Sandia National Laboratories (SNL).
Because the limitations of the Phase 3 test results are not appropriately stated in the Phase 3 test report, the report does not accurately depict the results from the tests and could potentially be misleading. In the Phase 3 tests, DNDO performed a limited number of test runs. Because of this, the test results provide little information about the actual performance capabilities of the ASPs. The report often presents each test result as a single value; but considering the limited number of test runs, the results would be more appropriately stated as a range of potential values. For example, the report narrative states in one instance that an ASP could identify a source material during a test 50 percent of the time. However, the narrative does not disclose that, given the limited number of test runs, DNDO can only estimate that the ASP would correctly identify the source from about 15 percent to about 85 percent of the time--a result that lacks the precision implied by DNDO's narrative. DNDO's reporting of the test results in this manner makes them appear more conclusive and precise than they really are. The purpose of the Phase 3 tests was to conduct a limited number of test runs in order to identify areas in which the ASP software needed improvement. While aspects of the Phase 3 report address this purpose, the preponderance of the report goes beyond the test's original purpose and makes comparisons of the performance of the ASPs with one another or with currently deployed portal monitors. In GAO's view, it is not appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs do not support many of the comparisons of ASP performance made in the Phase 3 report. As the report shows, if an ASP can identify a source material every time during a test, but the test is run only five times, the only thing that can be inferred with a high level of statistical confidence is that the probability of identification is no less than about 60 percent. Although DNDO states in the Phase 3 test report that the results will be relevant to the Secretary's certification that the ASPs represent a significant increase in operational effectiveness, it does not clarify in what ways the results will be relevant. Furthermore, DNDO offers no explanation as to why it changed its view from the Phase 3 test plan, which states that these tests will not be used to support a certification decision. The goal of SNL's special tests was, among other things, to identify potential vulnerabilities in the ASPs by using different test scenarios from those that DNDO planned to use in other ASP tests. SNL concluded in its test report that the ASPs' software and hardware can be improved and that rigor could be added to DNDO's testing methods. Furthermore, the report acknowledges that (1) a specific objective of the testing at the Nevada Test Site was to refine and improve the ASP's performance and (2) the special tests were never intended to demonstrate conformity of the ASPs with specific performance requirements. In GAO's view, these statements appear to accurately describe the purpose, limitations, and results of the special tests.
Recommendations
Our recommendations from this work are listed below with a Contact for more information. Status will change
from "In process" to "Implemented" or "Not implemented" based on our follow up work.
Director:
Team:
Phone:
Eugene E. Aloise
Government Accountability Office: Applied Research and Methods
(202) 512-6870
Recommendations for Executive Action
Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, the Secretary of DHS should use the results of the Phase 3 tests solely for the purposes for which they were intended--to identify areas needing improvement, not as a justification for certifying whether the ASPs warrant full-scale production.
Agency Affected: Department of Homeland Security
Status: In process
Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then the Secretary should direct the Director of DNDO to revise and clarify the Phase 3 test report to more fully disclose and articulate the limitations present in the Phase 3 tests--particularly the limitations associated with making comparisons between detection systems from a small number of test runs.
Agency Affected: Department of Homeland Security
Status: In process
Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then the Secretary should clearly state which "relevant insights into important aspects of system performance" from the Phase 3 report are factored into any decision regarding the certification that ASPs demonstrate a significant increase in operational effectiveness.
Agency Affected: Department of Homeland Security
Status: In process
Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and since there are several phases of additional ASP testing currently ongoing, the Secretary should direct the Director of DNDO take steps to ensure that any limitations associated with ongoing testing are properly disclosed when the results of the current testing are reported.
Agency Affected: Department of Homeland Security
Status: In process
Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
