This is the accessible text file for GAO report number GAO-08-1108R entitled 'Combating Nuclear Smuggling: DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates' which was released on September 22, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-08-1108R: United States Government Accountability Office: Washington, DC 20548: September 22, 2008: Congressional Requesters: Subject: Combating Nuclear Smuggling: DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates: Since the attacks of September 11, 2001, combating terrorism has been one of the nation's highest priorities. As part of that effort, preventing nuclear and radioactive material from being smuggled into the United States--perhaps to be used by terrorists in a nuclear weapon or in a radiological dispersal device (a "dirty bomb")--has become a key national security objective. On April 15, 2005, the president directed the establishment, within the Department of Homeland Security (DHS), of the Domestic Nuclear Detection Office (DNDO), whose duties include acquiring and supporting the deployment of radiation detection equipment. In October 2006, Congress enacted the SAFE Port Act, [Footnote 1] which made DNDO responsible for the development, testing, acquisition and deployment of a system to detect radiation at U.S. ports of entry. An important component of this system is the deployment of radiation portal monitors, large stationary detectors through which cargo containers and trucks pass as they enter the United States. Prior to DNDO's creation, another DHS agency--U.S. Customs and Border Protection (CBP)--managed programs for deployment of radiation detection equipment. In 2002, CBP began the radiation portal monitor project, deploying radiation detection equipment at U.S. ports of entry. This program initially deployed portal monitors, known as polyvinyl toluene monitors (PVT), and handheld detection technologies, such as radioactive isotope identification devices (RIID). CBP also established a system of standard operating procedures to guide its officers in the use of this equipment. Current procedures include conducting primary inspections with PVTs to detect the presence of radioactivity, and secondary inspections with PVTs and RIIDs to confirm and identify the source and determine whether it constitutes a threat. After its creation, DNDO assumed responsibility for the development, testing, and deployment of radiation detection equipment, while CBP maintained its role of operating the equipment at U.S. ports of entry. Currently deployed PVTs are capable of detecting radiation, but they have an inherent limitation because they are unable to identify specific radioactive isotopes and therefore cannot distinguish between dangerous and benign materials. CBP officers also use RIIDs to identify different types of radioactive material. However, RIIDs are limited in their ability to identify nuclear material. DNDO believes that these deficiencies may delay legitimate commerce at ports of entry, and that CBP may use an inordinate amount of inspection resources for radiation detection at the expense of other missions, such as drug interdiction. To address the limitations of the PVTs and RIIDs, DNDO sponsored the development of a next-generation portal monitor--the advanced spectroscopic portal (ASP). In contrast to PVTs, ASPs offer the ability to both detect and identify radioactive material, according to DNDO. The use of ASPs may eliminate, or at least greatly reduce, the need for RIIDs while minimizing both missed threats and false alarms. In September 2006, DNDO worked with CBP to produce a project execution plan that defines the radiation portal monitor project's objectives, scope, schedule, costs, and funding requirements to deploy radiation detection systems at U.S. ports of entry.[Footnote 2] DNDO plans to deploy ASPs at high-volume ports of entry that also experience high numbers of false positive and nuisance alarms. In these locations, ASPs would be deployed in primary and secondary inspection areas. In low- volume ports of entry--that can tolerate a higher false alarm rate-- PVTs would be placed in primary inspection areas and ASPs in secondary inspection areas. DNDO expects this approach to be cost effective and to provide a balance of cost and performance. DNDO is now testing a variation of the ASP, designed to screen cargo on standard sized trucks. Eventually, however, the project execution plan calls for DNDO to deploy several variations of the ASP in order to screen cargo on other types of conveyance, such as rail cars and at seaport terminals. In the meantime, DNDO is working toward deploying ASPs in secondary inspection locations, and possibly also in some primary inspection locations, perhaps as soon as the fall of 2008. Congressional committees have expressed concern over DNDO's estimates of ASP program costs. In February 2003, CBP completed the initial estimate of the total cost of the radiation portal monitor project. At that time, the agency reported that equipping U.S. ports of entry with PVT portal monitors would likely range from $399 million to $484 million. By September 2003, the scope of the program had increased and its estimated cost had grown to $496 million. In December 2004, when CBP began to anticipate deploying ASPs, the expected cost of the project rose again to $1.3 billion. This significant increase was due at least in part to the higher cost of the ASPs. DNDO took control of the program in April 2005, and in February 2007, raised the cost estimate to $1.7 billion. By March 2008, it had reached $2.1 billion. Further, between May 2006 and March 2008, DNDO's unit cost estimate for the standard cargo version of the ASP (including deployment costs) increased from about $576,400 to about $800,000. DNDO's current unit cost estimate of PVT standard cargo portals is about $425,000 (including deployment costs). Congressional committee concern regarding these costs was evident in the explanatory note accompanying DHS's 2008 appropriation: "While the current generation technologies may not have the expected capabilities of ASP systems, they are substantially less expensive, proven to work within known limits, and can be quickly deployed to POEs [ports of entry]." In an attempt to understand the cost of DHS's deployment of radiation detection portal monitors, we were requested to review DNDO's assessment of the program's costs and benefits. In October 2006, we reported, among other things, that DNDO's independent cost estimate of the program was incomplete.[Footnote 3] Specifically, we found the cost- estimating methodology used for the agency's analysis did not adhere to DHS guidelines and it omitted several key factors that could impact the ASPs' cost. For example, DNDO did not capture all of the ASPs' developmental costs, nor did the agency estimate the ASPs' full life- cycle costs. We concluded that DNDO's cost-benefit analysis did not provide sufficient justification for buying and deploying ASPs. In this context, you asked us to review (1) the projected costs to implement DNDO's 2006 project execution plan for the radiation portal monitor program, which includes the costs to develop, procure, deploy, operate, and maintain ASPs and other RPMs over a 10-year life cycle; and (2) the reliability of DNDO's projections given what is known about the costs to procure, install, and operate radiation detection equipment. During the course of our review, DNDO officials told us the agency's 2006 radiation portal monitor deployment strategy had been revised. According to these officials, the only ASP equipment DNDO now plans to deploy to U.S. ports of entry is the standard cargo portal.[Footnote 4] DNDO first notified us of this new approach in October 2007. Although we requested detailed documentation of the agency's revised portal monitor strategy, the extent of the documentation we received consisted of a 1-page spreadsheet of summary information regarding the quantity and costs of its revised deployment plans that DNDO provided in late July 2008. Based on this incomplete information, we are also providing a limited analysis of the summary data DNDO provided regarding its revised portal monitor deployment strategy, although this revised strategy has not been officially documented. To address these issues, we contracted with a company whose expertise is in estimating the life cycle costs of major federal acquisitions. Together we developed an independent cost estimate of 2006 DHS's strategy to deploy radiation detection portal monitors, covering fiscal years 2007 through 2017, to ensure that the Congress has authoritative information on all the life-cycle costs associated with a full scale acquisition of radiation portal monitors. We briefed your staffs on the results of our work in May, June, and July 2008. This report presents the details of those briefings, which focused mainly on our estimate of the life-cycle costs associated with the deployment of radiation detection portal monitors at U.S. ports of entry, and how our estimate compares with DNDO's estimate of those costs (see encl. I). To conduct our review and develop our independent cost estimate, we held discussions with officials from DNDO, CBP, and Pacific Northwest National Laboratory--which manages the deployment of radiation detection equipment for DHS. We compiled and analyzed information on the historical costs of PVTs and ASPs, as well as data on their development and deployment. In addition, we developed a cost-estimating model to generate specific estimates of present and future radiation portal monitor project costs. We analyzed the uncertainty associated with our estimates to help gauge their accuracy. In March 2008, we provided DNDO an extensive briefing on our cost-estimating methodology, including all supporting documentation and analysis, so that DNDO could better understand our approach and perhaps provide additional information to enhance our approach. DNDO offered suggestions to improve our cost-estimating methodology. We incorporated DNDO's suggestions as we deemed appropriate. A detailed explanation of our methodology is provided in enclosure II. We conducted this performance audit from October 2007 to August 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Summary: Our independent cost estimate suggests that from 2007 through 2017 the total cost of DNDO's program to equip U.S. ports of entry with radiation detection equipment will likely be about $3.1 billion, but could range from $2.6 billion to $3.8 billion. We based our estimate on the anticipated costs of DNDO implementing its 2006 project execution plan, the most recent official documentation of the program. According to this plan, DNDO will buy and deploy multiple types of ASPs, including those designed to screen rail cars, and airport and seaport cargo, as well as mobile ASPs--spectroscopic equipment mounted on vehicles--to provide greater flexibility in screening commerce. The project execution plan also targets several types of PVTs for purchase and deployment. DNDO's cost estimate of $2.1 billion to equip U.S. ports of entry with radiation detection equipment is unreliable because it omits major project costs and relies on a flawed methodology. For example, although the normal life expectancy of the standard cargo ASP is about 10 years, DNDO's estimate considers only 8 years--fiscal years 2006 through 2013. According to DNDO officials, OMB's budget submission software allows only a limited number of years of costs to be included. Furthermore, DNDO's cost estimate does not include all of the elements of the ASPs' life cycle, as it omits estimates for maintenance and operational sustainment of ASPs. Finally, contrary to OMB and DHS guidelines, DNDO did not provide detailed documentation of ASP costs, which raises questions about the adequacy and reliability of the agency's estimates. DNDO officials told us on several occasions during the course of our review the agency is no longer following the 2006 project execution plan. These officials told us the scope of the agency's current ASP deployment strategy has been reduced to only the standard cargo portal monitor. Although we repeatedly requested documentation of DNDO's current official deployment strategy, the agency did not provide such official information. In fact, DNDO officials continued to cite the 2006 project execution plan as the most recent official deployment documentation. In July 2008, the agency provided a 1-page spreadsheet of summary information outlining DNDO's current plans to buy and deploy ASPs and PVTs. Our analysis of these summary data indicates the total cost to deploy standard cargo portals over the period 2008 through 2017 will be about $2.0 billion, but could range from $1.7 billion to $2.3 billion. These data also indicate that between fiscal years 2008 and 2014, DNDO plans to deploy 717 ASP and 1,005 PVT standard cargo portals. Furthermore, agency officials acknowledged the program requirements that would have been fulfilled by the discontinued ASPs remain valid, including screening rail cars, airport cargo, and cargo at seaport terminals, but the agency has no current plans for how such screening will be accomplished. These officials told us the technology to accomplish these requirements likely will not be ASP monitors. We believe a comprehensive estimate of the cost to provide radiation detection equipment for U.S. ports of entry should account for meeting these objectives, even if DNDO decides that ASP technology is not suited to them. However, a DNDO official responsible for overseeing the agency's operations told us in August 2008 that DNDO's ASP deployment strategy could change dramatically depending on the outcome of ongoing ASP testing. In our view, it is difficult to assess the total costs of the ASP program because of the frequent changes in DNDO's deployment strategy. Furthermore, the Congress needs a complete understanding of DNDO's deployment strategy before approving additional ASP program funds. We provided a draft of this report to DHS for its review and comment. In its comments, the Department agreed with our recommendations and part of our conclusions, but strongly disagreed other parts of our conclusions. In addition, DHS commented that we used inconsistent terms in referring to "cost" and that as a result it is impossible to verify whether our assessments and conclusions are valid, or to conduct an "apples to apples" comparison. We disagree with DHS's assertion that we used inconsistent terms for "cost" and as a result it was impossible to verify our conclusions or to conduct a true "apples to apples" comparison. Our use of the various terms related to cost is accurate and consistent with the technical vocabulary of cost estimating. Even so, we made minor changes to the report to make the terminology more uniform and improve its readability. With regard to DNDO's comment about assessing the validity of our conclusions, over the course of our review, we provided extensive briefings and documentation on the components and results of our life-cycle cost estimate. In fact, according to a DNDO document (dated May 29, 2008) that summarized agency efforts to develop its own life-cycle cost estimate, DNDO used our cost estimate as its initial baseline model in starting to develop its own cost estimate. DHS commented that comparing our cost estimate to DNDO's is misleading because we included operation and maintenance costs--incurred by CBP-- in our analysis. We disagree that the draft report is misleading when it compares DNDO and GAO life-cycle costs. In fact, we believe our estimate helps clarify the program's true cost, while DNDO's estimate obscures it. The source for DNDO's $2.1 billion cost estimate is the agency's most recent OMB Exhibit 300, dated March 25, 2008. According to OMB's Exhibit 300 instructions for completing multi-agency investments, agencies should include: (a) the costs associated with the entire life cycle of the investment; and (b) the funding from the lead and all partner agencies [emphasis added]. By omitting major project costs (such as maintenance costs), DNDO has understated the life-cycle costs of the program and violated the intent of OMB policy and guidance. The effect of these omissions increases the probability of misinformed decisions and inadequate budget formulation for partner agencies. DHS offered several other comments which are discussed at the end of this letter. DNDO's Program to Deploy Radiation Detection Portal Monitors at U.S. Ports of Entry Is Likely to Cost About $3 Billion: Our independent cost estimate suggests the total cost of DNDO's program to equip U.S. ports of entry with radiation detection equipment will likely be about $3.1 billion, but could range between $2.6 billion and $3.8 billion. We based our estimate on the anticipated costs of DNDO implementing its 2006 project execution plan. According to this plan, DNDO plans to buy and deploy several types of ASPs, including those designed to screen rail cars and seaport cargo; as well as mobile ASPs to provide greater flexibility in screening commerce. The plan also provides for the deployment of several types of PVTs. Clearly, the numbers and types of portal monitors deployed will significantly affect the total cost of the radiation portal monitor project. In all, the project execution plan calls for the purchase of 2,754 portal monitors, with a total of 2,582 scheduled for deployment--approximately 1,034 ASPs and 1,548 PVTs--and 172 held in excess at the project's completion. In developing our estimate, we categorized radiation portal monitor project costs according to program phases--design and development, procurement, deployment, maintenance, and operational sustainment. We did not incorporate operational costs, in particular the cost of CBP officers operating the radiation detection equipment. DNDO and CBP believe that deploying ASPs will reduce the use of CBP staff resources for radiation detection tasks, but currently there are no usable estimates of how CBP's staffing would change with the deployment of ASPs. Finally, our analysis includes a period of 11 years, actual life- cycle expenses from fiscal year 2007 and estimated life cycle costs from fiscal year 2008 through 2017. The details of this analysis are presented in enclosure III. DNDO's Cost Estimates Are Unreliable and Could Result in Significant Cost Overruns: In submitting its budget request to OMB in March 2008, DNDO estimated the total cost of the radiation portal monitor project at $2.1 billion. However, DNDO's estimate is unreliable because it omits major project costs and relies on flawed methodology. As a result, DNDO's cost estimates and budget requests for the radiation portal monitor project are too low, which could lead to significant cost overruns later in the project. DNDO's estimate contains three major deficiencies. First, it does not appear to include the costs of all variations of ASPs contained in DNDO's project execution plan. DNDO's current baseline considers only the standard ASP cargo portal and ignores the costs of other types of ASPs, such as those designed to screen rail cars, airport and seaport cargo, and mobile detectors. According to senior DNDO officials, the current approved project execution plan no longer reflects the agency's procurement and deployment plans for ASPs. In acknowledging that the project execution plan should be revised and updated, these officials told us that DNDO currently plans to field only the standard cargo ASP portal--mainly at high-volume ports of entry--but DNDO did not provide us detailed documentation to support this major programmatic decision. Second, although the normal life expectancy of the ASP is about 10 years, DNDO's estimate considers only fiscal years 2006 through 2013. DNDO officials told us this was because OMB's budget submission software limits the number of years of costs that can be included. However, DHS cost guidance maintains that a project's life cycle can be estimated over that period of time during which equipment will remain available before it is exhausted, that is, decayed or deteriorated. The manufacturer of the sodium iodide crystals that comprise a key component of ASP systems expects an operational life of about 10 years for its crystals.[Footnote 5] DNDO officials agreed that a 10-year life cycle cost estimate would have been more appropriate, and added that they would have used a 10-year estimate had they not been constrained by the OMB software. Third, DNDO's cost estimate does not include all of the elements of the ASPs' life cycle. For example, the agency's estimate does not include cost estimates for maintenance or operational sustainment of ASPs. These costs are approximately $999.2 million and $364.9 million, respectively, under our cost estimate. Furthermore, DNDO's cost methodology and documentation contain significant weaknesses, which may further reduce the reliability of the agency's cost estimates. For example, DNDO was unable to provide detailed documentation of the costs used in its estimates. Such documentation is necessary, according to OMB, DHS, and GAO guidelines, to establish the basis of the estimates and to provide assurances that the estimates are credible.[Footnote 6] For fiscal years 2008 through 2012, the years included in both our analysis and the DNDO analysis, we estimate a $753 million budget shortfall for the radiation portal monitor project. Additionally, we estimate that DNDO will require another $833 million from fiscal year 2013 through fiscal year 2017 to complete the entire life cycle of the program. Our analysis projects some of the greatest deficits in fiscal years 2008, 2009, and 2010, when DNDO plans to acquire large numbers of portal monitors. DNDO's Revised Program to Deploy Radiation Detection Portal Monitors at U.S. Ports of Entry Is Likely to Cost about $2.0 Billion: We met with DNDO officials in March and June 2008 to discuss the preliminary findings of this report. At those times, they noted--as they had in October 2007--that the project execution plan no longer guides the radiation portal monitor project. Hence, in their view, because our cost estimate is based on the project execution plan, it likely will result in estimates higher than the program's current true cost since DNDO has eliminated all types of ASPs, except the standard cargo portal. In our view, however, estimates of any program's costs must be based on the agency's documented program. In the absence of more recent documentation, we believe our independent cost estimate must be based on the agency's most recent approved plan--the project execution plan. Furthermore, agency officials acknowledged the program requirements that would have been fulfilled by the discontinued ASPs remain valid, including screening rail cars, airport cargo, and cargo at seaport terminals, but the agency has no current plans for how such screening will be accomplished. These officials told us the technology to accomplish these requirements likely will not be ASP monitors. We believe a comprehensive estimate of the cost to provide radiation detection equipment for U.S. ports of entry should account for meeting these objectives, even if DNDO decides that ASP technology is not suited to them. At our June 2008 meeting, DNDO agreed to update its project execution plan, so that we could better estimate the costs of the agency's current plans. In addition, in light of the fact that DNDO has decided to procure and deploy only the standard cargo ASP monitor, we agreed to use DNDO's revised data to re-estimate the standard cargo portal's life- cycle costs. Furthermore, DNDO also agreed to provide its own updated estimate of the standard portal's life cycle costs, and to meet with us in mid-August 2008 to reconcile our two estimates. However, DNDO delivered neither the promised revised project execution plan nor a revised cost estimate. Instead, in July 2008, the agency provided a 1- page spreadsheet of summary information outlining DNDO's plans to buy and deploy portal monitors--ASPs and PVTs--for the 7-year period 2008 through 2014. DNDO's summary data indicate that during this time period the agency plans to deploy 717 ASPs and 1,005 PVTs. The summary data do not provide the breadth and depth of information needed to generate detailed and fully documented cost estimates. Furthermore, according to subsequent discussions with a senior DNDO official, if ongoing tests indicate the ASPs' performance warrants it, the agency may speed its deployment of ASPs over the next few months. In our view, the frequent changes in deployment plans, and the lack of available cost documentation, raises concerns about the overall management of the radiation portal monitor project, and whether it is guided by a sound and stable strategy. Despite DNDO's failure to provide revised detailed program data, we used the agency's summary data to perform a more limited cost estimate for only the standard cargo portal. The details of that analysis are presented in enclosure IV. In general, we found that from 2008 to 2017 the total program cost for buying and deploying standard cargo portals would likely be about $2 billion, but could range from about $1.7 billion to $2.3 billion. Conclusions: Although combating nuclear smuggling is one of the nation's highest priorities and the deployment of radiation detection portal monitors is one of our main weapons in that struggle, DNDO has not yet provided Congress the full scope or costs of the radiation portal monitor program. Our estimate of $3.1 billion is based on the cost to implement the program, as specified in the agency's September 2006 official program planning documentation. However, in maintaining that these plans no longer reflect the agency's goals and objectives for deploying portal monitors at the nation's ports of entry, and that agency plans currently include only the standard cargo portal, senior DNDO officials acknowledge a deployment program that is dramatically different in scope than the one presented to and approved by the Congress. Program officials now state the program includes only the standard cargo ASP-- a significant reduction in planned ASP equipment. However, DNDO officials agreed that while the program requirements to screen rail cars and extra-wide trucks have not been eliminated, the agency has no current plans for how such screening will be accomplished. These officials told us that the technology to accomplish these requirements likely will not be ASP monitors. Regardless, there will be costs associated with whatever technology DNDO adopts and those costs should be included in the agency's cost estimate--as ideally DNDO would report on the total costs to equip U.S. ports of entry with radiation detection capability, regardless of the technology used. If DNDO does attempt to implement its authorized 2006 project execution plan, we anticipate a $1.6 billion shortfall--$753 million for fiscal years 2008 through 2012, with an additional $833 million needed to complete the program in fiscal years 2013 through 2017. This represents additional financial risk for the U.S. government over the life of the program. DNDO will have to make up this shortfall somehow, either by returning to the Congress for additional funding, or by cutting expenses in other parts of the program, which may reduce the efficacy of the program. DNDO's decision to eliminate rail, extra-wide, and mobile ASPs from its program plans may be an effort to trim costs. In either case, the Congress will not be getting the radiation portal monitor program it initially approved. The Congress should have full knowledge and complete understanding of DNDO's deployment strategy and the cost-benefit trade-offs inherent in DNDO's portal monitor decisions before approving additional program funds. Recommendations for Executive Action: We recommend the Secretary of Homeland Security direct the Director of DNDO to take the following three actions: * Work with the Commissioner of CBP to update the projection execution plan to guide the entire radiation detection program at U.S. ports of entry. The new project execution plan should be based on documented requirements, and it should provide the agencies a flexible roadmap to acquiring, deploying, and using the most appropriate and cost-effective equipment available. * Revise DNDO's estimate of the program's cost and ensure that the estimate considers all the costs--design and development, sustainment, maintenance, deployment, and procurement--associated with its project execution plan. * Communicate this revised estimate to the Congress so that it is fully apprised of the program's scope and funding requirements. Agency Comments and Our Evaluation: We provided DHS with a draft of our report for its review and comment. In its comments, the Department agreed with our recommendations and part of our conclusions, but strongly disagreed with other parts of our conclusions. DHS's comments, and our responses, are provided below. DHS commented that the introductory pages of the draft report did not always place DNDO in its proper historical context and that the extent of growth in the portal monitor program was not properly attributed. Our draft report clearly stated that CBP managed the portal monitor program prior to the creation of DNDO. Nonetheless, we revised one paragraph of the final report to better clarify that DNDO was not managing the radiation portal monitor project prior to 2005. DHS also commented that we used inconsistent terms in referring to "cost" and that as a result it is impossible to verify whether our assessments and conclusions are valid, or to conduct an "apples to apples" comparison. We disagree with DHS's assertion that we used inconsistent terms for "cost" and, as a result, it was impossible to verify our conclusions or to conduct a true "apples to apples" comparison. Our use of the various terms related to cost is accurate and consistent with the technical vocabulary of cost estimating. Even so, we made minor changes to the report to make the terminology more uniform and improve its readability. With regard to DNDO's comment concerning assessing the validity of our conclusions, in March 2008, we provided DNDO a 5-hour briefing, with highly detailed briefing slides, which described our cost estimating methods, data, and preliminary results. At the conclusion of that meeting, we gave DNDO our slides and all the information we collected to that point, including the cost estimating models, spreadsheets, and data ready for the agency to use. Furthermore, according to a DNDO document (dated May 29, 2008) that summarized agency efforts to develop its own life-cycle cost estimate, DNDO used our cost estimate as its initial baseline model in starting to develop its own cost estimate. In fact, the structure of DNDO's estimate was, according to this document, being "Based on GAO ICE [independent cost estimate]." After providing DNDO with such extensive information and documentation, and having it use this information in support of the development of its own cost estimates, we believe that DNDO had all the data it needed to assess the validity of our analysis and conclusions. DHS commented that comparing our cost estimate to DNDO's is misleading because we included operation and maintenance costs--incurred by CBP-- in our analysis. We disagree that the draft report is misleading when it compares DNDO and GAO life-cycle costs. In fact, we believe our estimate helps clarify the program's true cost, while DNDO's estimate obscures it. The source for DNDO's $2.1 billion cost estimate is the agency's most recent OMB Exhibit 300, dated March 25, 2008. According to OMB's Exhibit 300 instructions for completing multi-agency investments, agencies should include: (a) the costs associated with the entire life cycle of the investment; and (b) the funding from the lead and all partner agencies [emphasis added]. According to OMB, the purpose of the Exhibit 300 is to provide OMB with the information it needs to make both quantitative decisions about budgetary resources consistent with the Administration's program priorities, and qualitative assessments about whether the agency's programming processes are consistent with OMB policy and guidance. By omitting major project costs (such as maintenance costs), DNDO has understated the life cycle costs of the program and violated the intent of OMB policy and guidance. The effect of these omissions increases the probability of misinformed decisions and inadequate budget formulation for partner agencies. Our independent life-cycle cost estimate is more comprehensive, and closer to OMB's guidelines, than DNDO's as our estimate includes a significant portion of CBP's expected expenses. DHS commented that our analysis included costs for ASP variants that DNDO and CBP no longer plan to deploy. Further, DHS noted that the 2006 Project Execution Plan--upon which we based our estimate--is outdated and that consequently our cost estimates are speculative. We disagree that we have mischaracterized the radiation portal monitor project as it has been presented to OMB and the Congress.[Footnote 7] DNDO's March 2008 OMB Exhibit 300 states that "RPM/ASP units will be deployed to screen cargo entering the United States across land crossings, seaports, rail lines, airports, and other ports of entry." Clear definition of a program's characteristics is a key component in developing a high-quality cost estimate. DNDO's March 2008 budget submission should provide an accurate picture of the agency's vision, while the Project Execution Plan (PEP) should provide many of the characteristics as it defines the project objectives, work scope, and schedules. The PEP was accepted in September 2006 and approved jointly by CBP and DNDO in January 2007. On multiple occasions, we asked DNDO to provide updates to that plan that have been endorsed by DNDO's management: * In October 2007, agency officials indicated that the PEP dated September 11, 2006, was the most current version. * In March 2008, agency officials stated that the 2006 PEP was out of date. We asked them to provide an updated PEP. Agency officials would not commit to providing such an update but stated that DNDO is currently developing an ASP life-cycle estimate, and that the specific quantity of ASPs procured will be based, in part, on ASP test results. * In June 2008, agency officials agreed to provide an updated PEP with detailed information and appropriate management signatures. However, this information was never provided. Instead, DNDO provided a one-page summary spreadsheet that listed the number of ASPs and PVTs that would be purchased over the next 7 years. This spreadsheet did not contain the detailed information that DNDO promised, nor did it carry the signature endorsement of any DNDO official. To date, we have not received any official plan from DNDO that outlines the technical and programmatic baseline of the program other than the 2006 PEP. We therefore relied on this document to provide programmatic characteristics. Indeed, had the Secretary certified ASPs last year as initially planned, the program would have included rail, and the other ASP variants. However, in response to DNDO's statement that the 2006 PEP no longer reflected DNDO's plans for the program, we generated a second independent cost estimate based on the simplified one-page spreadsheet that DNDO provided. It is important to note, however, that a senior DNDO official told us in August 2008 that the number of ASPs DNDO plans to buy is subject to dramatic change based on the results of on-going testing. DHS noted that our cost estimate included the same developmental costs for additional ASP variants that were required to develop the 4-panel standard cargo portal. DHS maintains that in so doing, we overstated the ASPs' developmental cost. On this point, we advised DNDO in March 2008 that we needed more information to be more precise in the area of developmental costs. Some of this information would need to come directly from the contractors developing the ASPs. Despite our request, and because of limitations DNDO place on the contractors in their contacts with GAO, we did not have meaningful contacts with contractor officials. Nonetheless, we estimate that developmental costs for the additional variants account for only about 3 percent of total life- cycle costs. DHS commented that our draft report states that there is a need to screen personally owned vehicles (POV) with ASP technology when, in DNDO's opinion, PVTs currently do an adequate job in screening POVs. While the 2004 version of the PEP included ASPs for POVs, the 2006 PEP did not. The narrative of our draft report inadvertently included POVs among the types of ASP variants that were included in the 2006 PEP. We have revised the narrative of our report to reflect this change. Importantly, this did not affect our cost estimate because the model for our life cycle cost estimate properly reflected the plans from the 2006 PEP for using PVTs to screen POVs. DHS commented that DNDO did not instruct its ASP contractors to refuse our requests for information and interviews, as was stated in the attachment to our draft report. DNDO instructed its contractors to respond to our requests for data through DNDO, and then DNDO would provide us access to these contractors' documents only in a "reading room" that was established in DNDO offices. The result of the agency's instructions to its contractors hindered our work in some respects. For example, in response to our requests for interviews and documents, two of the three ASP contractors followed DNDO's instructions and insisted that we make all such requests through DNDO. One of the contractors directly provided us with all of the information we requested. As is our normal practice, we sought to obtain documents directly from the contractors in order to assure complete reporting of all documents requested. In addition, we terminated one contractor interview because of repeated interruptions by DNDO officials. Other contractor interviews did not take place due to DNDO's insistence that agency officials also participate in those interviews, which we believed could have hindered or prevented candid responses to our questions. Although DNDO's actions at times hindered the collection of data from contractors, we were nevertheless able to collect sufficient information to support our reports findings, conclusions, and recommendations. Finally, DHS commented that DNDO's cost estimate for deploying only standard cargo portals--$2.1 billion--is consistent with our estimate of $1.9 billion. We disagree that DNDO's cost estimate is comparable to ours. For example, our estimate--which ranges from $1.7 billion to $2.3 billion--is based on different technical and programmatic baseline parameters than the DNDO estimate and includes costs that DNDO has omitted (such as maintenance). Furthermore, as of August 22, 2008, DNDO was unable to provide us with a documented life-cycle estimate to reconcile against our cost estimate. According to OMB guidance, such documentation would normally be expected to justify a $2.1 billion acquisition. We are sending copies of this correspondence to interested congressional committees and members, the Secretary of Homeland Security, and other interested parties. We will also make copies available upon request. In addition, this correspondence will be available at no charge on GAO's web site at [hyperlink, http://www.gao.gov]. Should you or your staffs have any questions, please contact me at (202) 512-3841 or by e-mail at aloisee@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this correspondence. Key contributors to this report include Jennifer Echard, Brian Octeau, Karen Richey, Benjamin Shouse, Eugene Wisnoski, and Ned Woodward. Signed by: Gene Aloise: Director, Natural Resources and Environment: Enclosures - 5: List of Requesters: The Honorable Joseph I. Lieberman: Chairman: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Henry A. Waxman: Chairman: Committee on Oversight and Government Reform: House of Representatives: The Honorable Bennie G. Thompson: Chairman: The Honorable Peter T. King: Ranking Member: Committee on Homeland Security: House of Representatives: The Honorable James R. Langevin: Chairman: The Honorable Michael T. McCaul: Ranking Member: Subcommittee on Emerging Threats, Cybersecurity, and Science and Technology: Committee on Homeland Security: House of Representatives: The Honorable Bart Gordon: Chairman: Committee on Science and Technology: House of Representatives: The Honorable Charles E. Schumer: United States Senate: [End of section] Enclosure I: Slide Presentation: GAO’s Independent Cost Estimate of DNDO’s Radiation Portal Monitor Program: Interim Briefing for Congressional Requesters: Agenda: * Objectives: * Summary: * DNDO Baseline: * Point Estimates and Uncertainty: * Evaluation of DNDO Estimates: * Scope & Methods: Figure: Photograph of portal monitors in use at the U.S. POE in Blaine, WA. Source: GAO. [End of figure] [End of section] Objectives: In a May 2007 letter to the Comptroller General, you asked us to review the: (1) projected costs to implement DNDO’s project execution plan for the radiation portal monitor program, which includes the costs to develop, procure, deploy, operate, and maintain ASPs and other RPMs over a 10- year life cycle, and the; (2) validity of DNDO’s projections given what is known about the costs to procure, install, and operate radiation detection equipment. Summary: * DNDO’s Program to Deploy Radiation Detection Portal Monitors at U.S. Ports of Entry Is Likely to Cost About $3 Billion or More: - The likely cost will be about $3.1 billion, but could range between $2.6 billion and $3.8 billion. - Our estimate is based on the anticipated implementation costs of DNDO’s most recent plans. * DNDO’s Cost Estimates Are Unreliable and Could Result in Significant Cost Overruns: - DNDO’s estimate of $2.0billion fails to take into account several major cost elements. - DNDO’s estimate includes only 8 years of life cycle costs. - DNDO did not document its cost estimating approach. Radiation Portal Monitor Program Baseline: Planned Deployments of ASP Equipment: * ASP configurations are consistent with those proposed and contracted for by DNDO in July 2006, and contained in the PEP. Table: Type of ASP Planned: Standard Cargo Portal (NaI); Number Planned: 717. Type of ASP Planned: Standard Cargo Portal (HPGe); Number Planned: 9. Type of ASP Planned: Wide Cargo; Number Planned: 39. Type of ASP Planned: Rail; Number Planned: 51. Type of ASP Planned: Mobile; Number Planned: 56. Type of ASP Planned: Mobile RIID (MRIID); Number Planned: 162. Total ASP Systems in 2006 PEP: Number Planned: 1,034. [End of table] Radiation Portal Monitor Program Baseline: Planned Deployments of PVT Equipment: * PVT configurations are consistent with those contained in the PEP. Table: Type of PVT Planned: 1 Panel; Number Planned: 3. Type of PVT Planned: 2 Panel; Number Planned: 338. Type of PVT Planned: 4 Panel; Number Planned: 367. Type of PVT Planned: 8 Panel; Number Planned: 23. Type of PVT Planned: 12 Panel; Number Planned: 1. Type of PVT Planned: 4 Panel conversion to 2 Panel; Number Planned: 184. Type of PVT Planned: 4 Panel refurbishment; Number Planned: 126. Type of PVT Planned: 8 Panel conversion to 4 Panel; Number Planned: 16. Type of PVT Planned: 2 Panel Mobile; Number Planned: 26. Total PVT Systems in 2006 PEP: Number Planned: 1,084. [End of table] Figure: GAO’s Estimates of Program Costs: [Refer to PDF for image] This figure is a pie-chart depicting the following data: GAO’s Estimates of Program Costs, Summary (dollars in millions): Maintenance: $999.2; Deployment: $689.1; Procurement: $603.7; Design and Develop: $464.5; Sustainment: $364.9. Total cost: $3.1 billion, with a minimum estimate of $2.6 billion and a maximum of $3.8 billion. In constant fiscal year 2008 dollars. [End of figure] Figure: ASP Unit Cost Estimates for Standard Cargo Portals (in thousands, constant fiscal 2008 dollars): [Refer to PDF for image] This figure is a multiple vertical bar graph depicting the following data: PVT: DNDO: $994; GAO: $308. ASP: DNDO: $658; GAO: $822. [End of figure] GAO’s Evaluation of DNDO’s Program Costs: DNDO’s Cost Estimates Are Unreliable: DNDO’s current baseline cost estimate omits major project costs. For example: * It only includes estimates for the ASP standard cargo portal, even though: - The PEP and DHS’s BY2009 OMB Exhibit 300 call for ASP deployments at land border crossings, as well as at seaports, rail lines, and other ports of entry. - DHS’s OMB Exhibit 300 requests funding for about 10,400 NaI crystals, enough to build the standard cargo, rail, extra-wide cargo, and mobile ASPs (less MRIIDs) called for in DNDO’s plans (see slide 5, infra). * Further, DNDO’s cost estimate contains only 8 years of life cycle costs, despite: - The sodium crystal maker estimates that its crystal should last about 10 years, and; - DHS guidance states that a project’s life cycle should be estimated over the period of time during which equipment will remain usable. * Finally, DNDO’s baseline cost estimate does not include the ASPs’ complete life cycle costs. - It omits cost estimates for maintenance and sustainment. - These are substantial costs—$999 million and $365 million, respectively—in our estimate. DNDO did not document its cost estimating methodology. * DNDO was unable to provide detailed documentation of the life cycle costs used in its estimate, even though such documentation is necessary, according to OMB and DHS guidelines, to establish the basis of the agency’s estimates and to provide assurances that the estimates are credible. * DHS’s OMB Exhibit 300 indicates the ASP program’s cost variance is greater than 10 percent, but does not provide the basis for the variance, or corrective actions taken. GAO’s Evaluation of DNDO’s Program Costs: Omissions from DNDO Estimates Could Lead to Cost Overruns: GAO’s independent cost estimate includes: * 10-year life cycle; * All variations of ASP portals; * All cost categories, i.e., development, deployment, procurement, maintenance, and sustainment. DNDO’s ASP baseline cost estimate includes: * 8-year life cycle; * Only the standard ASP cargo portals; * Three of the 5 cost categories, i.e., development, deployment, and procurement. GAO’s Evaluation of DNDO’s Program Costs: DNDO’s Estimates Could Result in Significant Cost Overruns: The differences between GAO and DNDO cost estimates suggests a potential cost overrun that could total about $1.1 billion by fiscal year 2017. Figure: In thousands, constant fiscal year 2008 dollars: GAO ICE: Design and Development: approximately $425,000; Procurement: approximately $600,000; Deployment: approximately $700,000; Maintenance: approximately $1,000,000; Sustainment: approximately $375,000; Total: $3.1 billion. DNDO Estimate: Design and Development: approximately $425,000; Procurement: approximately $775,000; Deployment: approximately $750,000. Total: $2.0 billion. Note: DNDO BY09 estimate allocation based on details provided within the BY08 OMB Exhibit 300. [End of figure] Even when GAO “normalized” the cost data, i.e., removed the maintenance and sustainment factors from our estimate, significant differences remained. The largest differences occur in fiscal years 2008, 2009, and 2010, when according to the PEP, DNDO would be buying large numbers of ASPs. Figure: Comparison of GAO and DNDO “Normalized” Costs: Includes only design, procurement, and deployment (in constant, fiscal year 2008 dollars): [Refer to PDF for image] This figure is a multiple line graph depicting the following approximated data: Year: FY 2007; DNDO: $320; GAO: $150. Year: FY 2008; DNDO: $140; GAO: $375. Year: FY 2009; DNDO: $210; GAO: $360. Year: FY 2010; DNDO: $275; GAO: $350. Year: FY: 2011; DNDO: $440; GAO: $210. Year: FY 2012; DNDO: $250; GAO: $140. Year: FY 2013; DNDO: $220; GAO: $45. [End of figure] For the 5 years that overlap the GAO and DNDO cost studies—fiscal years 2008 to 2012—we estimate a $753 million budget shortfall. Additionally, we estimate that DNDO will require another $833 million to complete the entire life cycle from fiscal year 2013 through fiscal year 2017. ASP contracts have already experienced unfavorable cost and schedule variances since DNDO awarded the contracts. * Earned value management analysis indicates that one ASP contractor is about 13 percent ($1.6 million) over budget and 3 percent behind schedule. Another ASP contractor is about 25 percent ($1.0 million) over budget and 23 percent behind schedule. These unfavorable variances are not likely to improve, but quite likely will worsen, over the course of the ASP contracts. Conclusions: * DNDO has not yet provided the full cost of the radiation portal monitor project to the Congress. * The anticipated $1.6 billion shortfall we estimate represents additional financial risk that the U.S. government will have to confront over the life of the program. * DNDO will have to make up this shortfall somehow, either by returning to the Congress for additional funding, or by cutting expenses in other parts of the program, which may reduce the efficacy of the ASP systems. * The Congress should have full knowledge and complete understanding of the cost-benefit trade-offs inherent in DNDO’s decisions. Scope and Methods: GAO’s Approach to Estimating Costs: * Establish DNDO’s technical & programmatic baseline: - Consult DNDO and PNNL officials; - Review DHS, CBP, DOE, and Congressional documents; - Extract historical cost, and technical/program data. * Develop cost estimate: - Develop estimating methodology; - Generate point estimate; - Estimate probability distributions; - Perform uncertainty analysis. * Documentation, Validation, and Verification: - Trace and verify all data and analyses to GAO standards; - Align GAO’s independent cost estimate with DNDO’s budget and ASP baseline; - Reference all data and inputs to source documentation. * Align GAO’s independent cost estimate with DNDO’s budget and baseline estimates and identify major differences; * Review and analyze DNDO documentation; * Present GAO’s cost model and documentation to DNDO for review and comment; * Respond to all DNDO comments. Scope and Methods: GAO’s Assumptions: GAO’s estimate of program costs: * Includes fiscal years 2007 through 2017; * Is consistent with DNDO’s most recent Project Execution Plan—the agency’s detailed plan for implementing the radiation portal monitor program; * Includes all program costs associated with development, procurement, deployment, maintenance, and sustainment, regardless of funds’ origin; * Does not include Port Radiation, Inspection, Detection and Evaluation system or operational costs; * Are estimated in total year and fiscal year 2008 constant dollars; * Includes an escalation rate of 3 percent per year. Scope and Methods: Limitations on GAO’s Analysis: * DNDO declined to provide any “pre-decisional” information, thus the estimate could be inconsistent with DNDO’s latest strategies and data. * DNDO instructed its ASP contractors to refuse GAO requests for interviews and data, thus GAO collected most contractor data through DNDO. * GAO’s cost model varied according to work breakdown structure element, and was dependent upon: - Data availability, and; - Granularity of available data. Figure: Scope and Methods: GAO’s Cost Model Architecture: [Refer to PDF for image] This figure is an illustration of GAO’s Cost Model Architecture. [End of figure] [End of enclosure] Enclosure II: Scope and Methodology: Our approach to generating an independent cost estimate of the radiation portal monitor project (RPMP) was to follow the twelve step process outlined within the GAO Cost Assessment Guide. That process, outlined below was followed in generating independent cost estimates for the total radiation portal monitor project, as well as for the limited cost estimate for only the standard cargo portal. Step 1: Define Estimate's Purpose: The purpose of the estimate was to satisfy the Congressional requests to GAO pertaining to life cycle costs. Step 2: Develop Estimating Plan: Upon receipt of the Congressional request, we identified resources and developed a design matrix that included identification of information required and sources, and the proposed scope and methodology for addressing the Congressional requests. Additionally, in accordance with Generally Accepted Government Auditing Standards (GAGAS), we planned an entrance conference with DNDO, issuance of data collection instruments, and interviews with program officials. Step 3: Define Program Characteristics: We collected information directly from DNDO that enabled us to define the program's technical and programmatic baseline. Specifically, the program's project execution plan (PEP) was provided to us, outlining the program's time-phased procurement quantity requirements, and detailed deployment schedules by port-of-entry. Additionally, we collected contract documents containing contractor statements of work (SOW) and system specification documents. Step 4: Determine Estimating Approach: We developed an estimating structure in the form of a work breakdown structure (WBS) reflecting the radiation portal monitor project outlined in DNDO's project execution plan as well as other program documents reviewed, such as contractor cost reports. For each WBS element, the most appropriate estimating approach was identified according to data availability and credibility. Estimating approaches applied included analogy, engineering build-up, expert opinion, and parametric methods. Step 5: Identify Ground Rules and Assumptions: Estimating ground rules and assumptions were identified and documented. Assumptions were minimized, relying instead on real data when possible. Step 6: Obtain Data: Cost, technical, and programmatic data was solicited and collected from a number of sources through the use of data collection instruments (DCIs). DCI's were provided to DNDO, PNNL, Raytheon, Thermo Eberline, and: Canberra. Additional data was collected through interviews and correspondence with program officials. Step 7: Develop Point Estimate: Data collected was thoroughly analyzed, resulting in the development of cost estimating methodology. The methodology was integrated into a comprehensive cost model using Microsoft ® Excel, resulting in a detailed and time-phased point estimate. Step 8: Conduct Sensitivity Analysis: A comprehensive sensitivity analysis was conducted on the cost model using Crystal Ball ® software. The contribution to forecast variance and rank correlation coefficients were examined for each cost model input, enabling the development of spider charts and tornado charts which aid in the identification of key cost drivers. Step 9: Conduct Risk and Uncertainty Analysis: Risk and uncertainty analysis was conducted in order to transform the static point estimate into a probabilistic range based on the risk and uncertainty inherent in the model input variables. For each model input variable, underlying data used for the basis of estimate was examined to establish an uncertainty range and probability distribution type. These parameters were entered into the Crystal Ball ® software which was then used to simulate 10,000 estimate trials using Latin Hypercube sampling.[Footnote 8] The simulation resulted in establishing probability distributions about the estimated values, enabling the identification of the confidence level of the point estimate. Step 10: Document the Estimate: Detailed documentation was generated that provides the cost estimate results, as well as ground rules and assumptions, and all underlying cost estimating methodology. The cost model was thoroughly referenced to source documents, traced and verified in accordance with GAGAS. Additional auto-generated documentation consisting of risk and uncertainty input parameters, forecasts, and statistics was produced. Step 11: Present Estimate to Management for Approval: The resulting cost estimates were presented to various levels of GAO management, as well as to Congressional staffers. Additionally, GAO presented the cost estimate and underlying methodology in detail to DNDO in March 2008. Step 12: Update the Estimate to Reflect Actual Costs and Changes: Upon presenting the cost estimate and underlying methodology in detail to GAO, DNDO provided comment, questions, and some additional data which became the basis for some minor cost estimate revisions. The revised cost estimate was integrated into a subsequent briefing to DNDO in June of 2008. DNDO then requested that GAO prepare a cost estimate excursion that addresses just the 4 panel PVT and ASP variants. That excursion was then prepared by GAO and included in this report. Because DNDO instructed its contractors to respond to our requests for data through DNDO, we collected most contractor data through DNDO. DNDO also declined to provide any "pre-decisional" information or produce a revised version of its 2006 PEP. As a result, our estimate may not be consistent with DNDO's latest procurement strategy. While these actions limited the data that were available to us, we were nonetheless able to develop sufficient information to support our findings, conclusions, and recommendations. We conducted this performance audit from October 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on out audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of enclosure] Enclosure III: Radiation Portal Monitor Program Independent Cost Estimate: Our independent cost estimate for the radiation portal monitor project was structured according to the level of program definition detail available within official program documents, contract documents, and historical RPM program data. In accordance with GAO's Cost Assessment Guide, a product-oriented work breakdown structure[Footnote 9] (WBS) tailored by program phase was generated at this level of detail. The following table provides the first 3 indenture levels of the WBS: [Footnote 10] Table: Element: 1.0; Description: Total RPM Program. Element: 1.1; Description: Design and Development. Element: 1.1.1; Description: ASP 4 Panel, Medium Resolution (Raytheon). Element: 1.1.2; Description: ASP 4 Panel, Medium Resolution (Thermo Eberline). Element: 1.1.3; Description: ASP 4 Panel, High Resolution (Canberra). Element: 1.1.4; Description: ASP 8 Panel, Medium Resolution. Element: 1.1.5; Description: ASP 12 Panel, Medium Resolution. Element: 1.1.6; Description: ASP 2 Panel, Mobile, Medium Resolution. Element: 1.1.7; Description: Multi-variant Engineering and Spiral Development. Element: 1.2; Description: Procurement. Element: 1.2.1; Description: RPM Procurement. Element: 1.2.2; Description: MRIID Procurement. Element: 1.2.3; Description: Computer Procurement. Element: 1.2.4; Description: Spares Procurement. Element: 1.3; Description: Deployment. Element: 1.3.1; Description: Integration/Installation/Site Design & Construction. Element: 1.3.2; Description: Initial Training and Operations Support. Element: 1.3.3; Description: Deployment Integration & Coordination. Element: 1.3.4; Description: Technical Support. Element: 1.3.5; Description: Project Operations. Element: 1.3.6; Description: GSA Construction. Element: 1.4; Description: Maintenance. Element: 1.4.1; Description: Maintenance (PVT Portal Monitors). Element: 1.4.2; Description: Maintenance (ASP Portal Monitors). Element: 1.4.3; Description: Maintenance (Computers). Element: 1.5; Description: Sustainment. Element: 1.5.1; Description: Program Office (In-House). Element: 1.5.2; Description: Program Office (In-House Contractor Support). [End of table] A cost estimating methodology was developed for each lowest level WBS element level based on historical data, contract data, and other official program data. Developing cost estimating methodology at the lowest levels facilitates cost estimate visibility, identification of cost drivers, and clarification of interrelationships among cost elements. For example, under element number 1.2.1 RPM Procurement, cost estimating methodologies were developed at the major component level for 14 unique RPM variants. Those variants span multiple configurations (fixed vs. mobile; 1, 2, 4, 8, and 12 panel configurations) and multiple technologies (polyvinyl toluene (PVT), sodium iodide (NaI), and high purity germanium (HPGe)). The resulting estimating framework for element 1.2.1 RPM Procurement therefore allows for the identification and categorization of cost by configuration or technology type, and enables sensitivity analysis for identifying and examining the effect of changing key model assumptions. The primary cost estimating methods applied were analogy[Footnote 11] and parametric.[Footnote 12] Cost estimates generated at the lowest WBS levels were summed up to each successive level of indenture to generate the $3.1 billion point estimate. Level 2 of the WBS provides a breakdown by program phase as illustrated by figure 1. Figure 1: Radiation Portal Monitor Project Costs by Program Phases: [Refer to PDF for image] This figure is a pie-chart depicting the following data: Radiation Portal Monitor Project Costs by Program Phases: Maintenance: $999,242,368.91 (32%); Deployment: $689,107,870.32 (22%); Procurement: $603,678,392.66 (19%); Design and Development: $464,477,572.69 (15%); Sustainment: $364,988,656.21 (12%). Source: GAO analysis. [End of figure] The $3.1 billion point estimate was translated into a cost estimate range through the application of cost risk and uncertainty analysis. [Footnote 13] The data underlying the cost model inputs was examined for variability, thereby establishing uncertainty bounds and probability distributions for most of the WBS elements. These distributions were combined into a simulation model, in which the distributions for each cost element were treated as individual populations from which random samples were taken. The model was recalculated 10,000 times by repeatedly drawing values from each WBS distribution using a Latin Hypercube Sampling (LHS) method.[Footnote 14] The resulting risk and uncertainty cumulative distribution function illustrates the probability range of the independent cost estimate along with the relative placement of the $3.1 billion point estimate on the distribution. Figure 2 shows this distribution. Figure 2: Risk and Uncertainty Range of the Radiation Portal Monitor Project: [Refer to PDF for image] This figure is a chart depicting the risk and uncertainty range of the radiation portal monitor project. Frequency is depicted from 0 to 10,000 and probability is indicated from 0 to 1. A minimum of $2.616 billion and a maximum of $3.804 for the project is indicated. The following specific data is depicted: 50% probability level: $3.116 billion; Point estimate (51%): $3.121 billion; 70% probability level: $3.196 billion. Source: GAO analysis. [End of figure] The independent cost estimate ranges from a low of $2.6 billion to a high of $3.8 billion. The $3.1 billion point estimate approximately equates to a 50 percent confidence level. An increase to the point estimate by approximately $0.1 billion results in a 70 percent confidence level. Program requirements collected from DNDO such as procurement quantity profiles, development plans and deployment schedules enabled a cost estimate profile to be developed. The GAO life cycle cost estimate profile is provided in figure 3 along with the corresponding DNDO profile extracted from the program's budget year 2009 Exhibit 300 submission. Figure 3: Funding Shortfall in the Radiation Portal Monitor Project, Fiscal Years 2007 to 2017 (dollars in millions): [Refer to PDF for image] This figure is a multiple line graph depicting the following data: Fiscal year: 2007; DNDO: $302; GAO ICE: $188. Fiscal year: 2008; DNDO: $145; GAO ICE: $466. Fiscal year: 2009; DNDO: $216; GAO ICE: $488. Fiscal year: 2010; DNDO: $289; GAO ICE: $511. Fiscal year: 2011; DNDO: $474; GAO ICE: $387. Fiscal year: 2012; DNDO: $285; GAO ICE: $330. Fiscal year: 2013; DNDO: $246; GAO ICE: $225. Fiscal year: 2014; GAO ICE: $210. Fiscal year: 2015; GAO ICE: $212. Fiscal year: 2016; GAO ICE: $214. Fiscal year: 2017; GAO ICE: $217. Source: GAO analysis. [End of figure] The chart illustrates a $753 million shortfall during the fiscal years 2008 through 2012. The DNDO estimate does not include any costs beyond fiscal year 2013, contributing to an additional shortfall of $833 million to complete entire life cycle of the program from fiscal year 2013 to 2017. [End of enclosure] Enclosure IV: Radiation Portal Monitor Program-GAO's Limited Cost Estimate for Only the Standard Cargo Portal. Our limited independent cost estimate for DNDO's radiation portal monitor project that only included standard cargo portals was structured according to the level of program definition detail available within official program documents, contract documents, and historical radiation portal monitor project data. In accordance with GAO's Cost Assessment Guide, a product-oriented Work Breakdown Structure[Footnote 15] tailored by program phase was generated at this level of detail. The following table provides the first 3 indenture levels of the WBS:[Footnote 16] Element: 1.0; Description: Total RPM Program. Element: 1.1; Description: Design and Development. Element: 1.1.1; Description: ASP 4 Panel, Medium Resolution (Raytheon). Element: 1.1.2; Description: ASP 4 Panel, Medium Resolution (Thermo Eberline). Element: 1.1.3; Description: Multi-variant Engineering and Spiral Development. Element: 1.2; Description: Procurement. Element: 1.2.1; Description: RPM Procurement. Element: 1.2.2; Description: Computer Procurement. Element: 1.2.3; Description: Spares Procurement. Element: 1.3; Description: Deployment. Element: 1.3.1; Description: Integration/Installation/Site Design & Construction. Element: 1.3.2; Description: Initial Training and Operations Support. Element: 1.3.3; Description: Deployment Integration & Coordination. Element: 1.3.4; Description: Technical Support. Element: 1.3.5; Description: Project Operations. Element: 1.3.6; Description: GSA Construction. Element: 1.4; Description: Maintenance. Element: 1.4.1; Description: Maintenance (PVT Portal Monitors). Element: 1.4.2; Description: Maintenance (ASP Portal Monitors). Element: 1.4.3; Description: Maintenance (Computers). Element: 1.5; Description: Sustainment. Element: 1.5.1; Description: Program Office (In-House). Element: 1.5.2; Description: Program Office (In-House Contractor Support). [End of table] A cost estimating methodology was developed for each lowest level WBS element level based on historical data, contract data, and other official program data. Developing cost estimating methodology at the lowest levels facilitates cost estimate visibility, identification of cost drivers, and clarification of interrelationships among cost elements. For example, under element number 1.2.1 RPM Procurement, cost estimating methodologies were developed at the major component level for 2 unique RPM variants. Those variants span multiple technologies (polyvinyl toluene (PVT), sodium iodide (NaI)). The resulting estimating framework for element 1.2.1 RPM Procurement therefore allows for the identification and categorization of cost by technology type, and enables sensitivity analysis for identifying and examining the effect of changing key model assumptions. The primary cost estimating methods applied were analogy[Footnote 17] and parametric.[Footnote 18] Cost estimates generated at the lowest WBS levels were summed up to each successive level of indenture to generate the $1.9 billion point estimate. Level 2 of the WBS provides a breakdown by program phase as illustrated by figure 4. Figure 4: Radiation Portal Monitor Project - Limited Cost Estimate for Only the Standard Cargo Portal, Costs by Program Phases: [Refer to PDF for image] This figure is a pie-chart depicting the following data: Radiation Portal Monitor Project - Limited Cost Estimate for Only the Standard Cargo Portal, Costs by Program Phases: Deployment: $761,370,624 (39%); Maintenance: $393,719,522 (20%); Procurement: $348,335,548 (18%); Sustainment: $337,224,628 (18%); Design and Development: $97,970,733 (5%). Source: GAO analysis. [End of figure] The $1.9 billion point estimate was translated into a cost estimate range through the application of cost risk and uncertainty analysis. [Footnote 19] The data underlying the cost model inputs was examined for variability, thereby establishing uncertainty bounds and probability distributions for most of the WBS elements. These distributions were combined into a simulation model, in which the distributions for each cost element were treated as individual populations from which random samples were taken. The model was recalculated 10,000 times by repeatedly drawing values from each WBS distribution using a Latin Hypercube Sampling (LHS) method.[Footnote 20] The resulting risk and uncertainty cumulative distribution function in figure 5 illustrates the probability range of the independent cost estimate along with the relative placement of the $1.9 billion point estimate on the distribution. Figure 5: Risk and Uncertainty Range of the Radiation Portal Monitor Project - Limited Cost Estimate for Only the Standard Cargo Portal: [Refer to PDF for image] This figure is a chart depicting the risk and uncertainty range of the radiation portal monitor project. Frequency is depicted from 0 to 10,000 and probability is indicated from 0 to 1. A minimum of $1.7 billion and a maximum of $2.289 for the project is indicated. The following specific data is depicted: 50% probability level (Point estimate): $1.939 billion; 70% probability level: $1.976 billion. Source: GAO analysis. [End of figure] The independent cost estimate ranges from a low of $1.7 billion to a high of $2.3 billion. The $1.9 billion point estimate approximately equates to a 50 percent confidence level. An increase to the point estimate by approximately $0.04 billion results in a 70 percent confidence level. Program requirements collected from DNDO such as procurement quantity profiles, development plans and deployment schedules enabled a cost estimate profile to be developed. The GAO life cycle cost estimate profile is provided in figure 6 along with the corresponding DNDO profile extracted from the program's budget year 2009 Exhibit 300 submission. The DNDO estimate does not include any costs beyond fiscal year 2013, contributing to a shortfall of $518 million over the entire life cycle of the program from fiscal year 2008 to 2017. Figure 6: Funding Shortfall in the Radiation Portal Monitor Project - Limited Cost Estimate for Only the Standard Cargo Portal, Fiscal Years 2008 to 2017: [Refer to PDF for image] This figure is a multiple line graph depicting the following data: Fiscal year: 2008; DNDO: $145; GAO ICE: $221. Fiscal year: 2009; DNDO: $216; GAO ICE: $265. Fiscal year: 2010; DNDO: $289; GAO ICE: $266. Fiscal year: 2011; DNDO: $474; GAO ICE: $255. Fiscal year: 2012; DNDO: $285; GAO ICE: $263. Fiscal year: 2013; DNDO: $246; GAO ICE: $259. Fiscal year: 2014; GAO ICE: $208. Fiscal year: 2015; GAO ICE: $143. Fiscal year: 2016; GAO ICE: $146. Fiscal year: 2017; GAO ICE: $148. Source: GAO analysis. [End of figure] [End of enclosure] Enclosure V: Comments from the Department of Homeland Security: U.S. Department of Homeland Security: Washington, DC 20528: [hyperlink, http://www.dhs.gov] September 9, 2008: Mr. Gene Aloise: Director, Natural Resources and Environment: Government Accountability Office: 441 G Street NW: Washington, DC 20548: Dear Mr. Aloise: Re: Draft Report GAO-08-1108R, Combating Nuclear Terrorism: DHS's Program to Procure and Deploy Advanced Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost Estimates. The Department of Homeland Security (DHS) has reviewed the above- referenced draft report. Although DHS generally concurs with the recommendations and with GAO's conclusion that the revised Radiation Portal Monitor Program (RPMP) is likely to cost about $1.9 billion, or a range from about $1.7 billion to $2.3 billion, we strongly disagree with the remaining conclusions in the report. First, the introductory section from pages 1-5 contains numerous inaccuracies. In an attempt to show steady cost growth, GAO notes that DNDO's estimates of cost had "grown" during the period from February 2003 to December 2004. Yet, the Domestic Nuclear Detection Office was not created until April 2005, and its first OMB-300 was not submitted until fiscal year 2006. In another attempt to show steady cost growth, GAO uses inconsistent terms for "cost" such as program cost, total cost, estimated cost, expected cost, cost, cost estimate, unit cost, deployment cost, program's costs, independent cost estimate, ASP's cost, developmental costs, and full life cycle costs. Because of this imprecise and varying use of the term "cost" throughout this section, it is impossible to verify whether the GAO's assessment or conclusions are valid, or to conduct a true "apples to apples" comparison. Second, the draft report also makes a very misleading comparison when it compares its estimated life-cycle cost of $3.1 billion for ASP to the DNDO budget submission of $2.1 billion. These numbers are essentially comparing "apples to oranges." GAO incorrectly assumes DNDO's budget submission for ASP should include operation and maintenance costs when in fact those are costs assumed by Customs and Border Protection (CBP). These costs are important but would never be included in a DNDO budget submission. We advised GAO of this during their March presentation to us and again during the exit conference held on June 12, 2008; however, GAO continues to make the comparison anyway. Third, the GAO cost estimate incorrectly includes costs for ASP variants that are not presently part of the joint deployment plan between CBP and DNDO. The 2006 Program Execution Plan (PEP), acknowledged by all to be outdated, contemplated the potential need for 162 systems called a mobile Radioactive Isotope Identification Device (RIID). Since that point in time, CBP no longer thinks that such a system will be required. The 2006 PEP also included 51 systems to screen international rail cars. It is also doubtful that ASP or any passive portal technology is a viable technology solution to this problem. It is not known what type (or cost) of technology might be used. Therefore, any costs included in the GAO estimate are purely speculative. We advised GAO of this during their March presentation to us and again during the exit conference held on June 12, 2008; however, GAO continues to include these costs. Fourth, the GAO cost estimate also included the same development costs for additional variants such as a 2-panel or 8-panel system as was required to develop the 4-panel cargo system. These systems are just different configurations of the same portal technology and do not require extensive development costs. We advised GAO of this during their March presentation to us and again during the exit conference held on June 12, 2008; however, GAO continues to include these costs. Fifth, the GAO report also states that there remains a need to cover "personally owned vehicle" (POV) sites. This is misleading since POV sites are currently screened with the existing polyvinyl toluene (PVT) systems. Since POV sites do not have substantial background suppression issues or Naturally Occurring Radioactive Material (NORM) traffic, the current PVT technology has sufficient sensitivity to the threat to meet mission needs at POV crossings and the next generation ASP technology is not expected to be necessary. Once again, the GAO was informed of this discrepancy. Finally, the GAO report states that "DNDO instructed its ASP contractors to refuse GAO requests for interviews and data, thus GAO collected most contractor data through DNDO." As DNDO advised GAO in March and again in June, this is incorrect. DNDO merely requested the opportunity to participate in contractor interviews, and that all data provided by the contractors be provided through DNDO so that DNDO could ensure that the data was current and accurate. No contractor was advised to refuse interviews, numerous interviews did take place, and written responses were provided to GAO questions when interviews could not be arranged. DNDO provided to the GAO its current buy plan to cover all commercial cargo crossings, including land borders and seaports. Using these numbers and scope, GAO calculated a-range of $1.7 billion to $2.3 billion. Despite the GAO's claims of cost increases, overruns, and unfunded requirements, DNDO's $2.1 billion budget submission falls directly within GAO's calculated range. In addition, DNDO is in the process of developing a life-cycle cost estimate (LCCE) and deployment plan to support the full-rate production decision Key Decision Point (KDP-3) and the Secretarial certification. We will not proceed to full-rate production on Advanced Spectroscopic Portal (ASP) systems until the Secretary certifies that they have a significant increase in operational effectiveness over current systems. The Department has been following a prudent path leading to certification of ASP systems and a full-rate production decision Thank you for the opportunity to review and provide comments to the draft report. Sincerely, Signed by: Jerald E. Levine: Director, Departmental GAO/OIG Liaison Office: [End of enclosure] Footnotes: [1] P.L. 109-347. [2] CBP and DNDO, Radiation Portal Monitor Project: Project Execution Plan (Revision 0), PIET-43741-PM-100 (Richland, Wash.: Sept. 11, 2006). [3] See GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the Purchase of New Radiation Detection Portal Monitors Was Not Based on Available Performance Data and Did Not Fully Evaluate All the Monitors' Costs and Benefits, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-06-133R] (Washington, D.C.: Oct. 17, 2006). [4] ASP and PVT "standard cargo portals" are 4-panel portal monitors intended to screen standard commercial trucks. [5] The ASP panels are comprised mainly of sodium iodide crystals. It is these crystals that give ASPs the ability to both detect radiation and identify its isotopes. [6] DHS, DHS Acquisition Planning Requirements, Appendix A - Chapter 3007 (Washington, D.C.: Oct. 26, 2004). OMB, OMB Circular A-11, Part 7, Capital Programming Guide (Washington, D.C.: June 2006). GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs--Exposure Draft, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-07-1134SP] (Washington, D.C.: July 2, 2007). [7] According to OMB Circular A-11, "The Exhibit 300 is designed to coordinate OMB's collection of agency information for its reports to the Congress as required by the Federal Acquisition Streamlining Act of 1994 (FASA Title V) and the Clinger-Cohen Act of 1996; to ensure the business case for investments are made and tied to the mission statements, long-term goals and objectives, and annual performance plans developed pursuant to the [Government Performance and Results Act of 1993]." [8] Latin Hypercube sampling is a statistical method that ensures the ensemble of sample points is representative of the real variability. [9] A Work Breakdown Structure is a hierarchical framework that reflects the requirements, resources, and tasks that must be accomplished to develop a program. [10] The complete WBS extends to five levels of indenture. [11] Analogy-based methods use actual costs from a similar program. Those costs may be adjusted to account for technical and/or programmatic differences to the program being estimated. [12] Parametric-based methods use statistical relationships developed between historical costs and program, physical, and performance characteristics. [13] Cost risk and uncertainty analysis is a cost-estimating best practice addressed in many guides and references, including the GAO Cost Assessment Guide. [14] Latin Hypercube sampling is a form of stratified sampling that ensures the distribution function is sampled evenly, thereby reducing the number of trials necessary for a Monte Carlo simulation to achieve a more accurate random distribution. [15] A Work Breakdown Structure is a hierarchical framework that reflects the requirements, resources, and tasks that must be accomplished to develop a program. [16] The complete WBS extends to five levels of indenture. [17] Analogy-based methods use actual costs from a similar program. Those costs may be adjusted to account for technical and/or programmatic differences to the program being estimated. [18] Parametric-based methods use statistical relationships developed between historical costs and program, physical, and performance characteristics. [19] Cost risk and uncertainty analysis is a cost estimating best practice addressed in many guides and references, including the GAO Cost Assessment Guide. [20] Latin Hypercube sampling is a form of stratified sampling that ensures the distribution function is sampled evenly, thereby reducing the number of trials necessary for a Monte Carlo simulation to achieve a more accurate random distribution. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: