This is the accessible text file for GAO report number GAO-08-46 entitled 'Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls' which was released on October 25, 2007. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Committees: October 2007: Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls: GAO-08-46: GAO Highlights: Highlights of GAO-08-46, a report to congressional committees. Why GAO Did This Study: The Department of Homeland Security (DHS) established the Automated Commercial Environment (ACE) program to replace and supplement existing cargo processing technology. According to the fiscal year 2007 DHS appropriations act, DHS is to develop and submit an expenditure plan for ACE that satisfies certain conditions, including being reviewed by GAO. GAO reviewed the plan to (1) determine whether the expenditure plan satisfies the legislative conditions, (2) determine the status of 15 open GAO recommendations, and (3) provide observations about the expenditure plan and DHS’s management of the program. To address the mandate, GAO assessed plans and related documentation against federal guidelines and industry standards and interviewed the appropriate DHS officials. What GAO Found: The ACE expenditure plan satisfies many—but not all—of the legislative conditions specified in the fiscal year 2007 DHS appropriations act. Specifically, the plan (with related program documentation and officials’ statements) complies with acquisition rules, requirements, guidelines, and management practices of the federal government; includes a DHS certification that an independent verification and validation agent is under contract; was reviewed and approved by DHS and the Office of Management and Budget (OMB); and was reviewed by GAO. In addition, it partially satisfies conditions for meeting the capital planning and investment control review requirements established by OMB in Circular A-11 (part 7), including information security, and for complying with the DHS enterprise architecture. DHS has implemented eight open GAO recommendations made during the past 4 years, including those related to performance measures and targets, independent verification and validation, cost estimation, and program reporting. Seven other recommendations made during this time are in the process of being implemented. With respect to these seven, DHS has taken steps to satisfy each, such as establishing an accountability framework, reducing overlap and concurrence among ACE releases, and completing a privacy impact assessment, and actions are under way or planned to more fully address them. GAO is making three new observations about the expenditure plan and the management of ACE. First, the program is taking needed steps to redefine requirements for several ACE releases because of limitations in the completeness of original requirements, but this redefinition is likely to introduce significant program schedule delays and cost increases. Second, the changes to ACE requirements have led to replacement of a key commercial product, but the new product carries the risk of negatively impacting user productivity. Third, the automated database used for managing ACE risks is incomplete and does not contain information needed to adequately inform program decisions. All told, DHS has continued to make progress on ACE, and the program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, GAO sees major program schedule delays and cost overruns on the horizon. To improve ACE management and minimize exposure to risk, it is important for DHS to remain vigilant in its efforts to satisfy ACE legislative requirements, fully implement prior GAO recommendations, and keep Congress fully informed about the program’s status, plans, and risk. What GAO Recommends: GAO is making recommendations to further strengthen ACE management and accountability by disclosing program information in quarterly reports to Congress related to unmet legislative conditions, open GAO recommendations, program changes, and risk management. DHS agreed with GAO’s findings and recommendations and described actions that it has under way and planned to address them. Most of the described actions are consistent with GAO’s recommendations. To view the full product, including the scope and methodology, click on [hyperlink, http://www.GAO-08-46]. For more information, contact Randolph C. Hite at (202) 512-3459 or hiter@gao.gov. [End of section] Contents: Letter: Compliance with Legislative Conditions: Status of GAO Recommendations: Observations on the Expenditure Plan and Management of ACE: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendixes: Appendix I: Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations: Appendix II: Comments from the Department of Homeland Security: Appendix III: Contact and Staff Acknowledgments: ACE: Automated Commercial Environment: CBP: U.S. Customs and Border Protection: CIO: chief information officer: COTS: commercial off-the-shelf: DHS: Department of Homeland Security: GBB: global business blueprint: ITS: internet transaction server: OMB: Office of Management and Budget: PIA: privacy impact assessment: October 25, 2007: The Honorable Robert C. Byrd: Chairman: The Honorable Thad Cochran: Ranking Minority Member: Subcommittee on Homeland Security: Committee on Appropriations: United States Senate: The Honorable David E. Price: Chairman: The Honorable Harold Rogers: Ranking Minority Member: Subcommittee on Homeland Security: Committee on Appropriations: House of Representatives: The U.S. Customs and Border Protection (CBP), as part of the Department of Homeland Security (DHS), submitted to Congress in February 2007 its fiscal year 2007 expenditure plan for the Automated Commercial Environment (ACE) program pursuant to the Department of Homeland Security Appropriations Act, 2007.[Footnote 1] Begun in 2001, ACE is to replace and supplement existing cargo processing technology. The system is being developed and deployed in a series of increments to about 300 ports of entry and is expected to be fully deployed by the end of 2011. The goals of ACE include (1) supporting border security by enhancing analysis and information sharing with other government agencies and providing CBP with the means to decide before a shipment reaches the border what should be targeted because it is a security threat and what should be expedited because it complies with U.S. laws and (2) streamlining time-consuming and labor- intensive tasks for CBP personnel and the trade community through a national trade account and a single Web-based interface. As required by the appropriations act, we reviewed ACE's fiscal year 2007 expenditure plan. Our objectives were to (1) determine whether the expenditure plan satisfies the legislative conditions, (2) determine the status of 15 open GAO recommendations for ACE,[Footnote 2] and (3) provide observations about the expenditure plan and DHS's management of the program. On July 26, 2007, we provided a briefing to the staffs of the Subcommittees on Homeland Security, Senate and House Committees on Appropriations on the results of our review. This report transmits those results. The full briefing, including our scope and methodology, is reprinted in appendix I. Compliance with Legislative Conditions: The ACE expenditure plan--including related program documentation and program officials' statements--satisfies or partially satisfies the six legislative conditions specified in the appropriations act. Specifically, the plan satisfies the conditions that it (1) comply with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government;[Footnote 3] (2) include a certification by the DHS Chief Information Officer that an independent verification and validation agent is currently under contract for the program; (3) be reviewed by the DHS Investment Review Board, the Secretary of DHS, and Office of Management and Budget (OMB); and (4) be reviewed by us. The plan and its supporting information partially satisfies the conditions that it (5) meet the capital planning and investment control review requirements established by OMB (including Circular A-11, part 7) and that it (6) comply with the DHS enterprise architecture. For example, although DHS determined that ACE was aligned with the DHS enterprise architecture, the agency's analysis did not address key aspects of an architectural alignment, such as alignment with the architecture's data reference model. Status of GAO Recommendations: DHS has implemented some, but not all, of the recommendations pertaining to ACE that we have made since 2003. These recommendations, along with their status, are summarized here. Eight recommendations have been implemented. For example, DHS has: * developed a range of ACE performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction with ACE; * aligned ACE program goals, benefits, desired business outcomes, and performance measures; * addressed those legislative conditions associated with measuring ACE performance and results and employing effective independent verification and validation practices; and: * ensured that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates. The remaining seven recommendations are in the process of being implemented, as described here. * Recommendation: Define and implement an ACE accountability framework that fulfills several conditions,[Footnote 4] to include ensuring the currency, relevance, and completeness of commitments made to Congress in expenditure plans and reporting in future expenditure plans the progress against commitments that were contained in prior expenditure plans. In progress. The program has established an accountability framework that is providing input for both the annual expenditure plan and quarterly congressional reports. However, as with prior expenditure plans, the fiscal year 2007 expenditure plan did not reflect the most current program information. For example, information on milestones, earned value management, and risks were about 4 months old when the plan was submitted to the appropriations committees in February 2007, and program commitments were no longer current. Further, while program officials continue to use the quarterly ACE status reports to provide the appropriations committees with more detailed information, these reports are generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance as well. CBP and DHS officials told us that the delays were due to the DHS review and approval process and that they are exploring ways to accelerate the process. The 2007 expenditure plan did not adequately report on progress against previous plan commitments. For example, the plan did not (1) report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved, (2) report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans, or (3) address progress against the milestone dates for each stage of a release that was included in the prior year's plan. According to program officials, the quarterly congressional reports provide more current information on the program's progress against prior commitments. However, these reports have also not fully addressed the commitments made in prior expenditure plans. Program officials stated that they intend to start providing this information in the last quarterly report of each year. * Recommendation: Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful. Planned. The program office has made changes that are to improve overall program management and, according to its December 2006 quarterly report to Congress, the program office plans to measure the impact of future management improvements. Moreover, this report stated that the program anticipates more changes, including creation of a cargo requirement management board to decide the disposition of all change requests to production systems; establishment of a new invoice review policy; and colocation of personnel within a given business area. However, program officials told us that they have yet to define measures to determine the impact of such changes and thus are not yet positioned to determine their success. * Recommendation: Minimize the degree of overlap and concurrency across ongoing and future ACE releases and capture and mitigate the associated risks of any residual concurrence. In progress. Since May 2006, the program office has reduced overlap and concurrence of ACE releases and has taken actions to reduce potential contention for limited resources by, for instance, decoupling (i.e., reducing dependencies among) certain program components; dividing releases into smaller subreleases to provide more flexibility in scheduling; improving planning for development, integration, testing, training activities, and milestones to better schedule use of development and test environments; and centralizing management of shared software services. Further, the program office conducts regular integration meetings with the teams supporting each release to discuss concerns, decisions, and schedules associated with resource availability and is using a software tool to track and mitigate release- specific concurrency risks. However, this tool contains vague or incomplete data relative to mitigating these risks. These data limitations make it difficult to determine the status or the effectiveness of the efforts to reduce the risks associated with overlap and concurrence among releases. * Recommendation: Direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment (PIA)[Footnote 5] and ensuring architectural alignment. In progress. One legislative condition states that the plan should meet OMB's capital planning and investment control review requirements, which include addressing security and privacy issues. The program office has developed a PIA for ACE release 4 (e-Manifest: Trucks), which is currently operational. DHS approved this PIA on July 14, 2006. This PIA addressed the major elements of DHS's guidance, and program officials stated that they would update the assessment for each ACE release. However, this PIA did not cover other recently completed screening releases. Program officials told us that these screening releases were considered to be part of the Automated Targeting System and were covered by the Automated Targeting System's PIA. However, this PIA does not specifically identify or address ACE screening releases. With respect to architectural alignment, DHS determined in May 2007 that all required ACE products and technologies were aligned with the DHS technical reference model and that ACE was thus aligned with the DHS enterprise architecture. However, we have yet to receive sufficient documentation describing the criteria and methodology used to make these determinations or verifiable analysis supporting the determinations. Moreover, the determinations were based on technical alignment and did not address other relevant aspects of program alignment to an enterprise architecture, such as data alignment. * Recommendation: Develop and implement key human capital management practices. In progress. In June 2006, the Office of Information Technology Strategic Human Capital Management Plan, which included an ACE-specific appendix as a road map for effective management of human capital resources, was approved. However, the plan does not address the basic tenets of effective human capital management, such as defining the positions needed (including core competencies) to perform core program functions, assessing and inventorying current workforce skills and abilities, assessing any gaps between needed and existing workforce levels and capabilities, and filling identified gaps. Officials acknowledged these limitations in the plans and stated that they are developing an implementation plan to address these shortfalls. * Recommendation: Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability. In progress. The June 30, 2006, quarterly report to the House and Senate Appropriations Committees included the ACE program's strategy for meeting its human capital needs and its accountability framework; however, the human capital strategy does not meet the basic tenets of strategic human capital management, as previously explained. * Recommendation: Accurately report to the appropriations committees on progress in implementing our prior recommendations. In progress. Quarterly reports to the House and Senate Appropriations Committees have contained information on the status of our open recommendations since November 2002, but recent reports remain outdated due to a 2-to 7-month time lapse between when the reports are produced and when they are provided to the appropriations committees. DHS and program officials stated that they are exploring ways to accelerate the review process and thereby improve the timeliness and accuracy of their reports to Congress. Observations on the Expenditure Plan and Management of ACE: We have three observations about ACE requirements, commercial product selection, and risk management. * Requirements: Redefinition of requirements for several ACE releases is now under way to address limitations in completeness of originally defined requirements, and this redefinition is likely to introduce program schedule delays and cost increases. In defining ACE requirements, the program office discovered that its original approach did not adequately engage all key stakeholders, such as software programmers and subject matter experts, for the legacy system ACE is intended to replace. To address this, key stakeholders are now collaborating and decomposing legacy code. However, this effort is expected to significantly delay some system releases and drops. Program officials have taken several actions that they say will minimize the impact of the delays, such as prioritizing shared functionality, dividing releases/drops into smaller increments, and changing release deployment strategies. However, these changes have yet to be approved, and the full extent of the cost and schedule implications is not yet known. Moreover, neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports have disclosed the requirements redefinition and its impact, and neither has addressed any changes to release deployment strategies. * Commercial product selection: Significant changes to ACE requirements have led to reevaluation and replacement of a key commercial off-the- shelf (COTS) product previously selected and being prepared for use. The program office conducted several analyses to determine which COTS products would best meet ACE system requirements, including a general review of various packages in 2002 to select a provider and a more detailed analysis in 2004 to define and allocate ACE requirements and select a specific product--the SAP Enterprise Portal. In December 2006, however, the ACE Chief System Architect determined through additional analysis that all planned SAP functionality could be provided by Internet Transaction Server (ITS) technology and recommended that ITS be adopted. The program office subsequently stopped work on SAP Enterprise Portal design and configuration efforts and reported that ITS would be used for release 5/drop A2 instead of the SAP Enterprise Portal. This decision is expected to have some near-term schedule impacts because much of the completed work for A2 had been based on the planned use of SAP Enterprise Portal. Further, use of ITS raises the risk of inadequate user response time, which would, in turn, negatively impact user productivity and introduce a high probability of significant cost and schedule impacts. Program officials reported that actions are under way to mitigate the risk through performance modeling and test planning. However, neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress disclose this COTS product change, its impact on release schedules and cost estimates, or the risk to future system performance. * Risk management: All program risks are not being effectively managed. The program office has developed a process guide and implemented an automated tool (database) for managing ACE risks in accordance with relevant guidance and best practices. Although the database contains fields to provide a description, level (high, medium, or low), and mitigation strategy (including start and end dates, exit criteria, and implementation status) for each risk, the completeness and quality of this information varies. Because of such database limitations, we could not determine the status of and mitigation progress on 17 risks. Moreover, these database limitations were not reflected in the documentation used at key program events, indicating that the program does not have the risk-related information that it needs to inform its program decisions and to reduce the chances of potential problems becoming actual problems. Program officials stated that they are taking steps to improve risk management, including establishing a group to ensure the quality and completeness of the database, holding regular group meetings with contract staff and team leads to discuss risks and their impacts, and conducting risk management training. To date, however, program risks have not been communicated to oversight organizations through the 2007 expenditure plan or recent quarterly reports to the House and Senate Appropriations Committees. Conclusions: Over the past 7 years, CBP and DHS have worked to fulfill legislatively mandated annual expenditure plan requirements and to implement dozens of our recommendations related to these plans and management of the program. Among other things, these requirements and recommendations have promoted effective program management and accountability for performance and results. As a result of these years of effort, the ACE program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appears to be on the horizon. To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays. Recommendations for Executive Action: To further strengthen ACE management and promote accountability for ACE performance and results, we are recommending that the Secretary of Homeland Security direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose: * the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all our prior recommendations; * the status and impacts on the program's estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different COTS product than originally selected; and: * the program's plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategies' implementation. Agency Comments and Our Evaluation: In written comments on a draft of this report signed by the Director, Departmental GAO/OIG Liaison, and reprinted in appendix II, DHS agreed with our findings and stated that it is committed to addressing them. Further, the department agreed with our recommendations and described actions that it said are under way or planned to address them. While most of the department's stated actions are consistent with our recommendations, in one case they may not be sufficient. Specifically, the department stated that it would ensure that future expenditure plans meet all OMB capital planning and investment control review requirements by attaching the required OMB budget submission for ACE, referred to as an Exhibit 300, to all future plans. However, we reviewed the fiscal year 2007 ACE Exhibit 300 as part of our review of the fiscal year 2007 ACE expenditure plan and found that it did not meet the OMB requirements cited above. Therefore, unless DHS improves the quality of future ACE Exhibit 300s by addressing the weaknesses we have identified, this action alone may not fully address our recommendation. We are sending copies of this report to the Chairmen and Ranking Minority Members of the other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We are also sending copies to the DHS Secretary, the CBP Commissioner and, on their request, to other interested parties. In addition, the report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3459 or at h [Hyperlink, hiter@gao.gov] iter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and key contributors to this report are listed in appendix III. Signed by: Randolph C. Hite: Director, Information Technology Architecture: and Systems Issues: [End of section] Appendix I: Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations: Information Technology: Management Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls: Briefing to the Staffs of the Subcommittees on Homeland Security, Senate and House Committees on Appropriations: July 26, 2007: Briefing Overview: Introduction: Objectives: Results in Brief: Background: Results: * Legislative Conditions: * Status of Recommendations: * Observations: Conclusions: * Recommendations for Executive Action: Agency Comments: Attachment 1. Scope and Methodology: Attachment 2. Related GAO Products: Introduction: The U.S. Customs and Border Protection[Footnote 6](CBP) is developing a new import and export processing system, referred to as the Automated Commercial Environment (ACE), to replace and supplement existing cargo processing technology. Begun in 2001, this system is being developed and deployed in a series of increments to about 300 ports of entry and is expected to be fully deployed by the end of 2011. The goals of ACE include: * supporting border security by enhancing analysis and information sharing with other government agencies and providing CBP with the means to decide before a shipment reaches the border what should be targeted because it is a security threat and what should be expedited because it complies with U.S. laws; and: * streamlining time-consuming and labor-intensive tasks for CBP personnel and the trade community through a national trade account and a single Web-based interface. The Department of Homeland Security Appropriations Act, 2007,[Footnote 7] states that DHS may not obligate $216.8 million of the $316.8 million appropriated for ACE until the House and Senate Committees on Appropriations receive a plan for expenditure that: 1. meets the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including Circular A-11, part 7;[Footnote 8]: 2. complies with DHS’s information systems enterprise architecture; 3. complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; 4. includes a certification by the Chief Information Officer (CIO) of DHS that an independent verification and validation agent (IV&V) is currently under contract for the project; 5. is reviewed and approved by the DHS Investment Review Board (IRB),[Footnote 9] the Secretary of Homeland Security, and OMB; and: 6. is reviewed by GAO. On February 6, 2007, DHS submitted its fiscal year 2007 expenditure plan for $316.8 million to the House and Senate Appropriations Subcommittees on Homeland Security. In addition, CBP submits quarterly reports to the House and Senate Appropriations Committees to keep them apprised of ACE progress and issues. Objectives: As agreed, our objectives were to: 1. determine whether the ACE fiscal year 2007 expenditure plan satisfies the legislative conditions, 2. determine the status of 15 open GAO recommendations for ACE,[Footnote 10] and: 3. provide observations about the expenditure plan and DHS’s management of the ACE program. We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from December 2006 through July 2007 in accordance with generally accepted government auditing standards. Details of our scope and methodology are provided in attachment 1. Related GAO products are in attachment 2. Results in Brief: Objective 1: Legislative Conditions: Table: Summary of satisfaction of legislative conditions: Legislative condition: Meets the capital planning and investment control review requirements established by OMB, including OMB Circular A-11, part 7; Status [A] [B]: Partially satisfied. Legislative condition: Complies with the DHS enterprise architecture; Status [A] [B]: Partially satisfied. Legislative condition: Complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; Status [A] [B]: Satisfied. Legislative condition: Includes a certification by the DHS CIO that an IV&V agent is currently under contract for the project; Status [A] [B]: Satisfied. Legislative condition: Is reviewed and approved by the DHS IRB, Secretary of DHS, and OMB; Status [A] [B]: Satisfied. Legislative condition: Is reviewed by GAO; Status [A] [B]: Satisfied. Source: GAO. [A] Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying many, but not all, key aspects of the condition that we reviewed. [B] Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every aspect of the condition that we reviewed. [End of table] Results in Brief: Objective 2: Open Recommendations: Table: Implementation of prior GAO recommendations GAO Recommendations[A]: GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures; Status [B,C,D]: [Empty]. GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: a. coverage of all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules; Status [B,C,D]: Complete. GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: b. currency, relevance, and completeness of such commitments made to Congress in expenditure plans; Status [B,C,D]: In progress. GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: c. reliable data relevant to measuring progress against commitments; Status [B,C,D]: Complete. GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: d. reporting in future expenditure plans progress against commitments contained in prior expenditure plans; Status [B,C,D]: In progress. GAO Recommendation: 1. Define and implement an ACE accountability framework that ensures: e. use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects, and document milestone decisions in a way that reflects the risks associated with proceeding with unresolved severe defects and provides for mitigating these risks.[E]; Status [B,C,D]: Complete. GAO Recommendation: 2. Develop the range of realistic performance measures and targets needed to support an outcome-based, results- oriented accountability framework, including user satisfaction; Status [B,C,D]: Complete. GAO Recommendation: 3. Explicitly align program goals, benefits, desired business outcomes, and performance measures; Status [B,C,D]: Complete. GAO Recommendation: 4. Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful; Status [B,C,D]: Planned. GAO Recommendation: 5. Fully address those legislative conditions associated with measuring performance and results and employing effective IV&V practices; Status [B,C,D]: Complete. GAO Recommendation: 6. Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates; Status [B,C,D]: Complete. GAO Recommendation: 7. Develop and implement a rigorous and analytically verifiable cost estimating program that embodies the tenets of effective estimating; Status [B,C,D]: Complete. GAO Recommendation: 8. Use earned value management (EVM)[F]in developing all existing and future releases; Status [B,C,D]: Complete. GAO Recommendation: 9. Have future expenditure plans specifically address any proposals or plans for extending and using ACE infrastructure to support other homeland security applications; Status [B,C,D]: Complete. GAO Recommendation: 10. Minimize the degree of overlap and concurrence across ongoing and future releases, and capture and mitigate the associated risks of any residual concurrence; Status [B,C,D]: In progress. GAO Recommendation: 11. Fully address those legislative conditions associated with having an approved privacy impact assessment (PIA) and ensuring architectural alignment; Status [B,C,D]: In progress. GAO Recommendation: 12. Develop and implement missing human capital management practices; Status [B,C,D]: In Progress. GAO Recommendation: 13. Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing human capital needs and the framework for managing performance and ensuring accountability; Status [B,C,D]: In progress. GAO Recommendation: 14. Report to House and Senate Appropriations Committees on a quarterly basis on efforts to address open GAO recommendations; Status [B,C,D]: Complete. GAO Recommendation: 15. Accurately report to the appropriations committees on CBP's progress in implementing our prior recommendations; Status [B,C,D]: In Progress. Source: GAO. [A] With respect to the fiscal year 2007 expenditure plan. [B] Complete means that actions have been taken to fully implement the recommendation. [C] In progress means that actions are under way to implement the recommendation. [D] Planned means actions are planned to implement the recommendation. [E] This is a combination of two related recommendations. [F] EVM is a management tool for measuring progress and is both an industry accepted practice and an OMB requirement. [End of table] Results in Brief: Objective 3: Observations: Summary of observations: * Limitations in the completeness of the original requirements for several ACE releases have resulted in changes to how requirements are being defined, as well as changes to the requirements themselves. These changes are likely to produce significant schedule delays and cost growth, and neither the expenditure plan nor the quarterly reports to Congress have disclosed these risks. * Requirements changes have led to reevaluation and replacement of a key commercial software product that was previously selected based on incomplete requirements. This change carries schedule and cost risks that neither the expenditure plan nor the quarterly reports have disclosed. * Management of ACE risks has not been effective, but improvements are planned to better anticipate and avoid problems that cause schedule delays and cost growth. ACE risks are not disclosed in either the expenditure plan or in the quarterly reports. Results in Brief: Recommendations and Agency Comments: To reduce ACE exposure to future schedule delays and cost growth and to promote greater accountability, we are making recommendations to DHS aimed at disclosing the nature of and progress in addressing the risks associated with not having fully satisfied expenditure plan legislative conditions, not having completed implementation of all prior GAO recommendations, and having to redefine ACE requirements and reselect a key commercial software product. In oral comments on a draft of this briefing, DHS and CBP officials agreed with our conclusions and recommendations, and provided clarifying information and technical comments that we incorporated in the briefing, as appropriate. Background: Program Overview: CBP is about 6 years into its trade processing modernization program, known as ACE. Among other things, ACE is to introduce reengineered business processes and next generation cargo processing technology, and it is to support CBP’s mission of (1) protecting the American public against terrorism and (2) enforcing the laws of the United States while fostering our nation’s economic security through lawful international trade and travel. ACE is also to support provisions of Title VI of the North American Free Trade Agreement, commonly known as the Customs Modernization Act. Subtitle B of the Act[Footnote 11] contains provisions that were intended to enable the government to modernize international trade processes and permit CBP to adopt an informed compliance approach with industry using automated systems. The goals of ACE are to: * enhance analysis and information sharing with other government agencies relative to new national security threats; * provide CBP personnel with the technology and information needed to decide, before a shipment reaches the border, what should be targeted because it is a security threat and what should be expedited because it complies with U.S. laws; * enable the efficient collection, processing, and analysis of commercial import and export data via an integrated, fully automated information system; * reduce costs for the government and the trade community by streamlining time-consuming and labor-intensive tasks; * enable government and trade communities to process, view, and manage accounts nationally and obtain historical information on cargo, conveyances, and crew, based on screening and targeting rules; and: * enable government to comply with legislative mandates to improve efficiency/effectiveness and provide better customer service to U.S. citizens. Background: Program Organization: Several CBP components support the execution of the ACE program. * The Cargo Systems Program Office (referred to in this briefing as the program office) is responsible for the implementation of ACE. This office has primary responsibility for ACE program management; is responsible for the day-to-day management of ACE modernization activities; and develops the policies, standards, processes, and metrics by which the program is managed and measured. * The ACE program office is located within CBP’s Office of Information Technology (OIT), which is headed by the Assistant Commissioner for Information and Technology. * The ACE program office is supported by the Targeting and Analysis Systems Program Office (a merger of ACE screening and targeting resources and the Office of Information Technology’s Interprocess Solutions Branch), which is responsible for acquisition and development of ACE screening and targeting functionality. * The program office is supported by CBP’s Office of Finance, Office of Strategic Trade, Office of Regulations and Rulings, and more than 100 participating government agencies and offices. Background: Contractors: CBP awarded a 15-year, indefinite delivery/indefinite quantity, prime integration contract (5-year base with two 5-year options) to IBM Global Services in April 2001 for development and implementation of ACE. CBP exercised the first option in April of 2006. * IBM and its subcontractors-collectively called the ACE Support Team[Footnote 12]-provide ACE with program management support, enterprise engineering, systems planning and development, and systems business process reengineering; technology architectures; prototyping; systems and application design; software development, testing, and evaluation; deployment; and developmental operations support. * IBM is also under contract for operations and maintenance of deployed ACE releases, as well as for ongoing enhancements (new capabilities, referred to as program baseline enhancements) implemented in sub- releases that are scheduled in-between major ACE releases. Background: Acquisition Approach and Cost: CBP also relies on support contractors to provide a range of management support, such as program management, financial management, process improvement, quality assurance, requirements and configuration management, program communication, organizational learning, human capital planning, specialized cost analysis and life cycle cost estimating/modeling services, and procurement support. In 2002, ACE was to be completed in four increments over 4 years at an estimated cost of between $1.5 billion and $1.6 billion. The ACE Program Plan (version 2.1, dated August 2006) states that ACE is expected to be fully deployed and operational by the end of 2011 at a cost of about $3.3 billion. However, program officials told us that cost and schedule estimates are being revised, but have not yet been approved and therefore were not provided for our review. In the fiscal year 2007 ACE expenditure plan, CBP reports that it has spent about $1.7 billion on the program. ACE is now being acquired and implemented through a series of 10 increments (referred to as releases), which are further divided into major system deliveries, known as “drops.” (See the following six slides for a description of the increments and their status.) Background: Description of ACE Increments: A description of the 10 ACE increments (releases and related drops) follows. Release 1 (ACE Foundation): Computer hardware and system software (infrastructure) to support subsequent system releases. Release 2 (Account Creation): Initial group of national account managers1 and 41 importers access to account information, such as trade activity. Release 3 (Periodic Payment): Additional account managers and importers, as well as brokers and carriers,2 with access to account information; provides initial financial transaction processing and revenue collection capability, allowing monthly payments of duties and fees. Release 4 (e-Manifest: Trucks): Electronic truck manifest3 processing and interfacing to legacy enforcement systems and databases. Background: Description of ACE Increments: Release 5 (Entry Summary, Accounts, and Revenue (ESAR)): SAP[Footnote 16] technologies to enhance and expend accounts management, financial management, and entry summary functionality. * Master Data and Enhanced Accounts (drop A1): SAP to deliver enhanced account creation and maintenance functionality and expand the types of accounts managed in ACE. * Entry Summary and Revenue (drop A2): Entry summary, interfaces with participating government agencies, calculation of duties and fees, reconciliation processing, and refunds. Release 6 (e-Manifest, All Modes, and Cargo Release): Electronic manifest capability for rail, air, and sea shipments; provides a multimodal manifest;[Footnote 17] enables full tracking of cargo, conveyances, individuals, and equipment; and enhances enforcement processes for rail, air, and sea. * e-Manifest: Rail and Sea Manifest (drop M1): Electronic manifest functionality for rail and sea shipments; rail, sea, and truck electronic manifests into the multimodal manifest. * e-Manifest: Air Manifest and Cargo Release (drop M2): Electronic manifest to air shipment and brings all modes of transportation into the multimodal manifest. * e-Manifest: Exports and Mail Entry Writing System (drop M3): Tracking of cargo, conveyances, individuals, and equipment for truck, sea, rail, and air manifests. Release 7 (Exports and Cargo Control): Remaining accounts management, revenue, manifest, release, and export functionality. * ESAR: Drawback, Protest, and Importer Activity Summary Statement (IASS) (drop A3): Import activity summary statement,[Footnote 18] drawback functionality, and enhanced protest; online processing for trade account applications. E-Manifest: Final Exports and Manifest (drop M4): Electronic manifest for mail, pipeline, and hand carry; electronic export processing. Screening S1 (Screening Foundation): Foundation for screening cargo and conveyances by centralizing criteria and results into a single standard database; allows user definition and maintenance of data sources and business rules for air, rail, sea, and truck modes of transportation. Screening S2 (Targeting Foundation): Platform and foundation for advanced targeting capabilities by enabling CBP’s National Targeting Center to search multiple databases for relevant facts and actionable intelligence and infer relationships between entities and data elements; architecture for integrating new data sources (including integrating external data sources and providing a single sign on capability), implementing analytical tools, and deploying analytical capabilities. Screening S3 (Advanced Targeting Capabilities): Screening for reconciliation, intermodal manifest, Food and Drug Administration data, and in-bond, warehouse, and foreign trade zone authorized movements; integrates additional data sources into targeting capability; and risk management capability. Background: Status of ACE Increments: According to CBP, as of July 2007,: * Five increments are fully operational: Releases 1, 2, and 3, and Screenings S1 and S2. * One increment (Release 4) has been deployed at 94 of the 99 truck land border ports. * Three increments are at various stages of development and/or deployment: Release 5/Drops A1 and A2, Release 6/Drop M1, and Screening Background: Status of ACE Increments: Figure: ACE Schedule: This figure is a bar chart showing the ACE schedule. [See PDF for image] - graphic text: Source: GAO analysis based on CBP data. [End of table] Background: Status of ACE Use: CBP reports increased use of operational ACE increments. For example: * The number of external accounts[Footnote 19] grew from 41 in June 2003 to 736 in August 2005, and stood at about 4,100 in October 2006. As of November 2006, about 4,500 corporate entities had been approved to pay monthly duties and fees. * Total revenue collections through ACE grew from $84,673 in June 2004 to $1.3 billion in July 2005, and as of October 2006 stood at about $8 billion. * The number of e-Manifests that were filed using ACE grew from 20,847 in December 2006 to 45,548 in January 2007. Background: Funding for ACE Expenditure Plans: Table: Funding for ACE Expenditure Plans: Release: 1: ACE Foundation:-through Operational Readiness; Budgeted amount: Prior years[A]: 114.8; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 114.8. Release: 2: Account Creation-through Operational Readiness; Budgeted amount: Prior years[A]: 114.8; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 114.8. Release: 3: Periodic Payment:-through Operational Readiness; Budgeted amount: Prior years[A]: 172.9; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 183.2. Release: 4: e-Manifest Trucks-through Operational Readiness; Budgeted amount: Prior years[A]: 10.3; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 183.2. Release: 5(A1): ESAR: Master Data and Enhanced Amounts; Budgeted amount: Prior years[A]: 172.3; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 233.6. Release: 5(A2): ESAR: Entry Summary and Revenue; Budgeted amount: Prior years[A]: 23.3; Budgeted amount: Fiscal year: 38.0; Budgeted amount: Total: 233.6. Release: 6(M1): e-Manifest- Rail and Sea Manifest; Budgeted amount: Prior years[A]: 46.3; Budgeted amount: Fiscal year: 38.3; Budgeted amount: Total: 95.0. Release: 6(M2): e-Manifest- Air and Cargo Release; Budgeted amount: Prior years[A]: 10.4; Budgeted amount: Fiscal year: 38.3; Budgeted amount: Total: 95. Release: 6(M3): Exports and Mail Entry Writing System; Budgeted amount: Prior years[A]: [Empty]; Budgeted amount: Fiscal year: 38.3; Budgeted amount: Total: 95. Release: 7(A3): ESAR: Drawback, Protest and IASS[B]; Budgeted amount: Prior years[A]: [Empty]; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: [Empty]. Release: 7(M4): e-Manifest Custodial Entities, Pipeline, and Batch Processes[C]; Budgeted amount: Prior years[A]: [Empty]; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: [Empty]. Release: S1: Screening Foundation; Budgeted amount: Prior years[A]: 48.7; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 48.7. Release: S2: Targeting Foundation; Budgeted amount: Prior years[A]: 39.3; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 39.3. Release: S3: Advanced Targeting; Budgeted amount: Prior years[A]: 21.8; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 21.8. Activity: ACE Operations and Maintenance; Budgeted amount: Prior years[A]: 143.1; Budgeted amount: Fiscal year: 44.2; Budgeted amount: Total: 187.3. Activity: ACE Production Baselines; Budgeted amount: Prior year[A]: [Empty]; Budgeted amount: Fiscal year: 22.6; Budgeted amount: Total: 22.6. Activity: ACE Foundation Architecture and Engineering; Budgeted amount: Prior year[A]: 90.4; Budgeted amount: Fiscal year: 35.7; Budgeted amount: Total: 126.1. Activity: ACE Implementation Infrastructure and Support; Budgeted amount: Prior year[A]: 269.5; Budgeted amount: Fiscal year: 62.9; Budgeted amount: Total: 332.4. Activity: ACE Foundation Program Management; Budgeted amount: Prior year[A]: 137.5; Budgeted amount: Fiscal year: 18.9; Budgeted amount: Total: 156.4. Activity: ACE Communications, Training, Outcome, and Deployment; Budgeted amount: Prior year[A]: 31.7; Budgeted amount: Fiscal year: 24.4; Budgeted amount: Total: 56.1. Activity: Other Tasks; Budgeted amount: Prior year[A]: 20.6; Budgeted amount: Fiscal year: [Empty]; Budgeted amount: Total: 20.6. Activity: Cargo Systems Program Office; Budgeted amount: Prior year[A]: 201.2; Budgeted amount: Fiscal year: 25.3; Budgeted amount: Total: 226.5. Activity: International Trade Data System; Budgeted amount: Prior year[A]: 59.4; Budgeted amount: Fiscal year: 16; Budgeted amount: Total: 75.4. Activity: Management Reserve; Budgeted amount: Prior year[A]: 92; Budgeted amount: Fiscal year: 4.6; Budgeted amount: Total: 96.6. Activity: International Trade Data System Funding for Development[D]; Budgeted amount: Prior year[A]: [Empty]; Budgeted amount: Fiscal year: -14; Budgeted amount: Total: -14. Appropriated total; Budgeted amount: Prior year[A]: 1707; Budgeted amount: Fiscal year: 317; Budgeted amount: Total: 2024. Source: GAO analysis of CBP data. [A] Prior funding consists of stopgap funding, approved by the appropriations committees in March 2001, and the seven previous expenditure plans. [B,C] Funding for A3 and M4 has not yet been included in the expenditure plans. [D] This amount consists of prior year unobligated funds that ITDS provided to ACE to address requirements for participating government agencies. [End of table] Objective 1: Legislative Conditions: Condition 1: Partially Satisfied: The 6 legislative conditions have been either satisfied or partially satisfied. Table: Legislative condition: Legislative condition: 1. Meets the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including OMB Circular A-11, part 7; Status [A,B]: Partially satisfied. Legislative condition: 2. Complies with DHS’s enterprise architecture; Status [A,B]: Partially satisfied. Legislative condition: 3. Complies with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; Status [A,B]: Satisfied. Legislative condition: 4. Includes a certification by the DHS CIO that an IV&V agent is currently under contract for the project; Status [A,B]: Satisfied. Legislative condition: 5. Is reviewed and approved by the DHS IRB, Secretary of Homeland Security, and OMB; Status [A,B]: Satisfied. Legislative condition: 6. Is reviewed by GAO; Status [A,B]: Satisfied. Source: GAO. [A] Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying many, but not all, key aspects of the condition that we reviewed. [B] Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every aspect of the condition that we reviewed. [End of table] Condition 1: The plan, including related program documentation and program officials’ statements, partially satisfies the capital planning and investment control review requirements established by OMB, including Circular A-11, part 7.[Footnote 20] The table that follows provides an overview of the results of our analysis and selected examples in areas where A-11 requirements have or have not been fully satisfied. Given that the A-11 requirements are intended to minimize a program’s exposure to risk, permit performance measurement and oversight, and promote accountability, any areas in which the program falls short of the requirements reduces the chances of delivering cost effective capabilities and measurable results on time and within budget. Objective 1: Legislative Conditions: Condition 1: Partially Satisfied: Table: Condition 1: Partially Satisfied: Table: Examples of A-11 Conditions Results of Our Analysis: Provide a brief description of the investment and its status in the capital planning and investment control review process, including major assumptions made about the investment; The expenditure plan and the ACE Exhibit 300 budget submission provide brief descriptions of the ACE investment and its releases. The Exhibit 300 also describes the status of ACE relative to DHS’s capital planning and investment control process, and states that ACE is in the “control” stage of the process; The Exhibit 300 and the ACE program plan also describe investment assumptions, such as (a) an annual budget of $305.5 million will be provided; (b) incremental deliveries within releases will mitigate risks and reduce uncertainty of program estimates; (c) ITDS funding will be timely, separate, and adequate; (d) stakeholder representation and resources will be adequate. Report performance goals and measures for existing investments and show how the Federal Enterprise Architecture (FEA) Performance Reference Model (PRM) applies to this investment; The expenditure plan reports the program’s performance goals, benefits, objectives, performance measures, and their relationships for some, but not all, increments, including actual performance for fiscal years 2005 through 2006. The plan includes the ACE accountability framework, which is the program’s overall tool for implementing CBP’s PRM, which in turn is based on the FEA PRM. CBP’s PRM includes, for example, performance measures for user satisfaction, efficiency, productivity, and data reliability, among other elements, and the ACE accountability framework incorporates a subset of these PRM measures, augmented by additional measures focusing on the program’s status; ACE’s Exhibit 300 also presents performance goals, targets, measures, and results for 2003 through 2012. However, these are not fully consistent with the expenditure plan. According to CBP, this is because the ACE performance measures were aligned with DHS’s goals, objectives, strategies, and desired results as of July 2006, which is after the fiscal year 2007 Exhibit 300 was submitted. In addition, the performance measures continue to evolve as more understanding is gained in the usefulness and meaning of measures. (See the open recommendations section of this briefing for more information on performance measures.) Provide a summary of the investment’s risk assessment, including how 19 OMB-identified risk elements are being addressed; The expenditure plan presents some, but not all, of the identified risks for the program’s releases and activities. The Exhibit 300 also presents program risks, assessments, mitigation strategies, and status, and it organizes them by OMB risk categories. However, the risks in the expenditure plan and the Exhibit 300 are not fully consistent with each other or with the risks being tracked in the program’s risk database. Further, the most recent risk assessment is limited to security risks for Release 4; The ACE program has a documented risk management process that includes identifying, classifying, reporting, and tracking risks. However, this process has not been fully implemented. For example, some risks that are identified in the risk database and the accountability framework have missing or outdated status information and mitigation strategies. As a result, the status of all risks is not clear. Program officials stated that risk management is not yet mature, but that they are taking steps to improve it. (See the open recommendation section of this briefing for more information on risk management.) Provide a summary of the investment’s status in accomplishing baseline cost and schedule goals through the use of an earned value management (EVM) system or operational analysis, depending on the life-cycle stage; The expenditure plan reported some, but not all, EVM data for ACE releases and activities. For example, the data includes planned, estimated, and actual schedule milestones, but it does not include schedule variances; The EVM data in the expenditure plan was from the ACE accountability framework that program officials rely on to manage the program. (See the open recommendations section of this briefing for more information on the accountability framework.) According to program officials, they are using EVM to manage all ACE releases and screenings under contract and EVM tracking begins with an Integrated Baseline Review, which forms the basis for a realistic plan against which to objectively measure work to be completed during the period of performance; The Exhibit 300 also includes EVM data and describes contractor requirements for EVM, including verification of EVM compliance with industry standards. According to program officials, these contractual requirements are still operative; Nevertheless, the EVM program has still not been certified, as required by OMB. According to program officials, this certification is scheduled for the beginning of fiscal year 2008. As an interim measure, the prime contractor performed an EVM compliance and internal surveillance audit against relevant EVM criteria[Footnote 21] in March 2006 and found no issues. Provide a description of the investment's privacy and security issues. Summarize the agency's ability to manage security at the system or application level. Demonstrate compliance with the certification and accreditation processes as well as the mitigation of IT security weaknesses; Privacy: The expenditure plan does not discuss privacy issues. However, the Exhibit 300 states that ACE is subject to the privacy provisions of the E-Government Act of 2002. It further states that a privacy impact assessment was conducted in 2005 for Release 4 and that separate assessments will be conducted for each subsequent release. This assessment was approved by DHS on July 14, 2006, and our analysis shows that it complied with relevant DHS guidance. This assessment does not cover other recently completed screening releases S1 and S2. According to program officials, S1 and S2 are considered to be part of the Automated Targeting System (ATS), and are therefore covered by the ATS PIA. However, our analysis of the ATS PIA showed that while it addresses screening and targeting functions, it does not specifically identify or address releases S1 and S2. (See the open recommendations section of this briefing for further information on privacy issues); Security: CBP has taken a number of actions related to ACE security management. It conducted a security self-assessment for Release 4 in September 2006 using the National Institute of Standards and Technology (NIST) Security Self-Assessment Guide for Information Technology Systems and did not report any major security vulnerabilities. In addition, CBP officials stated that another security self-assessment was conducted in April 2007. However, we have yet to receive this self- assessment. CBP also reports that it has conducted two full contingency plan tests at its disaster recovery facility; Further, CBP accredited ACE Release 4 on November 22, 2004; Release S1on July 18, 2006; and Release S2 on October 18, 2006. The program office plans to conduct its next ACE certification and accreditation of ACE in September 2007 for Release 5/Drop A1. However, since November 2004, the deployed ACE system has been subject to considerable change. Specifically, about 7,100 trouble tickets, trouble reports, change requests and install requests were generated—some of which have resulted in system changes. Federal guidance recommends reaccreditation whenever significant changes to the system or its operational environment are likely to affect a system’s security posture.[Footnote 22] However, CBP has not established explicit criteria for when to reaccredit/recertify a system in the face of changes, or documented a systematic procedure for determining the security risk from the cumulative impact of changes made to the system. As a result, CBP has not recertified or reaccredited the system; In addition, the ACE security plan and security risk assessment have not been updated to reflect key information, such as the results of the April 2007 security self-assessment or a system access control risk identified in May 2006 and included in the program’s risk database. Source: OMB and NIST criteria and GAO analysis of DHS documentation. [End of table] Objective 1: Legislative Conditions: Condition 2: Partially Satisfied: Condition 2: The plan, including related program documentation and program officials’ statements, partially satisfies the condition that it comply with the DHS information systems enterprise architecture (EA) as currently defined. According to federal guidelines and best practices, investment compliance with an EA is essential for ensuring that an organization’s investment in new and existing systems is defined, designed, and implemented in a way that promotes integration and interoperability and minimizes overlap and redundancy, thus optimizing enterprisewide efficiency and effectiveness. A compliance determination is not a one- time event that occurs when an investment begins, but is, rather, a series of determinations that occurs throughout an investment’s life cycle as changes to key aspects of both the EA and the investment’s architecture are made (e.g., data, business, services, and technology). The DHS Enterprise Architecture Board, supported by the Enterprise Architecture Center of Excellence, is responsible for ensuring that projects demonstrate adequate technical and strategic compliance with the department’s EA. During 2006, the DHS Enterprise Architecture Board conducted three reviews of ACE architectural alignment with the DHS EA Technical Reference Model (TRM)[Footnote 23] as part of key milestone reviews for three release/screening increments. * Master Data and Enhanced Accounts (Drop A1), Critical Design Review, June 2006; * E-Manifest: Rail and Sea (Drop M1), Critical Design Review, September 2006, and: * Targeting Foundation (S2), Production Readiness Review, November 2006. On May 1, 2007, DHS EA officials reported that all required products and technologies were aligned to the TRM, and thus that ACE was in alignment with the DHS EA. In addition, the DHS CIO certified as part of the above milestone reviews that each increment was in alignment with the TRM. More specifically, * In December 2006, the Center of Excellence determined that ACE was conditionally compliant with the DHS EA, contingent on actions taken in regard to four conditions. Three of the four conditions were resolved by March 2007. * On March 29, 2007, the EAB recommended approval of ACE’s decision request for program alignment, with one remaining condition—that by June 30, 2007, the program submit technology insertions packages for the products identified as needing alignment with the DHS TRM. Notwithstanding these EA compliance determinations, DHS did not provide us with sufficient documentation of its determinations to allow us to understand the methodology and criteria, or to verify the analyses, that were used to arrive at them. Moreover, documentation that was provided showed that the determinations focused on ACE technical alignment, and did not address, for example, alignment to the DHS EA Data Reference Model. Until DHS demonstrates, through verifiable documentation and methodologically- based analysis, that ACE is aligned with all relevant aspects of the DHS EA, including corporate data structures and standards, the program will be at risk of being designed and implemented in a way that does not support optimized departmental operations, performance, and achievement of strategic goals and outcomes, including those related to information sharing. Objective 1: Legislative Conditions: Condition 3: Satisfied: Condition 3: The plan, including related program documentation and program officials’ statements, satisfies the condition that it comply with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government.[Footnote 24] Federal acquisition rules, requirements, guidelines, and management practices provide an acquisition management framework that is based on the use of rigorous and disciplined processes for planning, managing, and controlling the acquisition of IT resources.[Footnote 25] These acquisition management processes are embodied in published best practices models, such as the Capability Maturity Models® developed by Carnegie Mellon University’s Software Engineering Institute (SEI). These models explicitly define, among other things, acquisition process management controls that are recognized hallmarks of successful organizations and that, if implemented effectively, can greatly increase the chances of acquiring software-intensive systems that provide promised capabilities on time and within budget. In our prior reviews of the ACE program,[Footnote 26] we reported that ACE had satisfied this condition based on SEI’s November 2003 assessment of the program against the Software Acquisition Capability Maturity Model (SA- CMM®).[Footnote 27] That assessment assigned the program an SEI level 2[Footnote 28] rating, indicating that CBP had instituted basic acquisition management processes and practices. Since receiving its 2003 rating, the program office has not conducted another acquisition capability assessment to ensure that it is continuing to employ these basic acquisition controls, and does not plan to do so. Furthermore, program officials told us that although the prime contractor was contractually required to have an SEI CMM® Level 3[Footnote 29] capability, they are no longer required to do so because funding contractor program management-related activities is now a lower priority relative to other competing demands for ACE funding. According to CBP officials, several steps have been taken to mitigate any impact of no longer focusing on SEI CMM® compliance, such as: * requiring the prime contractor and the program office’s support contractors to follow plans, processes, and procedures for such key areas as EVM, configuration management, and problem reporting; * having the program office’s quality assurance staff, in collaboration with the prime contractor’s quality assurance function, monitor adherence to key management controls through reviews of ACE processes and products, including 48 such reviews in fiscal year 2006; and: * having the program office’s contracting function continuously monitor adherence to contractual provisions; * leveraging OIT assistance, through the Process Asset Group, to conducts reviews of new and updated plans, policies, and procedures;[Footnote 30] and: * cataloguing job aids in a Process Asset Group library that specifies the required activities for program management processes and disseminating them to ACE staff. Objective 1: Legislative Conditions: Condition 4: Satisfied: Condition 4: The plan satisfies the condition that it include certification by the DHS CIO that an IV&V agent is currently under contract. On October 24, 2006, the DHS Deputy CIO certified in writing that an IV&V agent is under contract for ACE and that the agent met applicable requirements and standards. (See open recommendations section of this briefing for more information on IV&V.) Objective 1: Legislative Conditions: Condition 5: Satisfied: Condition 5. The plan, including related program documentation and program officials’ statements, satisfies the requirement that it be reviewed and approved by the DHS IRB, the Secretary of Homeland Security, and OMB. * The DHS IRB reviewed the program and approved the expenditure plan on December 12, 2006. * The DHS Under Secretary for Management approved the expenditure plan on behalf of the Secretary of Homeland Security on February 6, 2007. * OMB approved the expenditure plan on January 22, 2007. Objective 1: Legislative Conditions: Condition 6: Satisfied: Condition 6. The plan satisfies the requirement that it be reviewed by GAO. Our review was completed on July 26, 2007. Objective 2: Open Recommendations: Accountability Framework: Open Recommendation 1: Define and implement an ACE accountability framework that fulfills these conditions: a. Covers all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules. Status: Complete: Effective program management includes defining and measuring progress against program commitments and being held accountable for results. Such commitments generally cover expected or estimated (1) capabilities and their associated use and quality, (2) benefits and mission value, (3) costs, and (4) milestones and schedules. Since 2003, we have reported that such commitments for ACE have not always been defined, although improvements have been made. Most recently, we reported[Footnote 31] that the program office had developed an initial version of an accountability framework for measuring several program commitments, such as capabilities and milestones, but that other commitments, such as benefits, had not been as well defined. During the last year, the program office has continued to improve its coverage of program commitments, as evidenced by the content of key program documents— ACE accountability framework, ACE Program Plan (August 2006), the CBP/ACE performance reference model, periodic Program Management Review (PMR) reports, and the fiscal year 2007 expenditure plan. For example, * The accountability framework now captures and integrates data on ACE functional and performance capabilities, benefits, and mission value, estimated and actual costs, milestones and schedule, accomplishments, and earned value management status. Further, the framework provides visibility into each ACE release at several levels of detail, and it is being used by the ACE Executive Director and others as the means for monitoring program progress, issues, and decisions. * The PMR reports have included the accountability framework data. * The fiscal year 2007 ACE expenditure plan included an example of the accountability framework commitments from the September 2006 PMR report. b. Ensures currency, relevance, and completeness of such commitments made to Congress in expenditure plans. Status: In progress: We have previously reported that commitments made in expenditure plans relative to program capabilities, benefits, costs, and schedules need to be current, relevant, and complete. To the extent that they are not, the currency and relevance of the plan and its utility to Congress as an accountability mechanism are limited. ACE expenditure plans have not always included such information. To address this limitation, we reported last year[Footnote 32] that CBP was relying on quarterly reports to Congress to provide the appropriations committees with more current, relevant, and complete information about the program than could be provided in the expenditure plan. However, we also noted that the quarterly reports were generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance. The fiscal year 2007 expenditure plan continues to include information about program commitments that is not current. Specifically, * The latest expenditure plan was provided to the appropriations committees on February 6, 2007. However, information from the ACE accountability framework that was included in the plan, such as milestones, EVM values, and risks, were as of October 2006, making it about 4 months old. For example, the expenditure plan listed the milestone for the Production Readiness Review (PRR) for a key release, Release 5/Drop A1, as March 1, 2007. However, this milestone had already slipped by more than 4 months--to July 12, 2007--by the December 2006 PMR (held on January 5, 2007). According to program officials, they continue to use the quarterly ACE status reports to provide the appropriations committees with more current information.[Footnote 33] However, recent quarterly reports have not been current. For example, * The December 2006 quarterly report was not provided until February 26, 2007, and it contained the same outdated PRR milestone as the expenditure plan did for the release previously mentioned. * Other quarterly reports have been submitted to the appropriations committees as many as 7 months after the end of the quarter. CBP and DHS officials told us that the delays were due to the DHS review and approval process and that incorporating the most current program information would delay the reports even longer. According to the ACE and DHS officials, they are exploring ways to accelerate the review process. c. Ensures reliable data relevant to measuring progress against commitments. Status: Complete: The quality of the capabilities that a program is to deliver is a relevant program commitment. One measure of the quality of system capabilities is the trend in the number and severity of unresolved system defects or problems. Reliable data about these defects are needed so that system maturity can be understood and informed investment decisions can be made. We previously reported [Footnote 34] that ACE defect data were not always consistent because the two tools that the program used to track defects were not integrated and reconciled. As a result, the true status of ACE defects, and thus an important measure of system quality, was not known. Since then, the program office has integrated the two tools using a cross- referencing function and has instituted manual processes for reconciling the data in each tool. The December 2006 quarterly congressional report stated that these efforts have improved the ability to record, assess, and report on system quality and performance. d. Ensures reporting in future expenditure plans progress against commitments contained in prior expenditure plans. Status: In progress: The value and utility of an expenditure plan for congressional oversight and agency accountability depends in large part on whether the plan reports on progress against commitments--system capabilities, benefits, costs, and schedules--made in prior plans. Historically, the ACE expenditure plans have not reported adequately on progress against previous plan commitments. Most recently, we reported[Footnote 35] that the fiscal year 2006 expenditure plan contained several such reporting gaps, such as whether funding amounts were actually obligated and expended as planned and whether releases met schedules. In particular, we reported that the plan did not address whether the design and development for Release 5 was actually accomplished as planned. The fiscal year 2007 expenditure plan still does not adequately describe the program’s progress against the commitments that were made in the fiscal year 2006 plan. For example, * The plan does not report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved. * The plan does not report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans. * The plan did not address progress against the milestone dates for each stage of a release that was included in the prior year’s plan. As a case in point, the fiscal year 2006 expenditure plan described specific functionality planned for Release 5/Drop A1 (Master Data and Enhanced Accounts), such as online registration for trade representatives. However, the fiscal year 2007 expenditure plan does not include information on whether these functions were delivered, stating only that Release 5 functionality would be deployed in two phases. According to program officials, the quarterly congressional reports provide more current information on the program’s progress against prior commitments. However, we found that these reports have also not fully addressed the commitments made in prior expenditure plans. For example, recent quarterly reports describe progress against developmental milestones for each release, but do not report whether key functionality associated with each release was delivered. In particular, the report for the first quarter of fiscal year 2007 identified planned future capabilities and associated milestones for Release 5/Drop A1 but did not address the status of the key functions associated with this release. Program officials stated that they intend to start providing this information in the last quarterly report of each year. e. Ensures use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects, and document the milestone decisions in a way that reflects the associated risks and plans for mitigating them.[Footnote 36] Status: Complete: As noted earlier, one measure of the quality of system capabilities being delivered is the number and severity of unresolved system defects or problems. As such, information on such defects is an important consideration when program decisions, such as key milestone decisions, are made. We previously reported[Footnote 37] that several key milestones were passed that had severe open defects and that program officials were unable to provide documentation regarding how the risks associated with these defects were assessed. We also reported that these risks were not being tracked in the program’s risk database. Since then, the system life cycle gate review process for Office of Information and Technology projects, including ACE, has been amended to include milestone readiness decisions based on an assessment and acceptance of risks (including the risks related to unresolved defects). Further, the ACE Risk and Issue Management Process guidance provides for the systematic identification, analysis, prioritization, planning, execution, evaluation, and documentation of program risks and issues and requires that any risks associated with going forward should be identified as part of each life cycle gate review. This process has been applied in the following two major gate reviews: * Release A1 (ESAR) Critical Design Review (May 2006) and: * Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review (August 2006). In the case of Release 5/Drop A1, program officials reported that no severe defects existed, but that all known risks associated with proceeding past the milestone had been entered into the program’s risk database and documented in the certification package submitted to the DHS CIO. The quarterly reports to Congress similarly state that risks related to gate review decisions and the associated impacts are entered into a database to ensure visibility and mitigation and are included in the package submitted to the DHS CIO for review and certification to pass a given milestone. Going forward, officials stated that similar assessments will be part of the following planned gate reviews for Release 5/Drop A1: * Operational Readiness Review (planned for August 23, 2007) and: * Live operations (planned for August 25, 2007). Open recommendation 2: Develop the range of realistic ACE performance measures and targets needed to support an outcome-based, results- oriented accountability framework, including user satisfaction with ACE. Status: Complete: We have previously reported on both the absence of meaningful performance measures to understand ACE progress, quality, and results, and have raised concerns about the practicality and applicability of some measures that have been defined. Most recently, we reported[Footnote 38] that defined ACE performance targets are not always realistic and that goals, expected mission benefits, and performance measures are not fully defined and adequately aligned. In response, the program office has established an initial set of performance measures that are tied to program objectives and related performance goals, and it has established processes for the collecting, analyzing, and integrating performance data for each measure. Among other things, these performance measures address user satisfaction, efficiency, and productivity. Moreover, these measures have been made an integral part of the ACE accountability framework and they were approved by the CBP Commissioner and the DHS CIO on June 30 and July 6, 2006, respectively. Objective 2: Open Recommendations: Performance Measures: According to program officials, these initial measures will continue to be revised and augmented, and thus will evolve over time based on lessons learned, changes to existing release capabilities, and refinements of requirements for future releases. For example, the program office reported in March 2007 that new measures are still being developed, reviewed, and evaluated for Releases 2, 3, and 4 at the same time that performance data is being collected and reported for existing measures for these releases. According to program officials, the new measures will be combined with existing performance measures. In addition, CBP’s December 2006 quarterly congressional report stated that performance measures for future releases, such as Release 5, are being scheduled for identification and development and that changes are being made to existing measures when they are determined to be inappropriate. For example, the program office learned that the performance measure "percentage of truck manifests being filed electronically" did not recognize that empty trucks are not required to file manifests, and thus it revised the metric used to divide the number of electronically filed truck manifests by the total number of required truck manifests instead of the total number of trucks in order to get a more meaningful reflection of ACE performance. To assist in managing these performance measures, the program has established a life cycle process for performance measures as well as a database tool to record and manage modifications. Objective 2: Open Recommendations: Performance Measures Alignment: Open recommendation 3: Explicitly align ACE program goals, benefits, desired business outcomes, and performance measures. Status: Complete: As just discussed, the program office has developed an initial set of ACE program measures that are tied to program goals, among other things, and they intend to continue to update and evolve these as the program moves forward. In addition, the program office has developed an ACE Performance Reference Model that explicitly aligns the CBP strategic goals, objectives, strategies, and desired results applicable to the ACE program with specific performance measures. For example, Table: CBP Strategic Goal; Preventing terrorism at ports of entry: Prevent terrorists and terrorist weapons, including weapons of mass destruction and weapons of mass effect, from entering the U. S.. CBP Objective; Improve information and targeting. CBP Strategy; Use advanced passenger and cargo information (NTC, ATS-Air, ATS, Screening and Targeting-ACE) to pre-screen, target, and identify potential terrorists and terrorist shipments and any related activity. Desired Result; Increased use of targeting. Performance Measure; Number of security-focused selections generated by system (= or above intensive threshold) by type. Further, this model links the measures to specific ACE releases. For example, the above performance measure applies to Screening Foundation (S1) and Targeting Foundation (S2). Objective 2: Open Recommendations: Management Improvement Measures: Open recommendation 4: Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful. Status: Planned: As we have previously reported,[Footnote 39] investments in program management improvements should include defined measures of progress and results. To date, the program office has implemented a number of such improvements; however, it has not had measures or metrics to determine the success of the improvements. Moreover, the December 2006 quarterly congressional report stated that the program anticipates more changes, including: * creation of a cargo requirement management board to decide the disposition of all change requests to production systems; * establishment of a new invoice review policy; and: * co-location of personnel within a given business area. According to its December 2006 quarterly reports to Congress, the program office plans to measure the impact of future management improvements. However, program officials told us that they have yet to define them and thus are not yet using such measures. Objective 2: Open Recommendations: Legislative Conditions: Open recommendation 5: Fully address those legislative conditions associated with measuring ACE performance and results and employing effective IV&V practices. Status: Complete: Among the legislative conditions that ACE expenditure plans have been required to meet are satisfaction of OMB guidance, including that associated with measuring program performance and results and use of effective IV&V practices. These conditions reflect good program management practices and, if implemented properly, can reduce program risks. Performance and Results: As previously discussed in this briefing, the program office has developed a range of ACE performance measures and taken steps to align them with program, CBP, and DHS strategic goals and outcomes. According to program officials, they plan to continue to evolve and refine the performance measures and include the measures in the ACE accountability framework. IV&V Practices: In January 2006, the DHS CIO certified that an IV&V agent was under contract for the ACE program, but noted several issues, including these: * mechanisms were needed to ensure that products were complete, of sufficient quality, and met the needs of the user and: * a more explicit technical approach, describing when and how certain activities should (or should not be) performed, was needed. Moreover, we subsequently reported[Footnote 40] that the scope of the contractor’s activities did not extend to both IV&V of key system products and development processes. On October 24, 2006, the Deputy CIO again certified that an IV&V agent was under contract and stated that the previous issues had been addressed. Since then, the program office has also developed an IV&V Implementation Management Plan that addresses the concerns raised by the DHS CIO and us. In particular, the plan: * requires an IV&V program consistent with the industry standard; * provides a set of objectives, guidelines, and expectations for IV&V activities, including periodic independent reports on status, observations, recommendations, and activities; and: * addresses satisfaction of quality standards for ACE products and user needs. Program officials told us that IV&V has allowed early identification and correction of program process and product weakness. Objective 2: Open Recommendations: Reconciliation of Cost Estimates: Open recommendation 6: Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates. Status: Complete: It is important that expenditure plans be based on reliable estimates of costs, to include reconciling differences between government and independent cost estimates. We recently reported[Footnote 41] that the cost estimate in the fiscal year 2006 expenditure plan was based on government and independent cost estimates that had been compared and found to be consistent. For the fiscal year 2007 expenditure plan, our analysis showed that the government and independent cost estimates differed by about 15 percent. According to program officials, they reconciled these differences in January 2007, and concluded that the results did not warrant changes to the expenditure plan because the government estimates used in the expenditure plan were more accurate. According to program officials, two primary factors account for the difference in estimates. Specifically, * The estimates assumed different timelines for completing development of all releases. The independent estimate assumed development completion by fiscal year 2010, while the government estimate assumed fiscal year 2011. According to program office officials, this accounts for about five percent of the difference. * The government estimate included a number of items that the independent estimator did not. For example, the independent estimator did not include training and outreach items, such as the cost of conferences with trade associations, training materials, and associated travel. As a result, the government estimated training and outreach costs over the life of the program to be about $90 million, while the independent estimate put these costs at about $39 million. The independent estimate has since been amended to include the missing items, and the reconciliation process is continuing. Objective 2: Open Recommendations: Rigorous Cost Estimation: Open recommendation 7: Develop and implement a rigorous and analytically verifiable cost estimating program that embodies the tenets of effective estimating.[Footnote 42] Status: Complete: The reliability of cost estimates is largely a function of the quality of the estimating process used to derive them. We previously reported[Footnote 43] that the program did not have a well-defined cost estimating process, but that it has since made progress in strengthening its cost estimating program. Specifically, the program office has: * defined and documented processes for estimating program costs (including management reserve costs) and: * hired a contractor to develop costs estimates that were independent of the government estimates. In September 2006, an ACE support contractor reported that both the government and independent cost estimation processes demonstrated significant conformance to effective estimating practices and concluded that the program is using a rigorous and verifiable cost estimating approach. Objective 2: Open Recommendations: Earned Value Management: Open recommendation 8: Use EVM in developing all existing and future releases. Status: Complete: EVM is a program management tool to measure progress by comparing, during a given period of time, the value of work accomplished with the amount of work expected to be accomplished. This comparison permits performance to be evaluated based on calculated variances from the planned (baselined) cost and schedule. EVM is both an industry accepted practice and an OMB requirement. We recently reported[Footnote 44] that the program office was not using EVM for all releases (e.g., Release 5) due to changes to release baselines and the lack of familiarity on the part of program staff with EVM practices. Since then, the program office has established the performance baselines needed to implement EVM for Release 5/Drops A1 and A2, Screening 3, and Release 6/Drop M1. For these releases, the program office reports that it has also applied EVM standards in establishing baselines and included EVM data in the accountability framework for management and decision-making purposes. Program officials also reported that they plan to implement EVM on future releases and task orders. Objective 2: Open Recommendations: ACE Infrastructure: Open recommendation 9: Have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans. Status: Complete: Together, the fiscal year 2007 expenditure plan and the quarterly reports to Congress address ACE’s relationships with other trade processing and DHS applications. For example, the plan discusses efforts to work with participating government agencies in defining and deploying the International Trade Data System (ITDS). The stated goal is to deliver ACE/ITDS in an integrated and coordinated manner. In this regard, the plan discusses workshops to gather requirements from participating agencies for Release 6/Drop M2 (first held on July 19, 2006), a working group to address these agencies’ HAZMAT issues (established on July 17, 2006), and efforts to develop data element inputs for ACE from other government agencies. In addition, the quarterly reports to Congress cite coordination efforts with related homeland security programs. For example, the report for the first quarter of 2007 states that: * Container Security Initiative will be supported by ACE Release 6 and Screening 1-3 capabilities; * Customs-Trade Partnership Against Terrorism is coordinating with ACE Release 5/Drop A1 capabilities to provide both CBP and trade representatives with the ability to view the status of CBP programs; and: * U.S.-Mexico Border Partnership Plan will coordinate with ACE to implement any cargo screening standards derived from partnership plan agreements. In addition, CBP reports that ACE and other CBP applications[Footnote 45] share infrastructure (desktops, clients, and local area networks) at the ports of entry. This infrastructure is purchased and maintained by CBP’s OIT Program Integration Division. Each application, including ACE, is responsible for complying with OIT’s infrastructure standards. Objective 2: Open Recommendations: Overlap and Concurrence: Open recommendation 10: Minimize the degree of overlap and concurrency across ongoing and future ACE releases and capture and mitigate the associated risks of any residual concurrence. Status: In progress: Significant overlap and concurrency among major program activities, such as releases, introduce considerable cost and schedule risks as they can create contention for limited resources among the releases. We have continued to report on extensive overlap and concurrence in ACE releases and the cost overruns and schedule delays that have resulted. Most recently, we reported[Footnote 46] that the ACE schedule continued to provide for such overlap and concurrency and that the risks associated with doing so were not being effectively addressed. Since then, the program office has reduced this overlap and concurrence. For example, it has: * decoupled (i.e., reduced dependencies among) certain ACE program components by separating Screening 1-3 from Releases 4-6 and: * aligned the development and delivery of functionality for different releases with the availability of required hardware environments. Recent quarterly congressional reports stated that CBP has taken actions to reduce potential contention for limited resources. Examples include: * dividing releases into smaller subreleases, called drops, to provide more flexibility in scheduling; * improving planning for development, integration, testing, and training activities and milestones to better schedule use of development and test environments; and: * centralizing management of shared software services to address, among other things, allocation of resources and responsiveness to workload peaks and use of consistent technical management approaches across releases. Moreover, the program’s risk database identifies overlap and concurrency-related risks. However, the mitigation strategies for these risks contain vague or incomplete data. These data limitations make it difficult to determine the status or the effectiveness of the efforts to reduce the risks associated with overlap and concurrence among releases. (See the observations section of this briefing for additional information on program risk management.) The program office is also conducting regular integration meetings with the teams supporting each release to discuss concerns, decisions, and schedules associated with resource availability and is using a software tool to track and mitigate release- specific concurrency risks. Notwithstanding these steps, the program continues to face challenges in managing dependencies among releases, which any associated concurrency in the program’s development exacerbates. For example, the schedule slips expected with Release 5/Drop A2 will affect Release 6/Drop M1, as resources have been shifted from M1 to address the A2 delay. In addition, further delays in A1 and A2 will impact the implementation of M1. Objective 2: Open Recommendations: Legislative Conditions: Open recommendation 11: Direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment (PIA)[Footnote 47] and ensuring architectural alignment. Status: In progress: The department approved the ACE PIA for Release 4 in July 2006, however, this assessment does not cover other completed releases. Further, DHS has determined that ACE is aligned with DHS architecture, but we have yet to receive documentation to adequately understand and verify the determination. PIA (In progress): In March 2006, DHS developed guidance on the development and content of a PIA. As we have previously reported,[Footnote 48] the purpose of a PIA is to ensure that there is no collection, storage, access, use, or dissemination of identifiable personal or business information that is not both needed and permitted. Development and use of a PIA are both a requirement of OMB and the E-Government Act of 2002.[Footnote 49] The program office has developed a PIA for Release 4 of the ACE e- Manifest: Trucks and the International Trade Data System, and DHS approved it on July 14, 2006. As noted earlier, our analysis shows that this PIA addresses the major elements of DHS’s guidance. Program officials stated that they would update the assessment for each ACE release. This assessment does not cover other recently completed screening releases (i.e., S1 and S2). However, program officials told us that S1 and S2 are considered to be part of the Automated Targeting System (ATS), and are therefore covered by the ATS PIA. However, our analysis of the ATS PIA showed that while it addresses screening and targeting functions, it does not specifically identify or address releases S1 and S2. Architectural Alignment (in progress): As discussed in the legislative conditions section of this briefing, DHS has determined that ACE is aligned with the DHS EA. For example, * In December 2006, DHS’s EA Center of Excellence determined that ACE was conditionally compliant with the DHS architecture, but that the program needed to take actions to address four conditions. * On March 2007, the DHS EA Board reported that ACE had satisfied three of the conditions and recommended approval of ACE’s request for program alignment with one remaining condition—alignment with the DHS Technical Reference Model. * In May 2007, DHS EA officials reported that all required products and technologies were aligned with the Technical Reference Model and that ACE was thus aligned with the DHS EA. However, as was also discussed earlier in this briefing, we have yet to receive sufficient documentation describing the criteria and methodology used to make these determinations or verifiable analysis supporting the determinations. Moreover, the determinations were based on technical alignment and did not address other relevant aspects of program alignment to an EA, such as data alignment. Thus, the program is at risk of being designed and implemented in a way that is not consistent with all relevant aspects of the DHS EA, such as data structures and standards, and thus does not support optimized DHS-wide operations, performance, and results. Objective 2: Open Recommendations: Human Capital Practices: Open recommendation 12: Develop and implement key human capital management practices. Status: In progress: As we have previously reported,[Footnote 50] effective strategic management of program human capital includes, among other things, defining the skill sets needed to perform program functions, assessing and inventorying the skill sets of the program’s current workforce and assessing any associated skill gaps in meeting future needs, and developing strategies to fill any identified gaps. Moreover, it includes having a well-defined plan that provides for effective implementation of these processes. In June 2006, CBP executives approved the OIT Strategic Human Capital Management Plan. The plan is intended to be a road map for supporting CBP’s workforce vision and ensuring effective management of human capital resources across OIT to address enterprise-wide priorities. An ACE-specific plan was included as an appendix to the CBP plan. This appendix was intended to provide better near- and long-term human capital management practices for the OIT offices involved in ACE development. Neither the OIT nor the ACE-specific plan addresses the basic tenets of effective human capital management. For example, neither provides for: * defining the positions needed (including core competencies) to perform core program functions; * assessing and inventorying current workforce skills and abilities; * assessing any gaps between needed and existing workforce levels and capabilities; and: * filling identified gaps via such means as hiring new staff, training existing staff, and augmenting staff with contractor support. OIT officials acknowledged these limitations in the plans and stated that they are developing an implementation plan to address these shortfalls. The officials stated that the implementation plan will include accountability, timeframes, and metrics to carry out the larger OIT Strategic Human Capital Management Plan. According to these officials, this implementation plan was to be completed in January 2007, following presentation to the OIT Deputy Director's Council and approval by the OIT Office Directors and Assistant Commissioner. However, as of July 2007, it had not yet been approved and thus was not available for our review. In the interim, OIT and program officials reported taking various steps to address ACE human capital needs. These include, for example, * Using various methods to fill staff vacancies. As of May 2007, the program reports that it has 62 full-time employees and one vacancy and that additional staff are working on ACE either full- or part-time. * Lobbying OPM for more flexibility in recruiting, to include higher salaries and direct hiring authority. • Reorganizing OIT into six program offices aligned to major mission areas so that the number of government personnel responsible for ACE development activities can be augmented by IT functional program management expertise. * Offering training to improve the skills of staff. Nevertheless, these steps have not been guided by a well-defined plan and thus represent activities that are not tied to strategic goals and outcomes and thus cannot be measured against them. Without a plan, it is unlikely that the program will be able to adequately ensure that it has the right people at the right time to deliver ACE successfully. Objective 2: Open Recommendations: Quarterly Reports: Open recommendation 13: Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability. Status: In progress: The June 30, 2006 quarterly report to the House and Senate Appropriations Committees included the program’s strategy for meeting its human capital needs and its accountability framework. Human Capital Strategy (In Progress): The June 30, 2006, quarterly report to Congress included a description of the previously described ACE-specific appendix to the OIT Strategic Human Capital Management Plan, including the plan’s five goals and strategies. Moreover, the report states that this ACE-specific plan is aligned with OPM’s Human Capital Assessment and Accountability Framework.[Footnote 51] As previously stated, however, this ACE- specific plan does not meet the basic tenets of strategic human capital management, as defined in this framework and other relevant guidance. Objective 2: Open Recommendations: Quarterly Reports: According to program officials, they are developing a new OIT human capital strategy and implementation plan, but they have yet to provide us with a date when it will be completed. Accountability Framework (Complete): The June 30, 2006, quarterly report included a description of the ACE accountability framework. As previously discussed, this framework is the means by which the program measures performance relative to promised ACE capabilities, costs, schedules, earned value, risks, and mission values and benefits. The report also included an appendix illustrating the format and content of the accountability framework tool. At that time, CBP reported that it would continue to work with stakeholders to further enhance the format, readability, and utility of the framework as a program management and reporting tool. Since June 2006, the program office has refined and implemented the accountability framework. The framework covers all program commitment areas and key system aspects to support executive decision making and provides external program stakeholders with information on the program. Objective 2: Open Recommendations: Quarterly Report on Open Recommendations: Open recommendation 14: Report quarterly to the House and Senate Appropriations Committees on efforts to address open GAO recommendations. Status: Complete: CBP has submitted quarterly reports on ACE to both the House and the Senate Appropriations Subcommittees since November 2002, including reports for each quarter of fiscal year 2006 and the first and second quarters of fiscal year 2007. These reports have addressed CBP’s efforts to address open GAO recommendations. Objective 2: Open Recommendations: Accurately Report on Open Recommendations: Open recommendation 15: Accurately report to the appropriations committees on CBP's progress in implementing our prior recommendations. Status: In progress: As previously stated, CBP has included information on progress in meeting GAO’s recommendations in its quarterly reports[Footnote 52] to the appropriations committees since November 2002. However, some of this information is dated and thus inaccurate due to the time lapse between when the reports are produced and when they are provided to the appropriations committees. Recently, this time lapse has been between about two and seven months. According to program officials, they are exploring ways to accelerate the review process and thereby improve the timeliness and accuracy of their reports to Congress. DHS officials also told us that they have ongoing efforts to improve the review process; however, the process has not improved consistently or significantly. Objective 3: Observations: Requirements Limitations Likely to Cause Major Delays: Observation 1: Redefinition of requirements for several ACE releases is now under way to address limitations in completeness of originally defined requirements, and this redefinition is likely to introduce program schedule delays and cost increases. A key aspect of successful system acquisition programs is having well- defined requirements. Among other things, requirements should be complete, and to accomplish this, best practices advocate engaging all key stakeholders in the requirements definition and management process. In defining ACE requirements, the program office discovered that its original requirements definition approach did not adequately engage all key stakeholders, and to its credit, has since taken steps to address this. Specifically, * In the spring of 2004, the program office and the ACE support team conducted more than 300 business process workshops with the ACE user community to help define ACE requirements for the future releases. This requirements definition approach was referred to as the Global Business Blueprint (GBB) effort. Objective 3: Observations: Requirements Limitations Likely to Cause Major Delays: * In June and July of 2005, a key group of ACE stakeholders that were not originally involved in the GBB (application programmers familiar with the legacy system that ACE is to replace) raised questions about the completeness of the requirements. To address these questions, the program office examined the GBB’s completeness by first reverse- engineering the legacy software for a small number of legacy system programs and then comparing the reverse-engineered requirements to the GBB-derived requirements. Based on this, which is referred to as legacy code decomposition, the program office found that the GBB-derived requirements were missing about 20 percent of needed ACE functionality. * In August 2005, the program office decided that it needed to decompose all of the legacy system code in order to completely capture the requirements for all ACE releases. Work began with decomposition of the legacy code related to ACE Release 5/Drop A2 (Entry Summary Accounts and Revenue). * In September 2006, the program office determined that the legacy code decomposition approach alone was not sufficient to gain a full understanding of the requirements given the size and complexity of A2. As a result, the requirements definition process was expanded to engage another key stakeholder group (business process subject matter experts). Under the expanded approach, referred to as legacy code decomposition and collaboration, legacy system software programmers and subject matter experts were to examine the code line by line in defining ACE requirements. * In November 2006, the legacy system code decomposition and collaboration effort for Release 5/Drop A2 fell behind schedule because of lack of personnel with legacy system expertise and experience. The schedule for A2 was tentatively revised, but at the time of our review, the schedule was still under review, had not yet been approved, and thus was not yet available for our review. The program office has identified the A2 legacy code decomposition and collaboration process as a high risk item that will significantly impact the A2 schedule. For example, the decomposition and collaboration process for part of the functionality on A2’s critical path—the Authorized Broker Interface Entry Summary—is not expected to be completed before early 2008, at the earliest. According to program officials, delays for other releases/drops, such as Release 5/Drop A1 and Release 6/Drop M1, will not be as significant. They also said that, while they have not yet estimated how long A2 will be delayed and what the associated cost implications are, they do not expect the cost increases to breach the current acquisition program baseline of $3.3 billion, which translates into a cost increase of less than $200 million. Moreover, they said that several actions have been taken to minimize the impact of the delays. For example, * A2 functionality necessary for M1 has been given priority in order to support M1 deployment as originally planned and: * A2 scope is being divided into increments to allow some functionality to be delivered sooner and to minimize the impact on other drops. However, these actions carry consequences, such as missed opportunities to combine field training and an increase in the number of legacy interfaces, thus increasing the potential for introducing software problems. In addition, program officials told us that they are considering changes to the A2 and M1 deployment strategies to address stakeholder concerns, and they said that these changes could also minimize the magnitude of the A2 and M1 delays. Specifically, they said that they had planned to deploy on a national basis, meaning that A2 and M1 functionality would be deployed to all ports at the same time and concurrently adopted by all users nationwide. However, deployment may change to a filer-by-filer basis, meaning that A2 and M1 functionality would be deployed to all ports at the same time, but not all filers would begin using it at the same time. Objective 3: Observations: Requirements Limitations Likely to Cause Delays: According to program officials, they believe that the change in the deployment strategy would both address stakeholder concerns and minimize any A2 and M1 schedule delays caused by the redefinition of requirements. However, they added that these changes have yet to be approved, and the full extent of the cost and schedule implications are not yet known. Moreover, neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports have disclosed the A2 and M1 requirements redefinition and its impact, and neither has addressed any changes to their deployment strategies. Objective 3: Observations: Key COTS Product Being Replaced: Observation 2: Significant changes to ACE requirements have, in turn, led to reevaluation and replacement of a key commercial off-the-shelf product (COTS) previously selected and being prepared for use. When acquiring commercial component-based systems, like ACE, best practices advocate basing decisions on whether to employ a given COTS product on thorough, rigorous, and continuous analysis of a number of factors, including how well competing products satisfy defined system requirements. To the program office’s credit, it reports having followed the CBP system life cycle methodology to determine which COTS product would best meet the requirements for Release 5/Drop A2. These analyses include: * In 2002, the program office reviewed, in general terms, various COTS packages and determined that a solution using SAP (formerly Systems Application and Products), a COTS provider, combined with other commercial solutions and customized development, provided the best combination of capability, performance, and cost for ACE. * In 2004, a more detailed analysis was conducted as part of the previously mentioned GBB process, which was intended to define and allocate ACE requirements to future ACE releases and provide the basis for, among other things, selecting a specific SAP product. At that time, the SAP Enterprise Portal product was selected for Release 5/Drop A2. * In December 2006, the ACE Chief System Architect recommended that the Internet Transaction Server (ITS) technology already used by CBP should be adopted instead of the SAP Enterprise Portal, based on the determination that all currently planned SAP functionality could be presented using the ITS technology. This decision was based on improved understanding of the requirements and previous analyses of ITS. On the basis of these analyses and the Chief System Architect’s recommendation, the program office subsequently stopped work on SAP Enterprise Portal design and configuration efforts and, in March 2007, the program reported that the SAP Internet Transaction Server would be used for Release 5/Drop A2 instead of the SAP Enterprise Portal. While this decision was expected to have some near-term schedule impact because much of the completed work for A2 had been based on the planned use of SAP Enterprise Portal, the program office reports that these impacts are offset by the cost advantages of other ACE releases already using the Internet Transaction Server technology. However, in March 2007, program officials identified a risk of inadequate response time for the Internet Transaction Server—thus negatively impacting user productivity—and that there was high probability of significant cost and schedule impacts. Actions are underway to mitigate the risk through performance modeling and test planning. Neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress disclose this COTS product change, its impact on release schedules and cost estimates, or risk to future system performance. Objective 3: Observations: Risks Not Being Effectively Managed: Observation 3: All program risks are not being effectively managed. Risk management is a continuous, forward-looking process that is intended to either prevent program cost, schedule, and performance problems from occurring or to minimize the impact if they occur. According to relevant guidance and best practices, effective risk management involves proactively identifying, assessing, and disclosing risks; defining and implementing cost-effective strategies for mitigating these risks; and measuring and disclosing progress in doing so. To its credit, the program office has developed a process guide and implemented an automated tool (database) for managing ACE risks in accordance with relevant guidance and best practices. Among other things, the database contains the description, level (high, medium, or low), and mitigation strategy (including start and end dates, exit criteria, and implementation status) for each risk. However, the completeness and quality of the information on each of the risks in the risk database[Footnote53] vary. For example, * many risks were missing information on the status of efforts to implement the mitigation strategy and: * many risks (18) were missing criteria for completing mitigation steps, clear descriptions of what the risk entailed, and start and end dates for planned mitigation activities. Because of such database limitations, we could not determine the status of and mitigation progress on 17 risks. Moreover, these database limitations were reflected in the documentation used at key program events, such as PMRs. This means that the program does not have the risk-related information that it needs to inform its program decisions and to reduce the chances of potential problems becoming actual problems. Program officials, including the official responsible for risk management, stated that risk management is immature and needs to be strengthened. The reasons they gave for these risk management weaknesses are due to: * all staff not being trained on how to use the tool (last training was provided in 2003, and the since then a number of people have joined the program); * the tool is unique to the ACE program and thus no CBP guidance exists on its use); and: * each ACE group addresses risk differently in its weekly meetings. To improve ACE risk management, program officials told us that they are: * establishing a group to ensure the quality and completeness of the database; * holding regular group meetings with contract staff and team leads to discuss risks and their impacts; and: * conducting risk management training. If implemented effectively, such steps should result in more meaningful information about program risks that can be useful to DHS in managing the program and to Congress in overseeing it. To date, however, ACE program risks have not been communicated to oversight organizations through the 2007 expenditure plan or recent quarterly reports to the House and Senate Appropriations Committees. Conclusions: Over the past 7 years, CBP and DHS have worked to fulfill legislatively mandated annual expenditure plan requirements and to implement dozens of our recommendations related to these plans and management of the program. Among other things, these requirements and recommendations have promoted effective program management and accountability for performance and results. As a result of these years of efforts, the ACE program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appear to be on the horizon. To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep the Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays. Recommendations for Executive Action: To further strengthen ACE management and promote accountability for ACE performance and results, we are making the following recommendation to the Secretary of Homeland Security to direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose: (1)the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all our prior recommendations; (2)the status and impacts on the program’s estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different COTS product than originally selected; and: (3)the program’s plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategies’ implementation. Agency Comments: In oral comments on a draft of this briefing, DHS and CBP officials agreed with our conclusions and recommendations, and provided clarifying information and technical comments that we incorporated in the briefing, as appropriate. Attachment 1: Scope and Methodology: To accomplish our objectives, we analyzed the ACE fiscal year 2007 expenditure plan and supporting documentation, and compared them to relevant federal requirements and guidance, applicable best practices, and our prior recommendations. We also interviewed DHS and CBP officials and ACE program contractors. In particular, we reviewed * DHS and CBP investment management practices, using OMB A-11, part 7; * DHS and CBP certification activities for ensuring ACE compliance with the DHS enterprise architecture; * DHS and CBP acquisition management efforts, using SEI’s SA-CMM; * CBP cost estimating program and cost estimates, using SEI’s institutional and project-specific estimating guidelines;[Footnote 54] * independent verification and validation (IV&V) activities using the Institute of Electrical and Electronics Engineers standard for software verification and validation;[Footnote 55] * CBP actions to coordinate ACE with related programs; * CBP’s reorganization documentation, including the organizational charts and roles and responsibilities matrix; * ACE’s accountability framework; and: * cost and schedule data and program commitments from program management documentation. For DHS-, CBP-, and contractor-provided data that we did not substantiate, we have made appropriate attribution indicating the data's source. We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from December 2006 through July 2007 in accordance with generally accepted government auditing standards. Attachment 2: Related GAO Products: ACE Expenditure Plans: Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but IT Faces Long-Standing Management Challenges and New Risks. GAO-06-580. Washington, D.C.: May 31, 2006. Information Technology: Customs Automated Commercial Environment Program Progressing, but Need for Management Improvements Continues. GAO-05-267. Washington, D.C.: March 14, 2005. Information Technology: Early Releases of Customs Trade System Operating, but Pattern of Cost and Schedule Problems Needs to Be Addressed. GAO-04-719. Washington, D.C.: May 14, 2004. Customs Service Modernization: Automated Commercial Environment Progressing, but Further Acquisition Management Improvements Needed. GAO-03-406. Washington, D.C.: February 28, 2003. Customs Service Modernization: Third Expenditure Plan Meets Legislative Conditions, but Cost Estimating Improvements Needed. GAO-02-908. Washington, D.C.: August 9, 2002. [End of section] Appendix II: Comments from the Department of Homeland Security: U.S. Department of Homeland Security: Washington, DC 20528: [hyperlink, http://www.dhs.gov] October 10, 2007: Mr. Randolph C. Hite: Director, Information Technology Architecture and Systems Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Hite: Re: Draft Report GAO-08-46, Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls (GAO Job Code 310634) The Department of Homeland Security (DHS) appreciates the opportunity to review and comment on the draft report referenced above. Consistent with the Fiscal Year 2007 DHS appropriations act, DHS is to develop and submit an expenditure plan for the Automated Commercial Environment (ACE) that satisfies certain conditions, including being reviewed by the U.S. Government Accountability Office (GAO). The report correctly notes that the expenditure plan satisfies many of the legislative conditions specified in the act and that the Department continues to make progress. GAO recognizes that we have implemented eight GAO recommendations made during the last four years and that seven other recommendations made during that time are in the process of being implemented. Department and U.S. Customs and Border Protection (CBP) officials are committed to fully addressing these remaining open recommendations regarding the ACE accountability framework; measures and associated program management metrics for determining success of improvements; minimizing the degree of overlap and concurrency across ACE releases and mitigating associated risks; meeting the legislative conditions regarding a privacy impact assessment and ensuring architectural alignment; implementing key human capital management practices; managing human capital needs; and reporting progress implementing GAO recommendations to Congress quarterly. GAO recommends three actions involving program information disclosure to the House and Senate Appropriations Committees designed to further strengthen ACE management and accountability for ACE performance and results. U.S. Customs and Border Protection officials agree with the recommendations. The new recommendations and CBP's planned corrective actions follow. Recommendation 1: [Future quarterly reports disclose] the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all prior GAO recommendations. Response: GAO concluded that CBP satisfied four of six legislative conditions but only partially satisfied the following two conditions: (1) Meet the capital planning and investment control review requirements established by the Office of Management and Budget (OMB), including OMB Circular A-11, Preparation, Submission, and Execution of the Budget, Part 7 [Planning, Budgeting, Acquisition and Management of Capital Assets]. The risk of not completely satisfying this requirement as part of the expenditure plan is minimal as CBP does meet OMB's requirements, but must better demonstrate compliance in future expenditure plan submissions. Full alignment to OMB's investment management requirements can be demonstrated via the Office of Management and Budget Exhibit 300 form [ACE Exhibit 300] that will be attached to all future ACE Expenditure Plan submissions. (2) Comply with the DHS enterprise architecture. DHS officials determined in May 2007 that all required ACE products and technologies were aligned with the DHS technical reference model and that ACE was thus aligned with the DHS enterprise architecture. CBP agrees with GAO that a compliance determination is not a one- time event but a series of determinations that occurs throughout an investment's life cycle. CBP will continue to address DHS architectural requirements as DHS evolves its alignment methodology and criteria to meet GAO expectations. Recommendation 2: [Future quarterly reports disclose] the status and impacts on the program's estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different commercial off-the-shelf product than originally selected. Response: ACE program officials intend to disclose this information in future quarterly reports. Recommendation 3: Future quarterly reports disclose] the program's plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategies' implementation. Response: Officials responsible for ACE plan to strengthen risk management by: (1) Undertaking a new round of risk identification efforts to include the appropriate contractor technical and management personnel as well as government personnel. The first session was held on September 14, 2007. After these identification sessions, the list of risks will be analyzed and prioritized. A response strategy will be developed for each risk. (2) The risks will be categorized and linked to the 19 risk categories in the OMB 300 document for information technology programs. This effort will ensure that the critical areas of risk are covered. (3) Reports from the risk management software will be made available to the appropriate oversight personnel. The reports will include the mitigation strategy and status of its implementation. The foregoing actions should enable CBP to ensure that future Congressional reports provide complete and current information regarding program risks and attendant mitigation strategies, as well as the status of efforts to mitigate these risks. Sincerely, Signed by: Steven J. Pecinovsky: Director: Departmental GAO/OIG Liaison Office: [End of section] Appendix III: Contact and Staff Acknowledgments: GAO Contact: Randolph C. Hite, (202) 512-3459: Staff Acknowledgments: In addition to the person named above, Daniel Castro, Dawn Day, Neil Doherty, Nancy Glover, Paula Moore, Jamelyn Payan, Nik Rapelje, and Karen Talley made key contributions to this report. [End of section] Footnotes: [1] Pub. L. No. 109-295 (Oct. 4, 2006). [2] Two related open recommendations have been combined. [3] We did not evaluate the compliance of the ACE program with respect to the Federal Acquisition Regulation or OMB directives, but we did evaluate its compliance with established best practices models that incorporate IT investment requirements for federal agencies. [4] This recommendation has multiple conditions, of which three are completed and two are in progress. A sixth condition of this recommendation--that the ACE accountability framework clearly and unambiguously delineate roles and responsibilities of the government and the prime contractor--was previously completed. See Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but It Faces Long-Standing Management Challenges and New Risks, GAO-06-580 (Washington, D.C.: May 31, 2006). [5] 44 U.S.C. § 3501 note. [6] The U.S. Customs and Border Protection (CBP), formerly the Bureau of Customs and Border Protection, was formed in 2003 under the new Department of Homeland Security from the former U.S. Customs Service and other entities with border protection responsibilities. [7] Pub. L. No. 109-295, (October 4, 2006). [8] OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and management of federal capital assets. [9] The purpose of the IRB is to integrate capital planning and investment control, budgeting, acquisition, and management of investments. It is also to ensure that spending on investments directly supports and furthers the mission and that this spending provides optimal benefits and capabilities to stakeholders and customers. [10] Two related open recommendations have been combined. [11] 19 U.S.C. Section 1411. [12] This partnership was formerly known as the e-Customs partnership. It includes Lockheed Martin, Bearing Point, Sandler and Travis, Computer Sciences Corporation, and a number of small businesses. [13] CBP national account managers work with the largest importers to ensure their compliance with trade laws. [14] Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a bond; carriers are individuals or organizations engaged in transporting goods for hire. [15] Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane. [16] Systems Applications and Products (SAP) is a commercial enterprise resource planning software that has multiple modules, each performing separate but integrated business functions. ACE will use SAP to support many of its business processes and functions. In addition, the CBP’s Modernization Office is using SAP as part of a joint project with its Office of Finance to support financial management, procurement, property management, cost accounting, and general ledger processes. [17] The multimodal manifest involves the processing and tracking of cargo as it transfers between different modes of transportation, for example, cargo arrives by ship, is transferred to a truck, and then is loaded onto an airplane. [18] An import activity summary statement is a summary of an importer’s shipment activities over a specific period of time that is transmitted electronically to CBP on a periodic basis by importers and brokers. [19] These accounts include importers, brokers, carriers, authorized service providers, and other members of the trade community who utilize ACE. [20] OMB Circular A-11, part 7 establishes policy for planning, budgeting, acquisition, and management of federal capital assets. [21] EVM is a management tool for measuring progress and is both an industry accepted practice and an OMB requirement. [22] Guide for the Security Certification and Accreditation of Federal Information Systems, NIST Special Publication 800-37. NIST, Gaithersburg, MD, May 2004. [23] A technical reference model is list of approved IT industry standards, hardware, and software products that ensures that IT solutions developed by individual programs (such as ACE) are consistent within DHS as a whole. [24] For this condition, we did not evaluate the compliance of the ACE program with respect to the Federal Acquisition Regulation, OMB directives, or other governmentwide requirements. [25] See, for example, the Clinger-Cohen Act of 1996 (P.L. No. 104-106, 40 U.S.C. §§11101 through §§11704) and OMB Circular A-130. [26] GAO-06-580; GAO-05-267. As with our prior reviews of ACE expenditure plans, the evaluation does not include compliance with federal acquisition regulations or other federal rules and requirements beyond those encompassed by SEI’s Capability Maturity Models. [27] The SA-CMM® is consistent with the acquisition guidelines and systems acquisition management practices of the federal government, and provides a management framework that defines acquisition practices for such process areas as acquisition planning, solicitation, requirements development and management, project management, contract tracking and oversight, and evaluation. [28] ACE’s level 2 rating indicated that CBP had instituted basic acquisition management processes and practices consistent with the acquisition guidelines and management practices of the federal government in the following areas: acquisition planning, solicitation, requirements development and management, project management, contract tracking and oversight, and evaluation. [29] Level 3 capability is more advanced than level 2 and indicates that acquisition management processes have been defined throughout the organization. [30] The program office refers to these plans, processes, and procedures as assets. [31] GAO-06-580. [32] GAO-06-580. [33] Quarterly reports should be received by Congress approximately 60 days after the end of the quarter. [34] GAO-06-580. [35] GAO-06-580. [36] This is a combination of two prior recommendations. From GAO-05- 267, "Define and implement an ACE accountability framework that …ensures use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects” and from GAO-06-580, “Document key milestone decisions in a way that reflects the risks associated with proceeding with unresolved severe defects and provides for mitigating these risks." [37] GAO-06-580. [38] GAO-06-580. [39] GAO-04-719. [40] GAO-06-580. [41] GAO-06-580. [42] See, for example, models developed by the Carnegie Mellon University SEI, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95- SR-005 (Pittsburgh, Pa.: Carnegie Mellon University, 1995) and A Manager's Checklist for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa.: Carnegie Mellon University,1995). [43] GAO-04-719, GAO-05-267, GAO-06-580. [44] GAO-06-580. [45] CBP applications include IT systems that provide tools and information to help front-line officers ensure the security of our nation. This includes applications for the United States Visitor and Immigrant Status Indicator Technology (US-VISIT), Container Security Initiative and Automated Targeting System. [46] GAO-06-580. [47] 44 U.S.C. § 3501 note. [48] GAO-06-580. [49] OMB, Guidance for Implementing the Privacy Provisions of the EGovernment Act of 2002, OMB M-03-22 (Sept. 26, 2003). [50] GAO-06-580. [51] As revised by OPM in 2005, this framework reflects guidance from the collaboration of OMB, OPM, and GAO. [52] The reports are due to Congress approximately 60 days after the completion of a fiscal quarter. ACE quarterly reports are due to DHS approximately 30 days after the end of the quarter, and then DHS has 30 days to review the report and submit it to Congress. [53] As of May 23, 2007 the risk database contained 46 risks. [54] SEI's Institutional and project-specific estimating guidelines are defined in Robert E. Park, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University Software Engineering Institute, 1995) and A Manager's Checklist for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa: 1995), respectively. [55] Institute of Electrical and Electronics Engineers (IEEE) Computer Society, Standard for Software Verification and Validation 1012-1998 (June 8, 2005). GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "Subscribe to Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Gloria Jarmon, Managing Director, JarmonG@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: