This is the accessible text file for GAO report number GAO-08-1020 entitled 'Information Technology: SSA Has Taken Key Steps for Managing Its Investments, but Needs to Strengthen Oversight and Fully Define Policies and Procedures' which was released on October 14, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Ranking Member, Committee on Finance, U.S. Senate: United States Government Accountability Office: GAO: September 2008: Information Technology: SSA Has Taken Key Steps for Managing Its Investments, but Needs to Strengthen Oversight and Fully Define Policies and Procedures: GAO-08-1020: GAO Highlights: Highlights of GAO-08-1020, a report to the Ranking Member, Committee on Finance, U.S. Senate. Why GAO Did This Study: The Social Security Administration (SSA) spends about $1 billion annually to support its information technology (IT) needs. Given the size and significance of the agency's ongoing and future investments in IT, it is crucial that the agency manages these investments wisely. Accordingly, GAO was requested to determine whether SSA's investment management approach is consistent with leading investment management best practices. To accomplish this, GAO used its IT investment management framework and associated methodology, with a focus on the framework’s Stages 2 and 3, which are based on the investment management provisions of the Clinger-Cohen Act of 1996. What GAO Found: SSA’s investment management approach is largely consistent with leading investment management practices. It has established most of the practices needed to manage its projects as investments and is making progress towards managing IT investments as a portfolio; however, it is not applying its investment management process to all of its investments. Specifically: * The agency is executing a majority of the key practices needed to build the foundation for managing its IT projects as investments. Of the 5 processes and their 38 associated key practices, SSA is executing 31 practices. (See table below.) However, the agency’s investment board, which should provide executive oversight of investments, is not adequately monitoring the performance of IT projects. * SSA has made progress in establishing the key practices for managing investments as a portfolio—it is executing 18 out of 27 key practices. The agency has made important progress in defining and creating the investment portfolio, but it has not developed enterprisewide portfolio selection criteria. The agency also has not established procedures for evaluating the portfolio, and its postimplementation reviews do not determine whether projects meet the agency’s strategic goals. * SSA is not applying its investment management process to a major portion of its IT budget. Specifically, IT products and services acquired with its acquisition budget ($610 million of the $1 billion IT budget for fiscal year 2008) are not managed by the board as investments. SSA’s executive-level review board is not responsible for overseeing the acquisition budget. Consequently, executive management has limited insight into investments acquired with these funds, and the agency has limited ability to ensure that the budget is spent in the most efficient and effective manner. Until it establishes oversight of all investments and fully defines policies and procedures for overseeing both individual projects and an agencywide portfolio, SSA risks not being able to select and control these investments consistently and completely, thus increasing the chance that investments will not meet mission needs in the most cost- effective and efficient manner. Table: Social Security Administration’s IT Investment Management Capabilities: Stage 2: Building the investment foundation: Instituting the investment board; Key practices executed (percentage): 7/8 (88); Stage 3: Developing a complete investment portfolio: Defining the portfolio criteria; Key practices executed (percentage): 5/7 (71). Stage 2: Building the investment foundation: Meeting business needs; Key practices executed (percentage): 7/7 (100); Stage 3: Developing a complete investment portfolio: Creating the portfolio; Key practices executed (percentage): 7/7 (100). Stage 2: Building the investment foundation: Selecting an investment; Key practices executed (percentage): 9/10 (90); Stage 3: Developing a complete investment portfolio: Evaluating the portfolio; Key practices executed (percentage): 2/7 (29). Stage 2: Building the investment foundation: Providing investment oversight; Key practices executed (percentage): 2/7 (29); Stage 3: Developing a complete investment portfolio: Conducting postimplementation reviews; Key practices executed (percentage): 4/6 (67). Stage 2: Building the investment foundation: Capturing investment information; Key practices executed (percentage): 6/6 (100); Stage 3: Developing a complete investment portfolio: [Empty]; Key practices executed (percentage): [Empty]. Stage 2: Building the investment foundation: Overall; Key practices executed (percentage): 31/38 (82); Stage 3: Developing a complete investment portfolio: [Empty]; Key practices executed (percentage): 18/27 (67). Source: GAO analysis of SSA data. [End of table] What GAO Recommends: GAO is making recommendations to the Commissioner of Social Security related to strengthening the investment board’s role and responsibilities, improving project oversight for all major investments, defining project-level and portfolio-level policies and procedures for effective investment management, and improving postimplementation reviews. In commenting on a draft of this report, SSA agreed with most of GAO’s recommendations and identified actions initiated or planned to address them. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-1020]. For more information, contact Valerie Melvin, 202-512-6304, melvinv@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: SSA Has Taken Key Steps to Manage Investments, but Gaps Remain in Oversight and in Defining Policies and Procedures: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objective, Scope, and Methodology: Appendix II: Comments from Social Security Administration: Appendix III: GAO Contact and Staff Acknowledgments: Tables: Table 1: Key Participants and Roles and Responsibilities in SSA's Investment Management: Table 2: Stage 2 Critical Processes--Building the Investment Foundation: Table 3: Summary of Results for Stage 2 Critical Processes and Key Practices: Table 4: Instituting the Investment Board: Table 5: Meeting Business Needs: Table 6: Selecting Investments: Table 7: Providing Investment Oversight: Table 8: Capturing Investment Information: Table 9: Stage 3 Critical Processes---Developing a Complete Investment Portfolio: Table 10: Summary of Results for Stage 3 Critical Processes and Key Practices: Table 11: Defining the Portfolio Criteria: Table 12: Creating the Portfolio: Table 13: Evaluating the Portfolio: Table 14: Conducting Postimplementation Reviews: Table 15: Stages 4 and 5--Critical Processes Required for Improving the Investment Process and Leveraging IT for Strategic Outcomes: Figures: Figure 1: Organization of the Social Security Administration: Figure 2: The Five ITIM Stages of Maturity with Critical Processes: Figure 3: SSA's CPIC Process: Abbreviations: CIO: Chief Information Officer: CPIC: Capital Planning and Investment Control: IT: information technology: ITAB: Information Technology Advisory Board: ITIM: information technology investment management: OMB: Office of Management and Budget: SSA: Social Security Administration: [End of section] United States Government Accountability Office: Washington, DC 20548: September 12, 2008: The Honorable Charles E. Grassley: Ranking Member: Committee on Finance: United States Senate: Dear Senator Grassley: The Social Security Administration (SSA) manages and funds a variety of information technology (IT) initiatives ranging from those supporting the processing and payment of disability and supplemental security income benefits to those that facilitate the calculation and withholding of Medicare premiums. For fiscal year 2008, SSA plans to spend about $1 billion to support its IT needs. Given the size and significance of its ongoing and future investments in information technology, it is crucial that the agency manages these investments wisely. At your request, we conducted an evaluation to determine whether SSA's investment management approach is consistent with leading investment management best practices. These practices are identified in our IT Investment Management (ITIM) framework[Footnote 1] by which we evaluate the maturity of an agency's investment management processes focusing on the framework's Stages 2 and 3, based on the investment management provisions of the Clinger-Cohen Act of l996.[Footnote 2] To accomplish our objective, we analyzed SSA's self-assessment and supporting documents to determine whether the agency has developed the structures, policies, and procedures associated with executing those key practices in the ITIM framework. We also interviewed relevant agency officials about investment management practices. We selected three projects as case studies to determine if certain critical processes and key practices were applied. We conducted this performance audit from October 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. For more details on our objective, scope, and methodology, see appendix I. Results in Brief: SSA has established most--82 percent--of the basic practices needed to manage its projects as investments, including many of the foundational practices for selecting and controlling IT investments. The agency has also made progress in establishing the practices for managing IT investments as a portfolio, such as defining the portfolio criteria and creating the portfolio. Even with these capabilities, weaknesses remain in several areas. * The agency has implemented 31 of 38 key practices for managing projects as investments. The agency has established most of the key practices for instituting an investment board to manage its investments, and has implemented most of the practices for ensuring that investments meet business needs and for selecting investments. Also, the agency has established automated tools for capturing investment information about its projects. However, the agency has not fully established policies and procedures to guide investment management. For example, the agency has not established policies and procedures for the investment board and for prioritizing new investments. In addition, only 2 of 7 practices for providing investment oversight have been implemented. The agency has not fully developed policies and procedures for management oversight of IT projects and systems, such as elevating problems to the investment board. The agency also does not track corrective actions for underperforming investments and report them to the investment board. SSA officials said that aspects of its investment management approach, such as providing oversight of investments, do not always follow the key practices of our ITIM framework because SSA delegates decision rights to different executives and staff in the organization. Until SSA fully implements the basic foundational steps for managing projects, it cannot provide full assurance that the projects will meet organizational needs and be completed on time and within budget. * The agency has made progress in establishing the key practices for managing IT investments as a portfolio; it is executing 18 out of 27 key practices in this stage of the ITIM. Specifically, SSA has implemented most of the key practices for defining the investment portfolio, creating the portfolio, and conducting postimplementation reviews. However, the agency has not implemented all the policies and procedures for the key practices in this stage. For example, SSA lacks policies and procedures and other key practices for evaluating the portfolio to improve performance. In addition, although the agency is conducting postimplementation reviews of its investments, it does not evaluate quantitative data, limiting its ability to determine whether investments meet benefit expectations. * At the same time, SSA is not applying its investment management process to a major portion of its IT budget. Specifically, the budget portion allocated to its IT acquisitions--totaling about $610 million for fiscal year 2008--is not subject to the agency's investment management structures, policies, and procedures. This funding, used by the agency for acquisitions of IT-related products and services, is not allocated or overseen by the investment board and is not managed by established procedures, such as the ITIM management select and control process. Rather, this funding is managed through a deputy commissioner's office that is responsible for processing funding requests from the business units and handling subsequent negotiations. Consequently, SSA's executive management tasked with overseeing the agency's investments is not responsible for ensuring that this portion of the budget is spent in the most efficient and effective manner. Further, in the absence of such oversight, the agency is not positioned to ensure that its IT budget is being expended most effectively and that its IT investments best meet the organization's needs and objectives. To further strengthen SSA's investment management capability, we are recommending that the agency establish oversight of all investments and fully define investment management policies and procedures for both individual projects and the agencywide portfolio. Until it establishes oversight and defines policies and procedures, it risks not being able to select and control these investments in a way that is consistent and complete, which in turn increases the chances that these investments will not meet mission needs in the most cost-effective and efficient manner. The Commissioner of Social Security provided written comments on a draft of this report (reproduced in app. II). In the comments, SSA agreed with six of our seven recommendations and identified actions initiated or planned to address them. The agency disagreed with our recommendation that it develop policies and procedures for managing its IT acquisitions as investments and manage them using the investment board and investment management processes. The agency believed its budget development process already treats IT acquisitions as investments and maintains them under an investment management framework, though not one described by GAO's ITIM framework. However, under SSA's current process, these acquisitions are not subject to the agency's investment management select, control, and evaluate processes and are not managed by its investment board. By not applying the investment management processes to the acquisition budget, SSA limits the ability of its executive management tasked with overseeing the agency's investments to ensure that this portion of the budget is spent in the most efficient and effective manner. SSA also provided technical comments on a draft of this report, which we have incorporated where appropriate. Background: SSA's mission is to advance the nation's economic security through compassionate and vigilant leadership in shaping and managing America's Social Security programs. This includes one of the nation's largest entitlement programs--federal Old-Age, Survivors, and Disability Insurance benefits--commonly referred to as Social Security. The program provides monthly benefits to retired and disabled workers, their spouses and children, and the survivors of insured workers. SSA also administers Supplemental Security Income, a needs-based program for the aged, blind, and disabled that pays monthly benefits to individuals. Over 54 million people, one-sixth of the total U.S. population, receive monthly Social Security or Supplemental Security Income benefit payments. The agency's estimated 2008 budget of about $657 billion includes an administrative budget of $9.7 billion to support these programs, including about $1 billion for IT. Organizationally, SSA is headed by the Commissioner, who is assisted by a deputy commissioner and various other executive officials, including the Deputy Commissioner, Budget, Finance and Management; Chief Information Officer (CIO); Chief Strategic Officer; and nine deputy commissioners responsible for the agency's various business components. The organizational structure of the agency is depicted in figure 1. Figure 1: Organization of the Social Security Administration: [See PDF for image] This figure is an organizational chart of the Social Security Administration: Top level: Commissioner of Social Security; Deputy Commissioner; Chief of Staff; * Executive Secretary; * Office of Regulations; - Office of International Programs. Second level, all associated directly with top level: * Office of Chief Actuary; * Office of the General Counsel; * Office of the Inspector General; * Office of the Chief Information Officer. Third level, all associated with top level through second level: * Deputy Commissioner, Communications; * Deputy Commissioner, Human Resources; * Deputy Commissioner, Legislation and Congressional Affairs; * Deputy Commissioner, Retirement and Disability. Fourth level, all associated with top level through third level: * Deputy Commissioner, Operations; * Deputy Commissioner, Budget, Finance and Management; * Deputy Commissioner, Systems; * Deputy Commissioner, Disability Adjudication and Review; * Deputy Commissioner, Quality Performance. Source: Social Security Administration. [End of figure] The Commissioner is supported by approximately 60,000 employees located at headquarters and throughout a decentralized network of over 1,400 offices that include regional offices, field offices, teleservice centers, processing centers, state Disability Determination Services, program service centers, and hearing offices. Of these employees, approximately 3,300 IT staff and contractors are assigned to the Office of Deputy Commissioner, Systems. According to SSA, its organizational structure is designed to provide timely, accurate, and responsive service to the American public. SSA Relies on IT to Deliver Services: The agency relies extensively on information technology to administer its programs and to support related administrative needs. In this regard, IT is used to, among other things: * evaluate evidence and make determinations of eligibility for benefits on new claims, * pay monthly benefits, * issue new and replacement Social Security cards, * process earnings items for crediting to workers' earnings records, * handle millions of transactions on SSA's toll-free telephone number, * issue Social Security statements, * process continuing disability reviews, and: * process nondisability Supplemental Security Income redeterminations. The agency's IT budget for fiscal year 2008 is approximately $1 billion. Of this amount, $400 million is for work year[Footnote 3] support of software development projects in the Office of Deputy Commissioner, Systems and about $610 million is for acquisition of IT- related products and services.[Footnote 4] The agency expects to spend about 80 percent of its acquisition budget on infrastructure. Investment Management Is Critical to Effective Use of IT: A corporate approach to IT investment management is characteristic of successful public and private organizations. Recognizing this, Congress enacted the Clinger-Cohen Act of 1996,[Footnote 5] which requires the Office of Management and Budget (OMB) to establish processes to analyze, track, and evaluate the risks and results of major capital investments in IT systems made by executive agencies. In implementing the Clinger-Cohen Act and other statutes, OMB has developed policy and issued guidance for the planning, budgeting, acquisition, and management of federal capital assets.[Footnote 6] We have also issued guidance in this area[Footnote 7] that defines institutional structures, such as investment boards; processes for developing information on investments (such as cost/benefit); and practices to inform management decisions (such as whether a given investment is aligned with an enterprise architecture). IT Investment Management: A Brief Description: IT investment management is a process for linking IT investment decisions to an organization's strategic objectives and business plans. Consistent with this, the federal approach to IT investment management focuses on selecting, controlling, and evaluating investments in a manner that minimizes risks while maximizing the return on investment. [Footnote 8] * During the selection phase, the organization (1) identifies and analyzes each project's risks and returns before committing significant funds to any project and (2) selects those IT projects that will best support its mission needs. * During the control phase, the organization ensures that projects, as they develop and investment expenditures continue, meet mission needs at the expected levels of cost and risk. If the project is not meeting expectations or if problems arise, steps are quickly taken to address the deficiencies. * During the evaluation phase, expected results are compared with actual results after a project has been fully implemented. This comparison is done to (1) assess the project's impact on mission performance, (2) identify any changes or modifications to the project that may be needed, and (3) revise the investment management process based on lessons learned. Overview of GAO's ITIM Maturity Framework: Our ITIM framework consists of five progressive stages of maturity for any given agency relative to selecting, controlling, and evaluating its investment management capabilities.[Footnote 9] (See fig. 2 for the five ITIM stages of maturity.) This framework is grounded in our research of IT investment management practices of leading private and public sector organizations. The framework can be used to assess the maturity of an agency's investment management processes and as a tool for organizational improvement. The overriding purpose of the framework is to encourage investment processes that increase business value and mission performance, reduce risk, and increase accountability and transparency in the decision process. We have used the framework in many of our evaluations, [Footnote 10] and a number of agencies have adopted it. ITIM's five maturity stages represent steps toward achieving stable and mature processes for managing IT investments. Each stage builds on the lower stages and the successful attainment of each stage leads to improvement in the organization's ability to manage its investments. With the exception of Stage 1, each maturity stage is composed of "critical processes" that must be implemented and institutionalized in order for the organization to achieve that stage.[Footnote 11] These critical processes are further broken down into key practices that describe the types of activities that an organization should be performing to successfully implement each critical process. It is not unusual for an organization to perform key practices from more than one maturity stage at the same time. However, our research has shown that agency efforts to improve investment management capabilities should focus on implementing all lower stage practices before addressing the higher stage practices. Figure 2 provides an overview of the five ITIM stages of maturity and the critical processes associated with each stage. Figure 2: The Five ITIM Stages of Maturity with Critical Processes: [See PDF for image] This figure depicts the five ITIM stages as building blocks, starting with stage one, as follows: Maturity stage: Stage 1: Creating investment awareness; Critical processes: IT spending without disciplined investment processes. Maturity stage: Stage 2: Building the investment foundation; Critical processes: - Instituting the investment board; - Meeting business needs; - Selecting an investment; - Providing investment oversight; - Capturing investment information. Maturity stage: Stage 3: Developing a complete investment portfolio; Critical processes: - Defining the portfolio criteria; - Creating the portfolio; - Evaluating the portfolio; - Conducting postimplementation reviews. Maturity stage: Stage 4: Improving the investment process; Critical processes: - Improving the portfolio's performance; - Managing the succession of information systems. Maturity stage: Stage 5: Leveraging IT for strategic outcomes; Critical processes: - Optimizing the investment process; - Using IT to drive strategic business change. Source: GAO. [End of figure] In the ITIM framework, Stage 2 critical processes lay the foundation for sound IT investment management by helping the agency to attain successful, predictable, and repeatable investment management processes at the project level. Specifically, Stage 2 encompasses building a sound investment management foundation by establishing basic capabilities for selecting new IT projects. This stage also involves developing the capability to control projects so that they finish predictably within established cost and schedule expectations and developing the capability to identify potential exposures to risk and put in place strategies to mitigate that risk. It also involves instituting an IT investment board,[Footnote 12] which includes defining its membership, guidance policies, operations, roles, responsibilities, and authorities. The basic selection processes established in Stage 2 lay the foundation for more mature management capabilities in Stage 3, which represents a major step forward in maturity, in which the agency moves from project-centric processes to an agencywide portfolio approach. Stage 3 requires that an organization continually assess both proposed and ongoing projects as part of a complete investment portfolio--an integrated and competing set of investment options. It focuses on establishing a consistent, well-defined perspective on the IT investment portfolio and maintaining mature, integrated selection (and reselection), control, and evaluation processes. This portfolio perspective allows decision makers to consider the interaction among investments and the contributions to organizational mission goals and strategies that could be made by alternative portfolio selections, rather than focusing exclusively on the balance between the costs and benefits of individual investments. Organizations that have implemented Stage 2 and 3 practices have capabilities in place that assist in establishing selection; control; and evaluation structures, policies, procedures, and practices that are required by the investment management provisions of the Clinger-Cohen Act.[Footnote 13] Stages 4 and 5 require the use of evaluation techniques to continuously improve both the investment portfolio and the investment processes in order to better achieve strategic outcomes. At Stage 4, an organization has the capacity to conduct IT succession activities and, therefore, can plan and implement the deselection of obsolete, high-risk, or low- value IT investments. An organization with Stage 5 maturity conducts proactive monitoring for breakthrough information technologies that will enable it to change and improve its business performance. SSA's Current Investment Management Approach: SSA's investment management process is intended to meet the objectives of the Clinger-Cohen Act by providing a framework for selecting, controlling, and evaluating investments that helps to ensure it meets the strategic and business objectives of the agency. The investment management process is documented in the agency's Capital Planning and Investment Control (CPIC) Guide. The CPIC Guide assigns the responsibility for the investment management process to SSA executive-level managers. In this regard, the Information Technology Advisory Board (ITAB) is responsible for assigning resources to projects reported in the 2-year Agency IT Plan, which specifies which projects and systems the agency will build and operate. The board, which meets quarterly, is comprised of the deputy commissioners and other senior executives, such as the general counsel and the Deputy Commissioner, Budget, Finance and Management and it is chaired by the CIO. The CIO is the key decision maker in the CPIC process. He provides advice to the Commissioner and Deputy Commissioner of Social Security to ensure that IT is acquired and information resources are managed in a manner that is consistent with the policies and procedures of the Clinger-Cohen Act. The CIO is the chairman of the investment board and makes final IT budget recommendations to the Commissioner. The Deputy Commissioner, Systems is responsible for monitoring all development and operations projects included in the Agency IT Plan. Each deputy commissioner responsible for a portfolio has a portfolio manager and portfolio team to assist in the day-to-day management of the corresponding investment portfolio within each business component. Table 1 identifies the key participants that have a role in the agency's investment management process and their responsibilities. Table 1: Key Participants and Roles and Responsibilities in SSA's Investment Management: Key participants: Chief Information Officer (CIO); Membership/description: Heads the Office of the CIO; Examples of responsibilities: * Ensures that IT is acquired in accordance with CPIC procedures; * Chairs the investment board; * Reviews and approves the annual IT budget. Key participants: Deputy Commissioner, Systems; Membership/description: Heads the Office of Systems which employs approximately 3,300 staff who develop systems; Examples of responsibilities: * Oversees systems development and operations. Key participants: Deputy Commissioners and other top-level executives; Membership/description: Heads of organizational units responsible for business areas and corresponding portfolios; Examples of responsibilities: * Achieves portfolio objectives that correspond to the agency's strategic goals. Key participants: Information Technology Advisory Board (ITAB); Membership/description: CIO is the Chairman and members are deputy commissioner-level executives responsible for the business units; Examples of responsibilities: * Provides guidance on resources for each portfolio; * Approves the Agency IT Plan; * Oversees performance of IT projects. Key participants: Deputy Commissioner, Systems Planning Staff; Membership/description: Deputy Commissioner, Systems staff responsible for providing ITAB with investment information; Examples of responsibilities: * Publishes ITAB and portfolio material; * Schedules ITAB and cross-portfolio meetings. Key participants: Sponsor; Membership/description: Initiates IT proposals for new projects; Examples of responsibilities: * Describes the proposed project and business or user needs. Key participants: Portfolio team; Membership/description: Staff responsible for selecting investments; Examples of responsibilities: * Reviews sponsor proposals and recommends items for review; * Prepares a recommendation for specific IT proposals for the Agency IT Plan. Key participants: Portfolio team support staff; Membership/description: Staff responsible for supporting the portfolio team; Examples of responsibilities: * Arranges meetings and prepares meeting notes; * Completes portfolio team documents. Key participants: Portfolio team manager; Membership/description: Manager responsible for overseeing activities of the portfolio team within each business component; Examples of responsibilities: * Assures that the portfolio develops its internal processes and adheres to agencywide directives. [End of table] Source: GAO analysis of SSA data. SSA uses its established CPIC process to manage the work years associated with its in-house software development projects. (The acquisition budget is managed by a separate process discussed later in this report.) The CPIC process is as follows: * During the investment selection phase, new projects are proposed by a sponsor--either from a business unit for mission-related projects or from the Deputy Commissioner, Systems' organization for supporting acquisitions, such as telephone systems--and are assigned to 1 of 11 portfolios.[Footnote 14] Proposals that identify business needs are developed based on the Commissioner's priorities or gap analyses performed by each portfolio team that identify future business needs. The ITAB issues guidelines to the portfolio teams on the number of work years that each portfolio will have available for projects. In response, each portfolio team develops a prioritized list of proposed and ongoing projects within their work year allocations. Prioritization is based on a vote by portfolio team representatives. According to SSA's documented procedures, prioritization criteria can include relative benefits, costs, and risks. However, portfolio teams have discretion in how they weigh these and any other criteria. Next, the prioritized lists are combined into a proposed Agency IT Plan for approval by the ITAB. The plan is comprised of proposed investments for the next 2 fiscal years, and provides information on work year requirements. In addition, expected benefits and return on investment are included for new development projects. The ITAB approves or modifies the proposed plan once a year, including allocating work years to the portfolios. At this point, the selection phase of the annual cycle is basically complete, though portfolio teams can propose additional projects that arise in the middle of a cycle. During the control phase, the Deputy Commissioner, Systems holds monthly meetings with his staff who are assigned to monitor projects in development. During these meetings, projects that are not meeting cost and schedule expectations are identified, and corrective actions are initiated. According to SSA guidance, the objective of the Deputy Commissioner, Systems' meetings with his staff is to resolve problems related to underperforming projects without elevating them to the ITAB. During the months in which ITAB quarterly meetings are scheduled, the Deputy Commissioner, Systems meets with his staff prior to these meetings to prepare to address concerns about investments that may be raised during the meetings. If concerns are raised at the meeting, the Deputy Commissioner, Systems provides information about these investments. In addition, the ITAB receives investment profiles on the status of each of the agency's major IT investments. These profiles include reports on actual and expended work years, cost, schedule, and any variances. * During the evaluation phase, the CPIC Guide calls for the CIO to conduct postimplementation reviews on projects that have been completed and deployed for at least 3 months. The purpose of these reviews is to compare actual project results against planned results in order to assess performance and identify areas where future decision making can be improved. Figure 3 illustrates SSA's current investment management process as specified in agency guidance. Figure 3: SSA's CPIC Process: [See PDF for image] This figure illustrates the SSA CPIC Process. The process involves three steps: select, control, and evaluate. The process flows as follows: Select: * Agency Mission, Strategic Goals, and Objectives; feed into: * Agency Strategic Plan, supported by: - President's Management Agenda; - Legislation, Court Orders, Audits, feed into: * Performance Goals and Achievement Strategies, supported by: - Enterprise Architecture, (IT Architecture Review Board (ARB)) which is supported by: - IT Technology Advances and Standards; all contribute to: * IRM Strategic Plan, which feeds: * Prioritized Office IT Project Plans, which feed: * Business Case Review Using Defined Criteria: - Strategic Alignment; - Mission Effectiveness; - Organizational Impact; - Risk; - Return on Investment; - Benefit Value Score; all of which lead to: * Prioritized Agency IT Project Portfolio; - CIO/ITAB; - eCPIC; all lead to: * Agency Performance Plan; * Agency IT Budget; - Agency IT Investment Portfolio (Exhibit 53); - Capital Asset Plan and Business Case (Exhibit 300); - Milestone Review Schedule; - Designation of Projects for Post-Implementation Review; both performance plan and budget feed into: * IT Capital Plan. Control: * IT Capital Plan feeds into: * CIO IT Project Milestone; * Systems Development Management; * Quarterly ITAB IT Project Portfolio Review; * CIO IT Budget Execution Oversight. CIO IT Project Milestone and Systems Development Management feed into: * IT Project Implementation; - Proof of Concept; - Prototype; - Pilot; - Development; - Procurement; - IT ARB Review; - Implementation; - VISOR. Evaluate: IT Project Implementation leads to: * Post Implementation Reviews and Reports; - Compare Planned vs. Actual Cost, Schedule and Performance; - Evaluate Issues That Require Attention; - Document Effective Management Practice. Source: Social Security Administration. [End of figure] SSA Has Taken Key Steps to Manage Investments, but Gaps Remain in Oversight and in Defining Policies and Procedures: SSA has executed a majority of the key practices--82 percent--needed to effectively manage its IT projects as investments, but it has not fully implemented many of the related oversight responsibilities and procedures that our ITIM framework outlines. Of the five Stage 2 critical processes specified by the ITIM, it has (1) established most of the key practices needed for instituting the investment board, (2) developed procedures for ensuring that projects meet business and user needs, (3) established a process for selecting an investment, and (4) developed tools for capturing investment information. However, the critical process of providing oversight is not being fully executed. Also, the agency has made progress in establishing the critical processes and key practices for managing IT investments as a portfolio. It is executing 18 out of 27 key practices from this stage of the ITIM. However, it has not established enterprisewide portfolio selection criteria and has executed few key practices for evaluating the portfolio. In addition, its postimplementation reviews are not achieving key objectives. Further, a gap exists in the agency's management of its IT in that more than half of its budget--its acquisition budget--is not overseen as part of the agency's current investment management process. While SSA has taken key steps for managing its investments, until key practices are fully implemented and coverage of its management processes is extended to all investments, it will not be fully postured to ensure that its investments achieve their intended results and address the strategic goals, objectives, and mission of the organization. SSA Has Established Most of the Foundation for Managing IT Investments, but It Has Not Established Some Processes and Procedures: At the ITIM Stage 2 level of maturity, an organization has attained repeatable, successful IT project-level investment control and basic selection processes. Through these processes, the organization can identify expectation gaps early and take the appropriate steps to address them. According to ITIM, critical processes at Stage 2 include (1) defining IT investment board operations, (2) identifying the business needs for each IT investment, (3) developing a basic process for selecting new IT proposals and reselecting ongoing investments, (4) developing project-level investment control processes, and (5) collecting information about existing investments to inform investment management decisions. Table 2 describes the purpose of each of these Stage 2 critical processes. Table 2: Stage 2 Critical Processes--Building the Investment Foundation: Critical process: Instituting the investment board; Purpose: To define and establish an appropriate IT investment management structure and the processes for selecting, controlling, and evaluating IT investments. Critical process: Meeting business needs; Purpose: To ensure that IT projects and systems support the organization's business needs and meet users' needs. Critical process: Selecting an investment; Purpose: To ensure that a well-defined and disciplined process is used to select new IT proposals and reselect ongoing investments. Critical process: Providing investment oversight; Purpose: To review the progress of IT projects and systems, using predefined criteria and checkpoints, in meeting cost, schedule, risk, and benefit expectations and to take corrective action when these expectations are not being met. Critical process: Capturing investment information; Purpose: To make available to decision makers information to evaluate the impacts and opportunities created by proposed (or continuing) IT investments. Source: GAO. [End of table] Within these 5 critical processes are 38 key practices for effective project-level management. SSA has implemented 31 of these practices. Specifically, the agency has satisfied all the key practices associated with meeting business needs and capturing investment information and most of those associated with instituting an investment board and selecting an investment. However, the agency has not executed most of the key practices related to providing investment oversight. Moreover, the agency has not developed some policies and procedures required for the critical process areas, including providing investment oversight. Table 3 summarizes the status of SSA's Stage 2 critical processes, showing the number of associated practices that have been implemented, as they apply to the agency's management of its IT work year budget for in-house projects. Table 3: Summary of Results for Stage 2 Critical Processes and Key Practices: Critical process: Instituting the investment board; Key practices executed: 7; Total required by critical process: 8; Percentage of key practices executed: 88. Critical process: Meeting business needs; Key practices executed: 7; Total required by critical process: 7; Percentage of key practices executed: 100. Critical process: Selecting an investment; Key practices executed: 9; Total required by critical process: 10; Percentage of key practices executed: 90. Critical process: Providing investment oversight; Key practices executed: 2; Total required by critical process: 7; Percentage of key practices executed: 29. Critical process: Capturing investment information; Key practices executed: 6; Total required by critical process: 6; Percentage of key practices executed: 100. Critical process: Total; Key practices executed: 31; Total required by critical process: 38; Percentage of key practices executed: 82. Source: GAO. [End of table] SSA Has Established an IT Management Structure for Its Investments: The establishment of decision-making bodies or boards is a key component of the IT investment management process. At the Stage 2 level of maturity, organizations define one or more boards, provide resources to support their operations, and appoint members who have expertise in both operational and technical aspects of proposed investments. The board operates according to a written IT investment process guide that is tailored to the organization's unique characteristics, thus ensuring that consistent and effective management practices are implemented across the organization. Once board members are selected, the organization ensures that they are knowledgeable about policies and procedures for managing investments. Organizations at the Stage 2 level of maturity also take steps to ensure that executives and line managers support and carry out the decisions of the IT investment board. An IT investment management process guide should be an authoritative document that the organization uses to initiate and manage IT investment processes and should provide a comprehensive foundation for the policies and procedures that are developed for all of the other related processes. (The complete list of key practices is provided in table 4.) SSA has executed seven of the eight key practices for instituting the investment board. In particular, it has established the ITAB as its investment board. As previously discussed, the board is chaired by the CIO, and includes deputy commissioners and other agency senior executives, such as the Deputy Commissioner, Budget, Finance and Management. Further, the agency has a documented investment governance process and provides resources for the board. Management controls have been established for ensuring that the investment board's decisions are carried out. However, the agency is not executing one of the key practices associated with this process. The board is not implementing one of the three stages of the IT investment governance process based on the Clinger-Cohen Act. Specifically, it is not evaluating IT investments, including performing postimplementation reviews. Rather, the CIO alone is assigned this responsibility and the investment board does not receive the results of these reviews. Until all relevant IT governance becomes the responsibility of the ITAB, SSA may have insufficient high- level executive involvement in its investment management process and will not benefit from the contributions of those executives who are in the best position to make the full range of decisions needed for the agency to carry out its mission most effectively. Further, although SSA has established its investment board, the policies and procedures to define and implement the investment governance process are not fully established for all of the key practices. For example, the procedures for elevating underperforming investments to the board are not established. Further, although the CIO and Deputy Commissioner, Systems agree that the CPIC guide and other guidance they provided are official agency documents, these documents had not been officially approved by SSA's management. Without policy guidance that is agreed to and approved by all the appropriate levels of the organization, consistent and repeatable investment management practices cannot be assured. Table 4 summarizes our findings relative to SSA's execution of the eight key practices for instituting the investment board. Table 4: Instituting the Investment Board: Key practice: 1. An enterprisewide IT investment board composed of senior executives from IT and business units is responsible for defining and implementing the organization's IT investment governance process; Rating: Not executed; Summary of evidence: According to SSA's CPIC Guide and IT Planning Training Package, the agency investment management structure includes an investment board (ITAB). The ITAB is responsible for allocating IT staffing resources to the portfolios documented in the Agency IT Plan and overseeing control of IT investments. However, the ITAB is not responsible for evaluating IT investments. The CIO is assigned this responsibility and the board does not receive the results of project evaluations. Key practice: 2. The organization has a documented IT investment process directing each investment board's operations; Rating: Executed; Summary of evidence: The IT Planning Training Package and CPIC Guide outline SSA's IT investment process and direct the operations of the ITAB. The guides specify the roles of key entities involved in the organization's investment management process and explain procedures for assigning responsibility for investment decision making. The guidance assigns the ITAB decision-making authority for the allocation of work years for IT investments. Key practice: 3. Adequate resources, including people, funding, and tools, are provided for supporting the operations of each IT investment board; Rating: Executed; Summary of evidence: According to SSA officials, adequate resources are provided to support the operations of the ITAB. To support the ITAB, SSA has assigned portfolio teams to perform select and control activities for IT investments. Several tools are provided to support the process. Key practice: 4. The board members understand the organization's IT investment management policies and procedures and the tools and techniques used in the board's decision-making process; Rating: Executed; Summary of evidence: The ITAB members are kept informed of the organization's IT investment management policies and procedures and the tools and techniques used in the board's decision-making process. According to SSA officials, each board member has one or more staff with responsibility for preparing the members for meetings. Also, the members are updated on new investment management tools during the ITAB quarterly meetings. In addition, SSA maintains a Web site which includes information, forms, and guidelines supporting the agency's IT planning process. Key practice: 5. Each board's span of authority and responsibility is defined to minimize overlaps or gaps among the boards; Rating: Executed; Summary of evidence: SSA has one board, the ITAB, responsible for allocating resources to IT portfolios in accordance with the agency's goals and objectives. Key practice: 6. The enterprisewide investment board has oversight responsibilities for the development and maintenance of the organization's documented IT investment process; Rating: Executed; Summary of evidence: The ITAB is responsible for new and updated IT investment processes, such as procedures for how to calculate the cost- benefit analysis and benefit value score. Key practice: 7. Each investment board operates in accordance with its assigned authority and responsibility; Rating: Executed; Summary of evidence: The CPIC Guide outlines the roles and responsibilities of the ITAB. The board is performing the select and control responsibilities assigned to it in accordance with this guidance. Key practice: 8. The organization has established management controls for ensuring that investment boards' decisions are carried out; Rating: Executed; Summary of evidence: SSA has established management controls to help ensure that actions of the ITAB are carried out. For example, the CIO, ITAB's Chair, makes final IT budget recommendations to the Commissioner, that includes the work year resources allocated for the IT projects approved by the board. The Deputy Commissioner, Systems monitors the work year resources allocated and expended for these IT projects. Source: GAO. [End of table] SSA Has a Process for Ensuring Projects Align with Business Needs: Defining business needs for each IT project helps to ensure that projects and systems support the organization's business needs and meet users' needs. According to ITIM, effectively meeting business needs requires, among other things, (1) documenting business needs with stated goals and objectives; (2) identifying specific users and other beneficiaries of IT projects and systems; (3) providing adequate resources to ensure that projects and systems support the organization's business needs and meet users' needs; and (4) periodically evaluating the alignment of IT projects and systems with the organization's strategic goals and objectives. (The complete list of key practices is provided in table 5). SSA has in place all seven key practices for meeting business needs. The agency's CPIC Guide and IT Planning Training Package require that sponsors identify the current and future business needs for proposed and ongoing projects and systems. Business needs are to be aligned with the SSA Strategic Plan. Resources for ensuring that IT projects and systems support the organization's business needs and meet users' needs include the ITAB, project sponsors and reviewers, the Systems Planning and Reporting System (which documents business needs information on proposed and ongoing projects), and the project scope agreement (which documents the business needs that the developer agrees will meet user needs). In reviewing selected agency projects as part of our study, we verified that the new and ongoing projects had these scope agreements. Table 5 shows the analysis for each key practice of the critical process for meeting business needs and summarizes the supporting evidence. Table 5: Meeting Business Needs: Key practice: 1. The organization has documented policies and procedures for identifying IT projects or systems that support the organization's ongoing and future business needs; Rating: Executed; Summary of evidence: The CPIC Guide and IT Planning Training Package document SSA's policies and procedures for identifying and supporting ongoing and future business needs. Key practice: 2. The organization has a documented business mission with stated goals and objectives; Rating: Executed; Summary of evidence: The SSA Strategic Plan documents its business mission with stated goals and objectives. Key practice: 3. Adequate resources, including people, funding, and tools, are provided for ensuring that IT projects and systems support the organization's business needs and meet users' needs; Rating: Executed; Summary of evidence: According to SSA officials, the agency has adequate resources for ensuring that the projects and systems support the organization's business needs. They include the ITAB, which has overall responsibility for ensuring that projects meet SSA's business needs; sponsors, who input business needs information into the Systems Planning and Reporting System tool, which includes forms for capturing this information; and the Commissioner's executive staff, which reviews the business needs information for accuracy. Key practice: 4. The organization defines and documents business needs for both proposed and ongoing IT projects and systems; Rating: Executed; Summary of evidence: SSA's policy calls for business needs for both proposed and ongoing IT projects and systems to be specified in the Systems Planning and Reporting System. We verified that business needs were defined and documented in the system for the three projects in our study. Key practice: 5. The organization identifies specific users and other beneficiaries of IT projects and systems; Rating: Executed; Summary of evidence: SSA policy and procedures call for specific users and other beneficiaries of IT projects and systems to be identified. We verified that specific users and other beneficiaries were identified for two of the three projects in our study. For the third project, Mainframe Architecture, SSA did not identify specific business users. Key practice: 6. Users participate in project management throughout an IT project's or system's life cycle; Rating: Executed; Summary of evidence: SSA policy and procedures call for specific users to participate in project management throughout a project's life cycle. We verified that users participated in project management for the three projects in our study. Key practice: 7. The investment board evaluates the alignment of its IT projects and systems with the organization's strategic goals and objectives and takes corrective actions when misalignment occurs; Rating: Executed; Summary of evidence: The ITAB evaluates projects' alignment with goals and objectives during the annual review cycle for projects and takes corrective action when misalignment occurs. Source: GAO. [End of table] SSA Has Implemented Most of the Procedures for Selecting New and Continuing Investments: Selecting new IT proposals and reselecting ongoing investments requires a well-defined and disciplined process to provide the agency's investment boards, business units, and developers with a common understanding of the process and the cost, benefit, schedule, and risk criteria that will be used both to select new projects and to reselect ongoing projects for continued funding. According to ITIM, this critical process requires, among other things, (1) making funding decisions for new proposals according to an established process; (2) providing adequate resources for investment selection activities; (3) using a defined selection process to select new investments and reselect ongoing investments; (4) establishing criteria for analyzing, prioritizing, and selecting new IT investments and for reselecting ongoing investments; and (5) creating a process for ensuring that the criteria change as organizational objectives change. (The complete list of key practices is provided in table 6.) SSA has in place 9 of 10 key practices for selecting investments. For example, the agency has established policies and procedures for integrating funding with the process of selecting an investment; the IT Planning Training Package states that the ITAB is to specify the resources available to each portfolio team for its investments. According to SSA officials, resources are provided for selecting investments, including managerial attention and tracking systems. Criteria have been established for selecting and reselecting investments, including return on investment, the business value of the investment, and investment cost. The agency ensures that selection criteria reflect organizational goals by aligning project selection with the organizational priorities set by the Commissioner each year. The IT Planning Training Package and ITAB meeting notes document the predefined selection criteria and process for selection of new investments. We verified that the three case study projects we reviewed were selected using the predefined selection process and criteria, and that these funding decisions were based on the selection information for the projects. However, SSA is not fully executing the key practice requiring policies and procedures for selecting new IT investment proposals. While the CPIC Guide has policies for identifying and evaluating new IT proposals, the IT Planning Training Package does not have documented procedures for prioritizing investment proposals. SSA officials said they do not have documented procedures because predefined criteria might result in not selecting a proposal that a portfolio team determines is required for operations. However, without predefined criteria for prioritizing investments consistently in each portfolio, SSA risks having less critical investments selected over investments that are more critical to accomplishing the portfolio's objective. Table 6 shows the rating for each key practice required to implement the critical process for selecting investments at the Stage 2 level of maturity and summarizes the evidence that supports these ratings. Table 6: Selecting Investments: Key practice: 1. The organization has documented policies and procedures for selecting new IT proposals; Rating: Not executed; Summary of evidence: While SSA's CPIC Guide has documented policies for selecting new IT proposals and the IT Planning Training Package has documented procedures for part of the selection process, the procedures are incomplete because they do not address prioritizing investments. SSA officials acknowledged that they do not have documented procedures for prioritization because the procedures are delegated to each portfolio team. However, the portfolio teams have not documented the prioritization procedures. Key practice: 2. The organization has documented policies and procedures for reselecting ongoing IT investments; Rating: Executed; Summary of evidence: SSA has documented policies and procedures for reselecting investments in its IT Planning Training Package. Key practice: 3. The organization has policies and procedures for integrating funding with the process of selecting an investment; Rating: Executed; Summary of evidence: SSA has policies and procedures in the CPIC Guide and IT Planning Training Package for integrating funding for staff with the process for selecting investments. Key practice: 4. Adequate resources, including people, funding, and tools, are provided for identifying and selecting IT projects and systems; Rating: Executed; Summary of evidence: According to SSA officials, adequate resources are provided for selecting IT projects and systems. The IT Planning Training Package documents the roles and responsibilities of staff and officials involved in identifying and selecting IT projects, including the portfolio team manager and support staff, the IT planning executives, the Office of Systems planning staff, and the Deputy Commissioner, Systems. SSA's list of planning contacts identifies individuals to whom these responsibilities are assigned. In addition, SSA has system tools for identifying and selecting IT projects and systems. Key practice: 5. Criteria for analyzing, prioritizing, and selecting new IT investment opportunities have been established; Rating: Executed; Summary of evidence: The CPIC Guide establishes criteria for analyzing, prioritizing, and selecting IT development investments. The criteria include benefits to SSA, including return on investment and intangible benefits, and costs and risks. SSA said that portfolio teams have flexibility in their use of prioritization criteria, and provided evidence that it was occurring. Key practice: 6. Criteria for analyzing, prioritizing, and reselecting IT investment opportunities have been established; Rating: Executed; Summary of evidence: The CPIC guide establishes criteria for analyzing, prioritizing, and reselecting their IT development investments. The criteria include benefits to SSA, including return on investment and intangible benefits, and costs and risks. Key practice: 7. A mechanism exists to ensure that the criteria continue to reflect organizational objectives; Rating: Executed; Summary of evidence: SSA's portfolio teams adjust the selection criteria in response to changes in the agency's strategic objectives. Key practice: 8. The organization uses its defined selection process, including predefined selection criteria, to select new IT investments; Rating: Executed; Summary of evidence: SSA's IT Planning Training Package outlines the select process and specifies criteria for selecting new investments, focusing on return on investment. Investments are selected by portfolio teams, and are reviewed and approved by the ITAB. We verified that the new development project we reviewed was selected using this process. Key practice: 9. The organization uses the defined selection process, including predefined selection criteria, to reselect ongoing IT investments; Rating: Executed; Summary of evidence: SSA's IT Planning Training Package documents that projects are reselected using the same process that is used to select new IT investments. We verified that the two ongoing projects in our study were reselected using this process. Key practice: 10. Executives' funding decisions are aligned with selection decisions; Rating: Executed; Summary of evidence: SSA's ITAB makes funding decisions for new and ongoing investments through its review and approval of the Agency IT Plan. The board's decisions are based on cost and benefit information provided with each investment for approval. Source: GAO. [End of table] SSA's Investment Board Has Limited Involvement in Providing Investment Oversight: An organization should provide effective oversight for its IT projects throughout all phases of their life cycles. Its investment board should maintain adequate oversight and observe each project's performance and progress toward predefined cost and schedule expectations as well as each project's anticipated benefits and risk exposure. The investment board should also employ early warning systems that enable it to take corrective action at the first sign of cost, schedule, or performance slippages. This board has ultimate responsibility for the activities within this critical process. According to ITIM, effective project oversight requires, among other things, (1) having written policies and procedures for management oversight; (2) developing and maintaining an approved project management plan for each IT project; (3) providing adequate resources for supporting the investment board; (4) having regular reviews by each investment board of each project's performance against stated expectations; and (5) ensuring that corrective actions for each underperforming project are documented, agreed to, implemented, and tracked until the desired outcome is achieved. The agency is executing two of seven key practices for providing oversight. The agency provides resources for oversight, and the board reviews summary reports on projects' cost and schedule performance. Also, the agency maintains project plans, including cost and schedule milestones for its investments. However, the agency is not executing the remaining five key practices related to providing oversight of IT projects. Although SSA provides the investment board with summary data on projects' performance related to cost, schedule, and benefits, the board does not receive information on projects' risks. Also, the board does not regularly track the implementation of corrective actions for each underperforming project. The board's meeting agenda allows individual deputy commissioners to raise concerns about project performance at quarterly meetings, but, based on our analysis of the ITAB meeting minutes, this opportunity is infrequently exercised. Specifically, during 2007, the meeting minutes showed that underperforming investments were discussed at only one of the quarterly meetings. Also, SSA officials have not specified the criteria for terminating projects that are underperforming. The Deputy Commissioner, Systems told us that he takes corrective actions to address underperforming projects but does not document these actions or report them to the ITAB. Table 7 shows the status of each key practice required to provide investment oversight at the project level and summarizes the supporting evidence. Table 7: Providing Investment Oversight: Key practice: 1. The organization has documented policies and procedures for management oversight of IT projects and systems; Rating: Not executed; Summary of evidence: Three documents address oversight policies and procedures: the CPIC Guide, the IT Planning Training Package, and the Office of Systems Project Management Directive. However, SSA does not have documented procedures for referring project performance problems to the ITAB. Key practice: 2. Adequate resources, including people, funding, and tools, are provided for IT project oversight; Rating: Executed; Summary of evidence: According to SSA officials, the agency has adequate resources for IT oversight. Portfolio managers and support staff are assigned to each portfolio and Office of Systems staff meet monthly to discuss portfolio health. Supporting tools, such as the Systems Planning and Reporting System and the Vital Signs and Observations Report, provide information on IT project status. Key practice: 3. IT projects and systems, including those in steady state (operations and maintenance), maintain approved project management plans that include expected cost and schedule milestones and measurable benefit and risk expectations; Rating: Not executed; Summary of evidence: SSA policy requires that all projects have a project plan. SSA has approved project plans for the development projects in our study. However, although SSA does maintain project plans for some of its operations and maintenance projects, it did not have such a plan for the operations and maintenance project in our study. Key practice: 4. Data on actual performance (including cost, schedule, benefit, and risk performance) are provided to the appropriate IT investment board; Rating: Not executed; Summary of evidence: The Deputy Commissioner, Systems is responsible for performance monitoring of IT projects and for providing to the ITAB CIO quarterly performance data for cost and schedule information at a summary level. However, the ITAB does not receive risk information for IT investments. Key practice: 5. Using verified data, each investment board regularly reviews the performance of IT projects and systems against stated expectations; Rating: Executed; Summary of evidence: Project performance monitoring for IT projects is performed by the Deputy Commissioner, Systems at monthly meetings. The ITAB also reviews summary cost and schedule earned value management information for groups of IT projects at its quarterly meetings. Key practice: 6. For each underperforming IT project or system, appropriate actions are taken to correct or terminate the project or system in accordance with defined criteria and the documented policies and procedures for management oversight; Rating: Not executed; Summary of evidence: According to SSA officials, the Deputy Commissioner, Systems, is responsible for corrective actions for underperforming projects. However, those actions are not documented. SSA officials have not specified criteria for terminating underperforming projects and could provide no examples of projects terminated for underperformance. Key practice: 7. The investment board regularly tracks the implementation of corrective actions for each underperforming project until the actions are completed; Rating: Not executed; Summary of evidence: Corrective actions are directed by the Deputy Commissioner, Systems at monthly meetings; however, these actions are not tracked or reported to the ITAB. The agency's policy is to resolve problems at the level of the Deputy Commissioner, Systems. SSA officials agreed that they should track these actions. Source: GAO. [End of table] SSA Has a Structured Process for Capturing Investment Information and Is Using It to Support Investment Management: To make good IT investment decisions, an organization must be able to acquire pertinent information about each investment and store that information in a retrievable format. During this critical process, an organization identifies its IT assets and creates a comprehensive repository of investment information. This repository provides information to investment decision makers to help them evaluate the potential impacts and opportunities created by proposed or continuing investments. The repository can take many forms and need not be centrally located, but the collection method should, at a minimum, identify each IT investment and its associated components. According to ITIM, effectively managing this repository requires, among other things, (1) developing written policies and procedures for identifying and collecting the information; (2) assigning responsibilities for ensuring that the information being collected meets the needs of the investment management process; (3) identifying IT projects and systems and collecting relevant information to support decisions about them; and (4) making the information easily accessible to decision makers and others. (The complete list of key practices is provided in table 8.) SSA has in place all six key practices associated with capturing investment information. For example, the agency's Project Resource Guide documents policies and procedures for submitting, updating, and maintaining relevant project information. One policy document, the Office of Systems Project Management Directive, identifies project management activities and work products for all projects approved by the investment board. SSA's Systems Process Improvement team is responsible for developing and maintaining the monthly health reports on project performance that are provided to the Deputy Commissioner, Systems to track actual project work years. In addition, projects must be recorded in the Systems Planning and Reporting System and each item to be considered by the investment board must be documented, including project dollar and work year estimates. The automated project status reports provide comprehensive status information for all development projects, including activities completed, activities in progress, and activities planned. We verified that information for three of the agency's IT projects we examined was collected in the Systems Planning and Reporting System and they all had a project scope agreement, which described the business, user, customer, and systems functions required. Also, project performance was reported in the monthly IT project health reports for all three projects. Table 8 summarizes the status of the six key practices for capturing investment information. Table 8: Capturing Investment Information: Key practice: 1. The organization has documented policies and procedures for identifying and collecting information about IT projects and systems to support the investment management process; Rating: Executed; Summary of evidence: The Project Resource Guide system has documented policies and procedures for identifying and collecting information to support the investment management process. This includes the use of management tools to collect and maintain information on IT investments. Key practice: 2. An official is assigned responsibility for ensuring that the information collected during project and systems identification meets the needs of the investment management process; Rating: Executed; Summary of evidence: The Deputy Commissioner, Systems planning staff is responsible for facilitating meetings of all the portfolio managers to discuss and arrive at a strategy for preparing the materials needed by the ITAB to provide guidance, and to address and arrive at consensus on issues that cross portfolio boundaries or that impact development of the Agency IT Plan. Key practice: 3. Adequate resources, including people, funding, and tools, are provided for identifying IT projects and systems and collecting relevant investment information about them; Rating: Executed; Summary of evidence: According to SSA officials, the agency has adequate resources available. SSA's ITAB members are responsible for the overall IT planning process. The Deputy Commissioner, Systems has designated planning staff and customer relations representatives to support ITAB efforts. SSA also has supporting tools for tracking IT assets. Key practice: 4. The organization's IT projects and systems are identified, and specific information is collected to support decisions about them; Rating: Executed; Summary of evidence: SSA uses tools (the two tracking systems and monthly health reports) for maintaining information on its IT investments. These tools are used to collect information on SSA's new and development projects. For the three projects that we examined during our study, information was collected in SSA's automated management tools. Key practice: 5. The information that has been collected is easily accessible and understandable to decision makers and others; Rating: Executed; Summary of evidence: SSA maintains information on its IT investments in its tracking systems. In observing the use of these management tools the information collected was easily accessible and understandable to those involved in decision making. Key practice: 6. The information repository is used by investment decision makers and others to support investment management; Rating: Executed; Summary of evidence: SSA's Deputy Commissioner, Systems and portfolio teams receive reports on information contained in the tracking systems, project scope agreements, and monthly health reports. Source: GAO. [End of table] SSA Has Established Processes for Managing Investments as an Enterprisewide Portfolio, but Key Practices Remain Not Executed: Once an agency has attained Stage 2 maturity, it needs to implement critical processes for managing its investments as an enterprisewide portfolio (Stage 3). An investment portfolio is an integrated, agencywide collection of investments that are assessed and managed collectively based on common criteria. Managing investments as a portfolio is a conscious, continuous, and proactive approach to allocating limited resources among an organization's competing initiatives in light of the relative benefits expected from these investments. Taking an agencywide perspective enables an organization to consider its investments comprehensively, so that collectively the investments optimally address the organization's mission, strategic goals, and objectives. Managing IT investments as a portfolio also enables an organization to determine its priorities and make decisions about which projects to fund and continue to fund based on analyses of the relative organizational value and risks of all projects, including projects that are proposed, under development, and in operations. Although investments may initially be organized into separate portfolios--based on, for example, business lines or life-cycle stages- -and managed by subordinate investment boards, they should ultimately be aggregated into this enterprise-level portfolio. According to the ITIM, Stage 3 maturity includes (1) defining the portfolio, (2) creating the portfolio criteria, (3) evaluating the portfolio, and (4) conducting postimplementation reviews. Table 9 summarizes the purpose of each critical process in Stage 3. Table 9: Stage 3 Critical Processes--Developing a Complete Investment Portfolio: Critical process: Defining the portfolio criteria; Purpose: To ensure that the organization develops and maintains IT portfolio selection criteria that support its mission, organizational strategies, and business priorities. Critical process: Creating the portfolio; Purpose: To ensure that IT investments are analyzed according to the organization's portfolio selection criteria and that an optimal IT investment portfolio with manageable risks and returns is selected and funded. Critical process: Evaluating the portfolio; Purpose: To review the performance of the organization's investment portfolios at agreed-upon intervals and to adjust the allocation of resources among investments as necessary. Critical process: Conducting postimplementation reviews; Purpose: To compare the results of recently implemented investments with the expectations that were set for them and to develop a set of lessons learned from these reviews. Source: GAO. [End of table] Within these 4 critical processes are 27 key practices associated with portfolio-level management. For the work year budget managed by its investment review board, SSA has executed 18 of the 27 key practices. SSA has executed all of the key practices for creating the portfolio and most of those for defining the criteria and conducting postimplementation reviews. However, the agency has not executed nine key practices, including establishing enterprisewide selection criteria and managing all of its investments as an enterprisewide portfolio. SSA has implemented postrelease reviews of its investments, but does not include evaluations of quantitative data and analyses, such as the investments' contributions toward achieving both the strategy and the objectives of the organization's IT strategic plan. Table 10 summarizes the status of SSA's Stage 3 critical processes and key practices. Table 10: Summary of Results for Stage 3 Critical Processes and Key Practices: Critical process: Defining the portfolio criteria; Key practices executed: 5; Total required by critical process: 7; Percentage of key practices executed: 71. Critical process: Creating the portfolio; Key practices executed: 7; Total required by critical process: 7; Percentage of key practices executed: 100. Critical process: Evaluating the portfolio; Key practices executed: 2; Total required by critical process: 7; Percentage of key practices executed: 29. Critical process: Conducting postimplementation reviews; Key practices executed: 4; Total required by critical process: 6; Percentage of key practices executed: 67. Critical process: Total; Key practices executed: 18; Total required by critical process: 27; Percentage of key practices executed: 67. Source: GAO. [End of table] SSA Defines Portfolio Criteria on a Strategic, Enterprisewide Basis: Developing an IT investment portfolio involves defining appropriate investment cost, benefit, schedule, and risk criteria to ensure that the organization's strategic goals, objectives, and mission will be satisfied by the selected investments. Portfolio selection criteria reflect the strategic and enterprisewide focus of the organization and build on the criteria that are used to select individual projects. When IT projects are not considered in the context of a portfolio, criteria based on narrow, lower-level requirements may dominate enterprisewide selection criteria. SSA is executing five of seven key practices associated with defining the portfolio criteria, including assigning responsibility to the ITAB for developing and modifying portfolio guidance and providing thresholds for selecting investments to the portfolio teams. According to SSA officials, the agency also has adequate resources for portfolio selection activities, including people and tools. Further, project management personnel are aware of the portfolio selection criteria. However, SSA is not executing two key practices. The agency has not fully documented policies and procedures, such as key procedures for creating and modifying IT portfolio selection criteria. Further, the investment board approved the core criteria for selection, but it has delegated the weighting of core criteria to the portfolio teams. This delegated approach conflicts with the need articulated in the ITIM framework to manage investments in a strategic, enterprisewide manner so that the investments address not only the objectives of individual programs, or lines of business, but also the impact that projects have on one another and the IT portfolio's overall benefit to the organization. Lacking complete enterprisewide portfolio criteria, SSA risks optimizing individual business processes while producing stovepiped systems, as well as not maximizing overall benefits to the agency. Table 11 shows the status for each key practice required to implement the critical process for defining the portfolio criteria and summarizes the evidence that supports these ratings. Table 11: Defining the Portfolio Criteria: Key practice: 1. The organization has documented policies and procedures for creating and modifying IT portfolio selection criteria; Rating: Not executed; Summary of evidence: SSA has policies and procedures for creating and modifying enterprisewide IT portfolio selection criteria, including guidance and thresholds. However, the procedures lack information specified in the ITIM, including: key information required to modify selection criteria; a record of previous selection criteria, their weights and rankings, and how they were developed; and triggers for initiating a change in the selection criteria. Key practice: 2. Responsibility is assigned to an individual or group for managing the development and modification of the IT portfolio selection criteria; Rating: Executed; Summary of evidence: The IT Planning Training Package assigns responsibility to the ITAB for developing and modifying the resource guidance and for providing thresholds for the portfolio teams, and assigns responsibility to each portfolio team for tailoring the criteria to align with its portfolio's objective. Key practice: 3. Adequate resources, including people, funding, and tools, are provided for portfolio selection criteria activities; Rating: Executed; Summary of evidence: SSA officials said adequate resources are available for portfolio selection activities, including people and tools. For example, it uses the Systems Planning and Reporting System for preparing new investment proposals. Key practice: 4. A working group has been designated responsibility for developing and modifying the IT portfolio selection criteria; Rating: Executed; Summary of evidence: The ITAB is designated the responsibility for developing and modifying the guidance and thresholds for IT portfolio selection and each portfolio team is designated responsibility for tailoring the criteria. Key practice: 5. The enterprisewide investment board approves the core IT portfolio selection criteria, including cost, benefit, schedule, and risk criteria, based on the organization's mission, goals, strategies, and priorities; Rating: Not executed; Summary of evidence: The Capital Planning and Investment Guide states that cost, benefit, schedule, and risk are the core portfolio selection criteria. However, the portfolio teams are delegated the responsibility to decide how these criteria are used to prioritize investments for selection, without approval by the ITAB. Key practice: 6. Project management personnel and other stakeholders are aware of the portfolio selection criteria; Rating: Executed; Summary of evidence: SSA conducts cross-portfolio team meetings to ensure that portfolio team members are aware of the portfolio selection criteria, and documents the criteria in its IT Planning Training Package. Key practice: 7. The enterprisewide investment board regularly reviews the IT portfolio selection criteria, using cumulative experience and event-driven data, and modifies the criteria as appropriate; Rating: Executed; Summary of evidence: The ITAB reviews the portfolio selection criteria annually, based on its cumulative experience and SSA's strategic objectives. Source: GAO. [End of table] SSA Is Creating its Investment Portfolio but Lacks Performance Measures: At ITIM Stage 3, organizations create a portfolio of IT investments to ensure that (1) they are analyzed according to the organization's portfolio selection criteria and (2) an optimal investment portfolio with manageable risks and returns is selected and funded. According to ITIM, creating the portfolio requires organizations to, among other things, document policies and procedures for analyzing, selecting, and maintaining the portfolio; provide adequate resources, including people, funding, and tools for creating the portfolio; and capture the information used to select, control, and evaluate the portfolio and maintain it for future reference. In creating the portfolio, the investment board should also (1) examine the mix of new and ongoing investments and their respective data and analyses and select investments for funding and (2) approve or modify the performance expectations for the IT investments they have selected. (The complete list of key practices is provided in table 12.) SSA is executing the seven key practices associated with creating the portfolio. For example, according to SSA officials, the agency has adequate resources for selecting the portfolio, including the ITAB executives, other supporting staff, and a system that tracks proposal information. The ITAB also considers a list of proposed IT investments and assigns IT staffing resources to the investment portfolios. Table 12 shows the status for each key practice required to implement the critical process for creating the portfolio and summarizes the evidence that supports these ratings. Table 12: Creating the Portfolio: Key practice: 1. The organization has documented policies and procedures for analyzing, selecting, and maintaining the investment portfolio; Rating: Executed; Summary of evidence: SSA has policies in place for individual portfolios calling for vision statements for analyzing and selecting projects and conducting gap analysis for maintaining the investment portfolio. Key practice: 2. Adequate resources, including people, funding, and tools, are provided for the process of creating the portfolio; Rating: Executed; Summary of evidence: SSA has adequate resources for creating the portfolio. The ITAB is composed of senior managers who meet regularly, and they are supported by portfolio teams. The Systems Planning and Reporting System supports decisions about projects. Key practice: 3. Board members are knowledgeable about the process of creating a portfolio; Rating: Executed; Summary of evidence: The deputy commissioners, who are members of the ITAB, are responsible for achieving the objectives of the IT investment portfolios, and therefore are knowledgeable of projects that support creating the investment portfolio. They are also briefed by the CIO, Deputy Commissioner, Systems, and their staff. Key practice: 4. The organization has defined the common portfolio categories that will be used across the organization; Rating: Executed; Summary of evidence: The ITAB has established nine investment portfolios to align with the objectives in the agency's strategic plan. The remaining two portfolios are aligned to legislation and infrastructure objectives. Key practice: 5. Each IT investment board examines the mix of new and ongoing investments and their respective data and analyses and selects investments for funding; Rating: Executed; Summary of evidence: The ITAB considers lists of proposed investments submitted by portfolio teams and makes final approval decisions. Key practice: 6. Each investment board approves or modifies the performance expectations for its selected IT investments; Rating: Executed; Summary of evidence: The Agency IT Plan approved by the ITAB has performance expectations for return on investment, investment cost, and schedule for each new investment, and the ITAB approves or modifies investment performance thresholds each year. Key practice: 7. Information used to select, control, and evaluate the portfolio is captured and maintained for future reference; Rating: Executed; Summary of evidence: Information used to select, control, and evaluate the portfolio is kept in an electronic archive. Documents relating to a project are available from the repository. Source: GAO. [End of table] SSA Has Not Fully Established a Process for Evaluating the Investment Portfolio: This critical process builds upon the Stage 2 critical process related to providing investment oversight by adding the elements of portfolio performance to an organization's investment control capacity. Compared to less mature organizations, Stage 3 organizations will have the foundation they need to control the risks faced by each investment and to deliver benefits that are linked to mission performance. In addition, a Stage 3 organization will have the benefit of good performance data generated by Stage 2 processes. Expanding this focus to the entire portfolio provides the organization with longer-term assurances that the IT investment portfolio will deliver mission value at acceptable cost. SSA has executed two of the seven key practices associated with this process: ensuring adequate resources, including staff and tools for reviewing the investment portfolio, and ensuring that the ITAB is familiar with the process for evaluating and improving investments. The remaining five key practices were not executed, partly because SSA has delegated portfolio management and partly because it is not executing the Stage 2 prerequisite critical process, providing investment oversight, which collects information on projects. As we have discussed, the ITAB does not receive information on nonperforming projects, because performance monitoring has been delegated to the Deputy Commissioner, Systems. SSA officials agreed that they were not evaluating the portfolio as a whole. Until SSA executes all the key practices associated with this critical process, senior executives will not have the information they need to determine whether the investments they have selected are delivering mission value at the expected cost and risk. Table 13 shows the status for each key practice required to implement the critical process for evaluating the portfolio and summarizes the evidence that supports these ratings. Table 13: Evaluating the Portfolio: Key practice: 1. The organization has documented policies and procedures for reviewing, evaluating, and improving the performance of its portfolio(s); Rating: Not executed; Summary of evidence: Although SSA has procedures for reviewing project work years, it does not have procedures documented for reviewing and evaluating its key performance measure of return on investment. Specifically, SSA does not have procedures in place to evaluate whether expected returns were achieved. Key practice: 2. Adequate resources, including people, funding, and tools, have been provided for reviewing the investment portfolio and its projects; Rating: Executed; Summary of evidence: SSA has staff for reviewing the investment portfolio and its projects: the portfolio team manager, Deputy Commissioner, Systems' staff, and ITAB members. SSA has tools for reviewing the investment portfolio and its projects including the Vital Signs and Observations Report and the monthly health reports. Key practice: 3. Board members are familiar with the process for evaluating and improving the portfolio's performance; Rating: Executed; Summary of evidence: The ITAB is familiar with the process for evaluating and improving the agency's IT investments using data about projects' cost and schedule performance. Key practice: 4. Results of relevant Providing Investment Oversight reviews from Stage 2 are provided to the investment board; Rating: Not executed; Summary of evidence: The ITAB does not receive project risk-level summary information and reports of documented corrective actions for underperforming projects. Key practice: 5. Criteria for assessing portfolio performance are developed, reviewed, and modified at regular intervals to reflect current performance expectations; Rating: Not executed; Summary of evidence: SSA has not established criteria for assessing portfolio performance, such as actual versus expected performance. Further, the criteria are not established to measure the overall contribution of the portfolio to SSA's goals and objectives. Key practice: 6. IT portfolio performance measurement data are defined and collected consistent with portfolio performance criteria; Rating: Not executed; Summary of evidence: SSA does not define portfolio performance measurement data, such as for contribution to SSA's goals and objectives. Key practice: 7. Adjustments to the IT investment portfolio are executed in response to actual portfolio performance; Rating: Not executed; Summary of evidence: SSA's ITAB makes adjustments to work years based on portfolio goals. However, SSA does not define portfolio performance measures and therefore cannot make adjustments to the IT investment portfolios in response to actual portfolio performance. Source: GAO. [End of table] SSA Is Conducting Postimplementation Reviews, but Some Improvements Are Needed: The purpose of a postimplementation review is to evaluate an investment after it has completed development in order to validate whether the estimated return on investment was actually achieved. Specifically, the review is conducted to (1) examine differences between estimated and actual investment costs and benefits and possible ramifications for unplanned funding needs in the future and (2) extract "lessons learned" about the investment selection and control processes that can be used as the basis for management improvements. Postimplementation reviews should also be conducted for investment projects that were terminated before completion to readily identify potential management and process improvements.[Footnote 15] SSA has executed four of the six key practices associated with this process: policies and procedures are defined, adequate resources are provided, individuals assigned to conduct postimplementation reviews are familiar with the processes, and projects for which reviews will be conducted are identified. The remaining two key practices were not executed: quantitative investment data are not collected and analyzed and lessons learned are not conducted on investment processes for selection, control, and evaluation. Without analyzing quantitative data on benefits achieved, SSA cannot determine whether the project has delivered anticipated benefits. Further, without knowledge of what benefits are actually achieved from projects, the portfolio cannot be evaluated, and Stage 4 and 5 practices cannot be carried out effectively. Also, without developing lessons learned from postimplementation reviews to improve the CPIC's select, control, and evaluate phases, the agency will be unable to use the reviews to improve its investment management processes. Table 14 shows the status for each key practice required to implement the critical process for conducting postimplementation reviews and summarizes the evidence that supports these ratings. Table 14: Conducting Postimplementation Reviews: Key practice: 1. The organization has documented policies and procedures for conducting postimplementation reviews; Rating: Executed; Summary of evidence: SSA has policies and procedures for conducting postrelease reviews used for postimplementation reviews, in its Project Resource Guide. Key practice: 2. Adequate resources, including people, funding, and tools, have been provided for conducting postimplementation reviews; Rating: Executed; Summary of evidence: According to SSA, adequate resources are provided for conducting postrelease reviews. SSA designates people to conduct the reviews, including a facilitator and user representatives. The agency also uses tools including a template for surveying users. Key practice: 3. Individuals assigned to the investment board to conduct postimplementation reviews should be familiar with the policies and procedures for conducting such reviews; Rating: Executed; Summary of evidence: SSA provides guidelines that explain the purpose and steps for conducting postrelease reviews, and provides facilitators to assist participants in completing the reviews. Key practice: 4. The investment board identifies those projects for which postimplementation reviews will be conducted; Rating: Executed; Summary of evidence: SSA designates every development project for postrelease review 90 days after the software release is completed. Key practice: 5. Quantitative and qualitative investment data are collected, evaluated for reliability, and analyzed during the postimplementation reviews; Rating: Not executed; Summary of evidence: SSA conducts postrelease reviews that analyze qualitative data collected on user satisfaction, but does not conduct quantitative data analysis, such as determining whether benefits were achieved. Key practice: 6. Lessons learned and recommendations for improving the investment process are developed during the postimplementation review, documented, and then distributed to all stakeholders; Rating: Not executed; Summary of evidence: SSA documents lessons learned as part of the postrelease review process. The lessons learned identify improvements in the project development process, but not in the select, control, and evaluate processes. Source: GAO. [End of table] More Than Half of SSA's IT Budget Is Not Subject to Its Current Investment Management Process: Even though SSA is executing most Stage 2 and Stage 3 key practices for the work year budget managed by its investment board, IT products and services acquired with the acquisition budget ($610 million in acquisitions in fiscal year 2008--58 percent of the IT budget) are not managed as investments under SSA's CPIC process, and are not reviewed by the ITAB. These products and services include, among other things, engineering support services, network infrastructure, mainframe capacity infrastructure, hardware maintenance, software maintenance, local telecom services, telephone systems maintenance, and an agencywide support service contract. These acquisition budget expenditures are under the overall direction of the Deputy Commissioner, Systems and are determined by funding requests from the business units and subsequent negotiations. Each deputy commissioner and the associate commissioners who report to the Deputy Commissioner, Systems, submit requests for funds based on the unit's acquisition needs. These requests are analyzed by the Deputy Commissioner, Systems staff, requests are reconciled with the available resources, a budget is developed, and the CIO reviews and signs it. Although this process involves a large budget and important assets, it is not subject to the CPIC select, control, and evaluate phases. For example, acquisitions of IT products and services are not selected by a board in a disciplined fashion, such as using the agency's CPIC select and control procedures, but instead are largely selected by one individual---the Deputy Commissioner, Systems. While the ITAB is provided a list of proposed projects for the Agency IT Plan, the list does not include the acquisition budget expenses associated with projects. However, the investment board does receive a report summarizing the total amount of the funds expended. Agency officials gave several reasons why the acquisition budget is not managed by the investment board. Specifically, in SSA's view, just as the other deputy commissioners have discretion to manage funding allocated to their portfolios, the Deputy Commissioner, Systems should have the same discretion to allocate funding in the infrastructure portfolio. Further, the officials stated that many items included in this budget are very technical and might not be well understood by senior business management; thus, review at this level is not thought to be effective. In addition, officials said that many items in the acquisition budget (such as telephones) are not optional, but necessary to keep the agency running, and thus do not require a decision process. Given the large amount of funds involved, senior management involvement and oversight are essential to ensure effective management of and full accountability for acquisitions of IT products and services. Further, until the agency manages all of its investments from an enterprisewide perspective, it will be unable to consider its investments comprehensively, and ensure that the investments optimally address the organization's mission, strategic goals, and objectives. SSA Is Beginning Initiatives Intended to Address High-Level ITIM Processes: Organizations that achieve the Stage 4 level of maturity evaluate their IT investment processes and portfolios to identify opportunities for improvement. At the same time, these organizations are able to maintain the mature control and selection processes that are characteristic of Stage 3 in the ITIM model. At Stage 4, organizations are capable of systematically planning for and implementing decisions to discontinue or deselect obsolete, high-cost, and low-value IT investments and planning for successor investments that better support strategic goals and business needs. Organizations acquire Stage 5 capabilities when they create opportunities to shape strategic outcomes by learning from other organizations and continuously improving the manner in which they use IT to support and improve business outcomes. Thus, organizations at Stage 5 benchmark their IT investment processes relative to other best- in-class organizations and conduct proactive monitoring for breakthrough information technologies that will allow them to significantly improve business performance. Table 15 shows the purpose of each critical process in Stages 4 and 5. Table 15: Stages 4 and 5--Critical Processes Required for Improving the Investment Process and Leveraging IT for Strategic Outcomes: Critical process: Stage 4--Improving the Investment Process: Improving the portfolio's performance; Purpose: To assess and improve the performance of the IT investment portfolio and the investment management process. Critical process: Stage 4--Improving the Investment Process: Managing the succession of information systems; Purpose: To ensure that IT investments in operation are periodically evaluated and determine whether they should be retained, modified, replaced, or otherwise disposed of. Critical process: Stage 5--Leveraging Information Technology for Strategic Outcomes: Optimizing the investment process; Purpose: To identify and implement measurable improvements in the IT investment management processes so that the processes meet or exceed those used by best-in-class organizations. Critical process: Stage 5--Leveraging Information Technology for Strategic Outcomes: Using IT to drive strategic business change; Purpose: To dramatically improve business outcomes by strategically employing IT investments. Source: GAO. [End of table] Because the ITIM is cumulative, agencies cannot fully implement Stage 4 and 5 processes without first executing Stage 2 and 3. Nonetheless, SSA officials said they have begun two initiatives related to a Stage 4 objective (improving the investment process) and a Stage 5 objective (leveraging IT for strategic outcomes). The first initiative, Application Portfolio Management, was established to improve the agency's information technology decision-making process. When fully implemented, the initiative is intended to address the Stage 4 critical process (managing the succession of information systems). The Application Portfolio Management review is used to analyze and quantify the health of existing software applications to determine whether they are eligible to be retired, renovated, or maintained. According to the agency, SSA has released version 1.0 of Application Portfolio Management and has begun identifying software applications that are eligible to be retired, renovated, or maintained. The second initiative, the Technology Infusion Process, is beginning to address the second Stage 5 critical process--using IT to drive strategic business change. The Technology Infusion Process was established to evaluate and implement new technologies or new uses of existing technologies that will facilitate SSA's ability to achieve the agency's strategic goals. SSA has begun to identify various technologies for research and has begun to review technology projects submitted by a component sponsor as candidates for the Technology Infusion Process. However, Application Portfolio Management has not identified hardware or infrastructure projects for retirement, renovation, or maintenance. Conclusions: Given the importance of IT to SSA's mission, it is vital that the agency manages its investments effectively. To its credit, SSA has established many of the basic practices needed to build the foundation for managing its projects as investments and for managing its investments as a portfolio. However, weaknesses remain. For example, although the agency has established an investment board as the decision- making body that defines and implements the investment governance process, key policies and procedures for the investment management process are not fully defined, and the investment board does not provide oversight of underperforming investments. Moreover, the agency does not track corrective actions for its underperforming projects. SSA has also taken the important step of creating an investment portfolio. However, it has not fully established the policies and procedures essential to managing the portfolio, such as for reviewing, evaluating, and improving the performance of the portfolio. Further, the agency's postimplementation reviews do not evaluate whether the expected benefits were achieved or identified lessons learned for improving the investment management processes. Moreover, the agency's IT acquisition budget, used to acquire IT- related products and services, is not allocated or overseen by the investment board and is not managed using investment governance processes. Failure to apply these processes to the acquisition budget makes it impossible for SSA executive management tasked with overseeing the agency's investments to ensure that this portion of the budget is spent in the most efficient and effective manner. Recommendations for Executive Action: To strengthen SSA's investment management capability and address weaknesses discussed in this report, we recommend that the Commissioner of Social Security take the following actions: To fully implement the key practices for building the investment foundation (Stage 2) for current and project-level future IT investments' success, direct the Chief Information Officer to: * establish comprehensive policies and procedures for defining the investment governance process that specify (1) investment board operating procedures, (2) delegations of authority, and (3) criteria for prioritizing new and ongoing investments; * strengthen and expand the board's oversight responsibilities for underperforming projects and evaluations of projects; and: * establish a mechanism for tracking corrective actions for underperforming investments. To fully implement the key practices for developing a complete investment portfolio (Stage 3), direct the Chief Information Officer to: * establish policies and procedures for defining the portfolio criteria; * establish portfolio-level performance evaluation policies and procedures and criteria for assessing portfolio performance; and: * evaluate quantitative measures during postimplementation reviews, and lessons learned for improving select, control, and evaluate processes. To ensure senior management involvement and full accountability for the agency's investments, direct the Chief Information Officer to: * develop and implement policies and procedures to manage IT acquisitions as investments and manage them using the investment management framework. Agency Comments and Our Evaluation: The Commissioner of Social Security provided written comments on a draft of this report (comments are reproduced in appendix II). In its comments SSA agreed with six of our recommendations and disagreed with one. Regarding those recommendations with which it agreed, SSA stated that it had initiated actions to document existing investment management processes and that it plans to strengthen and expand the role of the investment board in the oversight of underperforming projects and in the evaluations of investments. The agency also stated that it plans to establish a mechanism for tracking corrective actions for underperforming investments. Further, to achieve a complete IT investment portfolio, SSA plans to establish procedures for defining the portfolio criteria within the context of the existing delegation of authority to the portfolio sponsors. In addition, regarding postimplementation reviews, the agency stated it plans to evaluate quantitative measures and lessons learned for improving select, control, and evaluate processes. SSA disagreed with our recommendation that it develop policies and procedures for managing its IT acquisitions as investments and manage them using the investment board and investment management processes. The agency stated that its existing budget development process already treats these acquisitions as investments and maintains them by using an investment management framework, though not the one described in our ITIM framework. However, under SSA's current process, these acquisitions are not subject to the agency's investment management select, control, and evaluate processes and are not managed by its investment board. Given that the IT products and services make up the majority of SSA's IT budget, the investment board's involvement is essential to helping ensure effective management of and full accountability for acquisitions of IT products and services. As we previously noted, by the agency not applying its investment management process to the acquisition budget, it limits the ability of SSA's executive management tasked with overseeing the agency's investments to ensure that this portion of the budget is spent in the most efficient and effective manner. SSA also provided technical and other comments, which we have incorporated as appropriate. Among the comments, the agency stated that it had pursued the adoption of industry best practices developed by institutions such as the Software Engineering Institute of Carnegie Mellon University and believed it had achieved comprehensive and mature IT management practices. SSA added that our assessment had provided an opportunity for the agency to think carefully about many aspects of its investment management processes, and had enabled it to better understand the strengths and weaknesses of its current approach to managing investments. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies of the report to interested congressional committees, the Director of the Office of Management and Budget, and the Commissioner of Social Security. Copies of this report will be made available to other interested parties on request. This report will also be available at no charge on our Web site at [hyperlink, http://www.gao.gov]. Should you or your staff have questions on matters discussed in this report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Sincerely yours, Signed by: Valerie C. Melvin: Director, Human Capital and Management Information Systems Issues: [End of section] Appendix I: Objective, Scope, and Methodology: Our objective was to determine whether Social Security Administration's (SSA) investment management approach is consistent with leading investment management best practices. Our analysis was based on best practices contained in GAO's Information Technology Investment Management (ITIM) framework[Footnote 16] and the framework's associated evaluation methodology, and focused on the agency's implementation of critical processes and key practices for managing its business systems investments. To address our objective, we asked the agency to complete a self- assessment of its investment management process and provide the supporting documentation. We then reviewed the results of the agency's self-assessment of Stages 2 and 3 practices and compared them against our ITIM framework. We focused on Stages 2 and 3 because these stages represent the processes needed to meet the standards of the Clinger- Cohen Act and they establish the foundation for effective acquisition management. We also validated and updated the results of the self- assessment through document reviews and interviews with officials, such as the CIO, Deputy Commissioner, Systems, and other staff in these offices. In doing so, we reviewed written policies, procedures, and guidance that provided evidence of documented practices, including SSA's IT Capital Planning and Investment Control (CPIC) Guide and IT Planning Training Package. We also reviewed the fiscal year 2008-2009 Agency IT Plan and the board's meeting minutes and other documentation providing evidence of executed practices. We compared the evidence collected from our document reviews and interviews to the key practices in ITIM. We rated the key practices as "executed" on the basis of whether the agency demonstrated (by providing evidence of performance) that it had met the criteria of the key practice. A key practice was rated as "not executed" when we found insufficient evidence of a practice during the review or when we determined that there were significant weaknesses in SSA's execution of the key practice. In addition, SSA was provided with the opportunity to produce evidence for key practices rated as "not executed." We did not assess investments made with SSA's IT acquisition budget because SSA acknowledged that the acquisition budget is not managed using SSA's investment management process. This budget includes items that are not projects, but are technology items that support projects, or general infrastructure such as mainframe computers, desktop computers, data storage, or telecommunications services. As part of our analysis, we selected three IT projects as case studies to verify whether certain critical processes and key practices were being applied. SSA officials participated in the selection of these case studies. We selected projects that (1) supported different SSA functional areas, (2) were in different life-cycle phases, and (3) involved different funding amounts. These three projects are described below. Ready Retirement is a project that automates the processing of retirement applications. It allows individuals to file for benefits using a Web interface. This investment is expected to increase online claims filing, minimize the number of recontacts required to complete an application, and provide progress indicators to inform applicants of where they are in the application process. Ready Retirement is intended to prepare the agency for the growing retirement workload expected as baby boomers become eligible for retirement by enabling applicants to prepare their own applications. According to the agency, this project is estimated to require about 27 staff years for fiscal year 2008, which corresponds to costs of about $3.1 million.[Footnote 17] Appeals Council Case Processing is a software development project that automates the handling of case files in appeals of disability determinations. It is intended to provide the capability to process all disability cases electronically at all adjudicative levels. Further, the system can obtain claims, medical evidence, and supporting documentation over the Internet in a secured environment. The users have the capability to complete all disability case-related actions electronically. This project is expected to eliminate backlogs, reduce reliance on paper folders, and increase decisional and documentation accuracy and decisional consistency. SSA estimates that this project will require about 56 staff years in fiscal year 2008, which corresponds to costs of about $6.4 million. Mainframe Architecture is a large infrastructure investment that involves both developmental and operations and maintenance components, and includes both software development and hardware. SSA's mainframes are the hardware platform for many critical systems. The agency states that its objective is to provide 100 percent reliability and availability to mainframe users. Tasks for the project include enhancements to hardware and software technology, annual upgrades to the operating system, routine additions to mainframe capacity dictated by workload growth, and migration to the current software versions of over 100 vendor products. The agency estimates that this project will require about 54 staff years for developmental projects and about 28 staff years for operations and maintenance work in fiscal year 2008, which corresponds to costs of about $9.5 million. In addition, the project is expected to require about $84 million from the acquisition budget for a total cost of about $94 million. For these projects, we reviewed project management documentation, such as project proposals, project plans, and performance reports on costs and benefits. We also conducted interviews with the agency's CIO and Deputy Commissioner, Systems, as well as other managers responsible for the agency's investment management processes. We conducted our work at SSA headquarters in Baltimore, Maryland from October 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. [End of section] Appendix II: Comments from Social Security Administration: Social Security: The Commissioner: Social Security Administration: Baltimore MD 21235-0001: September 4, 2008: Valerie C. Melvin, Director: Human Capital and Management Information Systems Issues: U.S. Government Accountability Office: 441 G Street NW: Washington, D.C. 20548: Dear Ms. Melvin: Thank you for the opportunity to review and comment on the Government Accountability Office (GAO) draft report, "Information Technology: SSA Has Taken Key Steps for Managing Its Investments, but Needs to Strengthen Oversight and Fully Define Policies and Procedures" (GAO-08- 1020). Our attached comments provide specific responses to the recommendations and identify technical corrections that should be made to enhance the accuracy of the report. If you have any questions, please contact Ms. Candace Skurnik. Director. Audit Management and Liaison Staff, at (410) 965-4636. Sincerely, Signed by: Michael J. Astrue: Enclosure: Comments On The Government Accountability Office (GAO) Draft Report, "Information Technology: SSA Has Taken Key Steps For Managing Its Investments, But Needs To Strengthen Oversight And Fully Define Policies And Procedures" (GAO-08-1020): Thank you for the opportunity to review and provide comments on this draft report. Recommendation 1: Establish comprehensive policies and procedures for defining the investment governance process that specify: (a) investment board operating procedures; (b) delegations of authority; and (c) criteria for prioritizing new and ongoing investments. Comment: We agree. We believe that the development of comprehensive policies and procedures to support our information technology (IT) investment management process would contribute to the stability and shared understanding of the investment process. We have already initiated efforts to document existing processes and establish charters for existing bodies. Recommendation 2: Strengthen and expand the board's responsibilities for providing investment oversight, including underperforming projects and evaluations of projects. Comment: We agree. We will strengthen and expand the Information Technology Advisory Board's (ITAB) role in the oversight of underperforming investments and evaluation projects. Recommendation 3: Establish a mechanism for tracking corrective actions for underperforming investments. Comment: We agree. We will establish a mechanism for tracking corrective actions for underperforming investments. Recommendation 4: To fully implement the key practices for developing a complete investment portfolio (Stage 3), direct the Chief Information Officer (CIO) to establish policies and procedures for defining the portfolio criteria. Comment: We agree. We will establish policies and procedures for defining the portfolio criteria within the context of the existing delegation of authority to the Portfolio Sponsors. Recommendation 5: Establish portfolio-level performance evaluation policies and procedures and criteria for assessing portfolio performance. Comment: We agree. We will establish portfolio-level performance evaluation policies and procedures and criteria for assessing portfolio performance. Recommendation 6: Evaluate quantitative measures during post-implementation reviews, and lessons learned for improving select, control, and evaluate processes. Comment: We agree. We will evaluate quantitative measures during post- implementation reviews, and lessons learned. To a great extent, this will entail simply pulling together data already available from various management information systems. Recommendation 7: To ensure senior management involvement and full accountability for the agency's investments, direct the CIO to develop and implement policies and procedures for managing IT acquisitions as investments and put under investment management framework. Comment: We disagree. Our existing information technology systems (ITS) budget development process already treats IT acquisitions as investments and maintains them under an investment management framework, though not one described by GAO's Information Technology Investment Management (ITIM) Framework. We agree, however, that the ITS budget development process can be further integrated with the ITAB-centered investment management process. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov: Staff Acknowledgments: In addition to the contact person named above, key contributors to this report were Cynthia Scott, Assistant Director; Faiza Baluch; Rebecca LaPaze; Sabine Paul; Tomás Ramirez; Glenn Spiegel; Niti Tandon; and Daniel Wexler. [End of section] Footnotes: [1] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G] (Washington, D.C.: March 2004). [2] 40 U.S.C. §§ 11301-11331. [3] A work year represents one full-time equivalent employee or contractor. The investment board approves work years for the investment portfolios included in the agency's Annual IT Plan. [4] The two figures add to more than $1 billion because some contractors are included in both numbers. [5] The Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11101-11704. This act expanded the responsibilities of OMB and federal agencies under the Paperwork Reduction Act with regard to IT management. See 44 U.S.C. 3504(a)(1)(B)(vi) (OMB); and 44 U.S.C. 3506(h)(5) (agencies). [6] This policy is set forth and guidance is provided in OMB Circular A- 11 (June 2008) directs agencies to develop, implement, and use a capital programming process to build their capital asset portfolios. [7] See, for example, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO- 04-394G] GAO, Information Technology: A Framework for Assessing and Improving Enterprise Architecture Management (Version 1.1), [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-584G] (Washington, D.C.: April 2003); and Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making, GAO/AIMD-10.1.13 (Washington, D.C.: February 1997). [8] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G]; [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-10.1.13]; GAO, Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-94-115] (Washington, D.C.: May 1994); and OMB, Evaluating Information Technology Investments, A Practical Guide (Washington, D.C.: November 1995). [9] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G]. [10] GAO, Information Technology: DHS Needs to Fully Define and Implement Policies and Procedures for Effectively Managing Investments, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-424] (Washington, D.C.: Apr. 27, 2007); Information Technology: Treasury Needs to Strengthen its Investment Board Operations and Oversight, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-865] (Washington, D.C.: July 23, 2007); Information Technology: Centers for Medicare and Medicaid Services Needs to Establish Critical Investment Management Capabilities, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-12] (Washington, D.C.: Oct. 28, 2005); Information Technology: HHS Has Several Investment Management Capabilities in Place, but Needs to Address Key Weaknesses, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-06-11] (Washington, D.C.: Oct. 28, 2005); Information Technology: FAA Has Many Investment Management Capabilities in Place, but More Oversight of Operational Systems Is Needed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-822] (Washington, D.C.: Aug. 20, 2004); Bureau of Land Management: Plan Needed to Sustain Progress in Establishing IT Investment Management Capabilities, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1025] (Washington, D.C.: Sept. 12, 2003); Information Technology: Departmental Leadership Crucial to Success of Investment Reforms at Interior, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-1028] (Washington, D.C.: Sept. 12, 2003); United States Postal Service: Opportunities to Strengthen IT Investment Management Capabilities, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-03-3] (Washington, D.C.: Oct. 15, 2002); and Information Technology: DLA Needs to Strengthen Its Investment Management Capability, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-02-314] (Washington, D.C.: Mar. 15, 2002). [11] Stage 1 is typified by the absence of an organized, executable, and consistently applied IT investment management process. [12] An IT investment board is a decision-making body made up of senior program, financial, and information officials that is responsible for making decisions about IT projects and systems on the basis of comparisons and trade-offs among competing projects and has emphasis on meeting mission goals. [13] 40 U.S.C. §§ 11312-11313. [14] The portfolios include nine that align with the objectives described in SSA's Strategic Plan and two that support infrastructure and mandated projects. [15] SSA refers to postimplementation reviews as postrelease reviews. The agency's postrelease reviews are similar to the activities described in our ITIM framework for postimplementation reviews. [16] GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-394G] (Washington, D.C.: March 2004). [17] SSA estimates an average cost per staff year of $115,500. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: