OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
GENERAL OBSERVATIONS

January 2005

A-15-05-25096

AUDIT REPORT

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

MEMORANDUM

Date: January 6, 2005

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: General Observations (A-15-05-25096)
We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 16 of the Social Security Administration's performance indicators established to comply with the Government Performance and Results Act. For each performance indicator, PwC's objectives were to:

Test critical controls over the data generation and calculation processes for the specific performance indicator;

Assess the overall adequacy, accuracy, reasonableness, completeness, and consistency of the performance indicator and supporting data; and

Determine if each performance indicator provides meaningful measurement of the program and the achievement of its stated objectives.

The evaluation of the 16 performance indicators resulted in 7 audit reports. The seven reports are listed below.
Report Title CIN
Employment for Disabled Beneficiaries A-02-04-14068
Earnings Suspense File A-15-04-14069
President's Management Agenda Related Initiatives A-15-04-14070
Management Information Systems Development and Protection A-15-04-14071
Processing Time A-02-04-14072
Productivity A-15-04-14073
Disability Determination Services Net Accuracy Rate A-15-04-14074

During the audits of 16 performance indicators included in the Agency's Fiscal Year 2003 Performance and Accountability Report, PwC identified several areas for improvement across multiple indicators. The findings noted in this report apply to the overall management of the performance indicator process for the 16 performance indicators reviewed.

Please provide within 60 days a corrective action plan that addresses each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at
(410) 965-9700.

Patrick P. O'Carroll, Jr.

Attachment

MEMORANDUM

Date: December 22, 2004

To: Inspector General

From: PricewaterhouseCoopers LLP

Subject: Performance Indicator Audit: General Observations (A-15-05-25096)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.

OBJECTIVE

We audited 16 of SSA's performance indicators reported in the Agency's Fiscal Year (FY) 2003 Performance and Accountability Report (PAR). For each performance indicator audited, our objectives were to:

1. Test critical controls over the data generation and calculation processes for the specific performance indicator.

2. Assess the overall adequacy, accuracy, reasonableness, completeness, and consistency of the performance indicator and supporting data.

3. Determine if each performance indicator provides meaningful measurement of the program and the achievement of its stated objectives.

In addition to completing the objectives noted above, we also identified areas for improvement to the overall SSA process over performance indicators. This report contains our observations related to the overall management, documentation, and reporting of SSA performance indicators.

BACKGROUND

Performance indicator measurement at SSA is the process that provides multiple points of performance information for both external and internal Agency reporting needs. Information on performance indicator results has been integrated into SSA's PAR. The information reported in the PAR is used to help manage SSA operations by providing the Agency with a process to show current performance in relation to past performance and project future performance for the next several years. As a result, SSA has been able to use the information gained from monitoring the performance indicators to plan for future Agency actions. In its FY 2003 PAR, SSA reported the results of 46 performance indicators.

GPRA and the Office of Management and Budget (OMB) require SSA (and other Federal agencies) to report on the performance of programmatic functions. OMB holds SSA accountable for performance indicator variances, and budgetary decisions are based in part on the information reported in the PAR. The intent of reporting the results of the performance indicators in the PAR is to inform OMB, Congress, and the United States public of the performance results of the overall SSA mission.

The Office of the Chief Strategic Officer/Office of Strategic Management (OCSO/OSM) is responsible for coordinating with each indicator "owner" to support the development, monitoring and reporting of performance indicators. OCSO/OSM also encourages SSA's managers to use performance indicator information as a part of their ongoing strategic and operational planning activities. The individual owners are responsible for ensuring that the progress of the indicator is tracked and reported on a periodic basis.

RESULTS OF REVIEW

We identified several areas for improvement during our audits of 16 performance indicators included in SSA's FY 2003 PAR. A number of our findings relate to the overall management of the performance indicator process for the 16 performance indicators we reviewed. These include:

SSA has not consistently developed or documented detailed policies, procedures, and controls to collect, review, and report information for the individual performance indicators. For the indicators included in our audits, SSA did not consistently document the detailed responsibilities of the indicator owners, data sources, data interfaces, data modifications, or controls to ensure that performance indicator data were complete, accurate, valid, and that an audit trail of data and calculations of indicator results were maintained. We identified this issue in 13 of the 16 indicators reviewed. In addition, we found that SSA employees and contractors had excessive access rights defined within the Top Secret security software, which allowed update and create access to the mainframe datasets used to calculate performance indicators. We identified this issue in 6 of the 16 indicators reviewed.

SSA had not ensured that the disclosures in the PAR clearly linked each performance indicator to the Agency's strategic goals and objectives and described each performance indicator completely, accurately, and consistently. In addition, for many of the indicators, the narrative information included within the PAR did not consistently provide clear linkage between the indicators and the Agency's strategic goals and objectives. We identified this issue in 11 of the 16 indicators reviewed.

In addressing these findings in our reports, we noted that responsibility for policies, internal controls and disclosures related to individual performance indicators is distributed among the various designated performance indicator "owners" and business units. SSA has created the OCSO/OSM to coordinate and facilitate the process of gathering the performance indicator information and results for internal and external reporting. The purpose of OCSO/OSM as stated on OSM's web site is to "Promote the strategic management of SSA's programs, resources and service delivery to improve SSA Mission performance."

However, SSA has not formally designated the Chief Strategic Officer or another single Deputy Commissioner with the responsibility to ensure that performance indictor internal controls are consistently designed and implemented, indicator results are validated, or that performance indicator policies and procedures are created and enforced. Furthermore, a single Deputy Commissioner has not been designated to ensure that the reported indicators are meaningful, appropriately linked to the mission of SSA, or developed in a manner that highlights the ultimate outcome and results of SSA's performance. OMB Circular Number A-123 Management Accountability and Control requires agencies and individual Federal managers to take systematic and proactive measures to develop and implement appropriate, cost-effective management controls for results-oriented management.

RECOMMENDATION

We recommend SSA formally designate responsibility to ensure performance indicator policies and internal controls are designed and implemented across the Agency. SSA should consider designating a single Deputy Commissioner to be responsible for ensuring indicator results are validated, and that overall performance indicator internal controls are designed and implemented. In addition, a single Deputy Commissioner should ensure that the reported indicators are meaningful, appropriately linked to the mission of SSA, and developed in a manner that highlights the ultimate outcome and results of SSA's performance.

This designated individual(s) can be responsible for ensuring that SSA develops and documents policies and procedures used to prepare and disclose the results of the individual performance indicators on a consistent basis. Such documentation should include the detailed responsibilities of the indicator owners, data sources, data interfaces, data modifications, and controls in place to ensure that performance indicator data are complete, accurate, and valid; that an audit trail of data and calculations of indicator results are maintained; and that access to data used to calculate indicator results is appropriately restricted.

The designated individual can also require that indicator owners ensure: each performance indicator is fully described in a complete, accurate, and explicit manner within the PAR; the owners provide a clear linkage between the indicators and the Agency's strategic goals and objectives; and indicator titles, trend information, and data definitions are consistent within the PAR.

SSA COMMENTS

SSA believes that performance management leadership already exists under the Office of the Chief Strategic Officer. In addition, it believes that core responsibility for performance indicators should rest with the Deputy Commissioners as sponsors or owners of the indicators since they are responsible for achieving them. SSA stated that it believed that the FY 2004 PAR included clear descriptions and definitions of each of its performance indicators, including a description of the linkages between the indicators and its strategic goals.

PwC RESPONSE

We agree SSA has taken steps to improve the descriptions of its indicators and their linkages to its strategic goals. However, we continue to believe the designation of a single Deputy Commissioner, with responsibility for enforcement of policies and procedures, internal controls, and disclosure and validation of results, would improve SSA's performance management process. During our review of FY 2004 performance measures, we will assess SSA's progress in ensuring its indicators are meaningful, linked to the mission of the Agency, and developed to highlight clearly SSA's performance. We will also assess the consistency and quality of documentation and indicator results reported in SSA's FY 2004 PAR.

Appendices

APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Agency Comments

Appendix A
Acronyms
DDS Disability Determination Services
FY Fiscal Year
GPRA Government Performance and Results Act
OCSO/OSM Office of the Chief Strategic Officer/Office of Strategic Management
OMB Office of Management and Budget
PAR Performance and Accountability Report
SSA Social Security Administration
SSI Supplemental Security Income

Appendix B
Scope and Methodology
We updated our current understanding of the Social Security Administration's (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and inquiry of SSA management. We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program.

Through inquiry, observation, and other substantive testing including testing of source documentation, we performed the following as applicable:

Reviewed prior SSA, Government Accountability Office, and other reports related to SSA GPRA performance and related information systems.
Met with the appropriate SSA personnel to confirm our understanding of each individual performance indicator.
Flowcharted the processes.
Where applicable, we tested key controls related to manual or basic computerized processes (e.g., spreadsheets, databases, etc.).
Conducted and evaluated tests of the automated and manual controls within and surrounding each of the critical applications to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
For those indicators with results that SSA determined using computerized data, we assessed the completeness and accuracy of that data to determine the data's reliability as it pertains to the objectives of the audit.
Identified and extracted data elements from relevant systems and obtained source documents for detailed testing selections and analysis.
Identified attributes, rules, and assumptions for each defined data element or source document.
Tested the adequacy, accuracy, reasonableness, consistency, and completeness of the selection.
Recalculated the metric or algorithm of key performance indicators to ensure mathematical accuracy.

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency's mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency's mission, goals, objectives, and processes were used to determine if the performance indicators being used appear to be valid and appropriate given our understanding of SSA's mission, goals, objectives and processes. We followed all performance audit standards.

We audited the following performance indicators in the SSA Fiscal Year 2003 Performance and Accountability Report:

1. Percent of commercial positions competed or converted.
2. "Get to Green" on all President's Management Agenda initiatives.
3. Percent improvement in the retention rate.
4. Provide the equivalent of 40 hours of training annually to all employees.
5. Maintain zero outside infiltrations of SSA's programmatic mainframes.
6. Milestone measures for Managerial Cost Accountability System and Social Security Unified Measurement Systems.
7. Milestones in developing new performance management systems (Implement new Senior Executive Service system).
8. Disability Determination Services (DDS) net accuracy rate (allowances and denials combined).
9. Percent increase in the number of Supplemental Security Income (SSI) disabled beneficiaries earning at least $100 per month.
10. Reduction in the size of the earnings suspense file.
11. Percent of incoming earnings items removed from the suspense file at the end of the annual earnings posting cycle.
12. Average processing time for initial disability claims.
13. Average processing time for all hearings.
14. Percent improvement in productivity.
15. DDS cases processed per work year.
16. SSI Aged claims processed per work year.

Appendix C
Agency Comments

SOCIAL SECURITY

MEMORANDUM

December 20, 2004

To: Patrick P. O'Carroll, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report "Performance Indicator Audit: General Observations" (A-15-05-25096)--INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Please let me know if you have any questions. Staff inquiries may be directed to Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.

Attachment:
SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: GENERAL OBSERVATIONS" (A 15-05-25096)

Thank you for the opportunity to review and comment on the draft report. Our achievement of the American Association of Government Accountant's Certificate of Excellence in Accountability Reporting every year since the award program began in fiscal year (FY) 1998, and the attainment of a "green status" in the area of financial management on the FY 2004 President's Management Agenda Scorecard clearly establishes that our financial statements, annual performance plans and reports, and individual performance measures are comprehensive and accurate. The realization of both of these accomplishments was achieved through the Social Security Administration's (SSA) and the OIG staff's continuous efforts and diligence in evaluating and improving our performance management and reporting processes. We appreciate your providing us your observations from the reviews conducted in FY 2004 and our response to the report contents and specific recommendations are provided below.

General Comments

Regarding the statement that SSA has not consistently developed or documented detailed policies, procedures, and controls to collect, review, and report information for the individual performance indicators, it should be noted that during the FY 2004 key performance indicator and Government Performance Results Act audit process, components spent a lot of time preparing the required documentation based on guidance received from PricewaterhouseCoopers (PwC). Now that we are aware of the type of documentation that is being required, we anticipate the next audit cycle will produce more consistent documentation. However, we also need to balance the degree of effort required to maintain an audit trail of data and calculations for indicator results. In some cases it may be cost prohibitive to maintain an audit trail of data and calculations. In those cases, we will continue to work with PwC to develop a documentation approach that will enhance the Agency's performance measure reporting, while assisting the auditors in understanding the processes used to arrive at quantitative data supporting the Agency's annual performance outcomes.

PwC found that SSA had not ensured that the disclosures in the Performance and Accountability Report (PAR) clearly linked each performance indicator to the Agency's strategic goals and objectives or described each performance indicator completely, accurately, and consistently. In addition, the narrative information included within the PAR for many of the performance indicators did not consistently provide clear linkage between the indicators and the Agency's strategic goals and objectives. These issues were addressed in the FY 2004 PAR, as each performance measure includes a narrative that describes how the measure links to the Agency's strategic goals and objectives.

Recommendation Comments

Regarding the recommendation that we formally designate responsibility to ensure performance indicator policies and internal controls are designed and implemented across the Agency, the Office of the Chief Strategic Officer (OCSO) was established to direct the administration of the Agency's comprehensive management programs including competitive sourcing, quality management, strategic planning and workforce analysis. They accomplish their mission by collaborating and coordinating with Agency components to address cross-cutting programmatic and administrative issues that include promoting the strategic management of SSA's programs, resources, and service delivery to improve mission performance. OCSO has a formal Agency planning process in place, as it prepares the Strategic Plan, Annual Performance Plan and the Annual Performance Report in the PAR. They also track the results of performance goals in SSA's Tracking Report. We believe performance indicator management leadership already exists under the Office of Chief Strategic Officer.

Regarding the recommendations that we designate a single Deputy Commissioner to:
1) Ensure indicator results are validated, and that overall performance indicator internal controls are designed and implemented; 2) ensure that the reported indicators are meaningful, appropriately linked to the mission of SSA, and developed in a manner that highlights the ultimate outcome and results of SSA's performance; and 3) ensure that SSA develops and documents policies and procedures used to prepare and disclose the results of the individual performance indicators on a consistent basis. We believe that core responsibility for performance measures should rest with the Deputy Commissioners as sponsors or owners of the measures since they are responsible for achieving them.

Finally, regarding the recommendation that would require indicator owners to: 1) Ensure that each performance indicator is fully described in a complete, accurate, and explicit manner within the PAR; 2) provide a clear linkage between the indicators and the Agency's strategic goals and objectives; 3) ensure that indicator titles, trend information, and data definitions are consistent within the PAR. As stated above, this issue was addressed in the FY 2004 PAR as each performance measure includes a narrative that describes how the measure links to the Agency's strategic goals and objectives.

[The Agency also provided a technical comment which has been addressed in the report.]

Overview of the Office of the Inspector General

The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Executive Operations (OEO). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit

OA conducts and/or supervises financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations

OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General

OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Executive Operations

OEO supports OIG by providing information resource management and systems security. OEO also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, OEO is the focal point for OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.