OFFICE
OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
PERFORMANCE
INDICATOR AUDIT:
GENERAL OBSERVATIONS
January
2005
A-15-05-25096
AUDIT REPORT
Mission
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations
relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation
and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems
in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
Vision
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
MEMORANDUM
Date: January 6, 2005
To: The Commissioner
From: Inspector General
Subject: Performance Indicator Audit: General Observations (A-15-05-25096)
We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 16 of the Social
Security Administration's performance indicators established to comply with
the Government Performance and Results Act. For each performance indicator,
PwC's objectives were to:
Test critical controls over the data generation and calculation processes for
the specific performance indicator;
Assess the overall adequacy, accuracy, reasonableness, completeness, and consistency
of the performance indicator and supporting data; and
Determine if each performance indicator provides meaningful measurement of the
program and the achievement of its stated objectives.
The evaluation of the 16 performance indicators resulted in 7 audit reports.
The seven reports are listed below.
Report Title CIN
Employment for Disabled Beneficiaries A-02-04-14068
Earnings Suspense File A-15-04-14069
President's Management Agenda Related Initiatives A-15-04-14070
Management Information Systems Development and Protection A-15-04-14071
Processing Time A-02-04-14072
Productivity A-15-04-14073
Disability Determination Services Net Accuracy Rate A-15-04-14074
During the audits of 16 performance indicators included in the Agency's Fiscal
Year 2003 Performance and Accountability Report, PwC identified several areas
for improvement across multiple indicators. The findings noted in this report
apply to the overall management of the performance indicator process for the
16 performance indicators reviewed.
Please provide within 60 days a corrective action plan that addresses each recommendation.
If you wish to discuss the final report, please call me or have your staff contact
Steven L. Schaeffer, Assistant Inspector General for Audit, at
(410) 965-9700.
Patrick P. O'Carroll, Jr.
Attachment
MEMORANDUM
Date: December 22, 2004
To: Inspector General
From: PricewaterhouseCoopers LLP
Subject: Performance Indicator Audit: General Observations (A-15-05-25096)
The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.
OBJECTIVE
We audited 16 of SSA's performance indicators reported in the Agency's Fiscal Year (FY) 2003 Performance and Accountability Report (PAR). For each performance indicator audited, our objectives were to:
1. Test critical controls over the data generation and calculation processes
for the specific performance indicator.
2. Assess the overall adequacy, accuracy, reasonableness, completeness, and
consistency of the performance indicator and supporting data.
3. Determine if each performance indicator provides meaningful measurement of
the program and the achievement of its stated objectives.
In addition to completing the objectives noted above, we also identified areas for improvement to the overall SSA process over performance indicators. This report contains our observations related to the overall management, documentation, and reporting of SSA performance indicators.
BACKGROUND
Performance indicator measurement at SSA is the process that provides multiple points of performance information for both external and internal Agency reporting needs. Information on performance indicator results has been integrated into SSA's PAR. The information reported in the PAR is used to help manage SSA operations by providing the Agency with a process to show current performance in relation to past performance and project future performance for the next several years. As a result, SSA has been able to use the information gained from monitoring the performance indicators to plan for future Agency actions. In its FY 2003 PAR, SSA reported the results of 46 performance indicators.
GPRA and the Office of Management and Budget (OMB) require SSA (and other Federal agencies) to report on the performance of programmatic functions. OMB holds SSA accountable for performance indicator variances, and budgetary decisions are based in part on the information reported in the PAR. The intent of reporting the results of the performance indicators in the PAR is to inform OMB, Congress, and the United States public of the performance results of the overall SSA mission.
The Office of the Chief Strategic Officer/Office of Strategic Management (OCSO/OSM) is responsible for coordinating with each indicator "owner" to support the development, monitoring and reporting of performance indicators. OCSO/OSM also encourages SSA's managers to use performance indicator information as a part of their ongoing strategic and operational planning activities. The individual owners are responsible for ensuring that the progress of the indicator is tracked and reported on a periodic basis.
RESULTS OF REVIEW
We identified several areas for improvement during our audits of 16 performance indicators included in SSA's FY 2003 PAR. A number of our findings relate to the overall management of the performance indicator process for the 16 performance indicators we reviewed. These include:
SSA has not consistently developed or documented detailed policies, procedures,
and controls to collect, review, and report information for the individual performance
indicators. For the indicators included in our audits, SSA did not consistently
document the detailed responsibilities of the indicator owners, data sources,
data interfaces, data modifications, or controls to ensure that performance
indicator data were complete, accurate, valid, and that an audit trail of data
and calculations of indicator results were maintained. We identified this issue
in 13 of the 16 indicators reviewed. In addition, we found that SSA employees
and contractors had excessive access rights defined within the Top Secret security
software, which allowed update and create access to the mainframe datasets used
to calculate performance indicators. We identified this issue in 6 of the 16
indicators reviewed.
SSA had not ensured that the disclosures in the PAR clearly linked each performance
indicator to the Agency's strategic goals and objectives and described each
performance indicator completely, accurately, and consistently. In addition,
for many of the indicators, the narrative information included within the PAR
did not consistently provide clear linkage between the indicators and the Agency's
strategic goals and objectives. We identified this issue in 11 of the 16 indicators
reviewed.
In addressing these findings in our reports, we noted that responsibility for policies, internal controls and disclosures related to individual performance indicators is distributed among the various designated performance indicator "owners" and business units. SSA has created the OCSO/OSM to coordinate and facilitate the process of gathering the performance indicator information and results for internal and external reporting. The purpose of OCSO/OSM as stated on OSM's web site is to "Promote the strategic management of SSA's programs, resources and service delivery to improve SSA Mission performance."
However, SSA has not formally designated the Chief Strategic Officer or another single Deputy Commissioner with the responsibility to ensure that performance indictor internal controls are consistently designed and implemented, indicator results are validated, or that performance indicator policies and procedures are created and enforced. Furthermore, a single Deputy Commissioner has not been designated to ensure that the reported indicators are meaningful, appropriately linked to the mission of SSA, or developed in a manner that highlights the ultimate outcome and results of SSA's performance. OMB Circular Number A-123 Management Accountability and Control requires agencies and individual Federal managers to take systematic and proactive measures to develop and implement appropriate, cost-effective management controls for results-oriented management.
RECOMMENDATION
We recommend SSA formally designate responsibility to ensure performance indicator policies and internal controls are designed and implemented across the Agency. SSA should consider designating a single Deputy Commissioner to be responsible for ensuring indicator results are validated, and that overall performance indicator internal controls are designed and implemented. In addition, a single Deputy Commissioner should ensure that the reported indicators are meaningful, appropriately linked to the mission of SSA, and developed in a manner that highlights the ultimate outcome and results of SSA's performance.
This designated individual(s) can be responsible for ensuring that SSA develops and documents policies and procedures used to prepare and disclose the results of the individual performance indicators on a consistent basis. Such documentation should include the detailed responsibilities of the indicator owners, data sources, data interfaces, data modifications, and controls in place to ensure that performance indicator data are complete, accurate, and valid; that an audit trail of data and calculations of indicator results are maintained; and that access to data used to calculate indicator results is appropriately restricted.
The designated individual can also require that indicator owners ensure: each performance indicator is fully described in a complete, accurate, and explicit manner within the PAR; the owners provide a clear linkage between the indicators and the Agency's strategic goals and objectives; and indicator titles, trend information, and data definitions are consistent within the PAR.
SSA COMMENTS
SSA believes that performance management leadership already exists under the Office of the Chief Strategic Officer. In addition, it believes that core responsibility for performance indicators should rest with the Deputy Commissioners as sponsors or owners of the indicators since they are responsible for achieving them. SSA stated that it believed that the FY 2004 PAR included clear descriptions and definitions of each of its performance indicators, including a description of the linkages between the indicators and its strategic goals.
PwC RESPONSE
We agree SSA has taken steps to improve the descriptions of its indicators
and their linkages to its strategic goals. However, we continue to believe the
designation of a single Deputy Commissioner, with responsibility for enforcement
of policies and procedures, internal controls, and disclosure and validation
of results, would improve SSA's performance management process. During our review
of FY 2004 performance measures, we will assess SSA's progress in ensuring its
indicators are meaningful, linked to the mission of the Agency, and developed
to highlight clearly SSA's performance. We will also assess the consistency
and quality of documentation and indicator results reported in SSA's FY 2004
PAR.
Appendices
APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Agency Comments
Appendix A
Acronyms
DDS Disability Determination Services
FY Fiscal Year
GPRA Government Performance and Results Act
OCSO/OSM Office of the Chief Strategic Officer/Office of Strategic Management
OMB Office of Management and Budget
PAR Performance and Accountability Report
SSA Social Security Administration
SSI Supplemental Security Income
Appendix B
Scope and Methodology
We updated our current understanding of the Social Security Administration's
(SSA) Government Performance and Results Act (GPRA) processes. This was completed
through research and inquiry of SSA management. We also requested SSA to provide
various documents regarding the specific programs being measured as well as
the specific measurement used to assess the effectiveness and efficiency of
the related program.
Through inquiry, observation, and other substantive testing including testing of source documentation, we performed the following as applicable:
Reviewed prior SSA, Government Accountability Office, and other reports related
to SSA GPRA performance and related information systems.
Met with the appropriate SSA personnel to confirm our understanding of each
individual performance indicator.
Flowcharted the processes.
Where applicable, we tested key controls related to manual or basic computerized
processes (e.g., spreadsheets, databases, etc.).
Conducted and evaluated tests of the automated and manual controls within and
surrounding each of the critical applications to determine whether the tested
controls were adequate to provide and maintain reliable data to be used when
measuring the specific indicator.
For those indicators with results that SSA determined using computerized data,
we assessed the completeness and accuracy of that data to determine the data's
reliability as it pertains to the objectives of the audit.
Identified and extracted data elements from relevant systems and obtained source
documents for detailed testing selections and analysis.
Identified attributes, rules, and assumptions for each defined data element
or source document.
Tested the adequacy, accuracy, reasonableness, consistency, and completeness
of the selection.
Recalculated the metric or algorithm of key performance indicators to ensure
mathematical accuracy.
As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency's mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency's mission, goals, objectives, and processes were used to determine if the performance indicators being used appear to be valid and appropriate given our understanding of SSA's mission, goals, objectives and processes. We followed all performance audit standards.
We audited the following performance indicators in the SSA Fiscal Year 2003 Performance and Accountability Report:
1. Percent of commercial positions competed or converted.
2. "Get to Green" on all President's Management Agenda initiatives.
3. Percent improvement in the retention rate.
4. Provide the equivalent of 40 hours of training annually to all employees.
5. Maintain zero outside infiltrations of SSA's programmatic mainframes.
6. Milestone measures for Managerial Cost Accountability System and Social Security
Unified Measurement Systems.
7. Milestones in developing new performance management systems (Implement new
Senior Executive Service system).
8. Disability Determination Services (DDS) net accuracy rate (allowances and
denials combined).
9. Percent increase in the number of Supplemental Security Income (SSI) disabled
beneficiaries earning at least $100 per month.
10. Reduction in the size of the earnings suspense file.
11. Percent of incoming earnings items removed from the suspense file at the
end of the annual earnings posting cycle.
12. Average processing time for initial disability claims.
13. Average processing time for all hearings.
14. Percent improvement in productivity.
15. DDS cases processed per work year.
16. SSI Aged claims processed per work year.
Appendix C
Agency Comments
SOCIAL SECURITY
MEMORANDUM
December 20, 2004
To: Patrick P. O'Carroll, Jr.
Inspector General
From: Larry W. Dye
Chief of Staff
Subject: Office of the Inspector General (OIG) Draft Report "Performance Indicator Audit: General Observations" (A-15-05-25096)--INFORMATION
We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.
Please let me know if you have any questions. Staff inquiries may be directed to Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.
Attachment:
SSA Response
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: GENERAL OBSERVATIONS" (A 15-05-25096)
Thank you for the opportunity to review and comment on the draft report. Our achievement of the American Association of Government Accountant's Certificate of Excellence in Accountability Reporting every year since the award program began in fiscal year (FY) 1998, and the attainment of a "green status" in the area of financial management on the FY 2004 President's Management Agenda Scorecard clearly establishes that our financial statements, annual performance plans and reports, and individual performance measures are comprehensive and accurate. The realization of both of these accomplishments was achieved through the Social Security Administration's (SSA) and the OIG staff's continuous efforts and diligence in evaluating and improving our performance management and reporting processes. We appreciate your providing us your observations from the reviews conducted in FY 2004 and our response to the report contents and specific recommendations are provided below.
General Comments
Regarding the statement that SSA has not consistently developed or documented detailed policies, procedures, and controls to collect, review, and report information for the individual performance indicators, it should be noted that during the FY 2004 key performance indicator and Government Performance Results Act audit process, components spent a lot of time preparing the required documentation based on guidance received from PricewaterhouseCoopers (PwC). Now that we are aware of the type of documentation that is being required, we anticipate the next audit cycle will produce more consistent documentation. However, we also need to balance the degree of effort required to maintain an audit trail of data and calculations for indicator results. In some cases it may be cost prohibitive to maintain an audit trail of data and calculations. In those cases, we will continue to work with PwC to develop a documentation approach that will enhance the Agency's performance measure reporting, while assisting the auditors in understanding the processes used to arrive at quantitative data supporting the Agency's annual performance outcomes.
PwC found that SSA had not ensured that the disclosures in the Performance and Accountability Report (PAR) clearly linked each performance indicator to the Agency's strategic goals and objectives or described each performance indicator completely, accurately, and consistently. In addition, the narrative information included within the PAR for many of the performance indicators did not consistently provide clear linkage between the indicators and the Agency's strategic goals and objectives. These issues were addressed in the FY 2004 PAR, as each performance measure includes a narrative that describes how the measure links to the Agency's strategic goals and objectives.
Recommendation Comments
Regarding the recommendation that we formally designate responsibility to ensure performance indicator policies and internal controls are designed and implemented across the Agency, the Office of the Chief Strategic Officer (OCSO) was established to direct the administration of the Agency's comprehensive management programs including competitive sourcing, quality management, strategic planning and workforce analysis. They accomplish their mission by collaborating and coordinating with Agency components to address cross-cutting programmatic and administrative issues that include promoting the strategic management of SSA's programs, resources, and service delivery to improve mission performance. OCSO has a formal Agency planning process in place, as it prepares the Strategic Plan, Annual Performance Plan and the Annual Performance Report in the PAR. They also track the results of performance goals in SSA's Tracking Report. We believe performance indicator management leadership already exists under the Office of Chief Strategic Officer.
Regarding the recommendations that we designate a single Deputy Commissioner
to:
1) Ensure indicator results are validated, and that overall performance indicator
internal controls are designed and implemented; 2) ensure that the reported
indicators are meaningful, appropriately linked to the mission of SSA, and developed
in a manner that highlights the ultimate outcome and results of SSA's performance;
and 3) ensure that SSA develops and documents policies and procedures used to
prepare and disclose the results of the individual performance indicators on
a consistent basis. We believe that core responsibility for performance measures
should rest with the Deputy Commissioners as sponsors or owners of the measures
since they are responsible for achieving them.
Finally, regarding the recommendation that would require indicator owners to: 1) Ensure that each performance indicator is fully described in a complete, accurate, and explicit manner within the PAR; 2) provide a clear linkage between the indicators and the Agency's strategic goals and objectives; 3) ensure that indicator titles, trend information, and data definitions are consistent within the PAR. As stated above, this issue was addressed in the FY 2004 PAR as each performance measure includes a narrative that describes how the measure links to the Agency's strategic goals and objectives.
[The Agency also provided a technical comment which has been addressed in the
report.]
Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations
(OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General
(OCCIG), and Office of Executive Operations (OEO). To ensure compliance with
policies and procedures, internal controls, and professional standards, we also
have a comprehensive Professional Responsibility and Quality Assurance program.
Office of Audit
OA conducts and/or supervises financial and performance audits of the Social
Security Administration's (SSA) programs and operations and makes recommendations
to ensure program objectives are achieved effectively and efficiently. Financial
audits assess whether SSA's financial statements fairly present SSA's financial
position, results of operations, and cash flow. Performance audits review the
economy, efficiency, and effectiveness of SSA's programs and operations. OA
also conducts short-term management and program evaluations and projects on
issues of concern to SSA, Congress, and the general public.
Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste,
abuse, and mismanagement in SSA programs and operations. This includes wrongdoing
by applicants, beneficiaries, contractors, third parties, or SSA employees performing
their official duties. This office serves as OIG liaison to the Department of
Justice on all matters relating to the investigations of SSA programs and personnel.
OI also conducts joint investigations with other Federal, State, and local law
enforcement agencies.
Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters,
including statutes, regulations, legislation, and policy directives. OCCIG also
advises the IG on investigative procedures and techniques, as well as on legal
implications and conclusions to be drawn from audit and investigative material.
Finally, OCCIG administers the Civil Monetary Penalty program.
Office of Executive Operations
OEO supports OIG by providing information resource management and systems security.
OEO also coordinates OIG's budget, procurement, telecommunications, facilities,
and human resources. In addition, OEO is the focal point for OIG's strategic
planning function and the development and implementation of performance measures
required by the Government Performance and Results Act of 1993.