A-02-02-12050 Alternate Format

MEMORANDUM

Date: April 30, 2003

To: The Commissioner

From: Inspector General

Subject: Assessment of the Social Security Administration's Performance Measures (A-02-02-12050)

Attached is a copy of our final report. Our objective was to determine whether the Social Security Administration's key programs and activities critical to achieving its strategic goals were addressed by its Fiscal Year 2002 performance indicators, and, if so, whether those indicators were objective, understandable, and outcome-based.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

ASSESSMENT OF THE SOCIAL
SECURITY ADMINISTRATION'S
PERFORMANCE MEASURES

April 2003

A-02-02-12050

AUDIT REPORT

Mission

We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.

Executive Summary
OBJECTIVE

Our objective was to determine whether the Social Security Administration's (SSA) key programs and activities critical to achieving its five strategic goals were addressed by its Fiscal Year 2002 performance indicators, and, if so, whether those indicators were objective, understandable, and outcome-based.

BACKGROUND

The Government Performance and Results Act (GPRA) of 1993 seeks to improve the internal management of the Government, as well as program effectiveness and public accountability, by promoting a new focus on results, service quality, and customer satisfaction. The Office of Management and Budget, Congress, and the President have called for high-quality, outcome-based performance indicators and goals. For example, the President's Management Agenda calls for the integration of performance review and budget formulation. As part of this integration process, agencies are being asked to "identify high quality outcome measures" that will be used to accurately monitor program performance.

RESULTS OF REVIEW

SSA has made great strides in meeting the objectives of GPRA and has shown continued refinements of its performance indicators from year to year. While SSA has improved the quality of its performance indicators over time, further refinements would create better indicators. We evaluated SSA's performance indicators to determine whether they met our criteria for being objective, understandable, and outcome-based. We found that the Agency's performance indicators are:

objective in all 11 key areas;

understandable in 10 of 11 key areas; and

partially outcome-based in 10 key areas and fully outcome-based in 1 key area.

CONCLUSIONS AND RECOMMENDATIONS

With further refinements, SSA can create indicators that would be more useful in measuring those key programs and activities where we identified issues. (See Appendix A for specific suggestions relating to individual performance indicators.)

AGENCY COMMENTS

SSA was pleased that we found the performance indicators for 11 key programs and activities objective and that we found the indicators for all but 1 key program or activity understandable. In response to our finding that SSA's performance indicators were only partially outcome-based for 10 of the 11 key programs and activities, the Agency stated it is not necessarily appropriate or feasible to have a majority of its indicators outcome-based. In response to our recommendations, SSA identified selected workloads for which efficiency indicators were found to be appropriate. SSA disagreed with several of our recommendations and stated it met the intent of GPRA in developing high-level, representative performance indicators-including outcome-based indicators for its major programs. (See Appendix B for the full text of SSA's comments.)

OFFICE OF THE INSPECTOR GENERAL RESPONSE

We appreciate SSA's comments and the continuing dialogue we have had with SSA staff on measuring SSA's performance. Through our audit work and the development of our recommendations, we intended to highlight opportunities for the Agency to improve the measurement of its key programs and activities. While we agree a balanced set of measures can include both output and outcome measures, we encourage SSA to develop additional outcome-based measures-when such measures will provide a better measure of the Agency's ability to meet its goals. We do not believe, nor advocate, that SSA establish efficiency indicators for every discrete workload. Our report focused on the workloads that are critical to achievement of SSA's goals. We continue to believe the implementation of our recommendations would enhance SSA's ability to fully measure and communicate on the obtainment of Agency goals. (See Appendix A for our specific responses to each of those recommendations.)

Table of Contents
Page
INTRODUCTION 1
RESULTS OF REVIEW 4
SSA's Key Programs and Activities 4
CONCLUSIONS AND RECOMMENDATIONS 9

APPENDICES
Appendix A - Analysis of the Social Security Administration's Key Programs and Activities and Associated Fiscal Year 2002 Performance Indicators
Appendix B - Agency Comments
Appendix C - OIG Contacts and Staff Acknowledgments

Acronyms
APP Annual Performance Plan
CDR Continuing Disability Review
DDS Disability Determination Services
DI Disability Insurance
ESF Earnings Suspense File
FY Fiscal Year
GAO General Accounting Office
GPRA Government Performance and Results Act
IVT Interactive Video Teletraining
MEF Master Earnings File
OASDI Old-Age, Survivors and Disability Insurance
OASI Old-Age and Survivors Insurance
OHA Office of Hearings and Appeals
OIG Office of the Inspector General
OMB Office of Management and Budget
RSI Retirement and Survivors Insurance
SSA Social Security Administration
SSI Supplemental Security Income
SSN Social Security Number
TY Tax Year

Introduction
OBJECTIVE

Our objective was to determine whether the Social Security Administration's (SSA) key programs and activities critical to achieving its five strategic goals were addressed by its Fiscal Year (FY) 2002 performance indicators, and, if so, whether those indicators were objective, understandable, and outcome-based.

BACKGROUND

Office of Management and Budget (OMB) guidance states that performance indicators should inform the President, Congress, other interested parties, and the public of the expected level of achievement for the program or activity. OMB also indicates that the performance goals and indicators to be included in an annual performance plan should be centered on a program's core purpose and be a blend of outcomes and outputs supporting outcomes.

Congress has stated that annual goals should be quantified as outcomes, rather than outputs. For example, a Senate Committee on Governmental Affairs report states "…agency leaders and managers should, where possible and practical, use outcome-oriented goals and measures that demonstrate how well a program or activity is doing in achieving its intended results."

The President has also indicated the need for outcome-oriented performance indicators and goals. The President's Management Agenda calls for the integration of performance review and budget decisions. "This integration is designed to begin to produce performance-based budgets starting with the 2003 Budget submission. Over time, agencies will be expected to identify high quality outcome measures, accurately monitor the performance of programs, and begin integrating this presentation with associated cost."

A General Accounting Office (GAO) guide states that outputs can be defined as the direct products and services delivered by a program, while outcomes are defined as the results of those products and services. Outcomes are linked to a program's overall mission-the program's reason for existing.

SCOPE AND METHODOLOGY

To assess SSA's performance indicators and its coverage of the key programs and activities, we:

Reviewed relevant Annual Performance Plan (APP) related documents, including SSA's:

strategic plan-Mastering the Challenge, Strategic Plan 2000-2005-which sets forth and defines SSA's strategic objectives and related program activities;

initial and revised FY 2002 APPs, which define its annual performance indicators and goals; and

initial FY 2003 APP to note any relevant changes from the revised FY 2002 APP.

Reviewed the Office of the Inspector General's (OIG) FY 2001 and FY 2002 Major Management Challenge Statements to determine whether the issues identified had adequate performance indicator coverage.

Reviewed prior OIG, PricewaterhouseCoopers LLP, and GAO reports related to SSA's performance indicators.

Reviewed relevant guidance from OMB, GAO, and academic sources to develop criteria for defining objective, understandable, and outcome-based performance indicators.

Assessed the 84 performance indicators in the revised FY 2002 APP and assigned them to 11 key programs and activities that are critical to SSA's delivery of its day-to-day services and achievement of its 5 strategic goals (see Appendix A).

Reviewed each of the 84 performance indicators and associated key programs and activities to determine whether the individual indicators met the established criteria for being objective, understandable, and outcome-based.

Identified key outcomes or results such as accuracy, timeliness, efficiency, or other appropriate measures of program performance for which SSA did not have performance indicators.

Discussed the development of SSA's performance indicators for FYs 2002, 2003 and 2004 with SSA staff.

Determined whether each of the key programs and activities were fully measured with regard to program results. A program or activity was determined to be fully measured if SSA's performance indicators addressed all critical elements of that program or activity.

Identified and commented on opportunities to use outcome-based indicators to assist SSA in increasing its focus on program results.

We performed our audit in New York and at SSA Headquarters in Baltimore, Maryland. We conducted our review between November 2001 and November 2002. The entity audited was the Office of Strategic Management. Our audit was performed in accordance with generally accepted government auditing standards.

Results of Review

SSA has made great strides in meeting GPRA's objectives and has shown continued refinement of its performance indicators from year to year. While SSA has improved the quality of its performance indicators over time, further refinements would create more results-based indicators. SSA has 11 key programs and activities that are critical to delivery of its day-to-day services and achievement of its 5 strategic goals and has performance indicators to measure all of these key areas. We evaluated SSA's performance indicators to determine whether they met our criteria for being objective, understandable, and outcome-based. We then determined whether the indicators, as a group, fully measured the results of SSA's key programs and activities. We found that the key programs and activities were covered by performance indicators that are objective in all 11 key areas,

understandable in 10 of 11 key areas,

partially outcome-based in 10 key areas and fully outcome-based in 1 key area.

SSA's KEY PROGRAMS AND ACTIVITIES

SSA has five strategic goals that, according to its Strategic Plan, are intended to support its mission "To promote the economic security of the nation's people through compassionate and vigilant leadership in shaping and managing America's social security programs." The links between SSA's strategic goals and its key programs and activities are based on SSA's revised FY 2002 APP and are shown in Figure 1.

Figure 1: SSA's FY 2002 Strategic Goals and Their Links
to SSA's Key Programs and Activities
SSA's Strategic
Goals Key Programs and
Activities
Responsive Programs Policy Development, Research, and Program Evaluation
World-Class Service Enumeration
Claims Processing
Post-entitlement
800-Number Telephone Service
Customer Service
Program Integrity Enumeration
Earnings Processing
Combating Fraud, Waste, and Abuse
Debt Management
Valued Employees Human Capital
Public Understanding Informing the Public About the Social Security Program
In performing our review, we determined whether the indicators, as a group, fully measured the results of each of SSA's 11 key programs and activities. To that end, we evaluated all of the performance indicators in SSA's initial and revised FY 2002 APPs to determine whether they met each of the three criteria indicated below.

Objective - The performance indicator should be measurable, reliable, and verifiable with quantifiable goals.

Understandable - The performance indicator should be well-defined and clearly stated.

Outcome-based - The performance indicator should identify and measure the intended results, effects, or impact of the program and its goals.

As part of our assessment, we also identified key outcomes or results such as accuracy, timeliness, efficiency, or other measures of program performance for which SSA did not have performance indicators.

Our results are listed in Figure 2. Our detailed assessment of the performance indicators is shown in Appendix A. This assessment takes into consideration SSA's significant changes in its performance indicators for FYs 2003 and 2004.

Figure 2: OIG Assessment of SSA's Performance Measure Coverage of Key Programs and Activities

SSA's Key Programs and Activities Were SSA's Key Programs and Activities Covered by Performance Indicators that Met the Following Criteria?

Objective
Understandable
Outcome-Based
1. Enumeration Yes Yes Partially
2. Earnings Processing Yes Yes Partially
3. Claims Processing Yes Yes Partially
4. Post-entitlement Yes Partially Partially
5. 800-Number Telephone Service Yes Yes Partially
6. Informing the Public About the Social Security Program Yes Yes Yes
7. Combating Fraud, Waste, and Abuse Yes Yes Partially
8. Debt Management Yes Yes Partially
9. Policy Development, Research, and Program Evaluation Yes Yes Partially
10. Human Capital Yes Yes Partially
11. Customer Service Yes Yes Partially

SSA's key programs were covered by performance indicators, which for the most part, met the criteria for being objective and understandable. However, we found that the indicators for the post-entitlement area were partially understandable. For example, the indicators, representative payee actions, Old-Age, Survivors and Disability Insurance (OASDI) post-entitlement automation rates and Supplement Security Income (SSI) post-entitlement automation rates were not well-defined or clearly stated.

SSA's key programs and activities were covered by performance indicators that fully measured outcomes in 1 of the 11 key areas and partially measured outcomes in the remaining 10 areas. While we do not expect every performance indicator to be outcome-based, the outcomes of key programs and activities should be measured. Without a balanced set of indicators that fully measures key outcomes, SSA's performance reporting will not completely demonstrate whether its key programs and activities accomplish the results intended. Many of SSA's FY 2002 indicators focused on outputs or numerical workloads-the number of claims processed, telephone calls answered, clients served, SSNs issued, etc. Measuring outputs may help demonstrate whether the Agency is complying with congressional mandates or be useful for internal budgeting purposes. However, such outputs do not state results, such as a program or activity's accuracy, timeliness, effectiveness, efficiency, level of customer satisfaction, and/or overall impact. Without these types of indicators, it is difficult to judge the program or activities' performance.

For instance, we found the following issues.

Enumeration - SSA did not have indicators that measured the efficiency or productivity in processing SSN requests. In its initial FY 2002 APP, SSA noted its plan to develop efficiency indicators to measure this area. However, the Agency subsequently removed these proposed efficiency indicators from its FY 2003 APP. An example of an outcome-based performance measure in this area could be an indicator that evaluated the integrity of SSA's enumeration process, such as the extent of SSN misuse.

Claims Processing - SSA has performance indicators that measure the timeliness of initial disability claims processed, but they combine data for the Disability Insurance (DI) and Supplemental Security Income (SSI) disability programs. As we have noted in a prior report, combining the data from both programs does not accurately measure the performance of either disability program. SSA also does not measure the efficiency of its claims processing. An efficiency indicator could measure, for example, the staff time or personnel cost needed to process a claim. The initial FY 2002 APP stated that SSA planned to develop accuracy and efficiency indicators, which had not yet been defined, to monitor both processes. However, SSA removed these proposed indicators from its FY 2003 APP. SSA also did not have an indicator that measured the overall accuracy of disability payments throughout the entire disability determination process.

Post-entitlement - Two of nine indicators measuring SSA's post-entitlement activities were fully outcome-based, and two others were partially outcome-based. Our review of SSA's major post-entitlement workloads noted the following.

Continuing Disability Reviews - SSA did not have indicators related to the key outcome of continuing disability reviews (CDR), such as the percent of accurate CDR decisions. The development of an outcome-based indicator would increase accountability of the CDR process.

Representative Payment Program - SSA did not have any outcome-based indicators for its representative payment program. SSA had only one indicator for this program-an output measure that counts the number of representative payee actions. As we stated in prior reports, this indicator is vague and combines different types of representative payee activities, resulting in ambiguous information. Because the representative payment program is vulnerable to fraud and abuse, it is critical that SSA develop effective outcome-based measures to better monitor and evaluate the activity's integrity. For example, SSA could develop an indicator to measure the number of representative payees found in compliance with SSA's reporting requirements.

Non-Disability Redeterminations - As we noted in a prior report, SSA had not developed outcome-based indicators to measure the results of its redetermination workloads. SSA should develop indicators that measure the results of this process, such as the dollar amount of over- or underpayments made to SSI claimants and the number of claimants properly removed from the SSI program as a result of the redetermination process. Although SSA tracks such data internally through its annual Index of Dollar Accuracy surveys, it could include performance indicators in its APP to report these results. SSA also could develop efficiency or productivity indicators to measure the administrative costs of performing redeterminations compared to the actual dollar amounts collected and/or saved.

800-Number Service - Only one of four indicators in this area was intended to measure outcomes. This one indicator alone is not sufficient to provide a complete measure of the results of SSA's 800-number service. The remaining three performance indicators were output or workload indicators. As we have indicated in prior reports, while workload indicators may be helpful in tracking the volume of 800-number calls, SSA should develop more meaningful, results-based indicators concerning its 800-number service. For example, SSA could develop indicators for customer waiting time once connected or whether a caller hung up before being served once connected.

Debt Management - SSA's performance indicators for debt management were not outcome-based. SSA developed two indicators for this area-Outstanding OASDI debt in a repayment agreement, under appeal, or newly detected and Outstanding SSI debt in repayment agreement, under appeal, or newly detected. However, SSA did not define goals for these indicators or measure key results or outcomes, such as the amount or percentage of debt actually collected. In its FY 2003 APP, SSA modified both of these indicators to measure outstanding OASDI and SSI debt that was not in a collection arrangement. These indicators, however, still do not measure what should be a key objective of SSA's debt management efforts: the percent of SSA's outstanding debt subject to recovery that the Agency actually recovered. Also, SSA could develop an indicator measuring the percent of total debt written off because it was uncollectible.

Conclusions and Recommendations

SSA has made great strides in meeting the objectives of GPRA and shown continued refinements of its performance indicators from year to year. While SSA has improved the quality of its performance indicators over time, further refinements would create more useful indicators. See Appendix A for our recommendations relating to specific individual performance indicators. While our recommendations were initially based on our review of indicators appearing in the revised FY 2002 APP, we have recently been informed by SSA staff of changes in performance indicators in the draft FY 2004 APP. Given the changing environment of SSA's performance planning, we took these changes into consideration when formulating our recommendations. Specifically, we did not offer recommendations for indicators SSA staff told us would not be included in the FY 2004 APP.

AGENCY COMMENTS

SSA was pleased that we found the performance indicators for 11 key programs and activities objective and that we found the indicators for all but 1 key program or activity understandable. In response to our finding that SSA's performance indicators were only partially outcome-based for 10 of the 11 key programs and activities, the Agency maintained it is not necessarily appropriate or feasible to have a majority of its indicators outcome-based. In response to our recommendations, SSA identified selected workloads for which efficiency indicators were found to be appropriate. However, the Agency maintained it was not the intent of GPRA to have Agencies develop efficiency indicators or other outcome-based indicators for every discrete workload. SSA disagreed with several of our recommendations and stated it met the intent of GPRA in developing high-level, representative performance indicators-including outcome-based indicators for its major programs. (See Appendix B for the full text of SSA's comments.)

OIG RESPONSE

We appreciate SSA's comments and the continuing dialogue we have had with SSA staff on measuring SSA's performance. Through our audit work and the development of our recommendations, we intended to highlight opportunities for the Agency to improve the measurement of its key programs and activities. While we agree a balanced set of measures can include both output and outcome measures, we encourage SSA to develop additional outcome-based measures-when such measures will provide a better measure of the Agency's ability to meet its goals. We do not believe, nor advocate, that SSA establish efficiency indicators for every discrete workload. Our report focused on the workloads that are critical to achievement of SSA's goals.
GPRA specifically states its purpose is, in part, to "…improve Federal program effectiveness and public accountability…" and "…improve congressional decision making by providing more objective information on achieving statutory objectives and on the relative effectiveness and efficiency of Federal programs and spending." We contend the development of outcome-based and/or efficiency indicators for key programs and activities is in line with GPRA's intent, and is a critical step in meeting the President's Management Agenda goal of integrating budget and performance. We continue to believe the implementation of our recommendations would enhance SSA's ability to fully measure and communicate on the obtainment of Agency goals. (See Appendix A for our specific responses to those recommendations.)

Appendices

Appendix A
Analysis of the Social Security Administration's Key Programs and Activities and Associated Fiscal Year 2002 Performance Indicators
We have assessed the performance indicators related to each of the Social Security Administration's (SSA) 11 key programs and activities. In each area, we provide an overall Office of the Inspector General (OIG) conclusion. In addition, in those areas where we found SSA did not meet our criteria, we highlighted either an earlier recommendation to correct this problem or made a new recommendation. In some areas, while we may have found certain weaknesses in some performance indicators, we did not make specific recommendations if our overall conclusion was that the performance indicators adequately measured the key outcomes for the program or activity. In addition, we did not make recommendations for those performance indicators that SSA staff informed us would not be included in the Fiscal Year (FY) 2004 Annual Performance Plan (APP).

We evaluated all of the performance indicators in SSA's initial and revised FY 2002 APPs to determine whether they met each of the three criteria indicated below.

Objective - The performance indicator should be measurable, reliable, and verifiable with quantifiable goals.

Understandable - The performance indicator should be well-defined and clearly stated.

Outcome-based - The performance indicator should identify and measure the intended results, effects, or impact of the program and its goals.

ENUMERATION

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

The enumeration performance indicators measure accuracy and timeliness. However, these indicators either had data reliability problems or did not measure all relevant performance data. Additionally, SSA did not measure the efficiency of its enumeration process, that is, the amount of personnel time or cost required to process a Social Security number (SSN) request. In its initial FY 2002 APP, SSA noted its plan to develop efficiency indicators to measure this area. However, SSA subsequently removed these proposed efficiency indicators from its FY 2003 APP. Further, SSA did not have an indicator to measure the integrity of its enumeration process, despite our identifying SSA's vulnerability to SSN misuse as a major management challenge. We do recognize that measuring the extent of SSN misuse is difficult and would welcome an effort by SSA to quantify and measure this area.

We have previously recommended that SSA correct the weaknesses in data reliability identified for the indicator, Percent of original and replacement SSN cards issued within 5 days of receiving all necessary documentation, and take steps to either include all measurable items in the indicator, Percent of SSNs issued accurately, or clarify those items that are not being counted in the indicator. SSA has since disclosed in its APPs that SSNs assigned via the Enumeration-at-Birth process are excluded from this measure.

Performance Indicators
Addressing Enumeration
Objective

Understandable Outcome-based

OIG Comments
SSN requests processed Yes Yes NoThe indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of SSN requests processed.
Percent of original and replacement SSN cards issued within 5 days of receiving all necessary documentation Partially Yes Yes
The indicator is partially objective because of data reliability issues involving the accuracy of the performance data.
Percent of SSNs issued accurately Yes Yes Partially The indicator is partially outcome-based. As we have reported previously, it does not include a significant portion of the SSN population-SSNs issued at birth-about 20 percent of the population.

Recommendation 1
SSA should reinstate its plans to establish indicators to measure the efficiency of the enumeration process, such as the personnel costs of processing SSN requests.

Agency Comment
SSA stated it has established efficiency indicators, and GPRA's intent was not to have agencies develop efficiency indicators or other outcome-based measures for every discrete workload.

OIG Response
We do not believe SSA needs efficiency indicators for every discrete workload. Our report focused on the workloads that are critical to achieving SSA's goals, such as the enumeration process. We continue to believe that a measure of the costs associated with the enumeration process, such as the cost of production per SSN card, will help the Agency better align its performance and budget. We recognize SSA's prerogative in the development of a specific measure, but continue to believe that an efficiency measure is necessary to enhance the Agency's ability to fully measure and manage this critical workload.

EARNINGS PROCESSING

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

SSA has four performance indicators for Earnings Processing. Three do not fully meet the outcome-based criteria, and the fourth does not fully meet the understandability criteria. For example, the indicator, Percent of wage items posted to individuals' records by September 30th, did not fully meet our definition of outcome-based because it omitted self-employment earnings. We have previously recommended that SSA take action to either include all measurable items in the Percent of wage items posted to individuals' records by September 30th or clarify which measurable items were not included in the indicators in future APPs and performance reports. The Percent of W-2s filed electronically is a workload indicator and is not outcome-based.

In a previous report on the indicator, Percent of earnings posted correctly, we noted SSA did not report that it used Tax Year (TY) 1996 earnings data to arrive at the posting accuracy rate and then applied this rate to the FY 1999 earnings accuracy rate. We recommended that SSA fully disclose the time period of the source data used in the computation of the accuracy rate in all future performance reports. Further, SSA did not have an indicator to monitor and/or measure the Earnings Suspense File (ESF), despite our identifying the growth of the ESF as a major management challenge. We previously recommended that SSA develop performance indicators to monitor and measure reduction of the ESF's growth.

Performance Indicators
Addressing Earnings Processing
Objective

Understandable Outcome-based

OIG Comments
Percent of employee reports (W-2s) filed electronically Yes Yes No This indicator is not outcome-based because it does not measure the results, effects, or impact of the percent of W-2s filed electronically.
Annual earnings items processed Yes Yes No The indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the number of annual earnings items processed.
Percent of wage items posted to individuals' records by September 30th Yes Yes Partially This indicator is partially outcome-based because, as we have previously reported, it omits a key component of the earnings population from measurement: self-employment earnings.
Percent of earnings posted correctly Yes No Yes This indicator is not understandable because, as we have previously reported, it was not accurately defined in the FY 1999 APP. SSA has not yet corrected this problem in the FY 2002 APP. The APP does not disclose that the indicator's calculation is based on wages posted to both the Master Earnings File (MEF) and the ESF. The APP implies, erroneously, that the indicator is based only on earnings posted to the MEF. We also previously reported that SSA did not fully disclose the time period of the source data used in the computation of the accuracy rate.

Recommendation 2

SSA should define performance goals and develop indicators that measure SSA's intended goals in the earnings area, such as the anticipated improvement in wage processing efficiency when employee reports (W-2s) are filed electronically.

Agency Comment

SSA agreed with Recommendation 2.

CLAIMS PROCESSING

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

While SSA has developed some outcome-based measures for claims processing in its Old-Age, Survivors and Disability Insurance (OASDI) and Supplemental Security Income (SSI) programs, the performance indicators do not fully measure the results of SSA's programs and activities in this area. Our conclusions are discussed below separately for each of the programs and activities related to claims processing.

OASDI Program

SSA has three performance indicators for the OASDI program, two of which are outcome-based-measuring accuracy and timeliness. We reported recently that the OASDI accuracy indicator overstates payment accuracy since SSA counts some overpayments as accurate when calculating this statistic. By definition, SSA excludes certain overpayments from its accuracy rate measure. For example, SSA does not include payments made after a beneficiary's death.

Performance Indicators
Addressing Claims Processing Objective
Understandable Outcome-based

OIG Comments
Retirement and Survivors Insurance (RSI) claims processed Yes Yes No

This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.
Percent of Old-Age and Survivors Insurance (OASI) claims processed by the time the first regular payment is due or within 14 days from the effective filing date, if later Yes Yes Yes This indicator is objective, understandable and outcome-based.
Percent of OASDI payment outlays "free" of over (under) payments (based on non-medical factors of eligibility) Yes Yes Partially This indicator is partially outcome-based because, as we have noted in recent audit work, it does not count all types of improper payments as "errors." This results in an overstatement of the OASDI payment accuracy rate.

SSI Program

SSA's performance indicators for the SSI aged program measure accuracy and timeliness. We reported recently that the SSI program accuracy indicator overstates payment accuracy since SSA counts some overpayments as accurate when calculating this statistic. By definition, SSA excludes certain overpayments from its accuracy rate measure.

Performance Indicators
Addressing Claims Processing
Objective

Understandable Outcome-based

OIG Comments
SSI aged claims processed Yes Yes No This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.
Percent of SSI aged claims processed by the time the first payment is due or within 14 days of the effective filing date, if later Yes Yes Yes This indicator is objective, understandable and outcome-based.
SSI overpayment and underpayment accuracy rate including both preventable and unpreventable errors (based on non-medical factors of eligibility) Yes Yes Partially These indicators are partially outcome-based because, as we have noted in recent audit work, they do not count all types of overpayments as "errors." This results in an overstatement of the SSI payment accuracy rate.
SSI overpayment and underpayment accuracy rate excluding unpreventable errors (based on non-medical factors of eligibility) Yes Yes Partially

Joint Performance Indicators

SSA has two performance indicators that address both of SSA's programs; however, neither of these indicators measure outcomes.

Performance Indicators
Addressing Claims Processing
Objective

Understandable Outcome-based

OIG Comments
Implement activities necessary to have the software and infrastructure in place for paperless processing of RSI and SSI aged claims Yes Yes No

This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the paperless processing activity and its goals.

Overpayment actions Yes No No

The indicator is not understandable because it is not well-defined or clearly stated. It is not outcome-based because it does not identify and measure the intended results, effects, or impact of overpayment actions.

Disability Insurance and SSI Disability Programs

SSA has several indicators that measure the timeliness of initial claims processing in its Disability Insurance (DI) and SSI disability programs. However, these indicators are of limited use because they combine data for both disability programs. Historically, it has taken SSA different lengths of time to process DI and SSI disability claims. We previously noted this issue and recommended that SSA develop separate indicators to measure claims processing times of DI and SSI initial disability claims rather than combining the data for both programs. SSA also does not have an indicator that measures the overall accuracy of disability payment outlays that would include determinations made at the initial Disability Determination Services (DDS) level, as well as decisions made at the Office of Hearings and Appeals (OHA) level. We also previously noted concerns in SSA's method of measuring the timeliness of processing DI claims. Specifically, the indicators did not measure the total time it takes an SSA claimant to receive a final determination, including decisions reached on appeal.

Performance Indicators
Addressing Claims Processing
Objective

Understandable Outcome-based

OIG Comments
Number of initial disability claims processed Yes Yes No

This indicator is not outcome-based because it does not specifically measure the intended results, effects, or impact of the disability claims processing activity and its goals. In particular, the indicator over-consolidates data by combining the number of DI and SSI disability claims processed and does not provide an accurate measure of performance for each of the two programs separately.
Percent of initial disability claims decisions issued within 120 days Yes Yes Partially These indicators are partially outcome-based because, as we have previously reported, they combine claims processing data from both the DI and SSI programs. This makes what is being measured less precise.
Initial disability claims average processing time (days) Yes Yes Partially
DDS net allowance accuracy rate Yes Yes Yes These indicators are objective, understandable, and outcome-based.
DDS allowance performance accuracy rate Yes Yes Yes
DDS denial performance accuracy rate Yes Yes Yes
DDS net denial accuracy rate Yes Yes Yes
Implement activities necessary to have the software and infrastructure in place for electronic processing of disability claims Yes Yes No
This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the electronic disability claims processing activity and its goals. Further, it does not track the progress made in implementing the paperless process.
Initial disability claims pending Yes Yes No This indicator is not outcome-based because it does not identify or measure a key intended outcome.

Hearings and Appeals Process

In the hearings and appeals area, SSA has 11 performance indicators, 6 of which measure results such as timeliness, accuracy, and efficiency. Two of the 11 indicators in this area had data reliability issues. We have previously recommended that SSA address the data reliability issues identified concerning performance indicators in the hearings and appeals process.

Performance Indicators
Addressing Claims Processing
Objective
Understandable Outcome-based

OIG Comments
Number of hearings processed Yes Yes No
This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.
Percent of hearing decisions issued within 180 days from the date the request is filed Partially Yes Yes
These indicators are partially objective because the performance data used to measure the hearing process is unreliable. The system used to measure performance did not contain all of the hearing requests in the year reviewed and data within the system were incorrect.
Hearings average processing time (days) Partially Yes Yes
Number of hearings pending Yes Yes No This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.
OHA decisional accuracy rate Yes Yes Yes
This indicator is objective, understandable and outcome-based.
Implement activities necessary to have the software and infrastructure in place for electronic processing of hearings and appeals Yes Yes No
This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the activity and its goals. This indicator does not measure the goal of electronic processing of hearings.
Number of hearing cases processed per workyear Yes Yes Yes
These indicators are objective, understandable, and outcome-based.
Percent of decisions on appeals of hearings issued by the Appeals Council within 105 days of the appeal filing date Yes Yes Yes
Average processing time for decisions on appeals of hearings issued (days) Yes Yes Yes
Number of decisions on appeals of hearings issued per work year Yes Yes Yes
Other appellate actions Yes No No
This indicator is not understandable because it is vague and not well-defined. It is not outcome-based because it does not identify and measure intended results, effects, or outcome of appellate actions or goals.

Recommendation 3
SSA should develop additional results-based performance indicators to measure the accuracy and efficiency of OASI and SSI aged claims processing, as was proposed in its original FY 2002 APP.

Agency Comment
SSA agreed with Recommendation 3.

Recommendation 4
SSA should count all types of improper payments as inaccurate when calculating its OASDI and SSI program payment accuracy rates.

Agency Comment
SSA deferred responding to Recommendation 4 pending the outcome of the discussion between the Agency and the Inspector General with respect to the definition of an erroneous payment.

Recommendation 5
SSA should develop an indicator that measures the overall accuracy of decisions for all DI payment outlays, which takes into account initial DDS, as well as OHA decisions.

Agency Comment
SSA disagreed with our recommendation. SSA stated that fundamental differences in the decisionmaking process at the initial claim and appeal levels would result in an inaccurate measure if the processes were combined in one indicator.

OIG Response
We disagree with SSA's response. A claim is a distinct process with a beginning and an end. Splitting this process based on different steps the Agency takes for performance measuring purposes is an artificial distinction. From a claimant's point of view, a claim that was accurate at the DDS level, but inaccurate at the OHA level, is an inaccurate claim. While the individual indicators of accuracy for different processes provide useful information on SSA processes, they do not provide an overall assessment of the accuracy of disability claims processed by SSA.

Recommendation 6
SSA should develop an indicator that measures the timeliness of DI claims processing so that the total amount of time it takes an SSA claimant to receive a final decision-including decisions reached on appeal-is measured.

Agency Comment
SSA agreed with Recommendation 6.

POST-ENTITLEMENT

OIG Conclusion: The performance indicators are objective, partially understandable, and partially outcome-based.

While the performance indicators for measuring post-entitlement programs were objective, they were partially understandable for measuring the Representative Payee Program and post-entitlement automation rates in the OASDI and SSI programs. The performance indicators did not measure outcomes for Continuing Disability Reviews (CDR), SSI Non-Disability Redeterminations, the Representative Payee Program, and post-entitlement in the title II and XVI programs. Our conclusions are discussed below separately for each of the programs and activities related to post-entitlement workloads.

Continuing Disability Reviews

SSA's CDR performance indicators concentrate mainly on monitoring workloads. SSA does not have indicators to measure the intended outcomes of CDRs, such as the accuracy of payments to beneficiaries.

Performance Indicators
Addressing
Post-Entitlement
Objective

Understandable Outcome-based

OIG Comments
CDRs processed Yes Yes No These indicators are not outcome-based because they do not identify and measure the intended results, effects, or impact of the program and its goals.
Percent of multi-year CDR plan completed through FY 2002 Yes Yes No
SSI Non-Disability Redeterminations

SSA has one performance indicator for measuring this activity. SSA has not developed an indicator to measure the cost effectiveness of or the number of ineligible individuals terminated through its SSI redeterminations. We have previously recommended that SSA develop an indicator to measure the results of this process, such as the dollar amount of incorrect payments.

Performance Indicators
Addressing Post-Entitlement
Objective

Understandable Outcome-based

OIG Comments
SSI non-disability redeterminations processed Yes Yes No This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.

Representative Payment Program

For monitoring representative payees, SSA has only one indicator, and it does not measure results. In addition, we have identified this area as a management challenge-vulnerable to fraud and abuse. We have identified numerous weaknesses in SSA's oversight of representative payees, including their selection, monitoring, and accountability. We have reported that SSA should develop measures to report on the Agency's efforts to screen potential representative payees and monitor their activities and the extent of representative payee misuse, although measuring the extent of fraud in this area is difficult.

Performance Indicators
Addressing Post-
Entitlement Objective
Understandable Outcome-based

OIG Comments
Representative payee actions Yes No No

The indicator is not understandable because it is not well-defined or clearly stated. It is not outcome-based because it does not identify and measure the specific intended results, effects, or impact of the program and its goals. In particular, the indicator combines different types of representative payee activities. We have noted in a prior report that the "…results of the measure are ambiguous because the inputs are obtained from such diverse activities."

Post-entitlement in the Title II and XVI Programs

The two indicators in this area are not understandable since it is unclear as to what is included in the measurements.

Performance Indicators
Addressing Post-
Entitlement Objective
UnderstandableOutcome-based

OIG Comments
OASDI post-entitlement automation rate Yes No Yes These indicators are not understandable because they are vague and poorly defined.
SSI post-entitlement automation rate Yes No Yes

Disability/Return to Work Program

SSA has developed three performance indicators in this area, two of which are expressed as outcomes. However, these indicators do not provide a complete picture of the actual success of the Return to Work program since they do not measure the percent of DI and SSI disabled beneficiaries who have returned to work.

Performance Indicators
Addressing Post-
Entitlement
Objective

Understandable Outcome-based

OIG Comments
Percent increase in the number of DI adult worker beneficiaries who begin a trial work period Yes Yes Partially These indicators are partially outcome-based. While they measure certain immediate outcomes, they do not fully measure the percent of disabled beneficiaries who have returned to work.
Percent increase in the number of SSI disabled beneficiaries, aged 18-64, whose payments are reduced because of work (that is, participating in 1619(a) status) Yes Yes Partially
Activities to implement provisions of the Ticket to Work and Self-Sufficiency Programs and other employment strategies Yes Yes No
This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program that are being sought.

Recommendation 7
SSA should, for CDRs, develop more outcome-oriented indicators that measure the key outcomes of this process, mainly the accuracy of CDR decisions.

Agency Comment
SSA questioned the full extent of Recommendation 7. The Agency stated it will consider exploring this recommendation if it is limited to "…the accuracy of CDR decisions," and if the recommendation is to establish an accuracy standard related to CDR decisions similar to that for initial claims.

OIG Response
The intent of our recommendation was to establish an accuracy standard related to CDR decisions, similar to that for initial claims. Therefore, we continue to recommend that SSA develop an indicator that measures the key outcomes of the CDR process.

Recommendation 8
SSA should, for the representative payment program (a) revise its current indicator to, at a minimum, separately track representative payee changes and accounting forms; and (b) develop indicators to measure the integrity of the representative payee process, the extent of misuse, and/or its efforts to screen and monitor representative payees. For example, an indicator could measure the percent of representative payee accounting forms found to be in compliance with SSA's regulations by implementing a quality assurance review of a sample of forms.

Agency Comment
SSA disagreed with Recommendation 8. The Agency stated it would need more information about the anticipated benefits of quality assurance reviews of representative payee accounting forms to determine the feasibility of implementing the recommendation since establishing an ongoing quality review of this type would be resource intensive.

OIG Response
Our proposed representative payment program performance indicator was only an example to demonstrate the type of indicator necessary to measure the stewardship of funds managed by representative payees. We note the Agency reviews fee-for-service payees, organizational payees serving 100 or more beneficiaries, and individual payees serving 20 or more beneficiaries on a 3-year cycle. The Agency could develop an indicator that reports the results of these activities. While we offered a suggestion that SSA could measure accounting form compliance, we note the Agency's prerogative to develop a different indicator to measure the integrity of the representative payment program.

Recommendation 9
SSA should, for OASDI post-entitlement actions, develop indicators to provide more outcome-based data, such as measuring processing time, accuracy, and efficiency.

Agency Comment
SSA stated it has established efficiency indicators, and GPRA's intent was not to have agencies develop efficiency indicators or outcome-based measures for every discrete workload. The Agency also stated it has established a goal to improve overall productivity by 2 percent each year.

OIG Response
We note SSA's progress in developing an overall productivity goal. Measuring overall productivity from year to year will allow the Agency to see whether it is positioning itself to meet the challenge of its ever increasing workloads. We continue to believe that efficiency indicators for critical workloads will serve the same function. For example, a measurement of the costs incurred and benefits derived per redetermination over time will allow the Agency to see how well it is managing this critical workload, which helps to ensure SSA is paying the right amount of payments to its beneficiaries. We recognize SSA's prerogative in the development of a specific measure, but continue to believe that an efficiency measure is necessary to enhance the Agency's ability to fully measure and manage this critical workload.

Recommendation 10
SSA should, in the return to work area, develop indicators that measure the percent of DI and SSI disabled beneficiaries who have returned to work.

Agency Comment
The Agency stated it is considering two new performance indicators for the proposed FY 2004 APP that better reflect its efforts to return disabled beneficiaries to work: Percent increase in the number of DI and SSI beneficiaries, with tickets assigned who work and Percent increase in the number of SSI disabled beneficiaries earning at least $100.00 per month.

OIG Response
We note and are heartened by SSA's progress in developing performance indicators to better measure its efforts to return beneficiaries to work. However, we believe that additional indicators will enhance measurement of the Agency's ability to help disabled individuals gain employment. The best measure of SSA's efforts in this critical area would be to measure the percent of the individuals receiving disability benefits who the Agency has helped return to gainful employment. By measuring the "percent increase," one cannot determine the percentage of disabled beneficiaries who have returned to work-the goal of return-to-work efforts. For example, SSA's current indicators in this area could allow for a 100 percent or greater improvement in the number of individuals working, while masking the fact that SSA currently returns less than 1 percent of disabled beneficiaries to work.

800-NUMBER TELEPHONE SERVICE

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

Only one of four indicators used to measure the performance of SSA's 800-number telephone service is intended to fully measure outcomes; two other indicators partially measure outcomes; and one indicator is a workload indicator. While these indicators may help in tracking call volume and workloads, SSA does not have a sufficient set of indicators to provide a complete measure of all key outcomes relating to its 800 number service.

We have previously recommended that SSA develop indicators to measure all key results of its 800-number telephone service. We also recommended that SSA consider indicators that would effectively measure the quality of service received by 800-number callers.

Performance Indicators
Addressing 800-Number Telephone Service
Objective

Understandable Outcome-based

OIG Comments
800-number calls handled Yes Yes No

This indicator is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.
Percent of callers who successfully access the 800 number within 5 minutes of their first call Yes Yes Partially These indicators are partially outcome-based. While they measure certain immediate outcomes, they do not fully identify and measure the intended results, effects, or impact of successfully accessing the 800-number. In a prior review, we found that the 800 number performance indicators emphasized "access" rather than "service."15
Percent of callers who get through to the 800 number on their first attempt Yes Yes Partially

Percent of 800-number calls handled accurately Yes Yes Yes This indicator is objective, understandable and outcome-based.

INFORMING THE PUBLIC ABOUT THE SOCIAL SECURITY PROGRAM

OIG Conclusion: The performance indicators are objective, understandable, and outcome-based.

Performance Indicators
Addressing Informing the Public about the Social Security Program
Objective

Understandable Outcome-based

OIG Comments
Percent of public who are knowledgeable about Social Security issues Yes Yes Yes These indicators are objective, understandable and outcome-based.
Percent of individuals issued SSA-initiated Social Security Statements, as required by law Yes Yes Yes

COMBATING FRAUD, WASTE, AND ABUSE

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

SSA has four performance indicators related to reducing fraud, waste, and abuse-three of which are expressed as outcomes. However, two of the outcome measures, OASDI dollar amounts reported from investigative activities and SSI dollar amounts reported from investigative activities, partially measured the major intended results in this area, such as the dollar amounts collected from investigative activities. We previously recommended that SSA modify its data collection so it can record and report on the dollars collected as a result of its investigative activities. This type of proposed indicator would be better aligned under the debt management program area since it measures the collection of debt as opposed to an investigative or anti-fraud activity.

Performance Indicators
Addressing Combating Fraud, Waste, and Abuse
Objective

Understandable Outcome-based

OIG Comments
Number of investigations conducted (that is, closed) Yes Yes No
This indicator was not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.
OASDI dollar amounts reported from investigative activities Yes Yes Partially These indicators partially measure certain outcomes from investigative activities. However, they do not measure certain key results in this area such as the dollar amounts collected from investigative activities.
SSI dollar amounts reported from investigative activities Yes Yes Partially
Number of judicial actions reported Yes Yes Yes This indicator is objective, understandable, and outcome-based.

Recommendation 11
SSA should set-up a process to identify and quantify areas that are more susceptible to fraud, and establish performance indicators that would help reduce and measure the reduction of fraud in those areas.

Agency Comment
SSA disagreed with our recommendation. The Agency stated that, since the total universe of fraud is unknown, it cannot fully and accurately measure the total amount of fraud or identify the amount prevented and deterred. Further, SSA stated that measuring progress against an estimate would be problematic.

OIG Response
Based on the Agency's response, we revised our original recommendation. We believe that an appropriate step in addressing fraud would be to establish a process to estimate the amount and total number of improper payments in areas identified as more susceptible to fraud and then to take steps and measure progress through the creation of appropriate indicators in addressing those areas.

DEBT MANAGEMENT

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

SSA has two indicators to measure this area, but it has not developed goals for either of these indicators. These indicators are critical because they measure SSA's stewardship of both its programs. We found that both indicators fail to measure the amount of debt actually collected. Combining too many different categories of funds also weakens the indicators. SSA does not have performance indicators dealing with the amount of OASDI debt collected; amount of SSI debt collected; OASDI debt collected as a percent of outstanding OASDI debt; or SSI debt collected as a percent of outstanding SSI debt.

Performance Indicators
Addressing Debt Management
Objective

Understandable Outcome-based

OIG Comments
Outstanding OASDI debt not in a collection arrangement (excluding due process) Yes Yes Partially
These indicators are partially outcome-based. While the indicators identify and measure some aspects of program results, they do not measure the amount of debt actually collected or what the amount collected represents in terms of a percentage of the total outstanding debt owed to SSA. SSA modified these indicators in its FY 2003 APP to measure outstanding debt not in a collection arrangement. The revised indicators, however, do not measure the percent of SSA's outstanding debt recovered.
Outstanding SSI debt not in a collection arrangement (excluding due process) Yes Yes Partially

Recommendation 12
SSA should clearly establish performance goals that measure the intended results of its debt management activities, such as the percentage of SSA's outstanding debt the Agency is legally entitled to collect that is actually recovered.

Recommendation 13
SSA should consider developing an indicator to measure the percent of total debt written off because it is uncollectible.

Agency Comments
SSA disagreed with Recommendations 12 and 13. SSA stated it had already established clear debt management performance goals that measure the intended results of its activities. For example, SSA has two indicators that track the percentage of outstanding OASDI and SSI debt in a collection arrangement (excluding due process requests). Further, SSA believes its current debt performance indicators are outcome-based, as they will result in a quantifiable improvement in its debt management program.

OIG Response
We continue to believe SSA should develop performance indicators that measure the overall goals of its debt management efforts. The goal of SSA's debt collection activities is to prevent overpayments from occurring, and to collect overpayments once they do occur. We do not believe that a goal of SSA's debt management efforts is to create payment agreements. Agreements are only a tool used to collect debt, not a valid measure of the desired results of SSA's debt collection activities. Accordingly, we believe SSA's performance indicators in this area should guide Agency efforts to limit the amount of growth in improper payments and reduce the amount of overpayments. These indicators should also guide efforts to increase the percentage of debt cleared and lower the amount of waivers granted. Specifically, we suggest that SSA measure the amount of overpayments that occur as a percentage of all payments, the percentage of overpayments that becomes uncollected debt, the increase/decrease in the amount of debt collected per total debt, the change in the amount of waivers provided, and the percent and/or amount of debt outstanding, actually collected, and written off each year.

POLICY DEVELOPMENT, RESEARCH, AND PROGRAM EVALUATION

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

We have concluded that SSA partially measures results for this program area because it has two indicators that address such outcomes as timeliness, responsiveness, and customer satisfaction. However, we have some concerns about a number of the other indicators SSA has developed in this area-barometer indicators. Barometer indicators are intended to measure the OASDI and SSI programs' effects on the economic security of recipients and to help decisionmakers by providing data to guide future research and policy development concerning both short- and long-term Social Security issues. We believe these indicators are of limited use as performance indicators because they focus on the conduct of the research itself rather than the value or impact of the research. While we recognize that policy development and research is a difficult area to measure for results, we believe the conduct of the research should not itself be the measure of performance-especially if the research is being regularly performed.

Performance Indicators
Addressing Policy Development, Research, and Program Evaluation
Objective

Understandable Outcome-based

OIG Comments
Identification, development, and utilization of appropriate barometer measures for assessing the effectiveness of OASDI programs Yes Yes Partially
These indicators are partially outcome-based. While these indicators measure immediate outcomes such as the identification of certain barometer measures or the preparation of research, they do not measure the intended results, or impact of the research. The indicators only track the completion of the research products. In addition, they do not measure the impact, timeliness, value, or other customer-oriented factors of the activities.
Preparation of analyses and reports on demographic, economic, and international trends and their effects on OASDI programs Yes Yes Partially
Preparation of research and policy analyses necessary to assist the Administration and Congress in developing proposals to reform and modernize the OASDI programs Yes Yes Partially
Identification, development, and utilization of barometer measures for assessing effectiveness of the SSI program Yes Yes Partially
Preparation of a report and completion of data collection on the National Survey of SSI Children and Families Yes Yes Partially
Preparation of reports on results of the National Study on Health Activity Yes Yes Partially
Percent of users assigning a high rating to the quality of SSA's research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness Yes Yes Yes This indicator is objective, understandable and outcome-based.
Percent of major statistical products that are timely Partially Yes Yes While this indicator is understandable and outcome-based, it is partially objective. We have previously reported that it measures only part of SSA's statistical products. Specifically, SSA only measures the timeliness of five publications from one division. The strategic goal relates to all information used by SSA, not just that published by one division.
Preparation of a research design to develop techniques for validating medical listings Yes Yes Partially These indicators are partially outcome-based. While these indicators measure immediate outcomes such as the identification of certain barometer measures or the preparation of research, they do not measure the intended results, or impact of the research. The indicators only track the completion of the research products. In addition, they do not measure the impact, timeliness, value, or other customer-oriented factors of the activities.
Preparation of analyses of alternative return-to-work strategies Yes Yes Partially

Recommendation 14
SSA should reconsider whether to continue to include barometer measures for this program area in its APP.

Agency Comment
SSA agreed with Recommendation 14.

HUMAN CAPITAL

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

Two of SSA's seven indicators in the human capital area provide outcome-based data. However, many of the remaining issues in this area are covered by output or milestone measures that, by themselves, do not provide measurable results to track SSA's progress in structuring its workforce for the future.

Performance Indicators
Addressing Human Capital
Objective

Understandable Outcome-based

OIG Comments
Increase the retention rate of new hires Yes Yes Yes This indicator is objective, understandable and outcome-based.
Continue to implement the SSA Future Workforce Plan Yes Yes No
This indicator is not outcome-based because it does not identify or measure key intended results. In a prior review, we noted that this indicator only describes means and strategies by which SSA intends to implement its Future Workforce Plan and not the goal of these activities.
Develop, test and implement desktop video nationally Yes Yes No This indicator is not outcome-based because it does not identify or measure key intended results, effects, or the impact of the activity. Further, the indicator does not focus on or measure key aspects of program performance, nor does it measure how this activity is improving the skills of SSA's staff or its delivery of services.
Percent of offices with direct access to Interactive Video Teletraining (IVT) Yes Yes No These indicators are not outcome-based because they do not identify and measure the intended results, effects, or impact of the program and its goals.
Number of job enrichment opportunities in formal management development programs Yes Yes No
Define competencies for technical training and career development and make them available for employee use Yes Yes Partially
This indicator is partially outcome-based because it does not identify and measure the intended results or effects of having a workforce with the appropriate competencies or technical training.
Percent of employees, who are satisfied with overall physical environment, that is, it is professional, accessible, safe and secure Yes Yes Yes This indicator is objective, understandable and outcome-based.

Recommendation 15
SSA should develop other performance indicators that measure key Human Capital outcomes, such as the percent of employees with the appropriate competencies and training and the percent of SSA components that are fully staffed with employees with the necessary knowledge, skills, and abilities.

Agency Comment
SSA agreed with Recommendation 15.

CUSTOMER SERVICE

OIG Conclusion: The performance indicators are objective and understandable and partially outcome-based.

We noted a number of customer service issues in which SSA's performance indicators were either partially outcome-based or not outcome-based at all. However, we concluded that SSA's performance indicators did measure the key outcomes in this program area and, therefore, we did not make any recommendations.

Performance Indicators
Addressing Customer Service
Objective

Understandable Outcome-based

OIG Comments
Percent of public with an appointment waiting 10 minutes or less Yes Yes Yes These indicators are objective, understandable and outcome-based.

Percent of employers rating SSA's overall service during interactions with SSA as "excellent" Yes Yes Yes
Percent of people who do business with SSA rating the overall service as "excellent," "very good" or "good" Yes Yes Yes
Percent of people who do business with SSA rating the overall service as "excellent" Yes Yes Yes
Percent of employers rating SSA's overall service during interactions with SSA as "excellent," "very good" or "good" Yes Yes Yes
Percent of the public's interactions with SSA, including citizen-initiated services available either electronically via the Internet or through automated telephone service Yes Yes Partially This indicator is partially outcome-based because it does not identify and measure the ultimate results, effects, or impact of the program and its goals. The indicator does not measure key results, such as usage or the number of people who access each of the specific Internet-based services.
Activities to establish the capability for the public interacting with SSA on the Internet to communicate with an SSA employee while on-line Yes Partially No This indicator is partially understandable because it is vague in referring to numerous plans to implement Internet-based activities. It is not outcome-based because it does not identify and measure the intended results, effects, or impact of the program and its goals.

Performance Indicators
Addressing Customer Service
Objective

Understandable Outcome-based

OIG Comments
Percent of States with which SSA has electronic access to human services and unemployment information Yes Yes No
These indicators are not outcome-based because they do not identify or measure key results.
Percent of States with which SSA has electronic access to vital statistics and other material information Yes Yes No

Milestones/deliverables demonstrating progress in increasing electronic access to information held by other Federal agencies, financial institutions and medical providers ----- ----- ----- This proposed indicator has not been defined, so we could not evaluate or comment on it.

Appendix B
Agency Comments

MEMORANDUM 31297-24-831

Date: April 8, 2003

To: James G. Huse, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Measure Review: Assessment of the Social Security Administration's Performance Measures" (Audit No. A-02-02-12050)-INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Staff questions may be referred to Laura Bell on extension 52636.

SSA Response

S1J-3:LBell:lb 03/26/03
IDRIVE: 220020003di
ICN: 31297-24-831

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE MEASURE REVIEW: ASSESSMENT OF THE SOCIAL SECURITY ADMINISTRATION'S (SSA) PERFORMANCE MEASURES" (A-02-02-12050)

We appreciate the opportunity to review and comment on the draft report. We are pleased that the Inspector General (IG) found that the performance measures for the 11 key programs and activities were objective and that the performance measures for all but one of the key programs and activities were determined to be understandable. Regarding the finding that performance measures were only partially outcome-based in 10 of the 11 key programs and activities, we maintain that it is not necessarily appropriate or feasible to have a majority of outcome-based measures. We acknowledge that in our discussions during the exit conference, the IG stated that this was not their intended message.

The IG recommended that we establish efficiency indicators for a number of program areas. We have done so for selected workloads, such as productivity for SSI aged claims. However, the intent of GPRA was not to make Agencies develop efficiency indicators or other outcome-based measures for every discrete workload. We have followed the intent of GPRA in developing high-level, representative performance measures, including outcome-based measures, for our major programs. The Agency has established a goal to improve overall productivity by 2 percent each year.

Our responses to the specific recommendations are provided below.

Recommendation 1
SSA should reinstate its plans to establish indicators to measure the efficiency of the enumeration process, such as the personnel costs of processing Social Security number requests.

Recommendation 2
SSA should define performance goals and develop indicators that measure SSA's intended goals in the earnings area, such as the anticipated improvement in wage processing efficiency when employee reports (W-2s) are filed electronically.

SSA Response
We agree. The intent behind the earnings measures is to improve the accuracy of earnings reports so that they can be posted to individual's earnings records. We continue to track the percent of employee reports (W-2s) filed electronically. In our proposed FY 2004 Annual Performance Plan (APP), we are considering two performance measures aimed at reducing the size of the Earnings Suspense File.

Recommendation 3
SSA should develop additional results-based performance indicators to measure the accuracy and efficiency of Old Age, Survivors and Disability Insurance (OASDI) and Supplemental Security Income (SSI) aged claims processing, as was proposed in the original FY 2002 APP.

SSA Response
We agree in part. Interim efficiency indicators (i.e., cases processed per work year-PPWY) for SSI aged claims, Disability Determination Service (DDS) cases and SSA hearings cases are included in our proposed FY 2004 APP as Performance Assessment Rating Tool (PART) measures. The Agency has established a goal to improve overall productivity by 2 percent each year. Finally, we will continue to have accuracy measures for both OASDI and SSI benefits in our proposed FY 2004 APP as follows:

Percent of SSI payment free of preventable error (overpayments and underpayments)
SSI overpayment and underpayment accuracy rate (including both preventable and unpreventable error)
Percent OASDI payments free of overpayments and underpayments

Recommendation 4
SSA should count all types of improper payments as inaccurate when calculating its OASDI and SSI program payment accuracy rates.

SSA Response
We defer responding to this recommendation pending the outcome of the discussions between the Agency and the Inspector General with respect to the definition of an erroneous payment.

Recommendation 5
SSA should develop an indicator that measures the overall accuracy of decisions for all Disability Insurance (DI) payment outlays, which takes into account initial DDS, as well as Office of Hearings and Appeal's (OHA) decisions.

SSA Response
We disagree. Due to the fundamental differences in the decision making process at the initial claims and appeals levels, a combined indicator would result in an inaccurate measure. The separate indicators provide data that is more realistic and useful for managing the disability program.

Recommendation 6
SSA should develop an indicator that measures the timeliness of DI claims processing so that the total amount of time it takes an SSA claimant to receive a final decision-including decisions reached on appeal-is measured.

SSA Response
We agree, and have already been discussing a new measure for overall processing time of disability claims. Unfortunately, we do not currently have the data available to do this. We are looking into establishing a system that will provide that information.

Recommendation 7
For Continuing Disability Reviews (CDR), SSA should develop more outcome-oriented indicators that measure the key outcomes of this process, mainly the accuracy of CDR decisions.

SSA Response
We do not agree with this recommendation, as the full extent of it is not clear. We will consider exploring this recommendation if it is limited to "the accuracy of CDR decisions," and the recommendation is to establish an accuracy standard related to CDR decisions (similar to that for initial claims). However, we would have objections if the recommendation goes further than that (i.e., the accuracy of the prior allowance or continuance determination).

Recommendation 8
For the representative payment program SSA should (a) revise its current indicator to, at a minimum, separately track representative payee changes and accounting forms; and (b) develop indicators to measure the integrity of the representative payee process, the extent of misuse, and/or its efforts to screen and monitor representative payees. For example, an indicator could measure the percent of representative payee accounting forms found to be in compliance with SSA's regulations by implementing a quality assurance review of a sample of forms.

SSA Response
We disagree. While the indicators to measure the integrity of the representative payee process could require a subsequent review to ensure we are in compliance with policies and procedures, currently, we are only required to monitor fee-for-service payees, large organizational payees serving 100 or more beneficiaries, and individual payees serving 20 or more beneficiaries. These payees must be fully reviewed once every 3 years. In the past, both SSA and OIG have completed reviews of the representative payee process and the results of the reviews indicate misuse of benefits at less than 1 percent of the payees. We would need more information about the anticipated benefits of such a review in order to determine the feasibility of implementing the recommendation given that establishing an ongoing quality review of this type would be resource intensive.

Recommendation 9
For OASDI post-entitlement actions, SSA should develop indicators to provide more outcome-based data, such as measuring processing time, accuracy, and efficiency.

SSA Response
We have established efficiency indicators for selected workloads, and as noted above, the intent of GPRA was not to make Agencies develop efficiency indicators or other outcome-based measures for every discrete workload. The Agency has established a goal to improve overall productivity by 2 percent each year.

Recommendation 10
In the return to work area, SSA should develop indicators that measure the percent of DI and SSI disabled beneficiaries who have returned to work.

SSA Response
In our proposed FY 2004 APP, we are considering two new performance indicators that better reflect our efforts to return disabled beneficiaries to work - "percent increase in the number of DI and SSI beneficiaries, with tickets assigned, who work" and "percent increase in the number of SSI disabled beneficiaries earning at least $100.00 per month."

Recommendation 11
SSA should define the magnitude of fraud to provide a regulatory baseline from which to estimate potential dollars lost to fraud. This would significantly improve the indicators and SSA's performance reporting of efforts to accomplish its strategic objective to "aggressively deter, identify, and resolve fraud."

SSA Response
We disagree. The fact that the Agency does not include a performance indicator to "aggressively deter, identify, and resolve fraud" in the FY 2004 APP has been frequently discussed including the issue of quantifying fraud and measuring progress against that baseline. Since the total universe of fraud is unknown, we cannot fully and accurately measure the total amount of fraud nor identify the amount we may be preventing and deterring. Measuring progress against such an estimate is problematic. For example, an increase in identified fraud could mean we were doing a better job of identifying fraud or a worse job of preventing fraud. Rather than trying to quantify and measure progress against a moving baseline, we believe our current strategic objectives for stewardship are measures that focus on the specific program and administrative areas that may be vulnerable, and allow us to accurately measure progress.

Recommendation 12
SSA should clearly establish performance goals that measure the intended results of its debt management activities, such as the percentage of SSA's outstanding debt the Agency is legally entitled to collect that is actually recovered.

SSA Response
We disagree. We have already established clear debt management performance goals that measure intended results of our activities. They include two indicators that track the percentage of outstanding OASDI and SSI debt in a collection arrangement (excluding due process requests). The FY 2004 goal for OASDI is 37 percent and the goal for SSI is 55 percent. We believe our current debt performance indicators are outcome-based, as they will result in a quantifiable improvement in our debt management program.

We also disagree with the suggested concept of using a percentage of debt that is actually collected as we have learned through experience that this approach can be flawed. For example, the percentage of outstanding debt can be negatively affected by successful enterprises and changes undertaken, such as the change in the annual earnings test several years ago. That change had the effect of reducing the amount of highly collectable current pay debt, which in turn prevented us from achieving our former goal of increasing debt collections by 7 percent a year. Other initiatives, such as debt detection initiatives that uncover hard-to-collect nonpay debt, can change the mix of our debt portfolio and also thwart our ability to collect a stated percentage of debt. In that case, we would have done something good--detected debt and prevented further benefit outlays--but the success would have a negative impact on our percentage of debt collections.

Recommendation 13
SSA should consider developing an indicator to measure the percent of total debt written off because it is uncollectible.

SSA Response
We disagree. We believe an indicator based on debt write-offs is inappropriate and unworkable for the same reasons explained above; i.e., the collectability of debt in an agency's portfolio constantly changes, thus affecting the amount of debt collections and write-offs. For example, if an agency has a lot of unproductive debt that is difficult and costly to collect in a particular year, then write-offs might be higher than average in that year. In that case, it would be inappropriate to establish an indicator that aims to decrease write-offs. In addition, we note that write-offs are conditional, and are collected in the future when the debtor becomes reentitled to Social Security benefits. We also send written-off debts to Treasury for collection by the Treasury Offset Program (TOP), and to credit repositories such as Equifax and Trans Union.

Rather than establish an indicator, we believe an agency's debt write-offs should emanate from adhering to the Federal Claims Collection Standards (FCCS), Part 903. Those standards prescribe the circumstances, which warrant debt write-off; e.g., cost of collection is expected to exceed the amount recoverable, and we have adopted the provisions of FCCS for our write-off policy.

Recommendation 14
SSA should reconsider whether to continue to include barometer measures for its Policy Development, Research, and Program Evaluation program areas in its APP.

SSA Response
We agree. Since preparation of the barometer measures is not an end in itself, it has been discontinued as a performance indicator in FY 2003. We do believe the continued publication of these measures is an important way of assessing SSA's effectiveness in achieving its mission--advancing the economic security of the nation's people. Recognizing that OASDI and SSI payments are not the only factors that affect the economic status of the people whom we serve, the barometer measures provide the best available quantitative indicators of the ultimate outcomes of the our programs, we will continue to update and publish the barometer measures internally.

Recommendation 15
SSA should develop other performance indicators that measure key Human Capital outcomes such as the percent of employees with the appropriate competencies and training and the percent of SSA components that are fully staffed with employees with the necessary knowledge, skills, and abilities.

SSA Response
We agree that additional Human Capital outcome measures would be useful. As we continue to update our Human Capital plans, we carefully evaluate and select the most appropriate performance indicators for our use. The proposed FY 2004 APP includes the Human Capital outcome measure: "Percent improvement in retention rate." This is a very important outcome indicator because it measures the success of our employee training and employee satisfaction efforts. Our other measures for FY 2004 are milestone indicators that measure our progress as we evaluate additional outcome measures for the future.

Appendix C
OIG Contacts and Staff Acknowledgments
OIG Contacts
Rona Rustigian, Director, Northern Audit Division, (617) 565-1819
Timothy F. Nee, Deputy Director, (212) 264-5295

Staff Acknowledgments
In addition to those names above:
Jeffrey Shapiro, Senior Program Analyst
Stephen Liebman, Senior Auditor

For additional copies of this report, please visit our web site at http://www.ssa.gov/oig or contact the Office of the Inspector General's Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-02-02-12050.

Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration's (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers' Act of 1990, assess whether SSA's financial statements fairly present the Agency's financial position, results of operations and cash flow. Performance audits review the economy, efficiency and effectiveness of SSA's programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency, rather than detecting problems after they occur.

Office of Executive Operations

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from SSA, as well as conducting investigations of OIG employees, when necessary. Finally, OEO administers OIG's public affairs, media, and interagency activities, coordinates responses to Congressional requests for information, and also communicates OIG's planned and current activities and their results to the Commissioner and Congress.

Office of Investigations

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA's programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel's office also administers the civil monetary penalty program.