SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
IMPROPER PAYMENTS

August 2007

A-15-07-17128

AUDIT REPORT

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA's programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA's programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

MEMORANDUM

Date: August 31, 2007

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Improper Payments (A-15-07-17128)

We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 13 of the Social Security Administration's (SSA) performance indicators established to comply with the Government Performance and Results Act. Attached is the final report presenting the results of one of the performance indicators PwC reviewed. For the performance indicator included in this audit, PwC's objectives were to:

Assess the effectiveness of internal controls and test critical controls over data generation, calculation, and reporting processes for the specific performance indicator.
Assess the overall reliability of the performance indicator's computer processed data. Data are reliable when they are complete, accurate, consistent and not subject to inappropriate alteration.
Test the accuracy of results presented and disclosed in SSA's Fiscal Year 2006 Performance and Accountability Report.
Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

This report contains the results of the audit for the following indicators:

Percent of Old-Age, Survivors and Disability Insurance payments free of overpayment and underpayment.

Please provide within 60 days a corrective action plan that addresses each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

Patrick P. O'Carroll, Jr.

MEMORANDUM

Date: August 23, 2007

To: Inspector General

From: PricewaterhouseCoopers, LLP

Subject: Performance Indicator Audit: Improper Payments (A-15-07-17128)

OBJECTIVE

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.

Our audit was conducted in accordance with generally accepted government auditing standards for performance audits. For the performance indicator included in this audit, our objectives were to:

1. Assess the effectiveness of internal controls and test critical controls over the data generation, calculation, and reporting processes for the specific performance indicator.

2. Assess the overall reliability of the performance indicator's computer processed data. Data are reliable when they are complete, accurate, consistent and not subject to inappropriate alteration.

3. Test the accuracy of results presented and disclosed in SSA's Fiscal Year (FY) 2006 Performance and Accountability Report (PAR).

4. Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.

BACKGROUND

We audited the following performance indicator as stated in the SSA FY 2006 PAR:

Performance Indicator
Goal
Reported Results

Percent of Old-Age, Survivors, and Disability Insurance (OASDI) payments free of overpayment (O/P) and underpayment (U/P) * FY 2005
Overpayment accuracy: 99.8%

Underpayment accuracy: 99.8%

FY 2006
Overpayment accuracy: 99.8%

Underpayment accuracy: 99.8%
FY 2005 Actual
Overpayment accuracy: 99.6%

Underpayment accuracy: 99.8%

FY 2006 Actual
Overpayment accuracy: Not Available

Underpayment accuracy: Not Available
* The performance indicator calculation does not include unavoidable overpayments and underpayments that may result from legal or policy requirements.

The OASDI Program, authorized by Title II of the Social Security Act, provides income for eligible workers, eligible members of their families and survivors, eligible workers who have qualifying disabilities and for eligible members of their families before those workers reach retirement age. Individuals (or qualifying survivors) may become entitled under OASDI programs based on the individual's taxable earnings during his or her lifetime. In addition, to qualify for DI, a worker must have a physical or mental impairment that prevents them from performing gainful work with earnings above a certain monthly amount. SSA has stated that OASDI benefit payments have an increased risk of overpayments due to: substantial gainful activity not reported timely to SSA, computation errors in reporting of additional earnings, annual earnings test, and relationship/dependency (e.g., unreported marriage, not having child-in-care, and students not in full-time attendance). In addition, SSA has stated that OASDI benefit payments have an increased risk of underpayments due to: computation errors in reporting of additional earnings, incorrect reporting of age/date of birth, workers' compensation not reported timely to SSA, and wages/self-employment income not posted correctly. "While the improper payment rate in the OASDI program is very low, SSA's annual outlays are so large that even small percentages of payment error can mean millions of dollars paid incorrectly. Accordingly, SSA seeks continuous improvement in its process to minimize improper payments." The establishment of the Office of Quality Performance (OQP) assists with this effort.

OQP conducts reviews (known as Stewardship reviews) to measure the quality and accuracy of recurring benefit payments provided by SSA. These reviews continue to be the primary measure of quality for Agency performance and provide the basis for reports to Congress and other monitoring authorities. OQP randomly selects OASDI beneficiaries each month to participate in the review. The Office of Assistance and Insurance Program Quality (OAIPQ), or regional OQP offices, receive the sample participants' information from Headquarters OQP, schedule and conduct interviews with the beneficiary if necessary, and determine whether there has been an over or underpayment based on changes to non-medical factors of eligibility. The results of the review are entered into the Retirement and Survivors Insurance (RSI) Modernized Assessment System (RMAS). See Appendix C for a workflow and description of the review process. OQP calculates the performance indicator results based on the outcome of the Stewardship reviews.

The performance indicator is calculated in the following manner:

Percent of OASDI payments free of overpayment (O/P)

* Total Weighted Overpayment Dollars

Total Weighted Dollars Paid

* This percentage is subtracted from 100 percent to attain the accuracy rate

Percent of OASDI payments free of underpayment (U/P)

* Total Weighted Underpayment Dollars
Total Weighted Dollars Paid

* This percentage is subtracted from 100 percent to attain the accuracy rate

For FY 2005, the OASDI Payment Accuracy (Stewardship) Report noted that OASDI Stewardship reviews were completed on 1,528 cases, with an overpayment accuracy rate of 99.6 percent and an underpayment accuracy rate of 99.8 percent. The FY 2005 overpayment and underpayment accuracy results were presented as actual results in the FY 2006 PAR. The actual results for FY 2006 will be reported in the FY 2007 PAR. SSA management continued to use the FY 2005 goal as the goal for FY 2006 indicator.

The payment accuracy rate performance indicator percentage is derived from sampled cases and does not generally equal the true accuracy rates for all payments. Accordingly, SSA calculates the precision, or margin of error, for each of the accuracy rate percentages. The margin of error is not used in the calculation of the payment accuracy rate indicator. Rather, the margin of error provides a range of values that are expected to contain the true population accuracy rate with some level of confidence. Appendix D provides more information on SSA's methodology for calculating the margin of error.

RESULTS OF REVIEW

Our assessment of the indicator included in this report did not identify any significant exceptions related to the meaningfulness of the indicator, the accuracy of presentation, or disclosure of information related to the indicator in the FY 2006 PAR. We were able to recalculate the indicator results and found them to be accurate.

However, our assessment did identify issues with internal controls and data reliability. We noted that two SSA OAIPQ programmers had direct data access that would allow them to update production performance indicator data.

Internal Controls and Data Reliability

Two members of OAIPQ's programming personnel had the "All" access designation (within the Top Secret security software) to RMAS datasets used to calculate the indicator results. This level of access allows users to create, delete and modify any of the data (or datasets) contained within the datasets we reviewed. Therefore, the data used to calculate the performance indicator could be inappropriately modified and could impact the results of this performance indicator. This level of access prevents SSA from ensuring the integrity of this production data. By allowing programming personnel to have the "All" access designation, SSA is not conforming to Office of Management and Budget Circular A-130, Management of Federal Information Resources, Appendix III, Security of Federal Automated Information Resources, principles of "least privileged access" or segregation of duties. Therefore, the data cannot be considered reliable since the access control issue created the potential for inappropriate alteration. It should be noted that SSA management removed the programmers' update access to RMAS during September 2006.

CONCLUSION AND RECOMMENDATIONS

We recommend SSA:

1. Continue to restrict access to personnel who should not have the ability to directly modify, create or delete the datasets used to calculate indicator results.

AGENCY COMMENTS

The Agency agreed with our recommendation. The Agency's comments are included in Appendix E.

Appendices
APPENDIX A - Acronyms
APPENDIX B - Scope and Methodology
APPENDIX C - Process Flowcharts
APPENDIX D - Statistical Methodology
APPENDIX E - Agency Comments

Appendix A
Acronyms
CO Central Office
CR Consistency Review
DEQY Detailed Earnings Query
DI Disability Insurance
FO Field Office
FY Fiscal Year
GAO Government Accountability Office
GPRA Government Performance and Results Act
MBR Master Beneficiary Record
OAIPQ Office of Assistance and Insurance Program Quality
OASDI Old-Age, Survivors and Disability Insurance
ODISP Office of Disability and Income Security Program
OQP Office of Quality Performance
O/P Overpayment
ORSIS Office of Retirement and Survivors Insurance System
OSM Office of Strategic Management
OSSAS Office of Statistics and Special Area Studies
PAR Performance and Accountability Report
PSC Program Service Center
PwC PricewaterhouseCoopers, LLP
RETAP Regular Transcript Attainment and Selection Pass
RMAS RSI Modernized Assessment System
RSI Retirement and Survivors Insurance
SSA Social Security Administration
U/P Underpayment

Appendix B
Scope and Methodology
We updated our understanding of the Social Security Administration's (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and questions to SSA management. We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program.

Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:

Reviewed prior SSA, Office of the Inspector General and other reports related to SSA's GPRA performance and related information systems.
Reviewed applicable laws, regulations and SSA policy.
Met with the appropriate SSA personnel to confirm our understanding of the performance indicator.
Flowcharted the process. (See Appendix C).
Tested key controls related to manual or basic computerized processes (e.g., spreadsheets, databases, etc.).
Conducted and evaluated tests of the automated and manual controls within and surrounding each of the critical applications to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
Identified attributes, rules, and assumptions for each defined data element or source document.
Recalculated the metric or algorithm of the performance indicator to ensure mathematical accuracy.
We assessed the completeness and accuracy of the data to determine the data's reliability as it pertains to the objectives of the audit.

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency's mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency's mission, goals, objectives, and processes were used to determine if the performance indicator appeared to be valid and appropriate given our understanding of SSA's mission, goals, objectives and processes.

We followed all performance audit standards in accordance with generally accepted government auditing standards.

In addition to these steps, we specifically performed the following to test the indicator included in this report:

Interviewed personnel in the Office of Quality Performance (OQP).
Assessed the sample selection methodology for Old-Age, Survivors and Disability Insurance (OASDI) Stewardship case files.
Reviewed a sample of OASDI Stewardship cases and determined if the review was in compliance with OQP policies and procedures.
Reviewed the process for controlling access to the datasets storing the indicator data and tested the appropriateness of the access privileges granted to the datasets for a selection of SSA personnel.
Reviewed the weights applied to the sample size and universe.
Re-performed the calculation of the payment accuracy rate, including case weights, for overpayment and underpayments.

Appendix C
Flowchart of OASDI Payment Accuracy

Flowchart of OASDI Payment Accuracy - Narrative
The Office of Retirement and Survivors Insurance Systems (ORSIS) uses the Regular Transcript Attainment and Selection Pass (RETAP) program to select 1 of every 500 cases eligible for selection from the Master Beneficiary Record (MBR) so the input file is a manageable size. The sample is selected monthly, usually on Friday, from cases in pay status on the current MBR. This will result in the sample size of 500 Disability Insurance (DI) and 1,000 Retirement and Survivors Insurance (RSI) cases.
Office of Statistics and Special Area Studies (OSSAS) updates Field Office (FO) cluster to identify those FOs that are eligible for sample selection.
RSI Modernized Assessment System (RMAS) transmits support materials and lists of cases electronically to the Office of Assistance and Insurance Program Quality Branch (OAIPQ) and Central Office (CO).
OAIPQ requests claim folders from the Program Service Center (PSC), gathers queries, e.g., MBR, Numident, Detailed Earnings Query (DEQY).
Upon receipt of the claim folders, OAIPQ will perform a desk review and recalculate benefits payments to verify accuracy of payment in sample period. OAIPQ will complete the portion of the RSI/DI Quality Review Case Analysis form (SSA -2930) related to the desk review and MBR documentation. Verification tables are used to guide the information collection process.
The desk reviewer contacts the beneficiary to verify existence and receipt of payment. In most cases, contact may be made either by phone or in person for a field review. The field interview is required to obtain certain types of verification, such as earnings for the Annual Earning Testing or the relationship between beneficiaries on the sampled Social Security Number.
If a field interview is required, the reviewer compares the information gathered from the sampled beneficiary to the information obtained and documented in the SSA-2930.
OAIPQ enters results regarding the accuracy of payments made in the sample period into the RMAS. RMAS is a record of the types of deficiencies, the number of occurrences, and the dollar impact.
If OAIPQ does not detect any discrepancies, a random sample of 10 percent of the non-error cases is selected for Consistency Review (CR) by Office of Quality Performance (OQP). Following CR, the Title II folder is returned to the PSC. OQP's case file is maintained in the OAIPQ for the specified period of time.
If OAIPQ does identify discrepancies, OAIPQ will document findings on the Quality Review Feedback Report (SSA-93) and forwards the report to OQP for a CR for all dollar errors over $5. Following the CR, the Title II folder and SSA-93 are returned to the PSC. Informational discrepancies, which do not affect the payment under review, are also reported.
If the PSC does not agree with deficiency, Feedback Coordinator rebuts the review findings either informally or through a potentially three-level process culminating with the Regional Commissioner. Resolution may occur at any level.
If the PSC agrees with the deficiency, operating component performs corrective action, indicating action on the SSA-93, and forwards the response to OAIPQ.
If OQP agrees with the rebuttal and RMAS is still open for the year, OQP removes the discrepancy from the database so that it is not reported as an error.
The OAIPQ maintains a separate file for each case that is reviewed. These files are maintained in the respective OAIPQ offices for approximately 18 months after the close of RMAS for that year. Following the retention period, SSA-8538 (Quality Review Collateral Contact List/Record of Disclosure) is removed from the file and retained for 55 years, and the rest of the case file is destroyed.
OQP extracts accuracy data from RMAS and applies case weights to sample size and universe.
OQP calculates the accuracy rate (error dollars divided by the total dollars paid for the fiscal year) for payments with overpayment and underpayment dollars and reports the rates in the Stewardship report.
Indicator data is sent to Office of Disability and Income Security Program for review of results in the Stewardship report and subsequently reports the annual Government Performance and Results Act (GPRA) results.
The Office of Strategic Management publishes the reported GPRA results in the annual Performance Accountability Report.

Appendix D

Statistical Methodology - Margin of Error Calculation

PricewaterhouseCoopers reviewed this performance indicator during the Fiscal Year (FY) 2005 Financial Statement audit. During the FY 2006 Government Performance and Results Act (GPRA) audit, we found no significant changes to the sampling and estimation models used to calculate this performance indicator. As such, our previous assessment of the sampling and estimation models used by the Social Security Administration (SSA) management in FY 2005 is consistent with the sampling and estimation models used in FY 2006. SSA uses a random group methodology to calculate the Title II payment accuracy margin of error. Per SSA documentation the margins of error methodology is described as follows:

1. The sample data is randomly assigned to 10 groups of equal size with any remainder assigned randomly, 1 to a group.
2. The accuracy rate (number of cases without error divided by the total number of cases) is computed independently for each of the 10 groups.
3. The variance is computed by subtracting the accuracy rate for each group from the accuracy rate established for the entire sample, squaring that difference, summing the squared differences, and dividing by 90.

The description provided by SSA is a valid method (dependent random groups) for standard error estimation.1 The standard error is calculated as the square root of the variance. To calculate precision at the 95 percent confidence level, SSA multiplies the standard error by 2.26 (the 97.5th percentile of the t-distribution with 9 degrees of freedom). To calculate the 95 percent confidence intervals, SSA multiplies the standard error by 2.26 and adds this number to the projected accuracy rate to get the upper 95 percent confidence limit, and subtracts this number from the projected accuracy rate to get the lower 95 percent confidence limit. We found the method for calculating the margin of error valid, and matched approximately SSA's reported margin of errors and confidence intervals. An exact match with SSA's reported margin of errors and confidence intervals was not possible due to differences in random seed number selection and group allocation.

Appendix E
Agency Comments

SOCIAL SECURITY

MEMORANDUM

Date: August 23, 2007

To: Patrick P. O'Carroll, Jr.
Inspector General

From: Larry W. Dye

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Improper Payments" (A-15-07-17128) -INFORMATION

We appreciate OIG's efforts in conducting this review. Our response to the recommendation is attached.

Please let me know if we can be of further assistance. Staff inquiries may be directed to
Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.

Attachment:
SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: IMPROPER PAYMENTS" (A-15-07-17128)

Thank you for the opportunity to review and comment on the draft report. Our response to the specific recommendation is provided below.

Recommendation 1

The Social Security Administration should continue to restrict access to personnel who should not have the ability to directly modify, create or delete the datasets used to calculate indicator results.

Response

We agree. The report accurately notes that the finding regarding the internal control problem was corrected in September 2006; therefore, no further action is necessary.

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Resource Management (ORM). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.

Office of Audit
OA conducts and/or supervises financial and performance audits of the Social Security Administration's (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA's financial statements fairly present SSA's financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA's programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.

Office of Resource Management
ORM supports OIG by providing information resource management and systems security. ORM also coordinates OIG's budget, procurement, telecommunications, facilities, and human resources. In addition, ORM is the focal point for OIG's strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.