OFFICE OF
THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:
SUPPLEMENTAL SECURITY INCOME
PAYMENT ACCURACY

September 2006 A-15-06-16107

AUDIT REPORT

Mission

By conducting independent and objective audits, evaluations and investigations, we inspire public confidence in the integrity and security of SSA’s programs and operations and protect them against fraud, waste and abuse. We provide timely, useful and reliable information and advice to Administration officials, Congress and the public.

Authority

The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:

 Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
 Promote economy, effectiveness, and efficiency within the agency.
 Prevent and detect fraud, waste, and abuse in agency programs and operations.
 Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
 Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

 Independence to determine what reviews to perform.
 Access to all information necessary for the reviews.
 Authority to publish findings and recommendations based on the reviews.

Vision

We strive for continual improvement in SSA’s programs, operations and management by proactively seeking new ways to prevent and deter fraud, waste and abuse. We commit to integrity and excellence by supporting an environment that provides a valuable public service while encouraging employee development and retention and fostering diversity and innovation.

MEMORANDUM

Date: September 18, 2006 Refer To:

To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Supplemental Security Income Payment Accuracy
(A-15-06-16107)

We contracted with PricewaterhouseCoopers, LLP (PwC) to evaluate 15 of the Social Security Administration’s performance indicators established to comply with the Government Performance and Results Act. The attached final report presents the results of one of the performance indicators PwC reviewed. For the performance indicator included in this audit, PwC’s objectives were to:
• Assess the effectiveness of internal controls and test critical controls over data generation, calculation, and reporting processes for the specific performance indicator.
• Assess the overall reliability of the performance indicator’s computer processed data. Data are reliable when they are complete, accurate, consistent and are not subject to inappropriate alteration.
• Test the accuracy of results presented and disclosed in the Fiscal Year 2005 Performance and Accountability Report.
• Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.
This report contains the results of the audit for the following indicator:
• Percent of SSI payments free of preventable overpayments and underpayments.
Please provide within 60 days a corrective action plan that addresses each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at
(410) 965-9700.

S
Patrick P. O’Carroll, Jr.

Attachment

MEMORANDUM

Date: September 6, 2006

To: Inspector General

From: PricewaterhouseCoopers, LLP

Subject: Performance Indicator Audit: Supplemental Security Income Payment Accuracy (A-15-06-16107)

OBJECTIVE

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance.

Our audit was conducted in accordance with generally accepted government auditing standards for performance audits. For the performance indicator included in this audit, our objectives were to:

1. Assess the effectiveness of internal controls and test critical controls over the data generation, calculation, and reporting processes for the specific performance indicator.

2. Assess the overall reliability of the performance indicator’s computer processed data. Data are reliable when they are complete, accurate, consistent and are not subject to inappropriate alteration.

3. Test the accuracy of results presented and disclosed in the Fiscal Year (FY) 2005 Performance and Accountability Report (PAR).

4. Assess if the performance indicator provides a meaningful measurement of the program it measures and the achievement of its stated objective.
BACKGROUND

We audited the following performance indicator as stated in the SSA FY 2005 PAR:

Performance Indicator
Goal
Reported Results

Percent of SSI payments free of preventable overpayment (O/P) and underpayment (U/P) FY 2004
Overpayment accuracy: 95.4%

Underpayment accuracy: 98.8%

FY 2005
Overpayment accuracy: 94.9%

Underpayment accuracy: 98.8%
FY 2004 Actual
Overpayment accuracy: 93.6%

Underpayment accuracy: 98.7%

FY 2005 Estimated
Overpayment accuracy: 93.6%

Underpayment accuracy: 98.7%

The Supplemental Security Income (SSI) Program, authorized by Title XVI of the Social Security Act, was designed as a needs-based program to provide or supplement the income of aged, blind, and/or disabled individuals with limited income and resources. To determine an individual's initial eligibility and benefit payment amounts, SSA relies on the individual's self disclosure of all income sources. SSI benefit payments have an increased risk of overpayments and underpayments because an SSI recipient's eligibility is based on medical and non-medical eligibility factors (e.g. financial status, marital status, living arrangements, etc.), and because these non-medical factors can change with some frequency.

Within SSA, the Office of Quality Performance (OQP) conducts reviews (known as Stewardship reviews) to measure the quality and accuracy of recurring benefit payments provided by SSA. OQP randomly selects SSI beneficiaries each month to participate in the review. The Assistance and Insurance Program Quality Branch (AIPQB), or regional OQP offices, receive the sample participants’ information from OQP, schedule and conduct interviews with the recipients if necessary, and determine whether there has been an over or underpayment based on non-medical factors of eligibility. The results of the review are entered into the SSI Quality Assurance (QA) System. See Appendix C for a workflow and description of the review process. OQP calculates the performance indicator results based on the outcome of the Stewardship reviews, excluding the "unpreventable" overpayments and underpayments.

SSA has identified two specific situations in which an overpayment or underpayment would be considered unpreventable. First, the Goldberg v. Kelly Supreme Court decision of 1970 ruled that the United States Constitution's Fourteenth Amendment Due Process Clause requires that there be a full evidentiary hearing before terminating certain government benefits. Accordingly, Due Process under the Fourteenth Amendment requires that SSI beneficiaries have the right to appeal before their benefit payments can be legally terminated. If the beneficiary is found to be ineligible for the SSI program, all payments received during the periods of ineligibility would be considered standard overpayments. However, payments that were made to ineligible beneficiaries while their cases were being reviewed through the hearings and appeals process would be considered unpreventable. If the beneficiary is found to be eligible for the SSI program and payments received during the periods of eligibility should have been a higher amount, the underpaid amount would be considered unpreventable.

Second, SSA relies on SSI recipients to report changes to their income in a timely manner. If a recipient fails to report a change to income in a timely manner, specifically a change that would affect SSI eligibility, payments made during the periods of ineligibility would be considered unpreventable. Furthermore, if the change of income which was not reported by the recipient caused an increase in the monthly payment amount, the underpaid amount would also be considered unpreventable.

The performance indicator is calculated in the following manner:

Percent of SSI payments free of preventable overpayment (O/P) and underpayment (U/P)

=

* Overpayments or underpayments (minus) - Unpreventable overpayments and underpayments
Total dollars paid

* This percentage is subtracted from 100 percent to attain the accuracy rate

For FY 2004, the SSI Payment Accuracy (Stewardship) Report noted that SSI Stewardship reviews were completed on 4,282 cases, with an overpayment accuracy rate of 93.6 percent and an underpayment accuracy rate of 98.7 percent. The FY 2005 overpayment and underpayment accuracy results were presented as an estimate. As of July 2006, the actual results were not yet available. SSA management used the actual indicator results for FY 2004 as the estimated results for FY 2005 indicator. Management indicated there were no legislative or operational changes that impacted the FY 2005 OQP process. As a result, management did not expect performance variances between FYs 2004 and 2005.

The payment accuracy rate performance indicator percentage is derived from sampled cases and does not generally equal the true accuracy rates for all payments. Accordingly, SSA calculates the precision, or margin of error, for each of the accuracy rate percentages. The margin of error is not used in the calculation of the payment accuracy rate indicator. Rather, the margin of error provides a range of values that are expected to contain the true population accuracy rate with some level of confidence. The margin of error was disclosed in the FY 2005 PAR. Appendix D provides more information on SSA’s methodology for calculating the margin of error.

RESULTS OF REVIEW

Overall, we found this performance indicator to be a meaningful GPRA measure. The performance indicator has a direct measure of outcomes and supports SSA’s strategic goal "Prevent fraudulent and improper payments and improve debt management." We found SSA’s methodology for calculating the accuracy rate for this performance indicator to be appropriate and statistically valid. We successfully recalculated the performance indicator results reported in the SSA FY 2005 PAR for FY 2004 actual results. The FY 2005 results were reported as estimates based on the actual FY 2004 results. Although we were able to recalculate the accuracy rate for FY 2004, we concluded that the Stewardship reviews were not consistently completed in full compliance with SSA's policies and procedures. We found the data used in the calculation was unreliable as a programmer had inappropriate access to the OQP production data. Lastly, we found that SSA management did not clearly define the data definition for the indicator and did not provide a clear linkage to other relevant information within the PAR related to the SSI Stewardship process.

Internal Controls

From the sample of cases we reviewed, we found that the SSI Stewardship reviews completed by the AIPQB were not completed in accordance with OQP's Quality Review Manual System (QRMS). We examined 45 case files for compliance with the QRMS. We concluded that 8 out of the 45 cases had non-payment deficiencies that were not communicated to the field offices. The QRMS requires that cases involving non-payment deficiencies such as incorrect mailing or residence addresses be reported to the appropriate field office. The lack of communication of these non-payment deficiencies to the field office does not change the accuracy of the reported performance indicator results. However, these deficiencies do represent areas of non-compliance with SSA policies and procedures.

AIPQB personnel record the results from the Stewardship reviews in the SSI QA database, which is used to calculate the overpayment and underpayment accuracy rate. During our review of the mainframe datasets used to calculate the indicator, we found that one programmer had the "ALL" access designation to the datasets. This level of access allows the user to create, delete, and modify any of the data contained within the dataset without detection. This level of access prevents SSA from ensuring the integrity of this production data, and consequently the accuracy of the reported indicator results. The Office of Management and Budget (OMB) Circular A-130 requires agencies to implement the practice of least privilege whereby user access is restricted to the minimum necessary to perform his or her job; and enforce a separation of duties so that steps in a critical function are divided among different individuals. Therefore, the data used in the calculation of the indicator cannot be considered reliable. SSA management reported that they have since taken the necessary steps to ensure that appropriate access to datasets for programmers have been designated.

PAR Presentation and Disclosure

SSA management did not clearly define the phrases "free of preventable" and "unpreventable," which were used interchangeably in the data definition for this indicator. In the SSI Payment Accuracy (Stewardship) Report, SSA management defines "unpreventable" as overpayments and underpayments resulting from legal or policy requirements (e.g. Goldberg v. Kelly Supreme Court decision). However, in SSA's tracking report, which is used to report progress of the indicator to the Office of the Chief Strategic Officer (OCSO), "free of preventable" is defined as overpayments and underpayments that are within the Agency's or beneficiaries' ability to prevent.
In addition, SSA management did not provide a clear linkage to other sections of the PAR that contain relevant information regarding the SSI Stewardship process. For example, the PAR contains the Improper Payment Information Act of 2002 Detailed Report, which provides information on the major causes of SSI improper payments and the corrective actions proposed by the SSA management. SSA management informed the audit team that they will be updating the data definition, and improving the linkage of this indicator to complementary areas of the FY 2006 PAR.

CONCLUSION AND RECOMMENDATIONS

We recommend SSA:

1. Reinforce the policies and procedures for management's review of case files in accordance with the SSI QRMS.

2. Continue to restrict access to personnel who should not have the ability to directly modify, create or delete the datasets used to calculate indicator results.

3. Clearly define the phrases "free of preventable" and "unpreventable" and ensure consistent usage of the wording throughout the narrative sections of the PAR.

4. Provide a clear linkage in the narrative sections of the PAR to other sections of the PAR that contain relevant information that support the performance indicator.

AGENCY COMMENTS

SSA agreed with our recommendations. See Appendix E for the full text of the Agency’s comments.

APPENDICES

APPENDIX A – Acronyms

APPENDIX B – Scope and Methodology

APPENDIX C – Process Flowcharts

APPENDIX D – Statistical Methodology

APPENDIX E – Agency Comments

Appendix A
Acronyms
AIPQB Assistance and Insurance Program Quality Branch
DEQY Detailed Earnings Query
FO Field Office
FY Fiscal Year
GAO Government Accountability Office
GPRA Government Performance and Results Act
OCSO Office of the Chief Strategic Officer
OMB Office of Management and Budget
OQP Office of Quality Performance
PAR Performance and Accountability Report
POMS Program Operations Manual System
PwC PricewaterhouseCoopers, LLP
QA Quality Assurance
QAMod Quality Assurance Modernization
QRMS Quality Review Manual System
ROC Regional Operations Component
ROQA Regional Office of Quality Assurance and Performance Assessment
SEQY Summary Earnings Query
SO Satellite Office
SSA Social Security Administration
SSI Supplemental Security Income
SSIRD SSI Record Display
SSN Social Security number
SSR Supplemental Security Record
U.S. United States Reports
U.S.C United States Code

Appendix B
Scope and Methodology
We updated our understanding of the Social Security Administration’s (SSA) Government Performance and Results Act (GPRA) processes. This was completed through research and questions to SSA management. We also requested SSA to provide various documents regarding the specific programs being measured as well as the specific measurement used to assess the effectiveness and efficiency of the related program.

Through inquiry, observation, and other substantive testing, including testing of source documentation, we performed the following:

• Reviewed prior SSA, Government Accountability Office, Office of the Inspector General and other reports related to SSA’s GPRA performance and related information systems.
• Reviewed applicable laws, regulations and SSA policy.
• Met with the appropriate SSA personnel to confirm our understanding of the performance indicator.
• Flowcharted the process. (See Appendix C).
• Tested key controls related to manual or basic computerized processes (e.g., spreadsheets, databases, etc.).
• Conducted and evaluated tests of the automated and manual controls within and surrounding each of the critical applications to determine whether the tested controls were adequate to provide and maintain reliable data to be used when measuring the specific indicator.
• Identified attributes, rules, and assumptions for each defined data element or source document.
• Recalculated the metric or algorithm of the performance indicator to ensure mathematical accuracy.
• We assessed the completeness and accuracy of the data to determine the data's reliability as it pertains to the objectives of the audit.

As part of this audit, we documented our understanding, as conveyed to us by Agency personnel, of the alignment of the Agency’s mission, goals, objectives, processes, and related performance indicators. We analyzed how these processes interacted with related processes within SSA and the existing measurement systems. Our understanding of the Agency’s mission, goals, objectives, and processes were used to determine if the performance indicator appeared to be valid and appropriate given our understanding of SSA’s mission, goals, objectives and processes.

We followed all performance audit standards in accordance with generally accepted government auditing standards.
In addition to these steps, we specifically performed the following to test the indicator included in this report:

• Interviewed personnel in the Office of Quality Performance (OQP).
• Assessed the sample selection methodology for SSI Stewardship case files.
• Reviewed a sample of SSI Stewardship cases and determined if review was in compliance with the OQP policies and procedures.
• Reviewed the process for controlling access to the datasets storing the indicator data and tested the appropriateness of the access privileges granted to the datasets for a selection of SSA personnel.
• Reviewed the weights applied to the sample size and universe.
• Recalculated the payment accuracy rate, including case weights, for overpayment and underpayments.
• Assessed estimation methodology for FY 2005 payment accuracy estimates.

Appendix C
Flowchart of SSI Payment Accuracy


Flowchart of SSI Payment Accuracy - Narrative
• The Quality Assurance Modernization (QAMod) randomly extracts two segments from the Supplemental Security Record (SSR).

• QAMod selects cases from sampled Field Offices (FO). FOs are divided into three groups determined by Regional Office of Quality Assurance and Performance Assessment (ROQA) travel costs. Groups 2 and 3 are randomly selected for each month in the Fiscal Year. Group 1 offices require no overnight travel.

• Electronic notification sent to Assistance and Insurance Program Quality Branch (AIPQB), or the Satellite Offices (SO), where the reviewer prepares/schedules in-home interview and conducts the interview.

• AIPQB obtains verification of recipients' eligibility factors, including the SSI Record Display (SSIRD), the Summary Earnings Query (SEQY), and other alerts. AIPQB reviewer analyzes all information and determines the accuracy of the payment.

• If AIPQB does not detect any deficiencies, the OQP case file is maintained in the regional AIPQB.

• If AIPQB does identify deficiencies, AIPQB completes SSA-93 form and forwards the form and claim folder (if available) to the FO.

• If the FO disputes the review, they resolve their differences with AIPQB. If the FO does not dispute the review, the FO must correct the deficiency within 30 days. The Regional Operations Component (ROC) ensures the correction was made within 30 days.

• AIPQB enters results regarding the accuracy of the payment made in the sample period and notes preventable and unpreventable payment deficiencies in the SSI QA database.

• OQP extracts data from QAMod and applies case weights to sample size and universe.

• OQP calculates the payment accuracy rate (number of cases without error divided by the total number of cases) and reports the rates in the Stewardship report.

• Indicator data is sent to Office of the Chief Strategic Officer (OCSO) and subsequently reported in the Performance and Accountability Report.

Appendix D
Statistical Methodology - Margin of Error Calculation

SSA uses a random group methodology to calculate the Title XVI payment accuracy margin of error. Per SSA documentation the margins of error methodology is described as follows:

1. The sample data is randomly assigned to 10 groups of equal size with any remainder assigned randomly, 1 to a group.
2. The accuracy rate (number of cases without error divided by the total number of cases) is computed independently for each of the 10 groups.
3. The variance is computed by subtracting the accuracy rate for each group from the accuracy rate established for the entire sample, squaring that difference, summing the squared differences, and dividing by 90.

The description provided by SSA is a valid method (dependent random groups) for standard error estimation.1 The standard error is calculated as the square root of the variance. To calculate precision at the 95 percent confidence level, SSA multiplies the standard error by 2.26 (the 97.5th percentile of the t-distribution with 9 degrees of freedom). To calculate the 95 percent confidence intervals, SSA multiplies the standard error by 2.26 and adds this number to the projected accuracy rate to get the upper 95 percent confidence limit, and subtracts this number from the projected accuracy rate to get the lower 95 percent confidence limit.

SSA reports the margin of error for the projected Title XVI SSI overpayment and underpayment accuracy rates, exclusive of unpreventable errors, as 0.68 and 0.29 percent, respectively.

We were unable to recalculate SSA’s reported margin of error and confidence intervals due to possible differences in random seed number selection and group allocation. However, we reviewed the method for calculating the margin of error and found it reasonable. In addition, we recalculated approximately SSA’s reported margin of error and confidence intervals.

Appendix E
Agency Comments


SOCIAL SECURITY

MEMORANDUM

Date: September 5, 2006 Refer To: S1J-3

To: Patrick P. O'Carroll, Jr.
Inspector General

From: Larry W. Dye /s/
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Supplemental Security Income Payment Accuracy" (A-15-06-16107)--INFORMATION

We appreciate OIG’s efforts in conducting this review. Our comments on the draft report’s recommendations are attached.

Please let me know if you have any questions. Staff inquiries may be directed to
Ms. Candace Skurnik, Director, Audit Management and Liaison Staff, at extension 54636.

Attachment:
SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL’S (OIG) DRAFT REPORT, “PERFORMANCE INDICATOR AUDIT: SUPPLEMENTAL SECURITY INCOME PAYMENT ACCURACY” (A-15-06-16107)

Thank you for the opportunity to review and provide comments on this draft report.

Recommendation 1

Reinforce the policies and procedures for management’s review of case files in accordance with the Supplemental Security Income (SSI) Quality Review Manual System (QRMS).

Comment

We agree. The Agency has discussed and reinforced the policies and procedures for management review of case files with the necessary staff.

Recommendation 2

Continue to restrict access to personnel who should not have the ability to directly modify, create or delete the datasets used to calculate indicator results.

Comment

We agree. We will continue to work to ensure that unauthorized access does not occur in the future.

Recommendation 3

Clearly define the phrases “free of preventable” and “unpreventable” and ensure consistent usage of the wording throughout the narrative sections of the Performance and Accountability Report (PAR).

Comment

We agree. The Agency has discussed with and received approval from the Office of Management and Budget to delete “unpreventable” from the performance measure. As included in the Fiscal Year (FY) 2006 – 2007 Annual Performance Plan and to be reported in the FY 2006 PAR, the new title of the performance measure will be, “Percent of SSI payments free of overpayments and underpayments.” The new data definition will be: “The SSI payment accuracy rate free of overpayments and underpayments is determined by an annual review of statistically valid sample of the beneficiary rolls; i.e., the findings are representative of the universe of the payments issued with 95 percent precision and confidence levels of +/-0.9 percent for overpayments and +/-0.3 percent for underpayments. Separate rates are determined for overpayment error dollars and underpayment error dollars. The accuracy rates are computed by dividing error dollars by the total dollars paid for the fiscal year. This percentage is subtracted from 100 percent to determine the accuracy rate. The current measuring system captures the accuracy rate of the non-medical aspects of eligibility for SSI payment outlays.”

Recommendation 4

Provide a clear linkage in the narrative sections of the PAR to other sections of the PAR that contain relevant information that support the performance indicator.

Comment

We agree. In the FY 2006 PAR discussion of SSI payments free of overpayments and underpayments, SSA linked these performance measures to SSA’s program initiatives addressing the Presidential Management Agenda program initiative, “Eliminating Improper Payments.” We refer the reader to the section of the PAR containing “SSA’s Improper Payments Act of 2002 Detailed Report.”

Overview of the Office of the Inspector General
The Office of the Inspector General (OIG) is comprised of our Office of Investigations (OI), Office of Audit (OA), Office of the Chief Counsel to the Inspector General (OCCIG), and Office of Resource Management (ORM). To ensure compliance with policies and procedures, internal controls, and professional standards, we also have a comprehensive Professional Responsibility and Quality Assurance program.
Office of Audit
OA conducts and/or supervises financial and performance audits of the Social Security Administration’s (SSA) programs and operations and makes recommendations to ensure program objectives are achieved effectively and efficiently. Financial audits assess whether SSA’s financial statements fairly present SSA’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs and operations. OA also conducts short-term management and program evaluations and projects on issues of concern to SSA, Congress, and the general public.

Office of Investigations
OI conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement in SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, third parties, or SSA employees performing their official duties. This office serves as OIG liaison to the Department of Justice on all matters relating to the investigations of SSA programs and personnel. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

Office of the Chief Counsel to the Inspector General
OCCIG provides independent legal advice and counsel to the IG on various matters, including statutes, regulations, legislation, and policy directives. OCCIG also advises the IG on investigative procedures and techniques, as well as on legal implications and conclusions to be drawn from audit and investigative material. Finally, OCCIG administers the Civil Monetary Penalty program.
Office of Resource Management
ORM supports OIG by providing information resource management and systems security. ORM also coordinates OIG’s budget, procurement, telecommunications, facilities, and human resources. In addition, ORM is the focal point for OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act of 1993.