To: The Commissioner

From: Inspector General

Subject: Performance Indicator Audit: Postentitlement Automation Rate (A 15-02-32092)

We contracted with PricewaterhouseCoopers (PwC) to evaluate the data used to measure 18 of the Social Security Administration's (SSA) performance indicators established to comply with the Government Performance and Results Act. Attached is the final report that presents the results of two of the performance indicators PwC reviewed. The objective of this audit was to assess the reliability of the data used to assess the reliability of the data used to measure the Old-Age, Survivors, and Disability Insurance and the Supplemental Security Income postentitlement automation rates.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you with to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

OFFICE OF

THE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE INDICATOR AUDIT:

POSTENTITLEMENT AUTOMATION RATE

February 2003

A-15-02-32092

AUDIT REPORT

MEMORANDUM

To: Office of the Inspector General

From: PricewaterhouseCoopers LLP

Date: January 22, 2003

Subject: Performance Indicator Audit: Postentitlement Automation Rate (A-15-02-32092)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity as set forth in its budget. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the reliability of the data used to measure the following Fiscal Year (FY) 2002 GPRA performance indicators:

Performance Indicator FY 2002 Goal

OASDI postentitlement automation rate. 89%

SSI postentitlement automation rate. 68%

See Appendix A for a description of the audit scope and methodology.

BACKGROUND

SSA offers retirement and long-term disability programs to the general public. Old-Age, Survivors, and Disability Insurance (OASDI) is authorized under title II of the Social Security Act (Act). Through the OASDI program, eligible workers and sometimes their family receive monthly benefits if they retire at an appropriate age or are found to have a disability that either prevents them from engaging in substantial gainful activity for at least 12 months or can be expected to result in death. Supplemental Security Income (SSI) is authorized under title XVI of the Act and provides monthly payments to aged, blind, and disabled individuals based on financial need and medical requirements.

One of SSA's strategic objectives is to "…have the capacity to take and process 99 percent of postentitlement (PE) actions in a paperless environment." PE actions refer to information SSA collects and manages after the recipient is approved for benefits under the OASDI or SSI programs. This on-going maintenance can include updating contact information, resources, living arrangements, or payment information.

In SSA's Annual Performance Plan (APP), the OASDI PE automation rate is defined as the "percentage of total OASDI PE transactions that do not create an exception or alert." OASDI PE transactions are entered into SSA systems through Field Offices (FO), Program Service Centers (PSC), batch files from other Government agencies, or recirculated transactions. The Daily Update Master Accounting System (DUMAS) creates an exception when the transaction is not able to update the Master Beneficiary Record (MBR). DUMAS creates an alert when a transaction requires additional correction after updating the MBR. The APP defines the OASDI PE automation rate as:

OASDI PE transactions that do not create an exception or alert OASDI PE automation rate = All OASDI PE transactions

The FY 2002 APP defines the SSI PE automation rate as the percentage of SSI PE transactions completed using modernized software compared to all SSI transactions. SSA is replacing its legacy system, Customer Information Controls System (CICS), with the Modernized SSI Claims System (MSSICS) for processing initial and PE SSI transactions. This performance indicator measures the percentage of transactions submitted through MSSICS versus CICS to update a recipient's Supplemental Security Record (SSR). The SSI PE automation rate is calculated with the following formula:

Core PE transactions through MSSICS SSI PE automation rate = All core transactions (MSSICS + CICS)

Appendix B provides more detail on the workflow and description of the OASDI and SSI PE automation rates.

RESULTS OF REVIEW

From May through August 2002, we reviewed the processes, controls, and data used to generate the FY 2002 PE automation rate performance indicators. We were not able to recalculate the overall OASDI PE automation rate because SSA does not retain the initial data. Instead, we matched the daily rate on two separate occasions and reviewed the process to calculate the yearly performance indicator. However, SSA's methodology to calculate the final performance indicator value does not match the definition published in the FY 2002 APP. We also matched the year-to-date value of the SSI PE automation rate. We have several findings regarding the calculation of the OASDI and SSI PE automation rate.

PERFORMANCE INDICATOR DATA WAS RELIABLE

We reviewed SSA's methodology to calculate the OASDI and SSI PE automation rate for FY 2002. At the time of our review, SSA had not completed calculation of the 2002 performance indicators. Because these are new performance indicators, we were not able to validate previous year results. For the OASDI PE automation rate, we were not able to validate the year-to-date calculation because SSA does not retain the data. Instead, we validated two daily counts and reviewed the methodology SSA uses to calculate the annual performance indicator. Our daily counts matched SSA's counts exactly. We then reviewed SSA's methodology to calculate the annual performance indicator. We found that SSA's definition from the APP is inconsistent with SSA's method of calculation. In addition, SSA does not include all exceptions and alerts in its calculation. Adding the excluded exceptions and alerts to SSA's monthly data summaries from FY 2000 and FY 2001, we recalculated the OASDI PE automation rate. The rate fell from 90.1 percent to 89.2 percent for 2001 and from 89.5 percent to 88.6 percent for 2000. Because 2002 data is not available, we are not able to determine the impact on 2002. Finally, SSA currently calculates the measure on a calendar year basis, although they report by FY. We describe our findings for the OASDI automation rate in more detail below.

For the SSI PE automation rate, we validated year-to-date results for the first 40 weeks of FY 2002 at 68.2 percent. The value we calculated was 68.2 percent, which matches SSA's calculations exactly. Overall, we found the calculations for the SSI PE automation rate to be reliable. During our review we noted some issues that we discuss below.

Definition of the OASDI PE automation rate is inconsistent with its measurement

Our audit identified several problems with the calculation of the OASDI PE automation rate, related to the data included in the numerator, data included in the denominator, and the period of performance being measured.

The Office of System Design and Development does not measure the number of alerts in the numerator SSA's FY 2002 APP defines the OASDI PE automation rate as the percentage of transactions that do not create an exception or alert. However, the Office of System Design and Development (OSDD), indicated that systems are not programmed to capture the number of alerts separately from the number of exceptions. As a result, OSDD measures the OASDI PE automation rate as the number of transactions that do not create an exception. To make the calculation consistent with the APP's definition, OSDD would need to identify and subtract the number of alerts in the numerator of their formula to calculate the OASDI PE automation rate.

The Office of System Design and Development does not include all appropriate transactions in the denominator

OSDD estimates the number of OASDI PE transactions (in the denominator of the equation) by adding the number of transaction that update the MBR and the number of exceptions or alerts that are sent to the PSC for correction. However, OSDD does not include the exceptions or alerts that are sent to the FO for correction. OSDD indicated these are not included because these alerts or exceptions are the result of keying errors. OSDD asserted that the OASDI PE automation rate is intended to measure progress in updating title II systems, the Title II Redesign, and does not include these exceptions or alerts because they are not created as a result of system limitations.

The Strategic Objective that this performance indicator supports is to "improve or maintain the accuracy, timeliness, and efficiency of processing postentitlement events." This Strategic Objective does not exclude keying errors from the calculation.

After adding in these additional exceptions/alerts, the OASDI PE automation rate decreased from 90.1 percent to 89.2 percent for 2001 and from 89.5 percent to 88.6 percent in 2000. Because the data was not available for FY 2002, we were unable to determine the impact on FY 2002.

Period of performance

In addition, OASDI currently calculates the OASDI PE automation rate on an annual basis, not by FY. Typically, SSA reports all performance indicators on a FY basis. SSA does not indicate in its APP that they are reporting the performance indicator on a calendar basis.

Data to recalculate the OASDI PE automation rate is not available

GPRA requires that Federal agencies retain documentation and data to allow verification of the performance indicator's value where possible. The files used to calculate the OASDI PE automation rate are purged from the systems after 60 to 90 days. OSDD does retain the Excel spreadsheets that calculate the monthly and annual total using summary data from the initial data files. However, it does not retain or archive the initial data used to create the spreadsheets.

Lack of internal controls in calculating the OASDI PE automation rate

During our audit, we noted SSA has limited internal controls to ensure accurate calculation of the OASDI PE automation rate. OSDD transfers data used to calculate the OASDI PE automation rate between multiple staff and Excel spreadsheets. In addition, OSDD has no quality reviews on data that is calculated or transferred between spreadsheets to ensure accuracy. There is, however, an informal, limited review of the general data to ensure that the monthly summary data is entered into the spreadsheets correctly and "looks right."

Definition of the SSI PE automation rate is inconsistent with its measurement

SSA's FY 2002 APP defines the SSI PE automation rate as the percentage of SSI PE transactions completed using the modernized software compared to all SSI transactions. However, the Office of Automation Support (OAS) measures the percentage of SSI PE transactions completed using the modernized software compared to all SSI "PE" transactions. The Strategic Objective that this performance indicator supports is to "…improve or maintain the accuracy, timeliness, and efficiency of processing postentitlement events." While the calculation supports the strategic objective, it does not match the published definition.

SSI PE automation rate may double-count some transactions

Both MSSICS and CICS count PE transactions that update the SSR. Some CICS or MSSICS transactions are not able to update the SSR. These transactions are rejected from the system and sent to a PSC or FO for correction. After the transaction is corrected and resubmitted, it is counted again for the performance indicator. As a result, both CICS and MSSICS transactions may be overstated. OAS estimated that due to improved system edits, the rejection rate for MSSCIS is close to zero percent. OAS estimated the rejection rate for CICS, the legacy system, at close to 10 percent. Because the rejection rate of CICS is higher than the rejection rate for MSSICS, the denominator of the SSI PE automation rate formula may be increased. As a result, the performance indicator may be understating the true SSI PE automation rate.

Lack of documentation in calculating both the OASDI and SSI PE automation rates

During our review, we noted that SSA lacks documentation of the methodology used to calculate both the OASDI and SSI PE automation rates. The process to calculate the OASDI PE automation rate is complex and involves calculations in multiple spreadsheets. OSDD lacks documentation to explain the overall process. As a result, SSA risks continuity in calculating the performance indicator, should key staff no longer be available.

For the SSI PE automation rate, OAS lacks documentation to define "core" transactions and explain how or when these transactions are identified or counted. As a result, OAS risks continuity in calculating the performance indicator and making future changes incorrectly.

OAS also lacks documentation to define its data source for both performance indicators. The FY 2002 APP lists the data source for both the OASDI and the SSI PE automation rate as Office of Systems Information Technology Plans and Office of Systems Management Information. OSDD indicated that the data source for the OASDI PE automation rate includes DUMAS, the Processing Center Action Control System, and the many application programs that update the MBR. OAS has indicated that the data source for the SSI PE automation rate does not include the Office of System Information Technology Plans.

OTHER MATTERS

We also determined if each performance indicator was an appropriate GPRA measure. We found both indicators were not outcome or output oriented as prescribed by GPRA. However, since the performance indicators measure progress for a key SSA initiative they meet the requirements for AAP performance indicators. The Office of Management and Budget Circular No. A-11, Part 6: Preparation and Submission of Strategic Plans, Annual Performance Plans, and Annual Performance Reports, section 220-8(e) states that APP performance indicators may include goals that measure the means or strategies that an agency will use to achieve its performance goals and indicators. This can include technologies applied to achieve a program or operational goal. We therefore found that these performance indicators were appropriate measures for the APP.

CONCLUSIONS AND RECOMMENDATIONS

We found that while the OASDI PE automation rate excluded some data that impacts the performance indicator's value, the methodology appears accurate. The methodology to calculate the SSI PE automation rate was correct, although some transactions may be double-counted due to system limitations. We noted issues with reporting both performance indicators related to their definition and calculation. Our audit identified six opportunities for improvement. Our recommendations are as follows.

Resolve discrepancy between the definition and calculation of the OASDI PE automation rate

According to the OASDI definition, SSA measures the number of exceptions and alerts that are created from PE transactions. However, OSDD has indicated that its systems are not currently programmed to count PE transactions that create alerts and does not subtract these from the numerator. We recommend that SSA resolve this discrepancy to align the measure with the definition.

In addition, OSDD does not include all PE transactions that create an exception or alert in the denominator. Specifically, they do not include exceptions or alerts sent to FOs for correction because they feel these are the result of keying errors. The APP does not make this distinction. We recommend that SSA add the transactions that create exceptions and alerts for FO resolution to the calculation or clarify the definition and purpose of the performance indicator.

Finally, we recommend that SSA either clarify the period of performance in the APP for reporting the OASDI PE automation rate or modify the calculation.

Retain relevant data for the OASDI PE automation rate

We recommend that SSA retain the original data used to calculate the OASDI PE automation rate. This will improve the ability of third parties to review and validate performance indicator accuracy rates in the future.

Implement internal control procedures for the OASDI PE automation rate

We recommend that SSA create and implement control procedures to ensure that the OASDI PE automation rate calculations are accurate. The process to calculate the OASDI PE automation rate is complex with data being copied between spreadsheets, manual calculations, multiple points of initial data entry into the spreadsheets, and transferring the file among multiple staff. Because there are limited internal controls and reviews, a risk for inaccuracies in data entry, calculations, or processing is created. In addition, SSA could increase the automation of calculating the performance indicator and, thereby, decrease the opportunities for errors.

Change the definition of the SSI PE automation rate

We recommend that SSA change the definition of the SSI PE automation rate to read "…the percentage of SSI PE transactions completed using the modernized software compared to all SSI PE transactions." We believe omission of "PE" in the definition was an oversight and SSA should clarify this in future APP publications.

Eliminate duplicate counts in calculating the SSI PE automation rate

We recommend SSA identify transactions that are recirculated in both MSSICS and CICS to get an accurate SSI PE automation rate. As SSA transfers cases from the legacy system, CICS, to the modernized system, MSSICS, this issue will be eliminated or reduced. However, SSA could not produce definitive information on the exact number of cases that are recirculated and double-counted. We recommend that SSA either identify and remove the duplicate transactions from the counts or produce a statistically valid estimate of the number of recirculated transactions.

Document methodology to calculate both the OASDI and SSI PE automation rate

We recommend that SSA document its methodology for calculating both the OASDI and the SSI PE automation rate. Documentation will help ensure continuity in calculating the performance indicator and to document reasons for any changes to the process. Because of the complexity in these performance indicators, SSA's documentation should include a description of why certain exceptions, alerts, or "core" transactions were included in the calculation. While completing this audit we found that OSDD had begun documentation of the methodology for the OASDI PE automation performance indicator. We recommend they continue and expand their documentation to include a description of the process, data source, methodology, and any known data limitations and their impact.

AGENCY COMMENTS

SSA partially agrees with Recommendation 1, but does not believe it should redirect information technology resources to address the discrepancy identified. In addition, SSA believes that the next release of the title II redesign, scheduled for April 2004, will largely resolve the issue. SSA disagrees with Recommendation 2 and 3, largely due to resource constraints and system limitations. SSA will consider both Recommendations 4 and 5. SSA agrees with Recommendation 6. The full text of SSA's comments can be found in Appendix D.

PWC RESPONSE

PwC acknowledges that SSA's information technology resources are limited, but believes that SSA should achieve the majority of our recommended improvements without having to redirect information technology resources from its redesign efforts. For Recommendation 1, PwC continues to believe that SSA should resolve the discrepancy between the definition and the calculation of the performance indicator. The solution could simply be to modify the definition in SSA's APP. For Recommendation 2, we continue to believe that SSA should save the daily summaries used to calculate the OASDI PE automation rate, not necessarily all daily transactions. With respect to Recommendation 3, we believe that the lack of adequate internal controls and reviews increases the risk of errors. By reducing the number of spreadsheets, manual calculations, multiple data entry points, and transfers of files among multiple staff, SSA should reduce the opportunity for errors, thereby improving internal controls. We also continue to believe that SSA needs to take action to address Recommendations 4 and 5.

Appendices

APPENDIX A - Scope and Methodology
APPENDIX B - Flowcharts and Descriptions
APPENDIX C - Acronyms
APPENDIX D - Agency Comments

Appendix A

Scope and Methodology

We conducted this audit to examine Social Security Administration's (SSA) Fiscal Year (FY) 2002 Old-Age, Survivor, and Disability Insurance (OASDI) and Supplemental Security Income (SSI) Postentitlement (PE) automation rate performance indicators. SSA developed these performance indicators to meet the requirements of the Government Performance and Results Act (GPRA). FY 2002 PE automation data and results were not complete at the time of this audit. Because historical data is not retained for the OASDI performance indicator, we recalculated SSA's daily calculation for 2 days and reviewed the methodology to calculate the monthly and annual totals. For the SSI PE automation rate, we calculated the year-to-date results. We evaluated SSA's internal controls and methodology for both indicators.

We performed our testing from May 1, 2002 through July 18, 2002 as follows:
Obtained Locate Beneficiary Indicator and counts for 2 days from the PE processing systems;
Recalculated the PE automation rate for these 2 days;
Reviewed the Office of System Design and Development's Excel data models that calculate the monthly and annual PE automation rate; and
Created flowcharts to document our understanding of the performance indicator.

To test the accuracy and reliability of the SSI PE performance data, we:
Obtained data used to calculate the performance indicator for the first 40 weeks of the FY;
Recalculated the SSI PE automation rate for the first 40 weeks of the FY;
Reviewed the Office of Automation Support's methodology to calculate the performance indicator; and
Created flowcharts to document our understanding of the performance indicator.

In conducting this audit, we also:
Reviewed SSA's Accountability Report for FY 1999, SSA's Annual Performance Plan (APP) for FY 2001, and SSA's Revised Final APP for FY 2002 to determine the baseline data, definition, and data source for the performance indicator;
Reviewed GPRA and related Office of Management and Budget regulations;
Interviewed Office of Systems staff to document the methodologies and procedures used to produce performance data for this indicator; and
Interviewed analysts to gain an understanding of the PE transaction process, the performance indicator calculation methodology used, and other relevant matters.

Our audit was limited to testing at SSA's Headquarters in Woodlawn, Maryland. The procedures we performed were in accordance with the American Institute of Certified Public Accountants' Statement on Standards for Consulting Services and the General Accounting Office's Government Auditing Standards ("Yellow Book") for performance audits.

Appendix B

Flowcharts and Descriptions

Supplemental Security Income (SSI) postentitlement (PE) automation rate data entry process:

1. Field Office (FO), Program Service Center (PSC) or Teleservice Center (TSC) have a SSI PE update.
2. Can the change be entered in the Modernized SSI Claims System (MSSICS)? If no, go to step 3; if yes go to step 8.
3. The FO or TSC enters the change in the Customer Information Controls System (CICS).
4. CICS updates the Supplemental Security Record (SSR) and receives credit for the transaction.
5. Did an error occur? If no, go to step 6; if yes, go to step 7.
6. SSR information is updated.
7. The transmitting office receives an exception for the correction.
8. FO, TSC, or PSC enters the change in MSSICS.
9. Does the data pass MSSICS edits? If no, go to step 10; if yes, go to step 11.
10. The FO or TSC must correct and resubmit the transactions.
11. MSSICS creates a new SSR in an overnight batch process.
12. The system compares the old and the new SSR records.
13. MSSICS receives credit for any changes that occurred to the SSR.
14. Did an error occur? If no, go to step 15; if yes, go to step 16.
15. The SSR is updated.
16. MSSICS sends and alert to the transmitting office for correction.
17. Go to step 2.

SSI PE automation rate calculation:
Office of Information Management (OIM) receives counts of MSSICS and CICS transactions.
OIM updates the post entitlement (PE) management information summary data.
The Office of Automation Support (OAS) runs a program that generates monthly counts.
OAS enters monthly counts into an Excel spreadsheet.
The excel spreadsheet calculates that Supplemental Security Income PE automation rate monthly and annually.

OASDI PE automation rate data production:
FO, PSC, central office - third-party) or title II applications enter data into a PE system.
PE system sends a "finder" to PE Search.
PE Search prioritizes transactions and attaches the highest priority finder to the original Master Beneficiary Record (MBR) record to create a locate beneficiary indicator (LBI).
PE Search also identifies transactions that will recirculate.
PE search puts LBIs and lower priority finders into an LBI-All file.
PE search sends the LBIs to the application programs.
The application programs build a PE Update Transaction (PUT) record to update the MBR record.
Applications send the PUTs and LBIs to the Daily Update Master Accounting System (DUMAS).
DUMAS receives and counts PUTs and LBIs.

Can the transaction be processed? If the transaction cannot be processed the following takes place:
DUMAS creates an exception.
DUMAS sends the exception to the Processing Center Action Control System (PCACS).

If the transaction can be processed the following takes place:
The transaction updates the MBR.
DUMAS creates a count of updated LBIs (ULBI), or transactions that update the MBR.

Does the transaction create an alert? If yes:
DUMAS creates an alert and sends the alert to the PSC for corrective action.
The MBR is updated.

OASDI PE automation rate calculation process:
Programmer A downloads monthly LBI and ULBI counts from the system into Excel spreadsheets.
Programmer A sends the Excel file to the Branch Chief.
Branch Chief reviews the Excel file and sends it to Programmer B.
Programmer B combines totals for each application program.
Programmer B enters the totals into a new preformatted Excel spreadsheet.
Programmer B pulls Title II Redesign totals from the Intranet and adds them to the Excel spreadsheet.
Programmer B adds the manual input totals to spreadsheet (See process to calculate the manual input totals below).
Programmer B sends the Excel spreadsheet to the Branch Chief.
Branch Chief copies the data into a new preformatted Excel spreadsheet.
The spreadsheet sums the data annually.
Branch Chief copies the annual summary to a final Excel spreadsheet.
Branch Chief adds PCACS Receipts to the final spreadsheet (see Computer Output Section (COS) report below).
The spreadsheet calculates the Calculated Input Accounts (PCACS Receipts plus Processed Accounts).
Spreadsheet calculates the processing percentage or PE automation rate (Processed Accounts/Calculated Input Accounts).
Branch Chief reports final processing percentage to the Division Director.

Title II Claims Processing Report:
The title II Claims Processing report is delivered to the Branch Chief.
Branch Chief enters monthly Modernized Claims Systems and Manual Adjustment Credit and Award Process (MADCAP) transaction volumes in an Excel spreadsheet.
Spreadsheet estimates the number of MADCAP Processed and Input Accounts for PE transactions.

COS Report:
The COS report is delivered in hardcopy format to the Division Director.
Division Director delivers the hardcopy COS report to Programmer B.
Programmer B keys PCACS receipts into an Excel Spreadsheet.

Appendix C

Acronyms

APP Annual Performance Plan
CICS Customer Information Controls System
COS Computer Output Section
DUMAS Daily Update Master Accounting System
FO Field Office
FY Fiscal Year
GPRA Government Performance and Results Act
LBI Locate Beneficiary Indicator
MADCAP Manual Adjustment Credit and Award Process
MBR Master Beneficiary Record
MSSICS Modernized SSI Claims System
OAS Office of Automation Support
OASDI Old-Age, Survivors, and Disability Insurance
OIM Office of Information Management
OSDD Office of System Design and Development
PCACS Processing Center Action Control System
PwC PricewaterhouseCoopers
PE Postentitlement
PUT PE Update Transaction
PSC Program Service Center
SSA Social Security Administration
SSI Supplemental Security Income
SSR Supplemental Security Record
TSC Teleservice Center
ULBI Updated Locate Beneficiary Indicator

Appendix D

Agency Comments

MEMORANDUM

Date: January 16, 2003

To: James G. Huse, Jr.
Inspector General

From: Larry W. Dye
Chief of Staff

Subject: Office of the Inspector General (OIG) Draft Report, "Performance Indicator Audit: Post Entitlement Automation Rate" (A-15-02-23092)-INFORMATION

We appreciate OIG's efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

Staff questions may be referred to Laura Bell on extension 52636.

SSA Response

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE INDICATOR AUDIT: POSTENTITLEMENT AUTOMATION RATE" (AUDIT NO. A-15-02-32092)

We appreciate the opportunity to review and comment on the draft report. We are pleased with your conclusion that the methodology used to calculate the Old Age Survivors and Disability Insurance (OASDI) and Supplemental Security Income (SSI) Postentitlement (PE) automation rates appear accurate. We constantly evaluate and modify the measures contained in the Agency Performance Plan (APP), and note that from year to year while some measures are added and others are taken out, we continue to track previous measures internally and use that data in the day-to-day management of the workloads. Our responses to the specific recommendations are provided below. We also include a technical comment that should be included in the final report.

Recommendation 1

The Social Security Administration should resolve the discrepancy between the definition and calculation of the OASDI PE automation rate.

SSA Response

We acknowledge your finding regarding the exclusion of certain elements in the measure's calculations. However, we do not believe that we should redirect information technology (IT) resources from our title II redesign efforts to address differences that occur due to unavoidable omission of alert data. We believe the title II system redesign, release 3, scheduled for
April 2004 will, to a large extent, address the concerns identified.

With respect to the intentional omission of Field Office (FO) generated exceptions into the denominator, we continue to support excluding them as they represent keying errors. We will consider adding a notation to the performance indicator to describe what the calculations do not include, why, and that the impact is insignificant (less than one percent).

As you noted, SSA does report most MI on a fiscal year basis. We can adjust the OASDI PE automation rate period to a fiscal year, rather than a calendar year to be consistent with other reporting periods.

Recommendation 2

SSA should retain relevant data for the OASDI PE automation rate.

SSA Response

We disagree. The information is collected from daily input runs that could represent as many as 500,000 to 3 million transactions. We believe the volume is just too large to retain it beyond our current 60-day period.

Recommendation 3

SSA should implement internal control procedures for the OASDI PE automation rate

SSA Response

We disagree. IT resources are very limited. If the results of the audit had found that the daily counts derived by auditors did not match SSA's exactly, we would be open to evaluate what else could be done beyond our current limited controls. With respect to automating the process, we believe the redesigned title II system release 3, scheduled for April 2004 will address the concerns identified.

Recommendation 4

SSA should change the definition of the Supplemental Security Income (SSI) PE automation rate

SSA Response

We will consider modifying the definition in future performance indicator documents.

Recommendation 5

SSA should eliminate duplicate counts in calculating the SSI PE automation rate

SSA Response

We will consider modifying the SSI PE system functions as we prioritize our future IT needs and resources.

Recommendation 6

SSA should document methodology used to calculate both the OASDI and SSI PE automation rate.

SSA Response

We agree. We completed the documentation for the methodology to calculate the OASDI PE automation rate in the summer of 2002 and will complete the documentation for the methodology to calculate the SSI PE automation rate by July 2003.

Technical Comments

The audit was conducted prior to the Deputy Commissioner for System's (DCS) reorganization, and the references to the Office of Systems Design and Development are no longer current. The responsibility for calculating the OASDI PE automation rate performance indicator, which is measuring the success of title II redesign, now resides in DCS' Office of Retirement and Survivors Insurance Systems.