Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure the Timely Processing of Disability Insurance Claims (A-02-99-11001)

The attached final report presents the results of our audit. Our objective was to assess the reliability of the Social Security Administration’s data used to measure the Fiscal Year 1999 Government Performance and Results Act of 1993 performance indicator dealing with the percent of Disability Insurance claims decided within 6 months after onset, or within 60 days after effective filing date, whichever is later.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

Office of the Inspector General
Larry G. Massanari
Acting Commissioner of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure the Timely Processing of Disability Insurance Claims (A-02-99-11001)

The Government Performance and Results Act (GPRA) of 1993 requires the Social Security Administration (SSA) to develop performance indicators that assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the reliability of SSA's data used to measure the following Fiscal Year (FY) 1999 GPRA performance indicator:

FY 1999 FY 1999 Performance Indicator Goal Actual

Percent of Disability Insurance (DI) claims decided within 6 months after onset or within 60 days after effective filing date, whichever is later 53 Percent 49.2 Percent

The SSA oversees two long-term disability programs. The DI program was established in 1954 under title II of the Social Security Act. The program is designed to provide benefits to wage earners and their families in the event the wage earner becomes disabled. The Supplemental Security Income (SSI) program was created as a result of the Social Security Amendments of 1972. SSI (title XVI of the Act) provides income to financially needy individuals who are aged, blind and/or disabled.

To determine disability eligibility, a claimant must first file a disability claim with SSA. Personnel in one of SSA’s approximately 1,300 field offices (FO) conduct the initial interview with disability applicants and assist them in completing the application. Applications for DI are prepared and input into the Modernized Claims System (MCS). Initial interviews are conducted to determine the applicant’s non-medical eligibility based on work history and to gather evidence needed to determine medical eligibility. At the time of filing, the interviewer tries to determine the date of onset (when the disability began) and if the claimant is performing substantial gainful activity (SGA). If the claimant is working and earning over the SGA limit, the interviewer may deny benefits without sending the case to the State Disability Determinations Service (DDS) office.

Disability determinations under both DI and SSI are performed by DDSs in each State according to Federal regulations. In carrying out its obligation, each DDS is responsible for determining claimants’ disabilities and ensuring adequate evidence is available to support its determinations. The DDS makes a disability determination after all necessary information is obtained and notifies the FO. A letter is sent informing the claimant of the determination (allowance or denial) and of his/her appeal rights.

DDS personnel input the determinations into MCS, which is an automated system. A small percentage of the monthly DI workload, 5 to 6 percent, is input into either the Manual Adjustment, Credit and Award Processing (MADCAP) or the Manual Adjustment, Credit and Award Data Entry (MACADE) systems, which are mostly manual systems. The Social Security Administration Claims Control System (SSACCS) is used to track non-MCS-processed case records for pending and completed title II claims filed with SSA. The Management Information Initial Claims Record (MIICR) reads claims processing information from both SSACCS and MCS (see Appendix C for a complete flowchart of the title II DI processing performance measure). The MIICR then writes data for the completed claim into the MIICR Master File, which creates a file of completed claims for weekly and monthly runs. A Service Delivery Objective (SDO) 14 report is generated from the MIICR Master File, which contains a monthly summary of processing time data. The SDO 14 report is provided to the Office of Strategic Management (OSM). OSM extracts monthly MIICR data, which is then summarized to create statistics for inclusion in SSA’s annual Accountability Report.

The data used to measure the percent of initial DI claims processed within 6 months after the disability began or within 60 days after effective filing date, whichever is later, was found to be reliable. However, we found that internal controls over date fields in MCS, SSACCS, MADCAP/MACADE and MIICR systems are needed to ensure future data reliability. SSA also lacks sufficient documentation of the process used to accumulate and generate the DI processing time statistic. In addition, we determined that SSA’s Office of Information Management (OIM) did not write its software used to measure the timely processing of disability claims in accordance with the Agency’s software standards and guidelines.

Our review disclosed that the reported performance of 49.2 percent was a reliable measurement of performance. This conclusion was based on a replication of the monthly statistics from the MIICR data for November and December 1999 and February 2000. We performed a parallel simulation of the performance measure using the criteria in effect at the time of our review. There were no differences between SSA’s reported statistics for the 3 months reviewed and our calculation of performance.

The results of our replication compared to SSA’s reported calculations are as follows:

Due to the unavailability of data for the entire FY 1999, we were only able to test the above 3 months. Since the process was the same for the entire period of our review, there was no indication that data from other months in FY 1999 would have yielded different results.

Our review of SSA’s monthly files containing data on the processing of initial DI claims indicates that the MIICR is accepting future dates or other apparently incorrect dates from MCS, SSACCS, and MADCAP/ MACADE. For MCS cases, the incorrect dates are input into the MCS system and accepted by MIICR. Other inaccurate dates from either excluded or non-MCS cases are posted to the SSACCS record that is also used by MIICR. This incorrect data from MCS and SSACCS is used by MIICR in its monthly reports regarding the percent of timely cases. This lack of controls over date fields in MCS excluded or non-MCS cases places the reliability of the data into question.

We analyzed 3 months of MIICR data consisting of 284,974 cases covering the period November and December 1999 and February 2000. We found a total of six records with a date of filing recorded as 2019, and three records with dates in the date of onset field of 2012, 2013, and 2015. While this small number of cases is immaterial to the calculation of percent of timely cases, these errors provide evidence that MIICR, the system that generates the statistical information, will accept future dates, which casts doubt on some of the data transferred to MIICR from MCS excluded or non-MCS-processed cases.

SSA did not include the international DI claims processed during FY 1999 in the statistics reported in its FY 1999 Annual Accountability Report. In our review of the monthly FO Initial Disability Claims Reports (Processing Times) for FY 1999, SSA reported that 1,203,567 initial DI claims were processed. The SDO 14 reports indicated that 1,201,729 claims were processed, which is a difference of 1,838 claims, or less than 1 percent of the total claims. Because the statistic for the percent of DI claims processed timely is taken from the SDO 14 report, these claims are not included in this statistic. If these claims were included in the GPRA indicator the revised statistic would represent a difference of plus or minus 0.1 percent and would be immaterial to the overall percent of claims processed timely. However, as these are completed initial DI claims, SSA should have included them when reporting the percent of DI claims processed timely.

SSA lacks sufficient performance measure documentation for the processes involved in the generation of the initial SSA DI claims processing time statistic.

SSA staff advised us that there is a lack of documentation of the SSA initial DI claims process from start to finish. Accordingly, we verified different phases of the process through discussions with personnel from the Office of Systems Requirements, Office of Systems Design and Development, and OIM. We compared the information obtained in these interviews with the criteria stipulated in the Management Information Manual, Part II. Based on all this information, we were able to design a flowchart of the initial DI claims process (see Appendix C).

The Office of Management and Budget Circular No. A-123, Management Accountability and Control states, "The documentation for transactions, management controls, and other significant events must be clear and readily available for examination." Further, GPRA requires agencies to "…describe the means to be used to verify and validate measured values…" (31 U.S.C. 1115(a)(6)). A significant lack of documentation does not provide the audit trail necessary for the verification of the performance measures.

OIM develops software programs to determine whether disability claims are processed timely. In reviewing three of these programs, we found that OIM did not develop software in accordance with guidelines included in Social Security’s Systems Engineering Technology (SET) manual. As a result, OIM software programs are inefficient and difficult to analyze and maintain. There is a risk that the programs could continue processing incorrect data without being detected. We found that OIM programmers:

• Repeat program instructions unnecessarily;
• Use different names for the same files in the same program;
• Do not check file status when reading and writing records to files;
• Do not use standard program termination routines; and
• Use, in the same program, two different methods for performing a discrete set of software instructions.

The software programs we analyzed are for computer applications that OIM uses to produce management information. The programs are stored in a program library on a mainframe computer that Social Security uses for software development. The programs are written in the COBOL programming language and maintained by OIM.

In the programs we reviewed, we found that OIM programmers do not always instruct the computer to return to, and execute, the same set of software instructions. Often, they have the computer execute copies of those instructions at different locations in the program, a practice not in compliance with SET standards. By duplicating software instructions rather than have the computer simply re-use the original, programmers increase the risk that inconsistencies could occur in developing and maintaining software. More frequently, programmers re-write and modify separate but identical software instructions, increasing the chance that differences will occur between the separate versions. Even if no differences develop, creating and modifying software is more expensive because repeat instructions have to be written or modified more than once.

Also, in each of the programs we reviewed, OIM programmers use different names for the same file. Programmers do this in referring to file description and defining the external name of the file. By not using the same name for a file whenever it appears, programmers make the task of writing and maintaining the software more difficult.

OIM programmers do not always check file status when reading and writing records to files, or use standard program termination routines to help identify why a program ends abnormally. These oversights increase the risk that OIM programs will continue processing despite errors and process incorrect data undetected. The absence of standard program termination routines also makes it harder and more time consuming to determine the cause of and correct an abnormal program end.

Finally, OIM programmers use, in the same program, two different methods for performing a set of software instructions, a practice not supported by SET guidelines. This practice makes program maintenance more complex and increases the risk of programming errors such as unintentionally bypassing or executing instructions.

The data used to measure the percent of initial DI claims decided within 6 months after onset or within 60 days after effective filing date, whichever is later, was found to be reliable. We did find, however, weaknesses in SSA’s internal controls over date fields in the MCS, SSACCS, MADCAP/MACADE and MIICR systems, which could cause an inaccuracy in the reported performance measure. We also found that there was a lack of an audit trail documenting the DI process from start to finish. Additionally, SSA’s OIM did not always conform to agencywide software policies and procedures. These conditions need to be corrected so complete reliance can be placed on the reported performance measure in the future.

Accordingly, we recommend SSA take the following corrective measures to improve the process used to measure the initial SSA DI claims processing time performance:

1. Establish controls in the MCS, SSACCS, MADCAP/MACADE and MIICR systems over date fields to limit data entry and processing errors;
2. Either include international DI claims in the measure or fully disclose in SSA’s Annual Performance and Accountability Report the number of international DI claims excluded;
3. Provide an adequate audit trail to document the processes involved in the generation and accumulation of the performance measure; and
4. Ensure OIM’s programming practices comply with SSA’s SET standards.

SSA agreed with all four of our recommendations. In response to Recommendation 1, SSA agreed that MCS, SSACCS, MADCAP/MACADE and MIICR systems should be programmed not to allow future year dates. By the end of December 2001, SSA will provide the Office of the Inspector General (OIG) with a detailed response of their findings and begin modifying the system to prevent future dates from being input as current dates. Regarding Recommendation 2, SSA agreed the Annual Performance and Accountability Report should disclose the number of international DI claims, which are excluded. For recommendation 3, SSA will address the issue of establishing an appropriate audit trail for this performance measure as it transitions MIICR functionality to the title II Workload Management Information Management System. Finally, SSA agreed that all software would be documented and developed according to SSA standards.

We are pleased that SSA agrees with our recommendations and we look forward to receiving an implementation plan detailing the steps the Agency will take to improve the performance measure relating to the processing of initial disability claims.

James G. Huse, Jr.

Appendices

Appendix A

Scope and Methodology

This audit was conducted to assess the reliability of Social Security Administration's (SSA's) performance data used to measure the timely processing of Disability Insurance (DI) claims. The measure reports the percent of DI claims decided within 6 months after onset or within 60 days after effective filing date, whichever is later. This review is part of a series of audits regarding the measurement of the disability process. One report has already been issued and another review concerning both the title II (DI claims) and title XVI (Supplemental Security Income) process is nearing the end of the field work phase.

To test the accuracy and reliability of SSA's performance data, we:

• obtained three monthly data base files for the periods November 1999, December 1999 and February 2000 for the title II DI claims which are contained in the Management Information Initial Claims Records (MIICR) system;

• validated a sample of 100 MIICR Service Delivery Objective (SDO) 14 DI records for December 1999 to the Modernized Claims System (MCS), comparing dates in key data fields in MIICR to the dates maintained in MCS;

• validated a sample of 100 MIICR non-SDO 14 DI records for December 1999 to MCS, comparing the records to ensure correct classification and appropriate exclusion from the SDO;

• checked that the 3 months of records contained valid entry codes and dates;

• compared key dates in MIICR to check for appropriateness for each record reviewed (i.e. date of filing, record establishment date, application receipt date, and date of onset) were prior to clearance date;

• performed a COBOL program systems language translation of MIICR comparing the data to procedures shown in SSA’s Management Information Manual;
• compared and reconciled the number of monthly SDO 14 cases reviewed to the numbers reported by SSA in their monthly SDO 14 reports. For the 3-month period under review, the number of initial disability claims processed were as follows:

• reconciled the number of DI claims reported on the 12 monthly SDO 14 reports to the amount reported on SSA’s FY 1999 Annual Accountability Report;

• reconciled the number of DI claims processed on the 12 monthly field office Initial Disability Claims Reports (Processing Times) to the numbers reported on the monthly SDO 14 reports;

• recalculated the number of timely cases for each of the 3 months reviewed and compared the results to SSA’s monthly SDO 14 reports;

• where appropriate, discussed our results with Office of Information Management (OIM) personnel and obtained clarification of discrepancies occurring in specific DI cases.

In conducting this audit, we also:

• reviewed SSA's Annual Accountability Reports for FYs 1997, 1998, and 1999, SSA's Annual Performance Plans for FYs 1999 and 2000, and SSA's revised Final Performance Plan for FY 2000 to determine the baseline data, definition, and data source for the performance indicator;

• reviewed pertinent Government Performance and Results Act and Office of Management and Budget laws and regulations;

• interviewed Office of Strategic Management policy and program staff to document the methodologies and procedures used to produce performance data for this indicator;

• interviewed SSA staff from the Office of System Requirements, Office of Systems Design and Development and OIM to gain an understanding of the DI process from the initial application phase to either allowance or denial, the composition of OIM's data base, the statistical methods used and reports generated, and other relevant matters; and

• flowcharted the entire title II DI process from initial claims taking until either allowance or denial (see Appendix C).

We reviewed those internal controls related to our audit objective. Our work was conducted at the Office of the Inspector General’s New York Field Office and SSA Headquarters in Baltimore, Maryland from July 1999 to March 2001. The entity audited was the Office of Systems. Our audit was performed in accordance with generally accepted government auditing standards, as applicable to a performance audit.

Appendix B

Acronyms

Appendix C

Flowchart of Title II Disability Processing Performance Measure

Appendix D

Agency Comments

Appendix E

OIG Contacts and Staff Acknowledgments

OIG Contacts

Frederick C. Nordhoff, Director
Financial Management and Performance Monitoring Audit Division (410) 966-6676
Timothy Nee, Deputy Director (212) 264-5295

In addition to those name above:

Stephen L. Liebman, Senior Auditor-in-Charge
Denise Ramirez, Program Analyst
Patrick Kennedy, Audit Manager
Carol Ann Frost, Computer Specialist
Annette DeRito, Program Analyst