The attached final report presents the results of our audit. Our objective was to assess the reliability of the Social Security Administration's (SSA) performance data used to measure the Office of Hearings and Appeals (OHA) decisional accuracy rate. The OHA decisional accuracy rate was one of the performance indicators developed by SSA to meet the requirements of the Government Performance and Results Act of 1993.

Please comment within 60 days from the date of this memorandum on corrective action taken or planned on each recommendation. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

Attachment

Mission

The Government Performance and Results Act (GPRA) of 1993, Public Law 103-62, requires the Social Security Administration (SSA) to develop performance indicators that measure or assess the relevant service levels and outcomes of each program activity. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. Our objective was to assess the reliability of SSA's data used to measure the following Fiscal Year (FY) 1999 GPRA performance indicator:

Our audit found that the data SSA used to measure the Office of Hearings and Appeals (OHA) decisional accuracy rate for FY 1999 was reliable. We reviewed the Office of Quality Assurance and Performance Assessment's (OQA) sample selection process and methodology for calculating FY 1999 OHA decisional accuracy and found that:

• data from the Quality Assurance (QA) function was accurately input into OQA’s database; and
• the substantial evidence support rate was reliable.

However, we found that OQA’s sampling plan needs to be updated to reflect current practices; the methodology for calculating the performance measure needs to be documented and stored for future reviews; and OQA’s policy of ensuring that Reviewing Judges (RJ) do not review cases from their own regions needs to be documented.

OQA performs a QA function under SSA's Deputy Commissioner for Finance, Assessment, and Management. RJs are Administrative Law Judges (ALJ) selected for a detail to conduct QA reviews on a sample of cases from Hearing Offices (HO) in SSA's 10 regions. RJs review appealed disability cases to consider whether the ALJ's final decision to allow or deny is supported by substantial evidence. If an RJ concludes that an allowance or denial decision is not supported by substantial evidence, the case is considered in error. The measure for OHA decisional accuracy reflects the percentage of accurate ALJ decisions issued by the HOs. The results of the FY 1999 substantial evidence QA reviews were tabulated and used to calculate the performance measure for the FY 1999 OHA decisional accuracy rate.

After a disability case is adjudicated and cleared through a HO, the status of the case is recorded on the OHA Case Control System (OHACCS). OQA selects approximately 1,400 cases (70 denials and 70 approvals for each of the 10 regions) each month from the OHACCS database. These cases are downloaded onto the OQA server and appended to the Disability Hearings Quality Review System (DHQRS) database. The RJs do not review every case in the DHQRS database—OQA selects a sample of cases for their review. To ensure objectivity, OQA’s practice is to assign cases to RJs from a different region than where the hearing was held. For the FY 1999 performance measure, RJs reviewed 3,915 cases—2,463 allowances and 1,452 denials—selected from cases adjudicated in FY 1998 or later. The cases actually reviewed were not equally distributed between favorable and unfavorable decisions in each region. Therefore, the selection of these cases was weighted when projected to the total cases occurring in the universe. For example, the allowances in a given region were projected to the total allowances in that region; and the same was done for denials in each region.

RJs assess whether an ALJ's ultimate decision to allow or deny is supported by substantial evidence. RJs review the evidence, the written hearing decision, and the hearing testimony cassette recording. Following their review, RJs complete a Data Collection Form (DCF) concerning adjudicative and procedural issues at each level of the decision making process. (For a description of the QA review process, see Appendix C.)

Once the RJ's review is completed, the sample cases and DCFs are returned to OQA headquarters. OQA staff input the DCF responses into the DHQRS database. The OHA decisional accuracy rate is calculated based on the RJ's response to one of two specific questions on the DCF, which asks:

Our audit found that the data SSA used to measure the OHA decisional accuracy rate for FY 1999 was reliable. We reviewed OQA’s sample selection process and methodology for calculating FY 1999 OHA decisional accuracy and found that:

• data from the QA function was accurately input into OQA’s database; and

• the substantial evidence support rate was reliable.

However, we found that OQA’s sampling plan needs to be updated to reflect current practices; the methodology for calculating the performance measure needs to be documented and retained to allow for future reviews; and OQA’s policy of ensuring that RJs do not review cases from their own regions needs to be documented.

GPRA requires that "...agencies describe the means to be used to verify and validate measure values," and the Office of Management and Budget (OMB) Circular A-123, Management Accountability and Control, requires that "…documentation for transactions, management controls and other significant events must be clear and readily available for examination."

For the FY 1999 performance measure, RJs reviewed 3,915 cases—2,463 allowances and 1,452 denials—selected from cases adjudicated in FY 1998 or later. We verified that the DHQRS database contained 1,400 records downloaded from OHACCS each month during this time period.

OQA sampled cases from the universe of allowances and denials from each of the 10 regions. The combined allowance and denial population for the regions ranged from 7,642 (4,844 allowances and 2,798 denials) in Region 8 to 78,071 (47,555 allowances and 30,516 denials) in Region 4. (The total for all 10 regions was 303,154.) Due to the variance in the number of decisions among regions, OQA properly employed a weighting methodology. A total of 3,915 decisions comprised the sample—an acceptable sample size for projection purposes in this analysis—and the evidence underlying each allowance or denial was reviewed by OQA to determine whether it was supportable.

We found that OQA’s sampling methodology has changed, but those changes have not been documented. Specifically, we found that the sampling plan did not show that OQA currently samples 1,400 cases per month. The sampling plan states that 1,250 cases should be sampled from the universe of approved and denied OHA disability cases. Additionally, the plan did not explain that only a sample of the 1,400 cases downloaded each month are reviewed by the RJs and used in calculating the performance measure. Also, the number of supportable and non-supportable denials—determined during our review of the data—did not always add to the number of sample counts drawn for the number of denials. Although the number of instances of this occurrence were small and did not materially affect the soundness of the accuracy rate, SSA should ensure these figures reconcile in the future, so that larger discrepancies, which could effect the soundness of the rate, do not occur.

We reviewed a random sample of 50 records from the 3,915 cases used in calculating the FY 1999 performance measure. Each record represented a decision reviewed by a RJ. The RJs used a standardized DCF for recording their decisions. We appraised 48 of the DCFs for the characteristic of accuracy. Accuracy was determined by the answer to either question 13 or 21 (depending on whether the decision was an allowance or denial) of the DCF matching the information input into the DHQRS database. For all 48 records reviewed, we found the RJs’ decisions were accurately entered into the OQA database. Since no errors were found during our probe sampling, we determined that further review of potential errors was unnecessary.

We also compared data in OHACCS against the DHQRS database, and assessed whether the 50 records in our sample were timely, reasonable, and complete. We verified whether the claim type, hearing type, issue, claimant, and disposition matched the criteria established by the DHQRS sample design. We compared the names and regions of the RJs with the names and regions of the ALJs who heard the case to determine whether RJs were reviewing cases within their own regions. We found one instance where a RJ from Region 4 reviewed a case heard by an ALJ from a different hearing office in the same region.

• The timeliness criteria required the cases in the DHQRS sample to be more than 60 days old. We found that all 50 records met the timeliness criteria.

• A record was determined to be reasonable if the claim type, hearing type, and disposition matched the criteria from the DHQRS sample plan. Each of the 50 records met the reasonableness measure.

• A record was determined to be complete if the RJ returned the claim to DDHQ, or if the DCF could be located. We determined that this measure was met. Although 2 of the DCFs could not be located, the remaining 48 were determined to be complete.

We requested OQA’s methodology and the actual data used to calculate the performance measure, so that we could re-calculate the performance measure ourselves. OQA did not have its methodology documented and had not retained sufficient data to allow us to re-calculate the performance measure. We informed SSA staff of our concerns that this lack of data would satisfy neither the GPRA requirement for describing the means to be used to verify and validate measure values, nor the OMB Circular A-123 documentation requirement. OQA staff told us they could reconstruct this information and subsequently provided the reconstructed data. OQA gave us information to support its methodology for calculating the performance measure, the universe counts, and the weights per case.

Using OQA’s data and weighting methodology, we were able to re-calculate an overall decisional accuracy rate of 88 percent. (The allowance accuracy rate was 85 percent and the denial accuracy rate was 92.7 percent.) Specifically, we found that the number of allowances and denials in the sample selected for each region was accurately divided into the total universe of allowances and denials for each region to develop a weighted figure. This weighted figure was then multiplied by the number of supportable and non-supportable allowances and denials in the sample to develop a weighted estimate of these characteristics in the population, by region. Simple division was employed correctly to develop allowance and denial support rate percentages by region, which were then combined into a single support rate by region, and varied from 82.6 percent in Region 1 to 90.6 percent in Region 4.

We determined that the data SSA used to measure the OHA decisional accuracy rate for FY 1999 was reliable. However, we found that OQA’s sampling plan had not been updated even though substantial changes had been made to the sampling plan. Further, OQA had not retained adequate documentation to support its calculation of the performance measure, and had to reconstruct information for our audit. We also found an unwritten policy in which OQA assigns cases to RJs from regions different than the ALJ who held the hearing. We recommend that SSA:

1. Update the sampling plan to reflect the current sampling methodology being used by OQA.
2. Document the methodology for calculating the performance measure and retain that documentation to allow for the timely verification of the performance measure values.
3. Document the policy whereby a RJ is not assigned to review cases heard by ALJs in the RJ’s region.

In response to our draft report, SSA agreed with our recommendations. Specifically, SSA plans to: update the sampling plan, document and retain the calculation methodology, and document the RJ review policy. (See Appendix E for SSA’s comments.)

Appendices

APPENDIX A – Scope and Methodology

APPENDIX B – Acronyms

APPENDIX C – Process for Determining OHA Decisional Accuracy

APPENDIX D – Other Matters

APPENDIX E – Agency Comments

APPENDIX F – OIG Contacts and Staff Acknowledgments

Scope and Methodology

Our objective was to assess the reliability of the Social Security Administration's (SSA) performance data used to measure the Office of Hearings and Appeals (OHA) decisional accuracy rate.

To accomplish our objective, we:

• Reviewed the Office of Quality Assurance and Performance Assessment’s (OQA) sampling methodology.

• Determined whether the data file from the Office of Hearings and Appeals Case Control System (OHACCS) was accurately downloaded to the OQA database.

• Selected a random sample of 50 records from the 3,915 cases reviewed by Reviewing Judges (RJ) in Fiscal Year (FY) 1999 from the OQA data file and examined the records for timeliness, reasonableness and completeness.

• Verified whether the RJs’ decisions on the data collection forms in the 50 cases selected were accurately entered in the OQA data file.

• Reviewed SSA's Accountability Report for FY 1999, Annual Performance Plans for FYs 1999 and 2001, and FY 2000 Annual Performance Plan Status Report to determine the baseline data, definition, and data source for the performance indicator.

• Interviewed OQA policy and program staff regarding the methodologies and procedures used to produce performance data for the OHA decisional accuracy rate.

• Reviewed the Government Performance and Results Act and Office of Management and Budget Circulars.

• Reviewed the system edits built into the Disability Hearings Quality Review System database.

Our audit did not include a test of OHACCS to verify the completeness of the data file provided by SSA. In addition, we did not determine the accuracy of the quality assurance (QA) review used by OQA to determine the OHA decisional accuracy rate due to the technical nature of the QA review. Our audit was limited to assessing OQA's methodology and data used for calculating the rate.

We performed our audit at the OQA office in Woodlawn, Maryland; and OHA headquarters in Falls Church, Virginia between October 2000 and January 2002. The entity audited was OQA within the Office of the Deputy Commissioner for Finance, Assessment and Management. We conducted our audit in accordance with generally accepted government auditing standards.

Other Matters

SSA has two Government Performance and Results Act performance indicators that report the accuracy of disability decisions. The first indicator, Disability Determination Services (DDS) decisional accuracy, reflects the percentage of correct initial determinations issued by State DDSs and the second indicator—discussed in this report—is Office of Hearings and Appeals (OHA) decisional accuracy, which reflects the percentage of correct disability hearing decisions issued by OHA. The Office of Quality Assurance and Performance Assessment (OQA) performs both DDS and OHA decisional accuracy reviews.

Currently, SSA does not have a method to assess the overall accuracy of payment outlays for disability-based benefits taking into account DDS and OHA case decisions and other non-medical factors of eligibility. According to SSA, such a measurement system is being developed.

According to SSA staff, there are many factors that result in different decisions at the Administrative Law Judge (ALJ) level and the initial level. These factors include presentation of additional medical evidence at the ALJ level, and worsened and/or new impairments.

Greater consistency between decisions made by the DDSs and OHA would result in either program or administrative savings. The following example illustrates this point. If the number of DDS denials in FY 1999 upheld by OHA increased by 5 percent, annual Disability Insurance program savings would be approximately $117 million, and annual Supplemental Security Income program savings would be approximately $69 million. Conversely, if 10 percent of the denials overturned by OHA were considered allowances at the DDS level, SSA could save approximately $19.9 million in administrative costs.

Agency Comments

We appreciate the OIG’s efforts in conducting this review. Our comments on the draft report content and recommendations are attached.

OIG Contacts and Staff Acknowledgments

In addition to those named above: