OFFICE OF

Assessing The Methodology Used to

The Government Performance and Results Act (GPRA) of 1993, Public Law 103-62, section 4, requires that the Social Security Administration (SSA) develop performance indicators that assess the relevant service levels and outcomes of each program activity set forth in SSA’s budget. GPRA also calls for a description of the means employed to verify and validate the measured values used to report on program performance. The objective of this audit was to assess the methodology used to derive the Fiscal Year (FY) 2000 baseline number of 97 for the following FY 2002 GPRA performance indicator:

Number of hearings cases processed per work year: 111

In its FY 2001 Annual Performance Plan (APP), SSA introduced "Percent Increase in Office of Hearings and Appeals (OHA) Production Per Work Year (PPWY) in the hearings process" as a performance indicator. In its FY 2002 APP, however, SSA changed this performance indicator to the "Number of hearings cases processed per work year." SSA believed this change better aligned the performance goal with the measurement and tracking systems it used to determine PPWY.

The Agency revised the OHA PPWY performance indicator after we initiated our audit of the FY 2001 measure. As such, we refocused our efforts to proactively address the new measure, "Number of hearings cases processed per work year." Specifically, we assessed the methodology used to derive the FY 2000 baseline number of 97 for this revised performance indicator.

SSA’s revised OHA PPWY indicator represents the average number of hearings cases processed per "direct" work year expended. A case is considered processed when a hearing office makes a decision or dismissal. The time involved in the work year includes the time worked by OHA employees in the 10 OHA regional offices, hearing offices within each region, Office of the Chief Administrative Law Judge (ALJ), and Medicare Part B Division. According to the FY 2002 APP, a direct work year represents actual time spent processing cases and does not include time spent on training, ALJ travel, leave, or holidays. The formula for this indicator can be expressed as follows.

SSA’s FY 2002 APP reflects a goal of 111 and shows the baseline years as follows.

In FY 2000, OHA reported that it processed 539,426 cases in 5,565 direct work years at the hearing office level.

In assessing the methodology used to derive the FY 2000 baseline of 97 for the revised performance indicator, we verified that the number of hearings cases processed matched the source data (that is, the numerator in the formula shown above). However, the data OHA used to derive direct work years (that is, the denominator in the formula) were inaccurate or incomplete. Specifically, OHA

• used an unofficial data source for its direct work hours and excluded some work hour categories;
• did not obtain or consider all training hours or require disclosure of all offices’ training activities;
• did not review the accuracy and validity of data used to estimate travel time and used an outdated formula; and
• used an electronic spreadsheet that contained errors to consolidate the various data components.

Because of incomplete and inaccurate information, we could not determine the actual PPWY for FY 2000. However, with information we were able to obtain, we did note that OHA understated the net direct work hours by at least 151,906 hours and understated the indirect hours by 132,936 hours. As shown in Table 1, these understatements resulted in a net change of 9.12 direct work years. Although this change did not significantly impact the reported PPWY of 97, inaccurate information used in calculating the figure could affect future years’ calculations. More importantly, without complete data with which to develop an accurate measure of the PPWY or of its baseline data, the figures reported as a performance indicator are not as meaningful.

Table 1: Effect of Errors on PPWY for FY 2000

Additionally, OHA did not sufficiently disclose all of the data sources for the PPWY performance indicator. While it listed the Monthly Activity Reports from the Hearing Office Tracking System (HOTS) as data sources, it did not list payroll records, travel reports, and the training formula needed to compute direct work years. Office of

Management and Budget (OMB) Circular A-11, Preparation and Submission of Budget Estimates, July 17, 2001, prescribes that the APPs indicate the source of the measured data.

To determine the baseline figure of 97, OHA obtained data in Monthly Activity Reports that were submitted by individual hearing offices from information maintained in HOTS. The number of hearings cases processed referred to the number of cases OHA completed at the hearing office level, regardless of whether an actual hearing was held to derive a decision or dismissal. We verified that OHA based the total number of hearings cases included in its FY 2000 baseline calculation on figures reported in the Monthly Activity Reports.

OHA understated the number of direct hours worked by 154,461 hours, and it understated the number of leave hours taken by a net of 34,810 hours. These errors occurred because OHA (1) used an unofficial data source for the work hours when preparing the cost consolidation spreadsheet and (2) did not include all work hours or leave categories in calculating the direct work year.

SSA’s official payroll record is the Payroll Analysis Recap Report (PARR). However, OHA used a report generated from the Time and Attendance Management Information System (TAMIS) to obtain data for the PPWY calculation. OHA staff informed us that they used TAMIS instead of PARR because PARR did not give the level of detail needed to complete the PPWY report. Also, in calculating direct work years, OHA did not include hours worked on holidays or exclude credit hours. Although hours worked on holidays were included on the TAMIS report, they were not considered in the calculation of direct work years. We noted that credit hours were not identifiable on the TAMIS report.

To determine the effect of using the TAMIS report instead of PARR, we compared the work hours based on PARR with the work hours derived from the TAMIS reports and found a net understatement of 119,651 direct hours. Table 2 summarizes the differences by category.

Table 2: Comparison of Cost Consolidation Hours to PARR Hours number of training hours was incomplete

OHA calculated the number of training hours based on training reports compiled by the regional offices. Each month, OHA asked the regional offices to submit the training reports for the number of training hours completed by employees. We determined that the number of hours charged to training was understated by at least 53,977 hours because OHA

• received only 96 of 120 training reports (80 percent) it requested from the regional offices during the year;

• received training reports late, did not include them in the monthly PPWY calculations, and OHA made no adjustments later to include these hours;

• did not consider the training hours applicable to the Medicare Part B Division when requesting the training reports; and

• did not disclose all offices’ training activities.

For the 96 reports OHA received, we identified 46,074 hours OHA did not include in the PPWY calculation because regional offices did not submit the training reports timely. OHA representatives stated they received some of the training reports after the monthly PPWY was calculated. However, once OHA received these late reports, it did not recalculate the monthly PPWY.

OHA did not consider the hours corresponding to the 24 missing training reports in the PPWY calculation. To determine the impact of the 24 missing training reports, we requested these reports from the respective regional offices. The regional offices later provided six of these reports, which reflected 7,903 training hours. We were unable to obtain the other 18  reports because the regions had not maintained copies. Accordingly, we were unable to determine the full impact of the missing training hours on the PPWY baseline calculation. We also could not determine the number of training hours applicable for the Medicare Part B Division because those records were not available.

The practice of not maintaining training reports is contrary to OMB Circular A-123, Management Accountability and Control, June 21, 1995, which states "documentation for transactions, management controls, and other significant events must be clear and readily available for examination." OMB defines management controls as the organization, policies, and procedures agencies use to reasonably ensure that reliable and timely information is obtained, maintained, reported and used for decisionmaking. Therefore, agencies must establish a clear methodology for verifying performance measure values and retain the appropriate documentation to enable an audit based on that methodology.

We also noted the training reports OHA regional offices submitted did not identify all component offices required to submit the reports. For example, some reports listed the regional office itself as a reporting entity, while others did not. Also, some reports listed all of the hearing offices in the regions and indicated whether any training was reported for those offices. Other reports neither listed all of the hearing offices in the region nor indicated whether an office had training hours to report. Without this information, OHA cannot be assured that regional offices are providing a complete and accurate report of its training hours. Unreported training activity impacts the accuracy of OHA’s PPWY calculation.

Travel Time Used in the Calculation was not Reliable

The General Accounting Office’s (GAO), Evaluator’s Guide, section III, Verification and Validation, states Agencies’ APPs ". . . should describe credible procedures to verify and validate performance information." GAO described procedures as credible when they provide for periodic review of data collection, maintenance, and processing procedures to ensure that they are consistently applied and continue to be adequate.

OHA did not review the accuracy and validity of data used to estimate travel time it included in the PPWY calculation. OHA calculated travel hours based on a formula that estimated the amount of ALJ and support staff travel time. OHA devised this formula in 1972, and it has not been reviewed or changed since. OHA did not maintain documentation to support the assumptions for the formula. Absent such documentation, OHA cannot be assured the premise used to calculate travel is valid. Additionally, OHA cannot be assured the formula represents the current travel practices of ALJs and support staff.

Using this formula, OHA estimated that each ALJ spent 275 hours in travel status, annually, to conduct hearings, and 10 percent of the time (27.5 hours, annually) a hearings clerk traveled with the ALJ. We believe travel hours should not be limited to ALJs and support staff conducting hearings but, rather, should include travel conducted by any OHA employee. The time an OHA employee spends traveling, regardless of the purpose of such travel, constitutes time not directly related to processing hearing cases and should be considered in determining the direct work year used in the PPWY calculation.

The Cost Consolidation Spreadsheet Contained Errors

OHA uses an electronic spreadsheet to capture and consolidate data when calculating the PPWY indicator. The spreadsheet contains extensive data entry cells and formulas. Each month, a staff member enters information into the spreadsheet to calculate PPWY by month and reporting office. We identified inaccurate figures in the calculation of direct work years. These inaccuracies occurred because OHA:

• Used an incorrect data field in one of the spreadsheet formulas. Specifically, the formula incorporated the number of holidays instead of the number of days in the month. This error resulted in the understatement of 43,862 sick leave hours.

• Estimated hours worked during 4 weeks of the year. Because each of these weeks occurred in the middle of a pay period and at the end of a month, actual employee time charges were not available when OHA entered the figures in the spreadsheet. Rather, OHA used estimates. The difference between the estimated and actual hours resulted in a net overstatement of 1,107 hours. Specifically, OHA understated regular hours by 1,417 hours, overstated overtime by 3,854 hours, overstated annual leave by 607 hours, and understated sick hours by 1,937.

• Could not provide documentation to support outside overtime worked during 2 months. As a result, OHA overstated overtime by 48 hours in the cost consolidation worksheet.

• Transposed one figure in the spreadsheet resulting in a 70-hour overstatement in overtime. Additionally, OHA incorrectly entered the number of employees in all 10 regions resulting in overstating holiday hours by 912.

• Duplicated one entry, resulting in a 131-hour overstatement of sick leave.

The errors resulted in a net understatement of 41,594 hours. Table 3 summarizes these inaccuracies by category.

GPRA, Public Law 103-62, section 4, requires agencies to " . . . describe the means to be used to verify and validate measured values. . . ." Verification and validation of values could detect errors and support the general accuracy and reliability of the performance information that is recorded, calculated, and reported in the cost consolidation spreadsheet.

We noted there was neither a process to ensure the continued accuracy of the formulas nor a crosscheck of entered data. Supervisory reviews were limited to the recalculation of select data. Also, OHA lacked policies and procedures addressing staff functions and responsibilities and documentation of supervisory reviews.

The data OHA used to calculate the number of hearings cases processed (that is, the PPWY formula’s numerator) matched the source data; however, the data used to derive direct work years (that is, the PPWY formula’s denominator) were inaccurate or incomplete. Specifically OHA:

• used an unofficial data source for its direct work hours and excluded some work hour categories;
• did not obtain or consider all training hours;
• used training reports that did not require disclosure of all offices’ training activity;
• did not review the accuracy and validity of the data used to estimate the travel time, and it used a formula that may have been outdated to estimate travel time; and
• used an electronic spreadsheet to consolidate the various data components that contained errors.

Additionally, OHA did not sufficiently disclose all of the data sources for its FY 2002 PPWY performance indicator. While it listed the HOTS as the source for the hearings processed, it did not list payroll records, travel reports, and the training formula needed to compute direct work years.

Accordingly, we recommend SSA take the following corrective actions to improve the process to measure production per work year in the hearing process:

• identify all components required to submit training reports;
• require consistency in regional reporting by showing a separate line item for each hearings office and the regional office;
• require the reporting of all training hours or explain the reason for excluding certain training hours;
• require negative reporting if employees do not engage in training during the reporting period; and
• establish policies and procedures for the retention of performance measure documentation.

1. Update the travel formula to ensure it better reflects a measure of the current travel of employees and includes all employees who travel. Also, document the underlying assumptions for the revised formula.
2. Establish procedures to detect errors in the electronic spreadsheet and document supervisory reviews.

In response to our draft report, SSA agreed with all of our recommendations. Additionally, the Agency provided technical comments, which we have included in this report. The full text of SSA’s comments is included in Appendix C.

Appendix A

Scope and Methodology

We conducted this audit to assess the methodology used to derive the Fiscal Year (FY) 2000 baseline number for the performance indicator "number of hearings cases processed per work year." This measure was one of the FY 2002 performance indicators SSA developed to meet the requirements of the Government Performance and Results Act of 1993.

To assess the methodology, we

• obtained FY 2000 electronic cost consolidation spreadsheets from the Office of Hearings and Appeals (OHA) Division of Budget and Financial Management (DBFM) containing monthly and year-to-date Production Per Work Year (PPWY) calculations in the hearings process;

• reviewed the OHA cost manual to gain an understanding of how it prepared the cost consolidation spreadsheet and the methodology for measuring productivity;

• verified that hearings disposition information on the Monthly Activity Reports and the Hearing Office Tracking System (HOTS) agreed with the hearings disposition information used to calculate PPWY;

• reviewed supporting documents to verify the accuracy of time and attendance hours and other pertinent data as recorded in DBFM’s electronic spreadsheets;

• recalculated and verified time and attendance information OHA reported on the cost consolidation spreadsheet;

• determined whether the number of training hours reported was complete and accurate;

• determined whether travel hours were accurately calculated; and

• reviewed DBFM electronic spreadsheets and pertinent documentation with DBFM analysts.

In conducting this audit, we also

• reviewed SSA's Annual Performance Plan for FYs 2001 and 2002 and SSA's Revised Final Performance Plan for FYs 2000 and 2001 to determine the baseline data, definition, and data source for the performance indicator;

• reviewed the Government Performance and Results Act, Office of Management and Budget laws and regulations, and General Accounting Office guidelines;
• interviewed OHA DBFM staff to document the methodologies and procedures used to produce performance data for this indicator;

• interviewed SSA’s Division of Administration Systems Development to gain an understanding of the Time and Attendance Management Information System (TAMIS); and

• interviewed the Office of Financial Policy and Operations staff to gain an understanding of the Payroll Analysis Recap Report (PARR).

We reviewed internal controls related to our audit objective. Our review did not include tests of the reliability of computer-processed data from TAMIS or PARR used to account for work and leave hours or HOTS for the number of hearings cases processed. We conducted our work at OHA Headquarters in Falls Church, Virginia, and the Office of the Inspector General’s Dallas Field Office. We performed field work from August 2001 to January 2002, and the entity we audited was OHA within the Office of the Deputy Commissioner for Disability and Income Security Programs. We performed our audit in accordance with generally accepted government auditing standards.

Appendix B
Acronyms

Appendix C

Agency Comments

OIG Contacts and Staff Acknowledgments

OIG Contacts

Kimberly Byrd, Director, Southern Audit Division (205) 801-1605
Paul Davila, Deputy Director (214) 767-6317

In addition to those named above:

Lela Cartwright, Auditor
John Bergamasco, Auditor

The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from the Agency, as well as conducting employee investigations within OIG. Finally, OEO administers OIG’s public affairs, media, and interagency activities and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.

The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.