OFFICE OF
THE INSPECTOR GENERAL
SOCIAL SECURITY ADMINISTRATION
Performance Measure Review:
Assessing The Methodology Used to
Determine the Number of Hearings
Cases Processed Per Work Year
August
2002
A-06-01-11037
AUDIT REPORT
Mission
We improve SSA programs and operations and protect them against fraud, waste, and abuse by conducting independent and objective audits, evaluations, and investigations. We provide timely, useful, and reliable information and advice to Administration officials, the Congress, and the public.
Authority
The Inspector General Act created independent audit and investigative units, called the Office of Inspector General (OIG). The mission of the OIG, as spelled out in the Act, is to:
Conduct and supervise independent and objective audits and investigations relating to agency programs and operations.
Promote economy, effectiveness, and efficiency within the agency.
Prevent and detect fraud, waste, and abuse in agency programs and operations.
Review and make recommendations regarding existing and proposed legislation and regulations relating to agency programs and operations.
Keep the agency head and the Congress fully and currently informed of problems in agency programs and operations.
To ensure objectivity, the IG Act empowers the IG with:
Independence to determine what reviews to perform.
Access to all information necessary for the reviews.
Authority to publish findings and recommendations based on the reviews.
Vision
By conducting independent and objective audits, investigations, and evaluations, we are agents of positive change striving for continuous improvement in the Social Security Administration's programs, operations, and management and in our own office.
SOCIAL SECURITY
MEMORANDUM
Date: August
1, 2002
To: The Commissioner
From: Inspector General
Subject: Performance Measure Review: Assessing the Methodology Used to Determine the Number of Hearings Cases Processed Per Work Year (A-06-01-11037)
The Government
Performance and Results Act (GPRA) of 1993, Public Law 103-62, section 4, requires
that the Social Security Administration (SSA) develop performance indicators
that assess the relevant service levels and outcomes of each program activity
set forth in SSA’s budget. GPRA also calls for a description of the means employed
to verify and validate the measured values used to report on program performance.
The objective of this audit was to assess the methodology used to derive the
Fiscal Year (FY) 2000 baseline number of 97 for the following FY 2002 GPRA performance
indicator:
Number of hearings cases processed per work year: 111
BACKGROUND
In its FY 2001 Annual Performance Plan (APP), SSA introduced "Percent Increase in Office of Hearings and Appeals (OHA) Production Per Work Year (PPWY) in the hearings process" as a performance indicator. In its FY 2002 APP, however, SSA changed this performance indicator to the "Number of hearings cases processed per work year." SSA believed this change better aligned the performance goal with the measurement and tracking systems it used to determine PPWY.
The Agency revised the OHA PPWY performance indicator after we initiated our audit of the FY 2001 measure. As such, we refocused our efforts to proactively address the new measure, "Number of hearings cases processed per work year." Specifically, we assessed the methodology used to derive the FY 2000 baseline number of 97 for this revised performance indicator.
SSA’s revised OHA PPWY indicator represents the average number of hearings cases processed per "direct" work year expended. A case is considered processed when a hearing office makes a decision or dismissal. The time involved in the work year includes the time worked by OHA employees in the 10 OHA regional offices, hearing offices within each region, Office of the Chief Administrative Law Judge (ALJ), and Medicare Part B Division. According to the FY 2002 APP, a direct work year represents actual time spent processing cases and does not include time spent on training, ALJ travel, leave, or holidays. The formula for this indicator can be expressed as follows.
Hearings cases per direct work year |
= |
Hearings Processed |
|
(Regular Time + Overtime) – (Leave + Travel + Training) 2,080 |
SSA’s FY 2002 APP reflects a goal of 111 and shows the baseline years as follows.
FY 2001 APP Goal |
FY 2000 Actual |
FY 1999 Actual |
103 |
97 |
98 |
In FY 2000, OHA reported that it processed 539,426 cases in 5,565 direct work
years at the hearing office level.
Results of Review
In assessing the methodology used to derive the FY 2000 baseline of 97 for the revised performance indicator, we verified that the number of hearings cases processed matched the source data (that is, the numerator in the formula shown above). However, the data OHA used to derive direct work years (that is, the denominator in the formula) were inaccurate or incomplete. Specifically, OHA
Because of incomplete and inaccurate information, we could not determine the actual PPWY for FY 2000. However, with information we were able to obtain, we did note that OHA understated the net direct work hours by at least 151,906 hours and understated the indirect hours by 132,936 hours. As shown in Table 1, these understatements resulted in a net change of 9.12 direct work years. Although this change did not significantly impact the reported PPWY of 97, inaccurate information used in calculating the figure could affect future years’ calculations. More importantly, without complete data with which to develop an accurate measure of the PPWY or of its baseline data, the figures reported as a performance indicator are not as meaningful.
Table 1: Effect of Errors on PPWY for FY 2000
Total Direct Work Hours Reported by OHA |
11,574,243.20 |
Errors |
|
Direct Hours |
|
Work Hours Understated 154,461.00 |
|
Spreadsheet Hours Overstated (2,555.00) |
|
Net Direct Hours Understated 151,906.00 |
|
Indirect Hours |
|
Leave Hours Understated 34,810.00 |
|
Training Hours Understated 53,977.00 |
|
Spreadsheet Hours Understated 44,149.00 |
|
Net Indirect Hours Understated 132,936.00 |
|
Net Effect of Errors (Direct Hours Less Indirect Hours) |
18,970.00 |
Adjusted Direct Work Hours |
11,593,213.20 |
Direct Work Years |
|
Adjusted Direct Work Years |
5,573.66 |
Direct Work Years OHA Used to Develop Baseline |
(5,564.54) |
Understatement in Number of Work Years |
9.12 |
PPWY with Adjusted Direct Work Years |
97 |
PPWY Reported as Baseline by OHA |
97 |
Additionally, OHA did not sufficiently disclose all of the data sources for the PPWY performance indicator. While it listed the Monthly Activity Reports from the Hearing Office Tracking System (HOTS) as data sources, it did not list payroll records, travel reports, and the training formula needed to compute direct work years. Office of
Management and Budget (OMB) Circular A-11, Preparation and Submission of Budget Estimates, July 17, 2001, prescribes that the APPs indicate the source of the measured data.
Number of hearings cases processed matched the source data
To determine the baseline figure of 97, OHA obtained data in Monthly Activity Reports that were submitted by individual hearing offices from information maintained in HOTS. The number of hearings cases processed referred to the number of cases OHA completed at the hearing office level, regardless of whether an actual hearing was held to derive a decision or dismissal. We verified that OHA based the total number of hearings cases included in its FY 2000 baseline calculation on figures reported in the Monthly Activity Reports.
Source DATA for hours worked was incomplete
OHA understated the number of direct hours worked by 154,461 hours, and it understated the number of leave hours taken by a net of 34,810 hours. These errors occurred because OHA (1) used an unofficial data source for the work hours when preparing the cost consolidation spreadsheet and (2) did not include all work hours or leave categories in calculating the direct work year.
SSA’s official payroll record is the Payroll Analysis Recap Report (PARR). However, OHA used a report generated from the Time and Attendance Management Information System (TAMIS) to obtain data for the PPWY calculation. OHA staff informed us that they used TAMIS instead of PARR because PARR did not give the level of detail needed to complete the PPWY report. Also, in calculating direct work years, OHA did not include hours worked on holidays or exclude credit hours. Although hours worked on holidays were included on the TAMIS report, they were not considered in the calculation of direct work years. We noted that credit hours were not identifiable on the TAMIS report.
To determine the effect of using the TAMIS report instead of PARR, we compared the work hours based on PARR with the work hours derived from the TAMIS reports and found a net understatement of 119,651 direct hours. Table 2 summarizes the differences by category.
Table
2: Comparison of Cost Consolidation Hours to PARR Hours number of training hours
was incomplete
Type of Hours |
Hours from PARR |
Hours from Cost Consolidation Spreadsheet |
Difference |
OHA Reporting Effect |
Direct Work Hours |
||||
Regular |
13,900,900 |
13,754,566 |
146,334 |
Understated |
Overtime |
479,546 |
471,419 |
8,127 |
Understated |
Subtotal |
14,380,446 |
14,225,985 |
154,461 |
Understated |
Leave Hours |
||||
Annual Leave |
1,145,160 |
1,154,538 |
(9,378) |
Overstated |
Sick Leave |
601,787 |
554,174 |
47,613 |
Understated |
Holiday |
531,656 |
550,136 |
(18,480) |
Overstated |
Admin Leave |
111,333 |
96,278 |
15,055 |
Understated |
Subtotal |
2,389,936 |
2,355,126 |
34,810 |
Understated |
Total Direct Hours Less Leave Hours |
11,990,510 |
11,870,859 |
119,651 |
Understated |
OHA
calculated the number of training hours based on training reports compiled by
the regional offices. Each month, OHA asked the regional offices to submit the
training reports for the number of training hours completed by employees. We
determined that the number of hours charged to training was understated by at
least 53,977 hours because OHA
For the 96 reports OHA received, we identified 46,074 hours OHA did not include in the PPWY calculation because regional offices did not submit the training reports timely. OHA representatives stated they received some of the training reports after the monthly PPWY was calculated. However, once OHA received these late reports, it did not recalculate the monthly PPWY.
OHA did not consider the hours corresponding to the 24 missing training reports in the PPWY calculation. To determine the impact of the 24 missing training reports, we requested these reports from the respective regional offices. The regional offices later provided six of these reports, which reflected 7,903 training hours. We were unable to obtain the other 18 reports because the regions had not maintained copies. Accordingly, we were unable to determine the full impact of the missing training hours on the PPWY baseline calculation. We also could not determine the number of training hours applicable for the Medicare Part B Division because those records were not available.
The practice of not maintaining training reports is contrary to OMB Circular A-123, Management Accountability and Control, June 21, 1995, which states "documentation for transactions, management controls, and other significant events must be clear and readily available for examination." OMB defines management controls as the organization, policies, and procedures agencies use to reasonably ensure that reliable and timely information is obtained, maintained, reported and used for decisionmaking. Therefore, agencies must establish a clear methodology for verifying performance measure values and retain the appropriate documentation to enable an audit based on that methodology.
We also noted the training reports OHA regional offices submitted did not identify all component offices required to submit the reports. For example, some reports listed the regional office itself as a reporting entity, while others did not. Also, some reports listed all of the hearing offices in the regions and indicated whether any training was reported for those offices. Other reports neither listed all of the hearing offices in the region nor indicated whether an office had training hours to report. Without this information, OHA cannot be assured that regional offices are providing a complete and accurate report of its training hours. Unreported training activity impacts the accuracy of OHA’s PPWY calculation.
Travel Time Used in the Calculation was not Reliable
The General Accounting Office’s (GAO), Evaluator’s Guide, section III, Verification and Validation, states Agencies’ APPs ". . . should describe credible procedures to verify and validate performance information." GAO described procedures as credible when they provide for periodic review of data collection, maintenance, and processing procedures to ensure that they are consistently applied and continue to be adequate.
OHA did not review the accuracy and validity of data used to estimate travel time it included in the PPWY calculation. OHA calculated travel hours based on a formula that estimated the amount of ALJ and support staff travel time. OHA devised this formula in 1972, and it has not been reviewed or changed since. OHA did not maintain documentation to support the assumptions for the formula. Absent such documentation, OHA cannot be assured the premise used to calculate travel is valid. Additionally, OHA cannot be assured the formula represents the current travel practices of ALJs and support staff.
Using this formula, OHA estimated that each ALJ spent 275 hours in travel status, annually, to conduct hearings, and 10 percent of the time (27.5 hours, annually) a hearings clerk traveled with the ALJ. We believe travel hours should not be limited to ALJs and support staff conducting hearings but, rather, should include travel conducted by any OHA employee. The time an OHA employee spends traveling, regardless of the purpose of such travel, constitutes time not directly related to processing hearing cases and should be considered in determining the direct work year used in the PPWY calculation.
The Cost Consolidation Spreadsheet Contained Errors
OHA uses an electronic spreadsheet to capture and consolidate data when calculating the PPWY indicator. The spreadsheet contains extensive data entry cells and formulas. Each month, a staff member enters information into the spreadsheet to calculate PPWY by month and reporting office. We identified inaccurate figures in the calculation of direct work years. These inaccuracies occurred because OHA:
The errors resulted in a net understatement of 41,594 hours. Table 3 summarizes these inaccuracies by category.
Table 3: Cost Consolidation Spreadsheet Errors
Type of Hours |
Number of Hours |
OHA Reporting Effect |
Direct Work Hours |
||
Regular |
1,417 |
Understated |
Overtime |
(3,972) |
Overstated |
Subtotal |
(2,555) |
Overstated |
Leave Hours |
||
Annual Leave |
(607) |
Overstated |
Sick Leave |
45,668 |
Understated |
Holiday |
(912) |
Overstated |
Subtotal |
44,149 |
Understated |
Direct Hours Less Leave Hours |
41,594 |
Understated |
GPRA, Public Law 103-62, section 4, requires agencies to " . . . describe the means to be used to verify and validate measured values. . . ." Verification and validation of values could detect errors and support the general accuracy and reliability of the performance information that is recorded, calculated, and reported in the cost consolidation spreadsheet.
We noted there was neither a process to ensure the continued accuracy of the formulas nor a crosscheck of entered data. Supervisory reviews were limited to the recalculation of select data. Also, OHA lacked policies and procedures addressing staff functions and responsibilities and documentation of supervisory reviews.
Conclusions and Recommendations
The data OHA used to calculate the number of hearings cases processed (that is, the PPWY formula’s numerator) matched the source data; however, the data used to derive direct work years (that is, the PPWY formula’s denominator) were inaccurate or incomplete. Specifically OHA:
Additionally, OHA did not sufficiently disclose all of the data sources for its FY 2002 PPWY performance indicator. While it listed the HOTS as the source for the hearings processed, it did not list payroll records, travel reports, and the training formula needed to compute direct work years.
Accordingly, we recommend SSA take the following corrective actions to improve the process to measure production per work year in the hearing process:
AGENCY COMMENTS
In response to our draft report, SSA agreed with all of our recommendations. Additionally, the Agency provided technical comments, which we have included in this report. The full text of SSA’s comments is included in Appendix C.
James
G. Huse, Jr.
Appendices
APPENDIX A - Scope and Methodology
APPENDIX B - Acronyms
APPENDIX C - Agency Comments
APPENDIX D - OIG Contacts and Staff Acknowledgments
Appendix
A
Scope and Methodology
We conducted this audit to assess the methodology used to derive the Fiscal Year (FY) 2000 baseline number for the performance indicator "number of hearings cases processed per work year." This measure was one of the FY 2002 performance indicators SSA developed to meet the requirements of the Government Performance and Results Act of 1993.
To assess the methodology, we
In conducting this audit, we also
We reviewed internal controls related to our audit objective. Our review did not include tests of the reliability of computer-processed data from TAMIS or PARR used to account for work and leave hours or HOTS for the number of hearings cases processed. We conducted our work at OHA Headquarters in Falls Church, Virginia, and the Office of the Inspector General’s Dallas Field Office. We performed field work from August 2001 to January 2002, and the entity we audited was OHA within the Office of the Deputy Commissioner for Disability and Income Security Programs. We performed our audit in accordance with generally accepted government auditing standards.
Appendix
B
Acronyms
ALJ |
Administrative Law Judge |
APP |
Annual Performance Plan |
DBFM |
Division of Budget and Financial Management |
FY |
Fiscal Year |
GAO |
General Accounting Office |
GPRA |
Government Performance and Results Act |
HOTS |
Hearing Office Tracking System |
OHA |
Office of Hearings and Appeals |
OMB |
Office of Management and Budget |
PARR |
Payroll Analysis Recap Report |
PPWY |
Production Per Work Year |
SSA |
Social Security Administration |
TAMIS |
Time and Attendance Management Information System |
Appendix
C
Agency Comments
SOCIAL
SECURITY |
Date: |
July 22, 2002 |
|
To: |
James
G. Huse, Jr. |
From: |
Larry
Dye |
Subject: |
Office of Inspector General (OIG) Draft Report "Performance Measure Review - Assessing the Methodology Used to Determine the Number of Hearings Cases Processed Per Work Year," A-06-01-11037—INFORMATION |
We appreciate OIG’s efforts in conducting this review. Our comments on the draft report contents and recommendations are attached.
Staff questions may be referred to Laura Bell on extension 52636.
COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFT REPORT "PERFORMANCE MEASURE REVIEW - ASSESSING THE METHODOLOGY USED TO DETERMINE THE NUMBER OF HEARINGS CASES PROCESSED PER WORK YEAR," A-06-01-11037
We appreciate the opportunity to comment on the draft report. Overall, we found value in the report contents and recommendations. Our responses to the specific recommendations are provided below. We are also providing a technical comment that we believe will improve the content of the report.
Recommendation 1
The Social Security Administration (SSA) should ensure the use of official data sources when capturing employee work hours and indicate the sources in the Annual Performance Plan (APP).
Comment
We agree. The Office of Hearings and Appeals (OHA) has included the additional sources used in the preparation of the hearings production per workyear (PPWY) on the data sheet(s) contained in the SSA Tracking Report. The Office of Strategic Management refers to the data sheets when writing the APP. The new elements are now included in the report.
Recommendation 2
SSA should include hours worked on holidays and credit hour categories to arrive at direct work years.
Comment
We agree. OHA staff were instructed in June 2002 to include hours worked on holidays in the calculations. Regarding credit hours, staff in the Office of Financial Policy and Operations (OFPO) believed that credit hours earned and credit hours used canceled each other out; therefore, we did not include them in the report. In the next few months, OHA will work with OFPO staff to explore the possibility of establishing a process to track credit hours, and we expect to maintain data on their use in fiscal year (FY) 2003.
Recommendation 3
SSA should establish procedures for the submission and retention of training data that will: identify all components required to submit training reports; require consistency in regional reporting by showing a separate line item for each hearings office and the regional office; require the reporting of all training hours or explain the reason for excluding certain training hours; require negative reporting if employees do not engage in training during the reporting period; and establish policies and procedures for the retention of performance measure documentation.
Comment
We agree. In November 2001, procedures were established that require each OHA component to submit a monthly training report listing each office, including the Regional Office, individually. Components are also required to submit negative reports if there are no training hours to report. The hearing offices must also provide a copy of the SSA-759 form for each training occurrence for the month. All records will be kept electronically for the current fiscal year, and hard copies will also be kept for the current and prior three fiscal years as well. OHA is establishing a training record data base, and we are confident that use of the data base and the new financial record keeping system will greatly enhance our ability to document and track training activity.
Recommendation 4
SSA should update the travel formula to ensure it better reflects a measure of the current travel of employees and includes all employees who travel. Also, document the underlying assumptions for the revised formula.
Comment
We agree. During FY 2003, OHA will study and propose changes to the travel formula.
Recommendation 5
SSA should establish procedures to detect errors in the electronic spreadsheet and document supervisory reviews.
Comment
We agree. As a result of this audit, staff corrected the errors found in the spreadsheets and has protected the cells containing formulas. Within the next 3 months, OHA plans to establish a procedure to detect errors in the electronic spreadsheet and will implement a process to document supervisory reviews of the monthly reports. OHA also plans to review the data entered on the electronic spreadsheet on a quarterly basis and document those reviews.
Appendix D
OIG Contacts and Staff Acknowledgments
OIG Contacts
Kimberly
Byrd, Director, Southern Audit Division (205) 801-1605
Paul Davila, Deputy Director (214) 767-6317
Acknowledgments
In addition
to those named above:
Lela
Cartwright, Auditor
John
Bergamasco, Auditor
For additional copies of this report, please visit our web site at www.socialsecurity.gov/oig or contact the Office of the Inspector General’s Public Affairs Specialist at (410) 966-1375. Refer to Common Identification Number A-06-01-11037.
Overview of the Office of the Inspector General
Office of Audit
The Office of Audit (OA) conducts comprehensive financial and performance audits of the Social Security Administration’s (SSA) programs and makes recommendations to ensure that program objectives are achieved effectively and efficiently. Financial audits, required by the Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present the Agency’s financial position, results of operations, and cash flow. Performance audits review the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-term management and program evaluations focused on issues of concern to SSA, Congress, and the general public. Evaluations often focus on identifying and recommending ways to prevent and minimize program fraud and inefficiency.
Office of Executive Operations
The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) by providing information resource management; systems security; and the coordination of budget, procurement, telecommunications, facilities and equipment, and human resources. In addition, this office is the focal point for the OIG’s strategic planning function and the development and implementation of performance measures required by the Government Performance and Results Act. OEO is also responsible for performing internal reviews to ensure that OIG offices nationwide hold themselves to the same rigorous standards that we expect from the Agency, as well as conducting employee investigations within OIG. Finally, OEO administers OIG’s public affairs, media, and interagency activities and also communicates OIG’s planned and current activities and their results to the Commissioner and Congress.
Office of Investigations
The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud, waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third parties, and by SSA employees in the performance of their duties. OI also conducts joint investigations with other Federal, State, and local law enforcement agencies.
Counsel to the Inspector General
The Counsel to the Inspector General provides legal advice and counsel to the Inspector General on various matters, including: 1) statutes, regulations, legislation, and policy directives governing the administration of SSA’s programs; 2) investigative procedures and techniques; and 3) legal implications and conclusions to be drawn from audit and investigative material produced by the OIG. The Counsel’s office also administers the civil monetary penalty program.