FY 2003 Annual Performance Plan - Performance Measurement

5.      Performance Measurement

The Department of Labor has implemented GPRA using a systematic, staged approach based on sound business practices.  Our first priority has been the establishment of enduring outcome goals that target continual improvement in the achievement of the core results the Department is committed to delivering on behalf of working men and women.  After several years of piloting and refining goals, a majority of the Department’s programs have stabilized their goals, and DOL’s focus has progressively shifted toward ensuring the reliability of our performance measures and effectively using performance data to enhance our program results.

The Department recognizes that transitioning to a fully performance-based organization depends upon the availability of reliable and timely information concerning the results of DOL’s programs.  The challenges to performance measurement vary significantly among DOL’s programs, with the data sources and the agencies’ level of control over the reporting systems being the primary factors influencing the reliability and usefulness of the Department’s performance information.  Many DOL agencies collect critical program data from third parties, including State and local government agencies, community-based organizations, private sector employers, and international organizations.  The Department’s authorities to increase the frequency of reporting, establish data standards or verify the accuracy of the information reported by third parties are limited in some cases.  In other programs, data important to fully assess the effectiveness of strategies adopted to achieve our GPRA goals are not readily available, or DOL’s information technology systems require enhancements to support routine access by program managers to key performance data.

Despite these measurement challenges, DOL is continuing its shift toward performance-based management through a number of initiatives in FY 2003.  First, the Department plans to continue sponsoring workshops for program staff on performance management and measurement issues.  Through these workshops, program managers will learn about measurement approaches from experts in the field and will also exchange ideas and best practices with each other.  Second, the Office of Inspector General will continue its reviews of the reliability and quality of the performance data used by DOL programs in FYs 2002 and 2003, and the Department will work closely with program staff in addressing problems identified in these reviews.  Finally, the Department plans to further increase the use of program evaluations, with a continuing emphasis on assessing and improving the effectiveness of those programs highlighted for attention by OMB.

Working together, DOL executives and program managers are continuously improving the quality of the Department’s performance information and the use of that information in management decisions.  However, the success of these efforts largely depends on DOL’s authorities to collect and verify data and the commitment of critical resources by OMB and the Congress.

5.1       Addressing Specific Performance Measurement Challenges

Within the larger Departmental framework, individual DOL agencies will address data challenges that are unique to their program environment and develop solutions that are consistent with the Department’s reporting requirements.  While some DOL programs currently have adequate systems in place, others must overcome barriers to the production of timely, accurate, and relevant performance data.  In FY 2003, DOL and its agencies will continue efforts to address four issues: the availability of data, validation of data, timeliness of reporting, and the use of data in managing for results.  Below are examples of several initiatives that agencies are undertaking to further improve the measurement of our program results.

5.1.1    Employment Standards Administration

FECA’s automated system is undergoing a complete redesign covering every major program operation.  The redesign will replace a patchwork of loosely linked programs each with its own database and rules, and provide a single automated system that will make data accessible to all users.  This redesign will allow workers’ compensation claims staff to work more efficiently while providing improved customer service to injured workers, medical providers, and employing Federal agencies.

5.1.2    Employment and Training Administration

The Department continues to address the need to ensure the accuracy and reliability of performance data submitted by our employment and training system partners which serves as the foundation for key program decisions.  The development of a comprehensive data validity system for the core indicators of the WIA program, Wagner-Peyser, and other key employment and training programs is expected to be substantially completed in FY 2002.

In FY 2003, the Department will focus attention on wide-scale training and technical assistance to deploy the data validity system among states and other grantees.  Office of Inspector General reports have raised issues about the accuracy and reliability of data used to measure the outcome performance of the Welfare-to-Work program and the recently implemented Trade Adjustment Participant Reports. Both programs are expected to be the subject of special attention with regard to the validity and verification system.

5.1.3    Veterans’ Employment and Training Service

Accurate outcome oriented performance data has been an issue for VETS since performance planning under the GPRA commenced in 1999 and placed emphasis on outcomes.  Reporting data prior to GPRA was developed to respond to the reporting requirements of Title 38, Section 4107, which is essentially activity based.  In addition, reporting of placements on the ETA 9002 has been subject to error and under-counting because it requires that a veteran specialist (DVOP or LVER) make contact with a registrant to confirm that the person got a job before their employment may be reported on the ETA 9002.

VETS initiated a special study in Maryland that utilized Unemployment Insurance wage records to gain a better understanding of the outcome of services provided by veterans specialists to veterans.  This study has shown that a much larger number of veterans get jobs than are likely to have been captured by the information collection and reporting systems now in place.  This result has been corroborated by other State studies conducted by ETA.   

VETS is an active partner with ETA in designing new reporting systems that will use the results of these studies to better capture outcomes. Both VETS and ETA will be implementing outcome oriented performance measures with the public labor exchange system, which in many states has been fully integrated into the One-Stop environment.  Unemployment Insurance wage records will be the key data source supporting the new performance measures.

VETS will continue to implement the recommendations made in the July, 2000 data capacity report on VETS programs prepared by a contractor.  This will help VETS ensure that its measurement of performance is accurate and verifiable to the best degree possible, considering VETS biggest programs (DVOP and LVER)  must rely on data from the States. 

In FY 2003, VETS intends to have a new program evaluation system in place that will capture performance data on services provided to veterans by the public labor exchange system.  A key consideration that the new system will address is how to effectively verify services to veterans when the old paradigm has changed.  In past years, all veterans entering a local office were registered and a formal file established.  This made statistical sampling of files an effective way to verify services to veterans that were reported on the VETS 200 and the ETA 9002.  However, in the One-Stop environment there has been an emphasis on self-service, where there is no registration.  In fact, in some States, there is no registration even if a mediated service is provided.  This may require replacing the traditional method of statistical sampling with other verification techniques, such as surveys.

5.1.4    Mine Safety and Health Administration

MSHA has a significant database and collection system that captures most of the information necessary to track performance under the strategic plan.  Data has been collected for many years and the database is well established for performance measures under the performance goal of reducing the number of mine fatalities and the nonfatal-days-lost injury incidence rate.  However,  MSHA relies on mine operators and contractors to comply with legal requirements to report accurately and timely employment, injuries and accidents.  MSHA conducts periodic audits to ensure compliance.  The number of audits conducted may influence the degree of compliance.  Audits were increased in FY2000.  Due to the results of these audits quarterly reports are being generated for managers showing active mines which do not report quarterly employment and production information.  Routine follow-up visits are being conducted at these mines and audits will be increased as appropriate depending upon findings.  If analysis of these visits/audits determine any pattern or problem with the quality or reliability of the data, an action plan to address these findings will be generated.

MSHA’s system for determining compliance with the coal respirable dust standard has been in place since the 1970s and procedures are well established to ascertain the accuracy and reliability of the data.  Automated devices are used to weigh the inspector dust samples and automatically enter the results into a custom designed program that updates the dust data files daily.  A quality control program developed jointly by MSHA and the National Bureau of Standards assures that the weighing process continues to produce reliable results over time, and computer edit checks assure the accuracy of the database.

Metal and Nonmetal inspectors have conducted industry-wide sampling since the 1970s.  Health policies and the management information system are well established and reliable.  Automated devices are used to weigh inspector dust samples at MSHA’s analytical lab that was certified by the American Board of Industrial Hygienists in FY 1998.  Computer edits assure the accuracy of MIS data input. The Agency workforce that inspects metal and nonmetal mines has steadily declined since 1994, and the number of metal and nonmetal mines has continually increased.  As a result, metal and nonmetal samples are generally collected at the discretion of the inspector based upon the conditions observed at the mine.  Designations of high-risk occupations and new sampling procedures have been established for collecting samples at Metal and Nonmetal Mines.  Using the new procedures, samples collected in FY 2001 are used to establish a baseline.  A new goal  was established in FY 2002 and continues in FY 2003. 

MSHA’s current database and collection system will be used to capture data and samples relating to noise exposures and diesel particulates.

5.1.5    Occupational Safety and Health Administration

In FY 2003, OSHA will continue to use program data more effectively to manage OSHA’s programs to improve the safety and health in the nation’s workplaces by ensuring the quality of safety and health data.  The agency will rely on the results of program evaluations to identify the best strategies to achieve its strategic and performance goals.

OSHA will maintain its efforts to improve data validation procedures. Validation of data generated by the agency for current performance measures will be addressed through a variety of means such as annual on-site audits of the injury and illnesses records of employers to determine the accuracy and reliability of the OSHA 200 Logs, the source of data for the OSHA Data Initiative and the BLS Annual Survey; information and outreach programs and enforcement of the injury and illness recordkeeping regulations; revision of injury and illness recordkeeping system (regulations, forms, and guidelines) to improve the quality of records by simplifying forms and regulations, providing clearer guidance for employers, and incorporating incentives for employers to maintain high quality records; and continuing the various methods OSHA developed for validating and verifying data in the OSHA Integrated Management Data System (IMIS).

Program evaluations will assess how well OSHA’s programs, policies and procedures are working, including the effectiveness of specific standards, customer satisfaction, and specific approaches towards reducing occupational injuries and illnesses. 

5.2       Linking Costs to Performance

The Department has a solid financial systems infrastructure from which a cost accounting capability is being developed using the resources of a reliable, established accounting system-the Department of Labor Accounting and Related Systems (DOLAR$). DOLAR$, serving as the system of record for financial results throughout the Department, has been modified to capture, aggregate, allocate and report costs. The Department's cost systems allow aggregation of costs across agency lines and the allocation of direct and indirect costs to the strategic and outcome goal levels established in the Department's Strategic Plan. In the FY 2001 Annual Report on Performance and Accountability, the Department provided cost data for FY 1999 through FY 2001 to accompany and provide perspective on each outcome goal presentation in the Performance Report section

The Department has maintained cost accounting information, beginning in FY 1999, for the outcome goals in the Department's Strategic Plan. In this FY 2003 plan, the Department continued its practice of linking budget authority and outlays to both strategic and outcome goals, as shown in Chapter 4. In addition, Appendix B provides an overview of the linkage between budget activities and outcome goals. DOL will continue to develop the capability to consolidate data from a variety of program and financial system sources and link that data as needed to meet the performance reporting requirements of GPRA.

During the remainder of FY 2002, the Department will undertake new initiatives to more closely link cost data to performance planning and analysis. For example, the Department will pilot an integrated budget and performance plan presentation for at least one program during the preparation of DOL's FY 2004 budget and performance plans. In addition, the Department will develop prototype goals to improve the cost effectiveness of one job training and one regulatory program, with the expectation that these prototypes could be adapted to other DOL job training and regulatory programs.