|
The Department of Labor has implemented GPRA using a systematic, staged
approach based on sound business practices. Our first priority has been the
establishment of enduring outcome goals which target continual improvement in
the achievement of the core results the Department is committed to delivering
on behalf of working men and women. After several years of piloting and
refining goals, a majority of the Department’s programs have stabilized
their goals, and DOL’s focus has progressively shifted toward ensuring the
reliability of our performance measures and effectively using performance data
to enhance our program results.
The Department recognizes that transitioning to a fully
performance-based organization depends upon the availability of reliable and
timely information concerning the results of DOL’s programs. The
challenges to performance measurement vary significantly among DOL’s
programs, with the data sources and the agencies’ level of control over
the reporting systems being the primary factors influencing the reliability and
usefulness of the Department’s performance information. Many DOL agencies
collect critical program data from third parties, including State and local
government agencies, community based organizations, private sector employers,
and international organizations. The Department’s authorities to increase
the frequency of reporting, establish data standards or verify the accuracy of
the information reported by third parties are limited in some cases. In other
programs, data important to fully assess the effectiveness of strategies
adopted to achieve our GPRA goals are not readily available, or DOL’s
information technology systems require enhancements to support routine access
by program managers to key performance data.
A number of initiatives to improve performance measurement and
performance based management will be undertaken in FY 2001 and additional
efforts are planned for FY 2002. During FY 2001, the Department will initiate a
series of workshops and seminars on GPRA implementation to facilitate an
exchange of best practices among DOL agencies, and performance measurement
related issues will be a featured topic. In addition, the Office of Inspector
General will expand reviews of the reliability and quality of the performance
data reported by DOL programs in FYs 2001 and 2002, and the Department will
work closely with our agencies in addressing problems identified by the audits.
The Department plans in FY 2002 to increase the use of program evaluations to
meet a variety of performance management objectives, including supplementing
information available through routinely collected performance information,
reviewing the effectiveness of strategies in meeting key performance goals and
analyzing the quality of data sources, indicators and reported program
results.
Working together, DOL executives and program managers will continuously
improve the quality of the Department’s performance information. However,
success in overcoming some of the challenges the Department faces in ensuring
the reliability of performance measures and effectively using performance data
to enhance the results of DOL’s programs will also require support for
data collection and verification authorities and a commitment of critical
resources by OMB and the Congress.
5.1 Addressing Specific Performance Measurement
Challenges
Within the larger Departmental framework, individual DOL agencies will
address data challenges that are unique to the agency’s program
environment and develop solutions that are consistent with the
Department’s reporting requirements. While some DOL programs currently
have adequate systems in place, others must overcome barriers to the production
of timely, accurate, and relevant performance data. In FY 2002, DOL and its
agencies will continue efforts to address four issues: the availability of
data, validation of data, timeliness of reporting and the use of data in
managing for results. The following examples describe several initiatives
completed or in progress to further improve the measurement of our program
results.
- The Employment Standards Administration will continue its efforts to
improve data measuring the results of three major programs in FY 2001. First,
the Wage and Hour program has introduced a new computer system, the Wage and
Hour Investigative and Reporting Database (WHISARD), permitting more timely
access to extensive information. Because there is no unbiased database on labor
standards violations or compliance, Wage and Hour faces a major challenge in
determining industry-wide levels of compliance and measuring changes in that
compliance. To determine the impact of Wage and Hour efforts, a statistically
sound method has been developed for establishing baselines and measuring
compliance using investigation-based compliance surveys of targeted industries
and areas. Data on the outcomes of repeat investigations will also be used to
evaluate the relative effectiveness, or return on investment, of the various
types of interventions.
The new WHISARD system provides many advantages
when compared to its predecessor. For example, data is entered into the system
directly at the source (by Wage and Hour investigators) rather than manually
batched and mailed to a central source for data entry later. As a result, data
is much more current, which facilitates tracking the progress of investigative
activity. Information on a complainant’s case is readily available which
enhances customer service and satisfaction. All users of the WHISARD system
have direct access, which was not previously possible and enhances efficiency
and enforcement effectiveness. In addition, WHISARD can produce data on a much
broader range of activities, such as local enforcement initiatives, which was
not possible with the prior system, and assists in the agency’s strategic
planning activities.
The Office of Federal Contract Compliance
Programs’ (OFCCP) Case Management System (CMS) is used to measure program
performance. Through software and hardware enhancements, field office managers
are now able to access CMS data to track the accomplishments of individual
organizational units. OFCCP has developed a database comprised of Federal
contractor-submitted personnel activity data and other required records, with
the contractor having the option to input directly to the database via the
Internet. Following the final refinement of the analytical model (scheduled for
completion in FY 2001) the response data will be analyzed and used in
determining the scheduling of technical assistance, outreach and compliance
evaluation actions, further increasing OFCCP’s efficiency and
effectiveness.
Increasingly sophisticated databases detailing
the case histories of injured Federal workers have been developed for the
Federal Employees’ Compensation Act (FECA) program. These systems
allow tracking of the goal of reducing lost production days, but also permit
precise evaluation of various program initiatives such as the impact of
alternative return to work techniques on various groupings of employees or
injury categories.
FECA’s automated system is undergoing a
complete redesign covering every major program operation. The redesign will
replace a patchwork of loosely-linked programs each with its own database and
rules, and provide a single automated system that will make data accessible to
all users. This redesign will allow workers’ compensation claims staff to
work more efficiently while providing improved customer service to injured
workers, medical providers, and employing Federal agencies.
- The Department continues to address the need to ensure the accuracy
and reliability of performance data submitted by our employment and training
system partners which serves as the foundation for key program decisions. The
development of a comprehensive data validity system for the core indicators of
the WIA program, Wagner-Peyser, and other key employment and training programs
is expected to be substantially completed in FY 2002.
- Accurate outcome oriented performance data has been an issue for VETS
since performance planning under the GPRA commenced in 1999 and placed emphasis
on outcomes. Reporting data prior to GPRA was developed to respond to the
reporting requirements of Title 38, Section 4107, which is essentially activity
based. In addition, reporting of placements on the ETA 9002 has been subject to
error and under-counting because it requires that a veteran specialist (DVOP or
LVER) make contact with a registrant to confirm that the person got a job
before their employment may be reported on the ETA 9002.
VETS initiated a special study in Maryland that
utilized Unemployment Insurance wage records to gain a better understanding of
the outcome of services provided by veterans specialists to veterans. This
study has shown that a much larger number of veterans get jobs than are likely
to have been captured by the information collection and reporting systems now
in place. This result has been corroborated by other State studies conducted by
ETA. VETS has initiated performance measure pilots in seven States that will
look at using UI wage records for a number of outcomes, including entered
employments, earnings gain, and retention in employment. VETS is an active
partner with ETA in designing new reporting systems that will use the results
of these studies to better capture outcomes.
VETS will continue to implement the
recommendations made in the July, 2000 data capacity report on VETS programs
prepared by a contractor. This will help VETS ensure that its measurement of
performance is accurate and verifiable to the best degree possible, considering
the fact that for its biggest programs (DVOP and LVER) VETS must rely on data
from the States. In FY 2001, VETS intends to have an updated evaluation system
in place for use by VETS field staff in their reviews of grantee performance.
This evaluation system will replace the current LESO (Local Employment Service
Office) Evaluation Manual that VETS has been utilizing to verify service to
veterans since 1989.
A key consideration that the new system will
address is how to effectively verify services to veterans when the old paradigm
has changed. In past years, all veterans entering a local office were
registered and a formal file established. This made statistical sampling of
files an effective way to verify services to veterans that were reported on the
VETS 200 and the ETA 9002. However, in the One-Stop environment there has been
an emphasis on self-service, where there is no registration. In fact, in some
States, there is no registration even if a mediated service is provided. This
may require replacing the traditional method of statistical sampling with other
verification techniques, such as surveys.
- Past reports by GAO and OIG raised concerns about the reliability and
accuracy of placement and job retention data used to measure the outcome
performance of Job Corps participants. Revisions to Job Corps' procedures have
satisfactorily addressed all of the issues cited by both reports. As an example
of enhanced procedures, neutral third party data verification is used to
validate reported measures. A random sample of placements (75% of those
reported) is verified using an independent placement verifier to ensure data
integrity. Additionally, a centralized data system with numerous management
information reports and system edit checks is used to minimize error in data
reporting. Each Job Corps contractor reporting participant achievements is
required to maintain systems to validate their data.
In FY 2002, progress will continue to be made
for indicators where data are currently unavailable or incomplete. For example,
the employment, retention and earnings gains outcomes for TAA/NAFTA program
participants are being revised to reflect the implementation of the WIA. In
addition, for indicators adopted with the implementation of WIA, baselines will
be reviewed and indicators refined. This will be done at the agency and program
level in conjunction with the Department-wide effort to improve the quality of
indicators.
- MSHA has a significant database and collection system that captures
most of the information necessary to track performance under the strategic
plan. Data has been collected for many years and the database is well
established for performance measures under the strategic goal of reducing the
number of mine fatalities and the nonfatal-days-lost incidence rate.. However,
MSHA relies on mine operators and contractors to comply with legal requirements
to report accurately and timely employment, injuries and accidents. MSHA
conducts periodic audits to ensure compliance. The number of audits conducted
may influence the degree of compliance. Audits were increased in FY2000. Due to
the results of these audits, beginning in FY 2001 quarterly reports are
generated for managers showing active mines which do not report quarterly
employment and production information. Routine follow-up visits will be
conducted at these mines and audits will be increased as appropriate depending
upon findings. If analysis of these visits/audits determine any pattern or
problem with the quality or reliability of the data, an action plan to address
these findings will be generated in FY 2002.
MSHA’s system for determining compliance
with the coal respirable dust standard has been in place since the 1970s and
procedures are well established to ascertain the accuracy and reliability of
the data. Automated devices are used to weigh the inspector dust samples and
automatically enter the results into a custom designed program that updates the
dust data files daily. A quality control program developed jointly by MSHA and
the National Bureau of Standards assures that the weighing process continues to
produce reliable results over time, and computer edit checks assure the
accuracy of the database.
Metal and Nonmetal inspectors have conducted
industry-wide sampling since the 1970s. Health policies and the management
information system are well established and reliable. Automated devices are
used to weigh inspector dust samples at MSHA’s analytical lab, which was
certified by the American Board of Industrial Hygienists in FY 1998. Computer
edits assure the accuracy of MIS data input The Agency workforce that inspects
metal and nonmetal mines has steadily declined since 1994, and the number of
metal and nonmetal mines has continually increased. As a result, metal and
nonmetal samples are generally collected at the discretion of the inspector
based upon the conditions observed at the mine. For FY 2002 and FY 2003 a new
goal and baseline have been established. Designation of high risk occupations
and new sampling procedures have been established for collecting samples at
Metal and Nonmetal Mines. This replaces the previously used indexing method.
Using the new procedures, samples collected in FY 2001 will be used to
establish a new baseline. For MSHA’s current database and collection
system will be used to capture data and samples relating to noise
exposures.
- In FY 2002, OSHA plans to continue to use program data more
effectively to manage OSHA’s programs in improving the safety and health
in the nation’s workplaces by ensuring the quality of safety and health
data, and continuing to use program evaluations.
The Occupational Safety and Health
Administration will continue to improve data validation procedures. Validation
of data generated by the agency for current performance measures will continue
to be addressed through a variety of means such as annual on-site audits of the
injury and illnesses records of employers to determine the accuracy and
reliability of the OSHA 200 Logs, the source of data for the OSHA Data
Initiative and the BLS Annual Survey; information and outreach programs and
enforcement of the injury and illness recordkeeping regulations; revision of
injury and illness recordkeeping system (regulations, forms, and guidelines) to
improve the quality of records by simplifying forms and regulations, providing
clearer guidance for employers, and incorporating incentives for employers to
maintain high quality records; and continuing the various methods OSHA
developed for validating and verifying data in the OSHA Integrated Management
Data System (IMIS).
Program evaluations will assess how well
OSHA’s programs, policies and procedures are working, including the
effectiveness of specific standards, customer satisfaction, and specific
approaches towards reducing occupational injuries and illnesses.
5.2 Linking Costs to Performance
The Department has a solid financial systems infrastructure from which a
cost accounting capability is being developed using the resources of a
reliable, established accounting system—the Department of Labor Accounting
and Related Systems (DOLAR$). DOLAR$, serving as the system of record for
financial results throughout the Department, has been modified to capture,
aggregate, allocate and report costs. A cost accounting module has been
developed to allow aggregation of costs across agency lines and to allocate
direct and indirect costs to the strategic and outcome goal levels established
in the Department’s Strategic Plan.
The Department has maintained cost accounting information, beginning in
FY 1999, for the outcome goals in the Department’s Strategic Plan. In FY
2002, the Department has for the first time linked budget authority and outlays
to both strategic and outcome goals—Appendix B provides an overview of the
linkage between budget activities that support the Department’s outcome
goals. DOL will continue to develop the capability to consolidate data from a
variety of program and financial system sources and link that data as needed to
meet the performance reporting requirements of GPRA.
|
