ANNUAL REPORT FY 2001

APPENDIX 3

SIGNIFICANT FY 2001 AUDITS AND EVALUATIONS

1. National Job Corps Study: The Impacts of Job Corps on Participants’ Employment and Related Agency:

Conducted by: Mathematica Policy Research, Inc.

Findings: The National Job Corps Study evaluated the program’s impact on students who applied to Job Corps between November 1994 and December 1995. Follow-up interviews were conducted over a 4-year period with youth in the randomly assigned participant and control groups. Findings included: (1) For every dollar spent for Job Corps, the benefit to society is $2.02; (2) Participation in Job Corps led to significant increases in education and training, including much greater attainment of GEDs and vocational certificates; (3) Job Corps generated positive employment and earnings gains, including higher paying jobs with more benefits; (5) Job Corps significantly reduced youths’ involvement with the criminal justice system; (6) Job Corps reduced the receipt of public assistance; and (7) the residential and non-residential components of Job Corps are each effective for the youths they serve.

Actions Taken: The evaluation affirms Job Corps’ program strategies. Thus, no actions have been taken to modify strategies based on these findings.

Additional Information: For additional copies of this report, please phone Dan Ryan at (202) 693-3649, e-mail him at dryan@doleta.gov, or send a request to the U.S. Department of Labor, Employment and Training Administration, Office of Policy and Research, Room N5637, 200 Constitution Ave., N.W., Washington D.C. 20210

2. American Customer Satisfaction Index (ACSI)

Agency: ETA

Conducted by: The ACSI is produced by a partnership between the University of Michigan Business School, the American Society for Quality, and the Federal Consulting Group.

Findings: The American Customer Satisfaction Index (ACSI) is an internationally accepted, measurement and diagnostic tool used by over 200 companies in the public and private sectors to gauge customer satisfaction with services they receive. Along with over 100 other Federal Government entities, the Office of Job Corps was selected to participate in the ACSI to gauge citizen satisfaction with Federal services. A national sample of parents/guardians of Job Corps graduates were asked their level of satisfaction with the services their children/students received while enrolled in Job Corps. Job Corps received a customer satisfaction rating of 80 percent, higher than both the federal government wide score (68.6 percent) and the private sector comparable score (71.2 percent). Additionally, 86% of the parents/guardians demonstrated confidence that Job Corps will continue to be effective in the future, and that they would be likely to recommend Job Corps to other parents/guardians.

Actions Taken: The ACSI survey affirms Job Corps’ program strategies. Thus, no actions have been taken to modify strategies based on these findings.

Additional Information: For additional information, contact the Federal Consulting Group, a franchise of the U.S. Department of the Treasury, by fax at (202) 906-6162, or e-mail Bernard Lubran at EBernie.Lubran@ots.treas.gov. by telephone at (202) 906-6068. The mailing address is 1700 G Street, NW, Third Floor, Washington, D.C., 20552.

3. Job Training Partnership Act, Title II-C Out-of-School Program Performance

Agency: ETA

Conducted by: Office of Inspector General (OIG), U.S. Department of Labor

Findings: (1) Positive outcomes for the Title II-C Out-of-School Youth Program were significantly lower than those reported. (2) Documentation supporting youth employability enhancements was seriously deficient. (3) Participants’ post-program earnings were affected by program interventions and level of participants’ participation.

Recommendations: (1) Inclusion of a review of implementing such a data validation effort in the monitoring program. (2) Notification of all states and the substate grantees to include: (a) audit results with emphasis on the necessity for states and their grantees to validate outcome data as part of their monitoring program and training provider eligibility determinations; (b) the importance of documenting not only reported outcomes but also specific services provided, dates services were provided, and actual program exit date; and, (c) the importance of not only enrolling youth in occupational skills training activities, but also finding ways to keep them actively participating in the program to completion, to maximize their post-program earnings.

Action Taken: In response, comprehensive guidance on youth services under WIA was issued in TEGL 18-00 on April 23, 2001. In addition, DOL notified the system of this audit report through TEGL 23-00 on June 11, 2001 and in this issuance emphasized some of the OIG recommendations. Finally, DOL is undertaking a data validation initiative to create both more precise programming specifications and more precise standards for validating data quality.

Additional Information: The complete report can be found on the Office of Inspector General website at
http://www.oig.dol.gov/public/reports/oa/2001/06-01-001-03-340.pdf.

4. Welfare-to-Work Competitive Grant Program Performance Audit (03-01-007-03-386)

Agency: ETA

Conducted by: Office of Inspector General (OIG), U.S. Department of Labor

Findings: (1) Competitive grantees need to improve services to WtW participants to ensure long-term employment. (2) WtW competitive grant program performance data are not reliable and the current evaluation approach may not be sufficient to assess the program as a whole. (3) Participants’ post-program earnings were affected by program interventions and level of participants’ participation.

Recommendations: OIG recommends that the Assistant Secretary for Employment and Training: (1) Direct the Office of Welfare-to-Work to require grantees to follow up on participants in unsubsidized employment to determine if additional services are needed to promote employment retention. (2) Direct the Office of Welfare-to-Work to conduct data validation reviews at the grantee level to ensure that the performance data being reported are accurate and complete, and to review for accuracy and completeness all performance data previously reported. (3) Consult with DHHS officials to reexamine the current approach being used to evaluate the WtW grants program, and to ensure that any national evaluation be designed to assess the innovative approaches used to move welfare recipients from welfare dependency to economic self-sufficiency.

Action Taken: Recommendation 1: ETA will focus its 2002 technical assistance activities specifically on retention and wage gains. By the end of the calendar year 2002, the Department will implement a cooperative venture with Temporary Assistance for Needy Families entitled, "Pathways to Advancement," to experiment with and pass on the lessons learned from selected grantees participating in this retention and wage-gain pilot program. Recommendation 2: ETA had already begun addressing data validation issues when the OIG conducted its audit. ETA has taken a number of steps to address the OIG’s concerns: (a) The Department has been working with Mathematica Policy Research, Inc. on a data validation project and has completed two parts of its long-planned three-part data validation strategy. The data validation process includes a review of all previously reported data for accuracy and completeness. (b) ETA has completed, posted on its website, trained regional office staff and provided hard copy guidance to grantees on a performance reporting package. [The package defines the specific data elements required to produce the WtW cumulative quarterly report; the precise calculations required to process the data elements to produce each count on the report; and also prescribes a file structure to enable WtW program managers and technical staff to analyze and validate performance outcomes.] (c) ETA has completed, posted on its website, trained regional office staff and provided hard copy guidance to grantees on reporting and performance software. The downloadable software calculates performance outcomes and generates all other required report information, among other functions. (d) ETA has issued a data validation guide, which prescribes the documentation that grantees are required to collect to support the participant eligibility and performance data they report. Recommendation 3: The Department has begun meeting with the Department of Health and Human Services to assure that their national evaluation design is more tailored to assess the innovative approaches being used to move welfare recipients from welfare dependency to economic self-sufficiency.

Additional Information: The complete report can be found on the Office of Inspector General website at
http://www.oig.dol.gov/public/reports.

5. Registered Apprenticeships: Labor Could Do More to Expand to Other Occupations GAO-01-940)

Agency: ETA

Conducted by: General Accounting Office (GAO)

Findings: Lack of systematic identification of occupations and the need to allay employers’ concerns slow expansion of apprenticeships. Specifically: (a) DOL lacks a systematic process to expand apprenticeship. (b) Employer perceptions present challenges to expanding apprenticeships; (c) DOL’s inadequate information system prevents measuring progress and sharing information. (d) Some apprenticeship programs that respond to current labor needs have overcome impediments.

Recommendations: (1) Lead a systematic effort to work with State apprenticeship councils and others interested to identify apprenticeable occupations that have shortages of skilled labor and establish plans for promoting apprenticeship programs in these occupations. (2) Work with other federal workforce development programs to identify funding for developing apprenticeships when additional support is needed. (3) Establish a mechanism for sharing among Labor representatives and employers information on apprenticeship programs, particularly those in occupations not traditionally apprenticed. (4) Ensure that the apprenticeship database contains detailed information on current programs so that accurate and complete information is shared and progress in meeting labor market needs can be evaluated.

Action Taken: In response, ETA has done or will do the following: take the lead on identifying apprenticeship opportunities for occupations with shortages in skilled labor as well as new and emerging industries; expand apprenticeship in high skilled occupations and high skilled industries to help meet 21st Century workforce [This includes working closely with the Department’s Office of the 21st Century Workforce to identify industries with skill shortages that could benefit by using the registered apprenticeship training system.] In partnership with the Federal Committee on Registered Apprenticeship, the National Association of State and Territorial Apprenticeship Directors, and State workforce leaders, identify opportunities to expand the apprenticeship system to non-traditional as well as new and emerging occupations.

These activities began in conjunction with FY 2002 and are being coordinated through a series of stakeholder meetings, industry association conferences, collaboration with other federal agencies, and a series of business forums. Through enhanced coordination with other federal programs involved as both mandatory and voluntary partners in the workforce investment system authorized by the Workforce Investment Act, ETA will explore additional financial, as well as technical and communications, support for expanding apprenticeship opportunities. ETA will facilitate increased employer and labor involvement in the full array of workforce development programs, including apprenticeship. ETA has initiated actions to build stronger relationships with Workforce Investment Boards, One-Stop Centers, businesses, other federal agencies, and community and faith-based organizations.

ETA has also placed informational materials in virtually all One-Stop Centers, made personal contact with 286 Workforce Investment Boards, and interacted with 793 Chambers of Commerce, business and labor organizations to build awareness of the value of the national apprenticeship system; engaged a contractor to manage a major public information initiative. The initiative provides for a new and innovative approach for registered apprenticeship in new and emerging industries and develops public information and training strategies to assist employers in identifying and developing high skill training through registered apprenticeship.

ETA has also begun to make the necessary improvements to correct weaknesses in its database. The current redesign of the apprenticeship information management system began with the Business Discovery Project conducted by NCR Corporation and the subsequent forums conducted through the Apprenticeship Impact Project. The final design, projected for completion in FY 2002, will provide accurate and complete information that will be shared with the Registered Apprenticeship System. This new system will allow ETA to more fully evaluate market participation in industries that have been identified as having significant skill shortages.

Additional Information: The complete report can be found on the General Accounting Office website at http://www.gao.gov.

6. GPRA Performance Measures Audit of the Dislocated Worker (DW), Trade Adjustment Assistance (TAA), and North American Free Trade Agreement Transitional Adjustment Assistance (NAFTA-TAA) Programs (23-01-001-03-330)

Agency: ETA

Conducted by: Office of Inspector General (OIG), U.S. Department of Labor

Findings: Regarding Federal, State and contractor monitoring, the OIG found that: (1) The primary contractor that assists ETA with program information for these programs did not perform on-site monitoring. (2) The monitoring policies, systems, and procedures in the five States OIG visited provided for regular and comprehensive monitoring of program performance and results information. (3) Federal monitoring of these programs needs to be significantly strengthened.

Regarding meeting GPRA and other requirements, the OIG found that: (1) The mission statements for ETA’s Division of Adults and Dislocated Workers and Division of Trade Adjustment Assistance were process oriented and not consistent with the purpose statements in the programs’ authorizing legislation. (2) The external goals, strategies, targets and external performance measures for each program were consistent with the program’s purpose(s) but not consistent with the existing mission statements which need to be revised. (3) Internal performance measures had not yet been identified but were reported to be under development. (4) The existing linkages between funding and goals and between costs and results are not adequate to meet current requirements.

Regarding ETA’s direction and management related to these programs, the OIG found that: (1) The States OIG visited were highly complimentary about ETA regional office help. However, they also expressed concerns about the clarity and consistency of ETA national office direction and procedures for WIA implementation, national emergency grants, and TAA and NAFTA-TAA petitions. (2) In implementing the GPRA, ETA has successfully created and maintained effective intergovernmental cooperation while establishing program goals, targets and measures, developing and implementing strategies, and designing monitoring and reporting mechanisms.

Recommendations: (1) Regarding Federal, State and contractor monitoring, the OIG recommends that the Assistant Secretary for Employment and Training ensure that the Office of Adults, Dislocated Workers, and TAAimplement the plans for the reinstitution and improvement of ETA monitoring of the DW, TAA and NAFTA-TAA programs in accord with the plans outlined by the Directors of the Division of Adults and Dislocated Workers, the Division of National Emergency Grants, and the Division of Trade Adjustment Assistance. (2) Regarding meeting GPRA and other requirements, the OIG recommends that the Assistant Secretary for Employment and Training ensure that the Office of Adults, Dislocated Workers, and TAA take the following actions: (a) Revise the mission statements for the Division of Adults and Dislocated Workers and the Division of Trade Adjustment Assistance. (b) Continue the development of internal performance measures for the Office of Adults, Dislocated Workers and TAA, and the Divisions of Adults and Dislocated Workers, National Emergency Grants, and Trade Adjustment Assistance. (c) Develop, in conjunction with the Office of Financial and Administrative Management and the Office of the Chief Financial Officer, the linkages between resources and goals as well as costs and results required by current legislation and directives. (3) The OIG did not have any recommendations regarding its third finding.

Action Taken: In response to the first recommendation, ETA is developing work plans, which when implemented this year, will improve the monitoring of dislocated worker programs. These work plans are being developed by the Office of Adult Services – which includes the Division of Adults and Dislocated Workers, National Emergency Grants, and Trade Adjustment Assistance – in consultation and coordination with the regional offices of ETA. In general, monitoring activities planned for the coming year will emphasize performance monitoring and targeting technical assistance to ensure the achievement of performance goals. In response to the second recommendation, ETA’s Office of Adult Services is consulting with the agency’s Office of Human Resources on the review of mission statements. Such reviews undertaken will be consistent with the agency’s overall initiative to implement Baldridge principles.

In terms of developing internal measures for the various programs, ETA concurs with the recommendation, with the understanding that there are currently internal measures in effect for both TAA/NAFTA and NEG. These measures include work processing deadlines stipulated in either legislation or current policy. Development of comprehensive internal measures for the adults, dislocated workers and trade programs will occur within the Baldrige organizational performance framework.

In terms of developing linkages between resources and goals as well as costs and results required by current legislation, ETA concurs with the recommendation to integrate performance and budget in its annual program performance plans. ETA’s Office of Financial Management and Office of the Chief Financial Officer have agreed to develop two pilot studies that will provide a model that may be applicable ETA-wide in establishing the linkages called for in the recommendation.

Additional Information: The complete report can be found on the OIG’s website at
http://www.oig.dol.gov/public/reports/oa/2001/main.htm.

7. Trends, Outcomes, and Management Issues in Trade Act Programs (GAO-01-59)

Agency: ETA

Conducted by: General Accounting Office

Findings: In its review of the Trade Adjustment Assistance (TAA) and North American Free Trade Agreement Transitional Adjustment Assistance (NAFTA­TAA) programs, GAO found the following issues: (1) It is unclear whether the TAA and NAFTA-TAAprograms have met the Department of Labor’s reemployment and wage retention goals for all workers, due to limited data on program outcomes. The Department of Labor and states have had long-standing problems in tracking TAA and NAFTA-TAA benefits and results. (2) Improving the Department of Labor's timeliness in processing certification petitions needs to be addressed because such delays can influence workers' decisions about training or career choices. (3) The absence of internal controls such as cross-checks between state and Department of Labor data may have resulted in ineligible workers receiving benefits under the programs. (4) Other challenges facing the Trade Act Programs stem from statutory requirements. Under current law, workers who experience breaks in training of more than 14 days lose their benefits, which state officials report as a barrier to effective service delivery. (5) The two trade adjustment assistance programs are administered under differing rules that make program administration challenging for state officials and confusing to workers.

Recommendations: GAO has the following recommendations for DOL, which relate to the first three findings above: (1) Establish an effective performance measurement system to track participant outcomes and then disseminate that information to the States and localities so that they can better serve participants. (2) Establish procedures that will enable the Department to certify workers for TAA and NAFTA-TAAwithin required time frames. (3) Establish more effective internal controls and improve oversight with regard to benefit payments by States in order to prevent workers ineligible for benefits from receiving them.

GAO also recommends that Congress consider simplifying administration of the TAA and NAFTA-TAA programs by standardizing time frames for workers to enter training, training waiver policies for certified workers; and time frames for completing certification investigations. These recommendations relate to the last two findings.

Action Taken: The Department has approved and implemented a revised reporting system that is an improvement by using performance indicators and definitions that are consistent with those of the Workforce Investment Act and using UI wage records (less costly and more consistently reliable) as the basis for reporting employment, earnings and retention. It has put new procedures in place and has hired contractor staff to assist in completing investigations timely. DOL has also collected information and corrections on benefit data from 28 of 31 States. In all cases, States made reporting errors, but did not pay benefits to ineligible workers.

Additional Information: The complete report can be found on the General Accounting Office website at http://www.gao.gov.

8. Improving Trade Act Programs (04-01-009-03-330)

Agency: ETA

Conducted by: Office of Inspector General (OIG), U.S. Department of Labor

Findings: OIG reviewed the TAA and NAFTA-TAA programs’ Fiscal Year 1999 activities and evaluated the programs’ performance by examining 724 program participants’ experiences in 16 states. OIG’s audit revealed the following findings: (1) While most program participants found jobs, fewer obtained suitable employment. (2) Less than one in four participants obtained program-assisted suitable employment. Participants did not find suitable employment for a variety of reasons, and many were outside of the service providers’control. (3) Front-end activities were emphasized, not program outcomes. Participant outcomes have received inadequate emphasis for a variety of reasons, including that there is no relationship between participants’ employment outcomes and the Trade programs’ funding. (4) Training was moderately successful. (5) Program data were unreliable. The reporting system used by the states to capture data on participants that had terminated from the program was unreliable. The system was often ignored or neglected by the states.

Recommendations: OIG has the following recommendations for DOL: (1) Ensure that States have suitable employment goals translated down to the local level of program services. (2) Ensure that States have clear, uniform procedures for program termination. (3) Ensure that States’ trade adjustment activities are an integral part of their program evaluation and monitoring systems. (4) Ensure that the Department and the States monitor participant data in the new TAA reporting system to ensure the data are accurate and complete.

Action Taken: The Department agrees with the OIG recommendations and has already made program improvements that address the OIG concerns. DOL has refined its performance measures and goals, and it will continue to work closely with States to ensure local offices are better aware of performance expectations. Ongoing training sessions and forthcoming directives will emphasize to State and local office Trade Act staff the importance of managing for performance and outcomes for participants, rather than simply for process. DOL’s revised Trade Act Participant Report system and new program review guide will further the implementation of the OIG recommendations.

Additional Information: The complete report can be obtained through the OIG’s website at http://www.oig.dol.gov/.

9. Flexibility and Accountability Needed to Improve Service to Veterans.

Agency: VETS

Conducted by: General Accounting Office (GAO)

Findings: (1) Veterans were receiving priority of service, however the effectiveness of service cannot currently be determined; (2) VETS does not adequately oversee DVOP and LVER Grants; (3) VETS monitoring is often unproductive; (4) DVOP and LVER programs do not always operate well in One-Stop Centers; (5) DVOP and LVER programs lack flexibility; and, (6) VETS has not adequately adapted to the One-Stop Center environment.

Recommendations: Matters for Congressional Consideration:(a) Provide States and local offices more discretion to decide where to locate DVOP and LVER staff and provide States the discretion to have half-time DVOP positions; (b) Allow VETS and/or States the flexibility to better define the roles and responsibilities of staff serving veterans. Instead of including these duties in the law, combine the DVOP and LVER grant programs into one staffing grant to better meet the States’ needs for serving veterans. (c) Provide VETS with the flexibility to consider alternative ways to improve administration and oversight of the staffing grants; (d) Eliminate the requirement that VETS report to the Congress a comparison of the job placement rate of veterans with that of non-veterans; and (e) Eliminate the requirement that VETS report on Federal Contractor Job Listings. GAO also noted that Congress should consider making the DVOP and LVER grant funding cycle consistent with that of other employment and training programs.

Recommendations for Executive Action:(a) Specify performance goals and expectations for serving veterans and allow States the flexibility to present a plan for how they intend to meet these goals and expectations; (b) Implement, as soon as possible, a performance measurement system that holds States accountable, reflects the agency’s goals and expectations, and defines how the performance data should be collected to ensure accuracy and reliability; (c) Implement a performance management system for the State grantees that provides incentives for meeting goals and penalties, beyond corrective action plans, for not meeting goals, and (d) Update oversight guidelines and improve staff training to ensure consistent monitoring of DVOP and LVER programs in one-stop centers.

Actions Taken: The Department has responded that it is willing to provide additional flexibility in those areas where it does not negatively impact on services to veterans. DOL has also agreed to work, in partnership, with stakeholders and Congress to develop legislation that ensures maximum employment and training opportunities for veterans, improves communications and coordination with the States, and provides them with more flexibility and coordination.

The Veterans Employment and Training Service has also developed new performance measures jointly with ETA to be effective July 2002, and planned development, during FY 2002 of a new procedural guide for VETS staff and a Grants Management course that will emphasize working with States and providing technical assistance.

However, the concepts of combining the DVOP and LVER grants, and enabling half-time DVOPs, although beneficial to States, may not be beneficial to our veteran customers and need to be given more consideration. VETS supports performance incentives and has established a task group to look at their feasibility and alternatives under existing statutory authority.

10. Proposed Performance Measurement System Improved, But Further Changes Needed.

Agency: VETS

Conducted by: General Accounting Office (GAO)

Findings: (1) VETS’ proposed measures hold promise, but some concerns remain; and (2) proposed data source for new measures is an improvement but will bring some challenges.

Recommendations: (1) Redefine staff-assisted services to include only those that may be considered staff intensive, such as case management, so that VETS will be able to evaluate the success of intensive staff-assisted services; (2) Clearly define target populations so that staff assisting veterans know where to place their priorities; (3) Eliminate the measure related to federal contractor jobs; (4) Establish and communicate guidelines that standardize how to count veterans for performance-reporting purposes so that VETS will be able to assess program performance nationwide.

Actions Taken: (1) The staff-intensive service definition is cast in the Workforce Investment Act (WIA) (2) VETS has identified veterans and disabled veterans as the target populations. (3) The federal contractor job listing measure will not be implemented. (4) ETA and VETS are working with the States to address the issue of standardizing applicant registration and enabling nationwide consistency, while observing the legislative intent of WIA to allow States flexibility in this area.

11. Government Information Security Reform Act Evaluation of the Bureau of Labor Statistics’ International Price Program (September 2001)

Agency: BLS

Conducted by: Office of Inspector General (OIG)/KPMG LLP

Findings: There were three findings related to: (1) procedural documentation for the appropriate handling and securing of security incident-related information; (2) monitoring completion of security training; and (3) physical access controls to the mailroom.

Recommendation(s): (1) Ensure that BLS institutes and enforces guidelines that document the procedures for the handling and storage of security incident-related information; (2) Ensure a tracking tool to follow the completion of security training; and (3) Improve physical access controls to the mailroom.

Actions Taken: BLS agrees with the recommendations and expects to close the recommendations in FY 2002.

Additional Information: A copy of the final evaluation may be obtained from the Bureau of Labor Statistics, Division of Management Systems, Room 3255, 2 Massachusetts Avenue, NE, Washington, DC 20212, or by calling 202-691-7628.

12. GAO Review of Consumer Price Index (CPI) Quality Adjustments (June 1999)

Agency: BLS

Conducted by: General Accounting Office (GAO)

Findings: BLS does not have a program to assess the decision-making patterns of commodity analysts. No evidence was found, however, to indicate whether errors or inconsistencies in commodity analysts’ decisions or lack of comprehensive reviews of those decisions has had a material effect on the calculation of the CPI.

Recommendation(s): There was one recommendation to evaluate, on a periodic basis, the degree of consistency and accuracy in analysts’ substitution determinations and the resulting effects on the CPI.

Actions Taken: During FY 2001, BLS completed a "Report on the CPI Pilot Project to Assess Commodity Analysts’ Substitution Decisionmaking" in response to the GAO recommendation. The analysis and conclusions in the report were discussed with GAO. Based on a recommendation in the report, a second, follow-up pilot project was begun during FY 2001, with the goal of improving decisionmaking by developing and testing documented product comparability criteria for use by commodity analysts. The second project is scheduled for completion in FY 2002.

Additional Information: A copy of the final GAO evaluation may be obtained from the General Accounting Office (GAO) Internet site: http://www.gao.gov in the GAO Reports section.