Detailed Information on the
Workforce Investment Act - Adult Employment and Training Assessment

Working with the Congress to consolidate this and other workforce programs, so that States and local communities receive one funding stream and have greater ability to be innovative and effective.

Eliminating unnecessary overhead by reducing Federal red tape and tightening State and local spending on administrative costs.

Conducting an evaluation to determine WIA services' impact on employment, retention, and earnings outcomes for participants.

Improving reporting efficiency, program management and accountability through the collection of new information with WISPR, a common reporting system for WIA, TAA, and Wagner-Peyser Act Programs.

Implementing efficiency measures that are linked to performance outcomes, account for all costs, and facilitate comparisons across Department of Labor training and employment programs.

Strengthening accountability by instituting common performance measures that will allow comparison between various job training programs. Measures will track employment, retention, and earnings.

Adopting efficiency measures that are linked to performance outcomes, account for all costs, and facilitate comparisons across Department of Labor training and employment programs.

Conducting an evaluation, using existing administrative data, to determine WIA services' impact on employment, retention, and earnings outcomes for participants. This evaluation will be completed by December 2008.

Measure: Entered Employment Rate

Explanation:Percent of participants employed in the first quarter after exit. This is a federal job training program common measure, which enables the Workforce Investment Act (WIA) Adult Program to describe in a similar manner the core purposes and results of the program compared to other education, employment and job training programs. For example, while the target population for the WIA Adult Program includes adults 18 years and older, the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures how many participants got a job according to the following formula: Of those who are not employed at the start of participation - The number of adult participants who are employed in the first quarter after the exit quarter divided by the number of adult participants who exited during the quarter. This definition for entered employment was adopted in PY 2001 at the onset of WIA reporting and was later adopted by other programs under the common measure initiative. Experience shows actual performance results for this measure tend to fluctuate from year to year.

Measure: Employment Retention Rate

Explanation:Percent of participants employed in the first quarter after program exit still employed in the second and third quarters after exit This is a federal job training program common measure, which enables the Workforce Investment Act (WIA) Adult Program to describe in a similar manner the core purposes and results of the program compared to other education, employment and job training programs. For example, while the target population for the WIA Adult Program includes adults 18 years and older, the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures how many participants retained their employment once placed in a job, according to the following formula: Of those who are employed in the first quarter after the exit quarter - The number of adult participants who are employed in both the second and third quarters after the exit quarter divided by the number of adult participants who exited during the quarter. This definition for employment retention was adopted in PY 2005, and made the measure more challenging by including an additional quarter in which a participant needs to be employed.

Measure: Average Earnings

Explanation:The average six-month earnings This is a federal job training program common measure, which enables the Workforce Investment Act (WIA) Adult Program to describe in a similar manner the core purposes and results of the program compared to other education, employment and job training programs. For example, while the target population for the WIA Adult Program includes adults 18 years and older, the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures participants' average six-month earnings once placed in a job (note: the average earnings for a year can be obtained by doubling the performance measure result), according to the following formula: Of those participants who are employed in the first, second, and third quarters after the exit quarter - Total earnings in the second quarter plus total earnings in the third quarter after the exit quarter divided by the number of participants who exited during the quarter. The common measure definition for average earnings was adopted in PY 2006, while prior year results were for a different earnings measure tracking earnings change.

Measure: Cost per Participant

Explanation:The program calculates the average cost for each participant by dividing the total annual appropriation for the program by the number of participants.

Is the program purpose clear?

Explanation: The program's purpose is to provide workforce investment activities such as employment counseling and assessment; and job search, training, and placement activities; that prepare adults who seek services to achieve successful employment outcomes including employment, job retention, and earnings increases. The program finances these activities through formula grants to States, and Statewide and local workforce investment systems provide the services.

Evidence: Workforce Investment Act of 1998 (WIA; Public Law 105-220), which is available at www.doleta.gov/usworkforce/wia/wialaw.pdf; WIA Sections 131-134 [codified at 29 U.S.C. 2861-2864]; and DOL's FYs 2003-2008 Strategic Plan define the purpose of the program. Strategic plans are accessible at www.dol.gov/_sec/media/reports/.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program, which traces its origins to similar programs that began more than 50 years ago, is intended to address the needs of adults 18 years or older who seek to develop skills and access resources to help them find or keep employment. In addressing these needs, the program also helps employers find skilled workers. Local communities may target services to certain adults based on the community's particular economic and labor market conditions. (There are over 600 WIA local workforce investment areas.) However, WIA requires that people who receive public assistance or otherwise have low incomes must be given priority for services if local resources for this program are limited.

Evidence: In 2003, the program served a total of 444,321 participants. About 65% of the participants served were low-income adults, and approximately 12% were public assistance recipients. 2003 Census data show that 35.9 million people are in poverty. WIA Section 134(d)(4)(E) says: "In the event that funds allocated to a local area for adult employment and training activities . . . are limited, priority shall be given to recipients of public assistance and other low-income individuals for intensive services and training services." Findings from the U.S. Chamber of Commerce's Center for Workforce Preparation survey, Rising to the Challenge: Business Voices on the Public Workforce Development System, published Spring 2003 indicate employers' needs that the program is designed to address. Thirty-three percent of respondents identified difficulties in finding and retaining qualified workers that affect their bottom line, 33 percent of respondents noted that job applicants were either poorly skilled or had the wrong skills, and 40 percent of respondents cited well-trained staff as a key to remaining competitive. The report, survey instrument, and data tables are accessible at www.uschamber.com/cwp/strategies/wia/survey.htm. As GAO noted in its February 2004 Report (GAO 04-282: Almost Half of States Fund Employment Placement and Training through Employer Taxes and Most Coordinate with Federally Funded Programs), as technological and other advances transform the U.S. economy, many of the nation's six million employers may have trouble finding employees with the skills to do their jobs well. GAO reports are accessible at www.gao.gov/docsearch/repandtest.html

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Although the program gives priority for services to public assistance recipients and other people with low incomes when local funds are limited, it provides similar services to adults (and outreach to employers) as those provided through the WIA Dislocated Worker program and the Employment Service. States receive separate grants, and local communities receive separate funding allocations, for each of the programs even though the resources finance similar services for adults. The programs all deliver services through "One-Stop Career Centers" authorized by WIA. The Administration's proposal to reauthorize and reform WIA would eliminate this duplication by consolidating these three programs and another one, the WIA Youth program. States would receive one grant to administer job training and employment services through the One-Stop Career Centers. In addition, the program duplicates some job training services offered through the Department of Health and Human Services' Temporary Assistance for Needy Families (TANF) program.

Evidence: WIA Sections 131-134, which can be accessed at www.doleta.gov/usworkforce/wia/wialaw.pdf. The Wagner-Peyser Act (which authorizes the Employment Service), as amended by WIA, is accessible at www.uses.doleta.gov/w-pact_amended98.asp. The Administration's WIA reauthorization and reform proposal is described at www.doleta.gov/Job_Training_WIA/.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program design has some major flaws, and reforms are necessary to improve its effectiveness and efficiency. First, the existence of separate funding sources for the Employment Service and the WIA Adult, Dislocated Worker, and Youth programs impedes States' and localities' efficiency, effectiveness, and flexibility to build and design a system or programs with innovative approaches to prepare participants for employment. Second, current regulations for the program include an inadequate and inaccurate definition of "administrative costs" that results in local offices' diverting significant amounts of resources from providing services to paying administrative overhead. Third, there is overlap with the WIA Dislocated Worker program and the Employment Service. The Administration's WIA reauthorization and reform proposal is designed to address these flaws by streamlining and consolidating multiple funding sources to provide Governors with the flexibility needed to better address workers' and employers' needs, while tightening the definition of "administrative costs" to ensure that fewer funds are spent on overhead and more funds are spent on training workers. This proposal provides an administrative framework through which workforce investment activities can be coordinated and duplication of administration, operations, and services can be eliminated.

Evidence: WIA Sections 131-134 (codified at 29 U.S.C. 2861-2864), accessible at www.doleta.gov/usworkforce/wia/wialaw.pdf. WIA regulations accessible at www.doleta.gov/reports/docs/regs/. DOL's FY 2003-2008 Strategic Plan-accessible at www.dol.gov/_sec/stratplan/main.htm. The Administration's WIA reauthorization proposal may be found at www.doleta.gov/Job_Training_WIA/. General Accounting Office (GAO) Report GAO-03-636, "Workforce Investment Act: Issues Related to Allocation Formulas for Youth, Adults, and Dislocated Workers," April 2003. GAO reports are accessible at www.gao.gov/docsearch/repandtest.html.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program offers three levels of service--core, intensive, and training--that are intended to be tailored to an individual's needs to achieve optimal employment outcomes. All adults are eligible for core services, which include outreach, job search and placement assistance, and labor market information. Intensive services include comprehensive assessments, development of individual employment plans and counseling and career planning, and are available to individuals who meet locally-determined eligibility criteria. Training services are available to those who have been unable to find employment through intensive services. Participants use an "individual training account" to select an appropriate training program from a qualified training provider. Customers are linked to job opportunities in their communities, including both occupational training and training in basic skills. WIA requires that public assistance recipients and other people with low incomes receive priority for intensive and training services when local resources for the program are limited. WIA includes provisions intended to help target program resources. For example, grantees are allowed to target resources to meet the locally-determined needs of participants in the WIA Adult and Dislocated Worker programs and may transfer up to 30 percent of funds between these two programs. Grantees also have the flexibility to determine how to allocate program funding allocated across the three levels of service. States and local workforce investment areas that decide to allocate higher resources levels to core and intensive services effectively may be duplicating operations and services performed by other programs and reducing the amount of funds available to participants for training services.

Evidence: WIA Section 131-134, which can be accessed at www.doleta.gov/usworkforce/wia/wialaw.pdf.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has three measures of its outcomes, both long-term and annual: (1) entered employment rate, (2) employment retention rate, and (3) average earnings. These measures are the ones adopted as a result of the Administration's initiative to establish common performance measures for Federal job training and employment programs that serve adults. They are very similar to the program's measures since 2000 under the WIA performance measurement system. DOL is implementing the "job training common measures," with revised methodologies, for this program in 2005.

Evidence: See the "Program Performance Measures" section for the program's measures and targets. See WIA Section 136 (29 U.S.C 2871), including proposed amendments to the same in the Administration's WIA reauthorization proposal. DOL's adoption of the job training common measures is discussed in its FYs 2003-2008 Strategic Plan, which is available at www.dol.gov/_sec/stratplan_2003-2008.htm. Guidance and information about implementation of the job training common measures and the WIA performance measurement system can be accessed at www.doleta.gov/performance/. The ETA Strategic Plan for FYs 1999-2004, which shows the program's measures and targets prior to the common measures, is accessible at www.doleta.gov/pdf/Final_f0.pdf

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: DOL is implementing the job training common measures for the program in 2005 but has established annual and long-term targets for the new measures based on data from its previous measures under WIA since 2000, which are similar. See the "Program Performance Measures" section for these targets. The previous and new long-term targets for this program's outcome measures are appropriately ambitious and reflect continued improvement over time with achievable efficiencies. It is worth special note that the Administration's WIA reform proposal includes a long-term goal that, within ten years, all States will achieve 100% entered employment rate for workers trained with program dollars.

Evidence: See the "Program Performance Measures" section for the program's targets. DOL's strategic plans and annual budgets, performance plans, and performance reports are available at www.dol.gov/dol/aboutdol/main.htm#budget. Prior to 2005, the program established targets for long-term performance measures using the WIA performance measurement system, as outlined in the ETA Strategic Plan for FYs 1999-2004. These long-term goals were to achieve 70% entered employment rate, 80% retention rate, and an average earnings increase of $3,423 by 2002.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has three measures of its outcomes, both annual and long-term: (1) entered employment rate, (2) employment retention rate, and (3) average earnings. These measures are the ones adopted as a result of the Administration's initiative to establish common performance measures for Federal job training and employment programs that serve adults. They are very similar to the program's measures since 2000 under the WIA performance measurement system. DOL is implementing the "job training common measures," with revised methodologies, for this program in 2005. The annual measures and targets support achievement of the program's long-term measures and targets.

Evidence: See the "Program Performance Measures" section for the measures and targets. DOL's FYs 2003-2008 Strategic Plan, at www.dol.gov/_sec/stratplan/strat_plan_2003-2008.htm, reflects DOL's adoption of the job training common measures. DOL's annual budgets, performance plans, and performance reports, including for this program, are available at www.dol.gov/dol/aboutdol/main.htm#budget.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: DOL is implementing the job training common measures for the program in 2005 but has established preliminary baselines and annual targets for the new measures based on data from the previous measures under the WIA performance measurement system, which are similar. The previous and new annual targets for this program's outcome measures are appropriately ambitious and reflect continued improvement over time with achievable efficiencies. Their achievement will support achievement of the long-term targets.

Evidence: Regarding targets established under the previous WIA performance measurement system: On average between 2000 and 2005, the program's annual targets for entered employment rate and retention rate increased one percentage point every year. Targets for the earnings change measure were also increased between 2000 and 2002. After decreasing the target in 2003 to reflect changing economic circumstances, the program raised the 2004 and 2005 annual targets above the 2003 result. The targets for the job training common measures also are appropriately ambitious and support achievement of the long-term targets. DOL's annual budgets, performance plans, and performance reports are available at www.dol.gov/dol/aboutdol/main.htm#budget.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Partners at all levels of the program work toward the program's goals. Under WIA, States are required to adopt specific goals and report on their progress in achieving these goals. DOL negotiates State performance targets for the program with the States. States have complied with this requirement each year since 2000. States that meet or exceed negotiated performance levels are eligible to receive what are called "incentive grants." Sixteen States were eligible to apply for an incentive grant based on 2001 performance; 23 States were eligible to apply for incentives based on 2002 performance; and 19 States were eligible to apply for incentives based on 2003 performance. Local communities and training providers also commit to and work toward the program's performance goals. States negotiate local performance goals with local workforce investment areas through a process that is similar to DOL's negotiations with the States. Local boards and One-Stop Career Centers build performance goals for job training providers into their contracts.

Evidence: DOL's guidance to States regarding negotiating annual goals is available at www.doleta.gov/performance/guidance/tegls.cfm. Guidance includes Training and Employment Guidance Letters (TEGLs). Important TEGLs include TEGL No. 18-04, Negotiating Workforce Investment Act (WIA) Title 1B Performance Goals for Program Year (PY) 2005 and 2006 for Inclusion in State Plans; TEGL No. 21-03, PY 2004 WIA State Planning Guidance; TEGL No. 8-99, Negotiating Performance Goals; and Incentives and Sanctions Process under Title I of the Workforce Investment Act (WIA); and TEGL No. 7-99, Core and Customer Satisfaction Performance Measures under Title I of the Workforce Investment Act (WIA). Federal Register Notices published on April 12, 2005, provide the Planning Guidance and Instructions for use by States in submitting two years of their Strategic Five-Year Plan for the job training common measures. These documents are accessible at www.doleta.gov/usworkforce/whatsnew/eta_default.cfm?id=1166. States' annual negotiated levels of performance for 2000  2005 are accessible at www.doleta.gov/performance/goals/st_neg_perf_level.cfm. Incentives and sanctions policy documents include TEGL No. 20-01, including Change 1 and Change 2, Application Process for Workforce Investment Act (WIA) Section 503 Incentive Grants; and TEGL 19-02, Sanctions Policy for Failure to Meet State Negotiated Performance Levels under Title I of the Workforce Investment Act (WIA), which are accessible at www.doleta.gov/Performance/results/incentives_sanctions.cfm. Examples of State-to-local performance goals and performance-based contract components for training providers are available at www.doleta.gov/Performance/goals/st_neg_perf_level.cfm

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Although there have been a number of independent evaluations of WIA programs, including this one, there have not been any of sufficient scope and quality to evaluate the program's effectiveness, especially its impact on participants' employment and earnings. Evaluations to date have focused on the administration and processes of WIA programs, including this one. These evaluations have provided some valuable insights about providing the employment services and job training, including several States' implementation strategies. ETA is considering conducting a new multi-year impact evaluation of this program or the proposed consolidated program for adults after WIA reauthorization. That evaluation would include a rigorous impact study with control or experimental groups.

Evidence: Evaluations of this program include: The Workforce Investment Act in Eight States, February 2005, conducted by the Nelson A. Rockefeller Institute of Government. This two-year study examined workforce delivery services in eight States, sixteen local areas, and more than thirty local One-Stop Career Centers. The purpose of the study was to provide information for national policy makers in the executive and legislative branches to inform future policy and legislation by providing detailed information to enhance understanding of the operations of the workforce delivery system. The Workforce Investment Act after Five Years: Results from the National Evaluation of the Implementation of WIA, June 2004, prepared by Social Policy Research Associates with contributions from TATC Consulting. This evaluation examined WIA implementation and operation over a four-year period in 21 States and 38 local workforce investment areas, with site visits conducted over three waves. The study highlights both progress and challenges and provides information that supports regional technical assistance efforts as well as policy and legislative proposals. Business as Partner and Customer under WIA: A Study of Innovative Practices, June 2004, prepared by Social Policy Research Associates. This report was added to the WIA implementation study noted above to investigate issues and challenges related to business involvement. Nine sites were visited between December 2003 and February 2004 to evaluate innovative approaches to business services and involvement, which is critical to a successful workforce development system. The Effects of Customer Choice: First Findings from the Individual Training Account Experiment, Final Interim Report, December 2004, by Mathematica Policy Research, Inc. This experiment was designed to provide information to local, State, and Federal policymakers with information on the effectiveness and tradeoffs of different individual training account (ITA) approaches.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The current extent of performance and cost integration does not permit an assessment of the precise impact of specific budget levels, policy, or legislation on performance outcomes. Nonetheless, DOL's 2005 and 2006 performance budget requests have provided a clear overview of DOL's priorities, strategic goals, and outcome goals. This information is presented in a transparent manner and is posted on the Web for public scrutiny. Also, this program is part of a DOL-wide initiative to use cost accounting to make it easier and faster to determine the amount of time, money, and human resources needed to manage the program effectively and achieve its goals.

Evidence: DOL's budget requests are available at www.dol.gov/dol/aboutdol/main.htm#budget.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: DOL is taking meaningful steps to correct this program's strategic planning deficiencies. The Administration's proposal to reauthorize and reform WIA would consolidate this program and three others??the WIA Dislocated Worker and Youth programs, and the Employment Service. This reform would provide Governors and local officials greater flexibility to plan, administer services, and use resources in a way that best meets their communities' specific needs. The major goals of the consolidation include providing flexibility to States and localities and reducing overhead so that more resources can be available for job training. In addition, DOL is implementing the Administration's job training common measures to focus on a few important outcomes and measures of effectiveness. Since 2000, the program has had measures that are very similar to these common measures. DOL's guidance to States regarding the WIA State planning process for 2005 and 2006 has set a national strategic direction for the workforce investment system, including enhanced integration of service delivery through One-Stop Career Centers. ETA is strengthening its budget and performance planning processes through implementation of the DOL Cost Analysis Manager. In addition, ETA plans to continue program improvements by commissioning an evaluation of the new comprehensive adult program once it has been implemented.

Evidence: DOL's FYs 2003-2008 Strategic Plan and DOL's FY 2005 Annual Performance Plan, which are accessible at www.dol.gov/_sec/stratplan/strat_plan_2003-2008. htm. DOL's budget requests are available at www.dol.gov/dol/aboutdol/main.htm#budget.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: States report program performance information quarterly and annually to DOL. ETA staff monitor grantee (State) performance and financial practices through analyses of these quarterly and annual data, as well as regular on-site reviews. Every year, ETA staff are required to review 50% of formula grantees as well as all grant activities identified as high risk. To assist staff in this monitoring, ETA developed an online tool for staff called "Grants E-Management System (GEMS)." GEMS helps staff to identify immediate and potential risks in grant activities and provides access to all program data through a related information tool called "Enterprise Information Management System (EIMS)." ETA staff use performance data to award incentive grants and identify States that may require technical assistance. States that fail to achieve one or more negotiated levels of performance are provided this technical assistance. Additionally, beginning in 2003, ETA provides a Performance Enhancement Project (PEP) that offers technical assistance to States that are having difficulty meeting the negotiated performance levels. In 2003, ETA implemented data validation for WIA, Wagner-Peyser and the Trade Adjustment Assistance programs to provide assurance regarding the integrity of performance data. States submitted results for 2003 data validation on February 1, 2005.

Evidence: Performance Management Plans for Managers and Supervisors (July 2003) prescribe the parameters for grant management activities--which includes the frequency of desk and on-site grant reviews; Employment and Training Order No. 1-03; ETA's Regional Accounting Policies & Procedures Manual provides Federal staff with guidance on grants management; DOL's Annual Report on Performance and Accountability for FY 2003 (Outcome Goal 1.1), which is accessible at www.dol.gov/%5Fsec/media/reports/annual2003/strat_goal1.htm; TEGL 3-03: Data Validation for Employment and Training Programs, wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1506; and DOL's OIG Report No. 06-02-006-03-390, "Workforce Investment Act Performance Outcomes Reporting Oversight," September 30, 2002. Information on the data validation initiative is accessible at www.uses.doleta.gov/dv/. DOL's OIG reports are accessible at www.oig.dol.gov/public/reports/oa/main.htm Performance Enhancement Project is accessible at: www.workforcetools.org/free_online_courses.asp. The EIMS, Dashboard, and Grants E-Management System (GEMS) of the Employment and Training Administration are tools that assist Federal staff in conducting grants management activities.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Federal managers and State partners are not fully held accountable for failing to meet important performance results, and no sanctions have been applied to grantees despite the fact that WIA provides sanction authority to DOL. ETA has policies and procedures in place to monitor results and reward good performance, and WIA authorizes the Secretary to apply sanctions to States that fail to meet agreed upon performance results. ETA has provided incentive awards to selected States based on 2001, 2002, and 2003 performance, and continues to provide technical assistance to States that fail one or more of the performance measures under the Performance Enhancement Project. The Administration is committed to strengthening accountability for poor performance.

Evidence: Sanctions and incentives are described in the following: WIA section 136(g); Training and Employment Guidance Letter (TEGL) No. 20-01, including Change 1 and Change 2, Application Process for Workforce Investment Act (WIA) Section 503 Incentive Grants; TEGL No. 19-02, Sanctions Policy for Failure to Meet State Negotiated Performance Levels under Title I of the Workforce Investment Act (WIA), which are accessible at www.doleta.gov/Performance/results/incentives_sanctions.cfm. Other evidence includes: States' Annual Progress Reports on WIA program activities; and revised performance management plans for senior executives (Form DL 1-2059, Rev. 10/2001) and for supervisors and managers (Form DL 1-382, Rev. 10/2001). States' Performance Reports are accessible at www.doleta.gov/Performance/results/Reports.cfm?#wiastann. Employment and Training Order No. 1-03; ETA Regional Accounting Policies & Procedures Manual (not accessible on the WWW; only hard copies are available at this time; this provides Federal managers guidance on grant management. DOL Annual Performance and Accountability Reports are available at www.dol.gov/dol/aboutdol/main.htm. Incentive awards to States were published in the Federal Register Notice and are available at: a257.g.akamaitech.net/7/257/2422/14mar20010800/edocket.access.gpo.gov/2004/04-8747.htm. Technical assistance to assist States that failed one or more of the performance measures is available under the Performance Enhancement Project and DOL online tutorial on performance measures at: www.workforcetools.org/free_online_courses.asp.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Each year, DOL promptly and fully obligates the program's funding to the States (makes it available for spending), consistent with WIA and the appropriations. For their part, States then obligate their funds in a timely manner and spend the funds for the intended purpose outlined in WIA. States are required to obligate at least 80 percent of their funds each year or the funds are subject to DOL's "recapturing" them for use by other States. (The States have continued to exceed the threshold requirement to obligate 80 percent of funds to their grantees; thus, it has not been necessary to recapture funds that are unobligated.) States submit to DOL quarterly financial status reports for obligations and expenditures, and expenditures are monitored by the Federal staff. Currently, there is no Federal reporting required for local level spending, which is a limitation, or the Governor's "reserve," which is 15 percent of a State's program funding. It is worth noting that most States do not spend the entire amount they receive each year and carry-over some funds to the next year. Under WIA, States have three years to spend the funds that they receive. In 2003, new resources provided to the States totaled about $932 million, while unspent resources carried over from previous years were about $394 million. From all of these funds, States spent about $973 million, approximately 5 percent more than the new resources.

Evidence: Training and Employment Guidance Letter (TEGL) No. 16-99, Change 1 titled, "Workforce Investment Act Financial Reporting," and TEGL 16-99, titled "Workforce Investment Act Financial Reporting," provide the instructions and clarifications for the financial reporting instructions. These TEGLs are accessible at wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1433 and wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1229. TEGL 23-02 titled "FY 2003 Congressional Rescissions for WIA Adults and Dislocated Workers, Workforce Investment Act (WIA) Allotments for Program Year (PY) 2003, Wagner-Peyser Act Final Allotments for PY 2003, Reemployment Services Allotments for PY 2003, and Work Opportunity and Welfare-to-Work Tax Credit Allotments for FY 2003", accessible at www.ows.doleta.gov/dmstree/tegl/tegl2k2/tegl_23-02.htm Additionally, the One-Stop Comprehensive Financial Management Technical Assistance Guide accessible at www.doleta.gov/sga/pdf/FinalTAG_August_02.pdf also provides information on the financial reporting requirements. WIA quarterly expenditure reports are available quarterly at: www.doleta.gov/budget/.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ETA and this program have in place some procedures related to efficiencies and cost effectiveness, but they are insufficient to fully measure and achieve efficiencies and cost effectiveness in program execution. ETA is working with other DOL offices, including the Office of the Chief Financial Officer (OCFO) and the Office of the Assistant Secretary for Administration and Management, to establish additional procedures that make use of budget, cost, and financial information to improve program execution. Currently, ETA captures information necessary to measure the program's efficiency in one sense: average cost per participant (program appropriation level divided by the number of program participants). This efficiency measure is one of the job training common measures. ETA is implementing this measure at the Federal level beginning in 2005, but cost and participant data from past years allows the calculation of average cost per participant looking backward: In 2001, about 399,000 participants at an average cost of $2,381 per participant. In 2002, 467,000 participants at an average cost of $2,033 per participant. In 2003, 444,000 participants at an average cost of $2,012 per participant. This information allows ETA to make some estimate of targets for 2005 and future years. In addition, DOL has begun to improve its use of financial data routinely for program management, in part through the Cost Analysis Manager, or "CAM," initiative coordinated by OCFO. ETA has developed cost models that reflect the nature of its work on this and other programs--grants management, chiefly--but has not yet routinized its use of the cost and financial information that this tool can provide managers and partners, including States. Although efficiency is an important factor in the assessment of program effectiveness, it is not the only one. Information on cost must be examined in the context of the outcomes achieved, characteristics of individual participants, and the types of training provided. ETA is pursuing strategies to support increased efficiency and effectiveness, particularly regarding coordination with other programs. Examples include projects to encourage participant co-enrollment and leverage resources with other Federal, State, and local employment and training programs.

Evidence: See the "Program Performance Measures" section for the program's new efficiency measure and targets. ETA Common Measures Training and Employment Guidance Letter (TEGL) No.15-03 clarifies the efficiency measure and is available at www.wdr.doleta.gov/dorectives/TEGL15-03.pdf. Information on program participants is found in the WIA Annual Report Data for PYs 2001  2003, which is accessible at www.doleta.gov/performance/results/wia_national_performance.cfm. Appropriation information is found in the DOL-ETA Budget Authority Charts accessible at www.doleta.gov/budget/bahist.cfm DOL's Financial Performance Improvement Plan discusses the Cost Analysis Manager initiative, including key cost drivers for this and other ETA programs. This plan is available at http://www.dol.gov/_sec/media/reports/main.htm

Does the program collaborate and coordinate effectively with related programs?

Explanation: Under WIA, the program is a required partner in the 1,900 comprehensive One-Stop Career Centers nationwide. States and localities administer these centers, which are the means by which people access this and other programs' services. Also, within DOL, this program, the Employment Service, the WIA Youth, WIA Dislocated Worker, and the Trade Adjustment Assistance programs are managed in the same national office. WIA also requires certain other Federal programs to be partners in the One-Stop Career Centers. Key issues of collaboration include Federal guidance for State planning and partners' sharing of the costs for operating the centers. Through memoranda of understanding, local workforce investment boards and States negotiate the deployment of staff and resources at the local level. DOL has urged States to manage these programs similarly to improve customer service and program performance. The Administration has proposed to reform WIA so that there are fewer Federal funding streams, greater State flexibility to administer these programs, and strengthened accountability for performance.

Evidence: DOL consults with Federal WIA partners, including the Departments of Education, Health and Human Services, Housing and Urban Development, Agriculture, and the Social Security Administration. DOL also works with the Department of Commerce on economic development strategies to incorporate in this program. States provide unified plans to DOL addressing the employment and training services providing by these various partners. DOL released planning guidance and instructions for current planning process for the WIA Title I and Wagner-Peyser Act strategic plan, and it is available at www.doleta.gov/usworkforce/wia-planning-guidance/standalone.cfm. Workforce Investment Act, Section 121(c) Memorandum of Understanding at www.doleta.gov/usworkforce/wia/wialaw.pdf provides guidance to One-Stop partners on allocation of resources, how to structure service delivery, etc. One-Stop Comprehensive Financial Management Technical Assistance Guide at www.doleta.gov/sga/pdf/FinalTAG_August_02.pdf; and www.servicelocator.org/partners/ The Administration's proposal on WIA Plus Consolidated Grant Program is posted on the web at: www.doleta.gov/pdf/Job%20Training%20Reform%20WIA%20Plus%20Consolidated%20Grant.pdf.

Does the program use strong financial management practices?

Explanation: In DOL's FY 2003 Performance and Accountability Report, DOL's Office of the Inspector General (OIG) identified grant accountability among the most serious management challenges facing DOL. In DOL's FY 2004 Performance and Accountability Report, OIG noted ETA's progress on grants management and stated that OIG had removed grant accountability from the top management challenges list. However, OIG said that "accountability over DOL awarded grants will continue to merit diligent attention." Because of the continuing concerns of DOL's OIG about ETA's grants management, this question was answered "NO." Direct Federal oversight of grants for employment and training activities is difficult because a large share of the funding is passed down through the States to sub-grantees (for example, local communities) and contractors. This program is required to report quarterly on its spending, and these requirements have been outlined in guidance to the grantees. The OIG recognized that the Employment and Training Administration (ETA) took a significant step in April 2003 when it issued an order implementing its Grant/Contract Administration Plan. Employment and Training Order No. 1-03 is designed to improve grantee accountability and compliance. In carrying out the order, ETA has issued a Regional Accounting Policies and Procedures Manual and is making improvements to post-award grant monitoring. ETA has developed an Internet-based grants management tool (GEMS) that allows the ETA regional staff to track the expenditures and identify those grants that are "at-risk." The OIG is encouraged by ETA's efforts, but OIG audits continue to highlight this area of concern.

Evidence: OIG's annual reports on "Top Management Challenges Facing the U.S. Department of Labor" are available at www.oig.dol.gov/topchallenges.htm. These reports are included in DOL's Annual Reports on Performance and Accountability, which are accessible at www.dol.gov/dol/aboutdol/main.htm. WIA quarterly expenditure reports are available quarterly at: www.doleta.gov/budget/. TEGL 16-99, Change 1 titled, "Workforce Investment Act Financial Reporting," and TEGL 16-99, titled "Workforce Investment Act Financial Reporting," provide the instructions and clarifications for the financial reporting instructions. These TEGLs are accessible at wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1433 and wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1229. Employment and Training Order No. 1-03; ETA Regional Accounting Policies & Procedures Manual, provides Federal managers guidance on grant management.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has taken steps to improve grants management and strengthen performance reporting and accountability. ETA created an Internet-based grants management tool that this program uses. ETA also developed, and this program uses, report validation software. This program participates in ETA's data element validation initiative. In addition, ETA regional staff provide technical assistance to grantees when deficiencies are identified in any aspect of program funds administration, reporting, performance, or financial operations. The Administration's WIA reauthorization and reform proposal would consolidate this program with others and streamline administrative procedures and oversight.

Evidence: Guidance on data validation for WIA Title I programs was published in TEGL 3-03 Change 1 published October 20, 2004; wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1599 Employment and Training Order No. 1-03; ETA Regional Accounting Policies & Procedures Manual; provides Federal staff with guidance on grants management.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: States are required to submit financial data to DOL every three months and report performance data through the data collection and reporting tools contained in ETA's Enterprise Information Management System (EIMS). Regional Federal staff use the Grants E-Management System (GEMS) as a comprehensive grant management tool. GEMS allows staff to track and document grant management activities, including on-site monitoring and technical assistance reviews. It also includes a risk assessment module and a case notes tool. It is a regional office responsibility to monitor, review and provide assistance to grantees (States). Grantee performance and financial practices are monitored by regional Federal staff through data analyses and regular on-site reviews. Additionally, Federal staff participate in bi-weekly conference calls on performance management. Every fiscal year, regional Federal staff are required to review 50 percent of grantees as well as all any grant activities identified as high risk. If necessary, regional Federal staff and grantees jointly develop corrective action plans to address program deficiencies. Grantees, in turn, monitor sub-grantees (for example, local communities) in the accomplishment of Federal and State performance measures and fiscal responsibility. Despite these oversight procedures and tools, ETA and this program face limitations in their knowledge of grantees' financial activities, but the Administration has proposed changes to WIA to improve this situation. Due to WIA financial tracking and reporting standards, ETA does not know the extent to which State and local commitments, or obligations, to spend program funds that they receive from DOL are firm. Therefore, it is difficult for DOL to say confidently the amount of program funds that remain available at any given time for States to provide local services or whether funds have been spent as intended.

Evidence: The GEMS user guide is available at: http://www.etareports.doleta.gov/gems/help/table_of_contents.htm. Performance Management Plans for Managers and Supervisors (07/03); Employment and Training Order No. 1-03 titled "Improving Administration of Grants within the Employment and Training Administration" (04/03). DOL's Office of the Inspector General (OIG) and the Government Accountability Office (GAO) have issued findings and reports on the limitations of financial reporting for this and other WIA State grant programs that ETA administers. DOL's FY 2003 Performance and Accountability Report contains a summary of the OIG's findings on WIA obligations reporting within the "Management and Performance Challenges" section, which is available at www.oig.dol.gov/public/topchallenges/2003.pdf. GAO Report GAO-03-239: "States' Spending Is on Track, but Better Guidance Would Improve Financial Reporting" is available at www.gao.gov.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: DOL collects standard performance and financial reports from States quarterly and annually. Currently, States' performance is summarized quarterly, and the data are made available to the public via the Internet and through the Quarterly Report Workforce System Results. Additionally, DOL Annual Reports are published and posted on the DOL website for public accessibility. DOL is improving performance reporting for this and other job training programs by moving toward having grantees submit performance information on a quarterly basis using a "rolling four quarter" basis so that a full year of information is provided whether or not the end of the particular quarter represents the end of the program year.

Evidence: National data for PY 2000 through PY 2003 in the Department of Labor's Annual Performance Reports and ETA's Quarterly Report of Workforce System Results can be viewed at www.doleta.gov/performance/results/Reports.cfm?#etaqr. Guidance on WIA data collection for performance and accountability was published in TEGL 14-00 Change 1 and is accessible at: wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1343.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has adopted new performance measures and is establishing new long-term goals as part of the Administration's job training common measures initiative. The first year of implementing these measures is 2005. However, the program has achieved most of its previous long-term performance goals, which are based on measures that are quite similar to the common measures. ETA's Strategic Plan for FYs 1999-2004 established the program's previous long-term goals: 70% entered employment rate; 80% job retention rate; and an average earnings increase of $3,423 for participants, by 2002. Performance results between 2000 and 2003 demonstrate that the program exceeded all three of these long-term goals but average earnings performance declined in the same period after the goal was achieved.

Evidence: 2002 and 2003 entered employment rate results of 74% exceeded the long-term goal by four percentage points. 2002 and 2003 retention rate results of 84% and 85%, respectively, exceeded the long-term goal by four and five percentage points. For the earnings increase measure, 2002 and 2003 results of $2,900 and $3,260, respectively, did not meet the long-term goal; however, the 2001 result of $3,555 exceeded the long-term goal of $3,423. ETA's Strategic Plan for FYs 1999-2004 can be accessed at www.doleta.gov/pdf/Final_f0.pdf. DOL's Strategic Plans and Annual Performance and Accountability Reports are accessible at www.dol.gov/dol/aboutdol/main.htm.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has adopted new performance measures and is establishing new annual goals as part of the Administration's job training common measures initiative. The annual goals will support achievement of the long-term goals. The first year of implementing these measures is 2005. However, the program has achieved most of its previous annual performance goals, which are based on measures that are quite similar to the common measures. Between 2000 and 2003, the program achieved all of its annual performance goals (for participants' entered employment rate, job retention rate, and average earnings increase) except the average earnings increase goal in 2002.

Evidence: 2003 results were as follows: 74% entered employment rate against a goal of 71%; 85% employment retention rate against a goal of 82%; and $3,260 average earnings change at six months against a goal of $3,100. 2004 results are 76% entered employment rate against a goal of 75%; 85% employment retention rate against the goal of 85%; and $3,703 earning change at six months against a goal of $3,300. ETA's Quarterly Reports of Workforce System Results and WIA Annul Reports are accessible at www.doleta.gov/performance/results/Reports.cfm?#wiastann. Annual Performance and Accountability Reports are accessible at www.dol.gov/dol/aboutdol/main.htm.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: ETA and this program have in place some procedures related to efficiencies and cost effectiveness, but they are insufficient to fully measure and achieve efficiencies and cost effectiveness in program execution, and the program has only partially demonstrated improvement. ETA is working with other DOL offices, including the Office of the Chief Financial Officer (OCFO) and the Office of the Assistant Secretary for Administration and Management, to establish additional procedures that make use of budget, cost, and financial information to improve program execution. Currently, ETA captures information necessary to measure the program's efficiency in one sense: average cost per participant (program appropriation level divided by the number of program participants). This efficiency measure is one of the job training common measures. ETA is implementing this measure at the Federal level beginning in 2005, but cost and participant data from past years allows the calculation of average cost per participant looking backward: In 2001, about 399,000 participants at an average cost of $2,381 per participant. In 2002, 467,000 participants at an average cost of $2,033 per participant. In 2003, 444,000 participants at an average cost of $2,012 per participant. DOL has begun to improve its use of financial data routinely for program management, in part through the Cost Analysis Manager, or "CAM," initiative coordinated by OCFO. ETA has developed cost models that reflect the nature of its work on this and other programs--grants management, chiefly--but has not yet routinized its use of the cost and financial information that this tool can provide managers and partners, including States. ETA is pursuing strategies to support increased efficiency and effectiveness, particularly regarding coordination with other programs. Examples include projects to encourage participant co-enrollment and leverage resources with other Federal, State, and local employment and training programs.

Evidence: ETA Common Measures Training and Employment Guidance Letter (TEGL) No.15-03 clarifies the efficiency measure and is available at www.wdr.doleta.gov/dorectives/TEGL15-03.pdf. Information on program participants is found in the WIA Annual Report Data for PYs 2001 - 2003, which is accessible at www.doleta.gov/performance/results/wia_national_performance.cfm. Appropriation information is found in the DOL-ETA Budget Authority Charts accessible at www.doleta.gov/budget/bahist.cfm. DOL's Financial Performance Improvement Plan discusses the Cost Analysis Manager initiative, including key cost drivers for this and other ETA programs. This plan is available at www.[INSERT LOCATION]. DOL's FY 2006 budget request proposes the WIA Consolidated Program, and therefore does not include a target average cost per participant for this program. Information on the proposal for a consolidated reporting system is accessible at www.doleta.gov/Performance/EMILE/EMILE.cfm.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The program compares favorably to similar programs on some measures. For example, in 2003, the program's participants had a higher entered employment rate (72%) than ETA's Trade Adjustment Assistance (TAA) and Employment Service (ES) programs (62% and 63%, respectively), both of which also are available through local One-Stop Career Centers. It is important to understand the context when comparing these ETA programs: TAA serves certain displaced workers impacted by major factory shut downs, and some proportion of those impacted workers may retire or leave the labor market. Also, one might expect entered employment rates for the WIA Adult program to be higher than those for the ES program because ES provides less intensive and lower cost services like information on job openings and group sessions on finding work. Also in 2003, the WIA Adult program had a higher employment retention rate (81%) than the ES program (72%). (The WIA Adult program's retention rate is lower than that for TAA, however; see below.) The WIA Adult program compares less favorably to the WIA Dislocated Workers program, though it is important to understand the different participant populations these programs serve. The WIA Adult program's entered employment rate of 72% is lower than the 87% for dislocated workers. Dislocated workers typically have much stronger attachment to and experience in the labor market and higher wages than the typical WIA Adult program participant. Dislocated workers are likely to return to the labor market after receiving services at a significantly higher level than participants of other programs. Not unexpectedly, the WIA Adult program employment retention rate of 81% is lower than that for WIA Dislocated Worker and the TAA programs with retention rates of 87% and 88%, respectively. Again, dislocated workers and those TAA participants who return to the workforce are more likely to have strong labor market ties and significantly higher wage levels than WIA Adult program participants and are much more likely to stay employed. Unfortunately, no other Federal agency has a program that is a close fit with WIA Adults, and some do not have sufficient performance data for a comparison. Nevertheless, one can make some comparison of the WIA Adult program to Temporary Assistance for Needy Families (TANF), which is administered by the Department of Health and Human Services' Administration for Children and Families (ACF). Under WIA, TANF is an optional partner in the One-Stop Career Center system. It is important to note that the WIA Adult program requires "universal access," although States and localities are required to give low-income individuals priority if resources are limited. TANF, on the other hand, is targeted to low-income individuals. In addition, the WIA Adult program is intended to provide job training and employment services to facilitate employment, job retention, and earnings gains. TANF's purposes and services and assistance are broader, though they include promotion of job readiness and employment. Finally, the programs are not competitors, and some TANF recipients participate in the WIA Adult program. During 2002, of approximately 20,000 WIA Adult participants who were also TANF recipients, their entered employment rate was nearly 69% and their employment retention rate was 79%. The methodology for corresponding TANF measures is not the same--though it is moving to adopt the job training common measures--but a rough comparison can be made. During 2002, 36% of adult TANF recipients become newly employed, and 59% of current or former TANF recipients were employed for three consecutive quarters.

Evidence: ETA Performance and Results information is available at http://www.doleta.gov/performance/results/Reports.cfm?#etaqr. See "WIASRD Summary Reports," "National Summary Reports" for Adults and Dislocated Workers. Also, annual Workforce System Results, including for Program Year 2003 and Fiscal Year 2004, are accessible at http://www.doleta.gov/performance/results/Edition-09_12-30-04.pdf. Quarterly results also are available from the performance and results site. WIA Adult program performance for 2002, by demographic and service groups (including TANF recipients), is available at http://www.doleta.gov/performance/results/Reports.cfm?#etaqr. The FY 2003 Annual Performance Report for ACF, including TANF, is available at http://www.acf.hhs.gov/programs/opre/acf_perfplan/ann_per/index.html. It is dated February 2004.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Although there have been a number of independent evaluations of WIA programs, including this one, there have not been any of sufficient scope and quality to evaluate the program's effectiveness, especially its impact on participants' employment and earnings. Evaluations to date have focused on the administration and processes of WIA programs, including this one. These evaluations have provided some valuable insights about delivering employment services and job training, including implementation strategies that certain States have adopted. ETA is considering conducting a new multi-year impact evaluation of this program or the proposed consolidated program for adults after WIA reauthorization. That evaluation would include a rigorous impact study with control or experimental groups.

Evidence: Evaluations of this program include: The Workforce Investment Act in Eight States, February 2005, conducted by the Nelson A. Rockefeller Institute of Government. This two-year study examined workforce delivery services in eight States, sixteen local areas, and more than thirty local One-Stop Career Centers. The purpose of the study was to provide information for national policy makers in the executive and legislative branches to inform future policy and legislation by providing detailed information to enhance understanding of the operations of the workforce delivery system. The Workforce Investment Act after Five Years: Results from the National Evaluation of the Implementation of WIA, June 2004, prepared by Social Policy Research Associates with contributions from TATC Consulting. This evaluation examined WIA implementation and operation over a four-year period in 21 States and 38 local workforce investment areas, with site visits conducted over three waves. The study highlights both progress and challenges and provides information that supports regional technical assistance efforts as well as policy and legislative proposals. Business as Partner and Customer under WIA: A Study of Innovative Practices, June 2004, prepared by Social Policy Research Associates. This report was added to the WIA implementation study noted above to investigate issues and challenges related to business involvement. Nine sites were visited between December 2003 and February 2004 to evaluate innovative approaches to business services and involvement, which is critical to a successful workforce development system. The Effects of Customer Choice: First Findings from the Individual Training Account Experiment, Final Interim Report, December 2004, by Mathematica Policy Research, Inc. This experiment was designed to provide information to local, State, and Federal policymakers on different individual training account (ITA) approaches.