TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 8-99

TO:         ALL STATE WORKFORCE LIAISONS
            ALL STATE WORKER ADJUSTMENT LIAISONS
            ALL STATE WORKER ADJUSTMENT LIAISONS
            ALL STATE EMPLOYMENT SECURITY LIAISONS
            ALL ONE STOP CAREER-CENTER SYSTEM LEADS                   

FROM:	    Lenita Jacobs-Simmons
            Deputy Assistant Secretary
  
SUBJECT:    Negotiating Performance Goals; and Incentives and Sanctions Process
            under Title I of the Workforce Investment Act (WIA).

1. Purpose. To inform States of ETA's implementation guidelines for the Negotiation and Goal Setting Process as well as for the Incentive and Sanction Process.

2. References. Workforce Investment Act of 1998 (Pub. L. 105-220), Section 136, Workforce Investment Act, Interim Final Rule, 20 CFR Part 666, published at 64 Federal Register 18662 (April 15, 1999).

3. Background. Under the Workforce Investment Act of 1998, States submitting a State Plan under Title I, subtitle B must propose expected levels of performance for each of the 17 indicators of performance consisting of the adult, dislocated worker and youth programs respectively and the two customer satisfaction indicators. While States may develop additional performance measures for State use, such measures will not be negotiated with the Department or included in the incentives and sanctions process.

In order to ensure an optimal return on the investment of the activities authorized, the Secretary and Governor must reach agreement in the State plan on the levels of performance for each of the first three program years. The requirements of the Government Performance and Results Act (GPRA) will also be a factor in negotiating the expected levels of performance. A critical component of annual reports to Congress will be representations of the extent to which States have met these negotiated performance levels. This process begins with each State negotiating performance levels with their Regional Office for all measures in each program area (adults, dislocated workers, and youth) and for customer satisfaction and contains a commitment that incentives and sanctions provided for under the law will be based on meeting those negotiated performance levels. This directive provides the requirements for that process, from negotiation between States and ETA's Regional Offices to subsequent review of levels achieved to determine appropriate incentives and sanctions.

4. Reaching Agreement on State Performance Levels. With the statutory inclusion of performance negotiations in WIA comes an expectation that States will use this provision in a pro-active manner to support their efforts for continuous improvement and customer satisfaction. Therefore, ETA has developed requirements that provide a uniform framework for negotiations and give States flexibility in pursuing continuous improvement for each program area. In order to obtain negotiated levels of performance for all 17 measures that demonstrate this commitment, the following negotiation process will be utilized. Please note that early implementing States will be negotiating performance levels for their 2^nd and 3^rd years of performance. Early implementing States should conform such negotiations to the guidance in this directive.

A. Process For Reaching Agreement on State Performance Levels

The DOL process for reaching agreement with States on performance levels will include these steps:

     1) States will develop baseline data following the instructions within this directive.

     2) States will analyze the baseline data and then offer expected State levels of performance, submitted as part of their State Plan, for each of the 17 indicators for each of the first three program years. States are encouraged to share this advance data and work with their Regional Office throughout this process.

     3) The information will go to the ETA Regional Office serving the State to review and the Regional Office will negotiate with the State to obtain mutually agreed upon expected levels of performance.

     4) The Regional Office will submit the negotiated performance levels⁽¹⁾ to the National Office for ETA for inclusion in the State Plan approval process.
______________________
    ¹In the Interim Final Rule, 20 CFR part 666, the outcome of the reaching agreement on State expected levels of performance was termed "adjusted levels of performance." In this document and in the Final Rule, the outcome of reaching agreement of State expected levels of performance will be termed "negotiated performance level." Actual performance for each program year will then be compared to these negotiated performance levels to determine whether States are eligible for incentives or subject to sanctions.

     6) Prior to the fourth program year, States will submit their expected performance levels for the fourth and fifth years of their program following this guidance in the form of plan modifications.

     7) ETA will also follow this guidance in the review, negotiation, and submission of the performance levels for program years four and five, although adjustments in the process may occur based upon experience.

B. Baselines

To the extent available, baselines for each of the 17 measures will be developed by each State and will be a key factor used to determine the level of performance that is negotiated with ETA. Summary definitions of the measures are presented in Attachment A.

Baselines are intended to give an indication of the past outcomes of a performance measure. For performance negotiations to be data-driven and reality based, the development of baselines is a critical aspect of the negotiation process. Because WIA authorizes new programs, there is not a perfect match of baseline data available to predict program performance under WIA. Instead, baselines must be based on data from preexisting programs. This situation will change in the coming years once WIA data is accumulated.

Initially, developing baselines using JTPA data is more appropriate for some of the core performance measures than for others. Baselines can be developed for the entered employment, employment retention, and earnings gain measures based on wage records because the wage record data, although not used under JTPA, can be accessed retrospectively by most States.

To assist States in the development of baseline data and expected performance levels, ETA has developed parameters to be used by all States in their baseline calculations. The Department anticipates that States will submit expected levels of performance that are above the baseline figures computed for the State. The parameters are outlined in Attachment B. The matrix in Attachment B will give States general instructions for each measure and will provide for comparable data nationwide. However, for specific operational definitions and calculation instructions on the core performance measures and the customer satisfaction measures, States should reference the TEGL 7-99, "Core and Customer Satisfaction Measures For the Workforce Investment System."

C. State Expected Levels of Performance

States will submit expected State levels of performance as part of the State Plan. States should be prepared to provide support for their expected performance levels by providing information on how baseline performance levels were developed as well as by providing other information States believe may affect performance. States will include in their plan:

• Baseline performance data and expected levels of performance for each of the 17 measures.
• The extent to which the expected levels of performance will result in the State attaining a high level of customer satisfaction.
• How the expected levels of performance compare with projected national averages (see Attachment B for projected national averages), taking into account how factors such as economic conditions, the characteristics of participants when the participants entered the program, and the services to be provided contributed to the development of the expected performance levels?
• The extent to which such expected levels of performance promote continuous improvement and ensure optimal return on investment of Federal funds.

In addition, the methodology for developing the baseline data and expected levels of performance should be included when possible. This should include a description of data sources, calculations, factors (see below for further discussion on factors), and values of factors used for expected levels of performance. For example, if a factor such as unemployment rate is used, the level of unemployment should also be included.

ETA has developed projected national average performance levels for each performance measure which should be utilized in developing State expected levels of performance. The national average for each performance measure is included in the matrix in Attachment B. While most of these national averages are based on the experiences of a limited number of early implementation States, ETA has determined these levels to be representative of estimated performance under WIA.

Performance levels may vary, up or down, based on environmental factors beyond the control of the State. Examples of possible factors to consider in negotiating expected levels of performance are listed below. This list of factors is not intended to be prescriptive or exhaustive, but to suggest the kinds of information that might be considered in the negotiation process.

Differences in Economic Conditions

• Unemployment rate
• Rate of job creation/loss
• New business start-ups

Characteristics of Participants

• Indicator of welfare dependency
• Indicator of educational level
• Indicator of poor work history
• Indicator of basic skills deficiency
• Indicator of disability
• Indicator of age
• Creation of "hardest-to-serve" index

Services to be Provided

• % WIA Title I to be spent on each category of service (core, intensive, training)
• Extent of follow-up services planned
• Extent of experimental/pilot programs
• Extent of non-Title I training money available

Other Factors That May Be Considered

• Community factors, like availability of transportation and daycare
• Policy-objective factors, like:
• evidence of application of Malcomb Baldridge criteria
• pursuit of new or enhanced partnerships
• piloting of new programs

D. Negotiation of Expected Levels of Performance

The Regional Office will review the information contained in the State plan and will compare the expected performance levels with the national averages, baseline information from other States, and the negotiated levels of performance established for other States, taking into account factors including differences in economic conditions, the characteristics of participants when the participants entered the program, and the services to be provided as discussed above. [It is understood that comparative information on negotiated levels may not be fully available during the negotiation process].

A discussion of the appropriateness of each performance level, and the adequacy of any information States offer to substantiate each performance level, will be the core of the negotiation process. The Regional Office will analyze the quality of the data presented by States, including the relevance of the data, the source of the data, the time period from which the data was drawn, and if the data is part of a trend or anomalous. When the Regional Office finalizes its analysis and determines that the State could increase their expected performance levels to support State continuous improvement and customer satisfaction strategies, they will negotiate with the State to obtain mutually agreed upon expected levels of performance.

E. Revisions

WIA allows for the renegotiation of performance levels if circumstances arise that result in a significant change in the factors used to develop the original performance levels. Since States have limited baseline data to project performance levels for PY 2000, 2001 and 2002, it is anticipated that States may want to renegotiate performance for PY 2001 and 2002 as more baseline data becomes available. DOL will collaborate with the system to develop methods for making such revisions. Such criteria could include significant changes in economic conditions, changes in service mix, or changes in client characteristics.

5. Incentives and Sanctions Policies.

Under 20 CFR 666.200, a State is eligible to apply for an incentive grant if its performance for the immediately preceding year exceeds:

• The State's negotiated levels of performance for the following areas: adult measures, dislocated worker measures, youth measures, and customer satisfaction measures for WIA Title I;

• The negotiated levels of performance included in plans submitted to the Department of Education for Title II Adult Education and Literacy programs; and

• The negotiated levels of performance under Title I of the Carl D. Perkins Vocational and Technical Education Act.

This guidance addresses the initial requirements for a State to be eligible for the WIA Title I incentives grant awards. Additional guidance will be provided at a future date about the relationship between the Department of Labor and Department of Education in the incentive process, as well as a detailed description of the incentive award application process.

A. Evaluating Performance for Incentives and Sanctions

The extent to which each State exceeds, meets, or falls below its negotiated performance levels will be examined. For each core performance measure, the percentage by which each State met its negotiated performance level will be calculated (e.g., if a State had an 80% negotiated performance level for the adult entered employment rate and the State's actual performance was 80%, they would have achieved 100% of their negotiated performance level and receive a score of 100% on that measure. If the State's actual performance was only 40%, the would have achieved only 50% of their negotiated target level and receive a score of 50% on that measure).

This analysis will be clustered by program area (adults, dislocated workers, and youth - both older and younger youth measures) and for customer satisfaction, so that each of the four groups will receive an aggregate performance score calculated from their scores for each measure in each program area and in the customer satisfaction group. The exact aggregation approach to be used will be determined as actual performance information becomes available and will likely include simple or weighted averaging. The resulting score will be the cumulative program area score for each of the four areas.

To be eligible to apply for an incentive grant, a State must meet two criteria. A State must achieve at least a 100% cumulative program area score for each of the program areas and for the customer satisfaction group. In addition, a State may not have any of their 17 measures fall below 80% of their negotiated performance levels in order to be eligible to apply for an incentive grant.

It is understood that as part of a continuous improvement strategy, States may wish to emphasize improvement on a particular measure (e.g., entered employment) which may have a slightly adverse effect on other measures in the same program area. To allow for these strategies, and to avoid consideration of sanctions for only missing their negotiated goal by a few percentage points, States will not trigger the sanctions process as long as they have achieved 80% of their negotiated performance levels for each measure.

Example 1 shows the four performance measures for the adult area, and the extent to which a state exceeded or fell below the negotiated performance levels on each of those measures. Note that the State did not achieve two of the targets. However, since the performance on each of the measures was above the 80% threshold, the State is within the acceptable performance range. This allows States the flexibility to target certain measures within a program area for improvement, without concern for minor decreases in other measures within the program area.

The following two examples shows how the cumulative program scores would be used to determine a State's eligibility for an incentive award. In each example, the bars for each program area represent the cumulative program score.

To encourage States to strive to achieve the negotiated level of performance, rather than just to meet the lower acceptable level (80%) allowed, incentive grants will only be available for a State that achieves a cumulative score in each area (adults/youth/dislocated worker/customersatisfaction) that is 100% or greater and where a State does not fall below 80% ofthe negotiated level on any single measure. Example 2 shows the 100% incentive threshold and cumulative program scores for each program area. A State has a cumulative program score of 115% of the negotiated target for the adult program, 110% for the dislocated workers, 100% for the youth program, and an 110% cumulative score for the two customer satisfaction measures (and did not fall below 80% of negotiated target on any of the measures). The State would be eligible for an incentive award.

In Example 3: A State has cumulative program scores of 100% or higher of negotiated target performance levels in all program areas but less than 100% cumulative score on customer satisfaction. The State would not be eligible for an incentive award.

Cumulative scores will NOT be averaged across areas; a State cannot make up a deficit in one area by exceeding performance in another.

B. Qualifying for Incentive Grants.

In order to promote equity and uniformity, only the Federally required core and customer satisfaction measures will be considered.

C. Amount of the Incentive Award.

Awards will range from $750,000 to $3,000,000 with a proportionate reduction in the minimum and maximum when total available funds are insufficient.

D. Incentive Application Process.

The final approach to distributing incentive awards is still under development, but may include the following process steps:

• Performance reports from each State will be submitted by a date specified by the Secretary.
• The Department of Labor will partner with the Department of Education to review the performance of each State and determine which States are eligible for an incentive award.
• For those States that are determined eligible, the following criteria may be used to determine the amount each State is eligible to receive:
  • The extent to which goals were exceeded;
  • The extent of improvements from previous year; and
  • The relationship of actual performance levels to actual performance levels in other states
• These adjusted award amounts will be published and applications for awards will be solicited.

E. Failing to Meet the Negotiated Levels.

Negotiated levels will be established for each of the 17 performance measures. The lowest acceptable level (lower limit) will be calculated as 80% of the established State negotiated level. States must attain 80% of the target performance level on each measure for performance to be determined acceptable. If a State falls below this threshold of 80% on any of the performance measures, the State may be subject to sanction. Although WIA and this guidance provide some specific requirements for sanctions, the Department will review possible sanctions on a case by case basis.

Example 4: A State has exceeded its negotiated target performance on adult entered employment by performing 110%, on employment retention by 105%, on earnings change by 100%, but only achieved 75% of the negotiated performance target for credential attainment. The State may be subject to sanction.

Example 5: A State did not achieve 100% of the negotiated performance targets on any of the adult measures. However, the State did achieve at least 80% of the negotiated performance target on each of these measures.

The state would not be subject to sanction.

F. Unacceptable Performance in Year One

The first year a State experiences unacceptable performance, ETA will make technical assistance available. This technical assistance will be provided upon request. One approach to this technical assistance may be helping the State to develop a performance improvement plan with the following components:

• Statement regarding which performance indicators will be improved through implementation of plan;
• Analysis of performance problem, including a description of problem solving techniques used to determine the most likely causes of the problem and a description of the most likely causes;
• Identification of possible actions to improve performance, and a descriptions of what actions the State will take to improve; and
• A timetable for implementing the selected solutions.

G. Unacceptable Performance in Two Consecutive Years.

If performance is unacceptable for two consecutive years on the same performance indicator, monetary sanctions may be imposed.

H. Amount of Monetary Sanctions.

Sanctions will be determined on a case by case basis, and may range from 1% up to a maximum 5% reduction.

I. Sanctions for Failure to Submit Annual Performance Progress Reports.

Under 20 CFR 667.300(e), States that are more than 45 days late may be sanctioned up to 1% plus up to an additional 1% for each additional 45 day period of lateness up to a total reduction of 5%.

6. Action. States are to provide these requirements to the appropriate State staff and use this guidance to:

a) develop baseline data and expected performance levels PY2000, PY2001 and PY2002 and submit this information in the State's WIA State Plan.

b) negotiate performance levels to be used in the State Plan approval process.

7. Inquiries. Questions concerning this issuance should be directed to your appropriate Regional Office.

8. Attachments.
Attachment A: WIA Core and Customer Satisfaction Measures-at-a-Glance;
Attachment B: Common Approach for Establishing Negotiated Levels of Performance;
Attachment C: Summary of Important Terms.

RESCISSIONS

EXPIRATION DATE    Continuing

DISTRIBUTION

Attachment A - WIA Core Measures-at-a-Glance

Adult Measures

1. Entered Employment Rate

Of those who are not employed at registration:
# of adults who have entered employment by the end of the 1^st quarter (Qtr.) after exit
# of adults who exit during the quarter

2. Employment Retention Rate

Of those who are employed in 1^st Qtr. after exit:
# of adults who are employed in 3^rd Qtr. after exit
# of adults who exit during the quarter

3. Earnings Change in Six Months

Of those who are employed in 1^st Qtr. after exit:
[Total Post-Program Earnings (earnings in Qtr 2 + Qtr 3 after exit)] - [Pre-Program Earnings (earnings in Qtrs 2 + 3 prior to registration)]
# of adults who exit during the quarter

4. Employment and Credential Rate

Of those who received training services:
# of adults who were employed in the 1^st Qtr. after exit and received a credential by the end of 3rd quarter after exit
# of adults who exited services during the quarter

Dislocated Worker Measures

5. Entered Employment Rate
# of dislocated workers (DW) who have entered employment by the end if the 1^st Qtr. after exit
# of dislocated workers who exit during the quarter

6. Employment Retention Rate

Of those who are employed in the 1^st Qtr. after exit:
# of DW who are employed in 3^rd Qtr. after exit
# of DW who exit during the quarter

7. Earnings Replacement Rate
Of those who are employed in the 1^st Qtr. after exit:
Total Post-Program Earnings (earnings in Qtr 2 + Qtr 3 after exit)
Pre-Dislocation Earnings (earnings in Qtrs 2 +3 prior to dislocation)

8. Employment and Credential Rate

Of those who received training services:
# of DW who were employed in the 1^st quarter after exit and received a credential by the end of 3rd quarter after exit
# of DW who exit during the Qtr.

Older Youth (19-21 years old) Measures

9. Entered Employment Rate

Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the 1^st Qtr. after exit:
# of older youth (OY) who have entered employment by the end of the 1^st Qtr. after exit
# of OY who exit during the quarter

10. Employment Retention Rate

Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the 3^rd Qtr. after exit:
# of OY who are employed in 3^rd Qtr. after exit
# of OY who exit during the quarter

11. Earnings Change in Six Months

Of those who are employed in 1^st Qtr. after exit who are not enrolled in post-secondary education or advanced training in the 3^rd Qtr. after exit:
[Total Post-Program Earnings (earnings in Qtr 2 + Qtr 3 after exit)] - [Pre-Program Earnings (earnings in Qtrs 2 + 3 prior to registration)]
# of OY who exit during the quarter

12. Credential Rate
# of OY who were in employment, post-secondary education, or advanced training in the first Qtr. after exit and received a credential by the end of 3rd Qtr. after exit
# of OY who exit during the quarter

Younger Youth (14-18 years old) Measures

13. Skill Attainment Rate

Of all in-school youth and any out-of-school youth assessed to be in need of basic skills, work readiness skills, and/or occupational skills:

Total # of attained basic skills + # of attained WR skills + # of attained Occ. skills
Total # of basic skills goals + # of WR skills goals + # of Occ. skills goals

14. Diploma or Equivalent Attainment Rate

Of those who register without a diploma or equivalent:
# of younger youth (YY) who attained a secondary school diploma or equivalent by the end of the 1^st Qtr. after exit
# of YY who exit during the Qtr. (except those still in secondary school at exit)

15. Retention Rate

# of younger youth found in one of the following in the 3^rd Qtr. after exit:
-- post secondary education
-- advanced training
-- employment
-- military service
-- qualified apprenticeships
# of YY who exited during the Qtr. (except those still in secondary school at exit)

Across Funding Streams

16. Participant Customer Satisfaction

The weighted average of participant ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale.

17. Employer Customer Satisfaction

The weighted average of employer ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale.

Attachment B

Common Approach for Establishing Negotiated Levels of Performance

1. General Instructions

States are requested to calculate State averages of past outcomes on the performance indicators where possible. These averages are to be calculated according to the following specifications:

• Source data:
  • JTPA data for individuals who terminated from JTPA between October 1, 1997 to September 30, 1998. This time period was chosen so that all states would have access to the necessary wage record data. Use of a common time period assures consistency among States.
  • Adult measures should be based on terminees from Title II-A.
  • Dislocated worker measures should be based on terminees from Title III, including substate and Governor's Reserve programs.
  • Older youth measures should be based on terminees from Title II-C who were age 19 or older on the date of application.
  • Younger youth measures should be based on terminees from Title II-C who were under age 19 on the date of application.
  • Wage records data will be required for most measures. Wage record data should be acquired for:
    • The second and third quarters before application for adults and older youth
    • The second and third quarters before the date of dislocation for dislocated workers. NOTE: if the dislocation date is not available, use the application date instead; and
    • The first, second, and third quarters after termination for adults, dislocated workers and older youth.

• Exclusions:
  • Individuals who received only objective assessment should be excluded from all calculations.
  • Individuals terminated with other termination codes for health/medical or institutionalized should be excluded from all calculations. This includes individuals whose other-termination code is institutionalized or health/medical (codes 1 or 2 in SPIR Item 39, Other terminations).
  • Dislocated workers served exclusively with National Reserve Funds should be excluded from all calculations
  • Entered employment measures for adults and older youth exclude individuals employed at application to JTPA. This data should be based on information collected from the participant and normally included in the JTPA data system in the field "labor force status." It should not be based on wage records.
  • Inclusions
    • Individuals served by 5%-incentive funded programs are to be included even though they were exempt from JTPA performance standards.

Detailed specifications for calculating measures are to be found in TEGL 7-99, Core and Customer Satisfaction Measures for the Workforce Investment System.

2. Matrix showing how projected averages are derived

The column labeled 'Projected Average' presents projected average values for WIA performance indicators based on available data.

The column labeled 'Explanation ...' describes how the projected averages were calculated.

The column labeled 'How to ...' describes how to calculate a comparable past value for the State based on JTPA data or indicates other data that might be useful for the negotiation process.

Indicators of Performance		Projected Average	Explanation of the method used for determining the average:	How to determine a State average using JTPA data:
A d u l t s	Entered employment	71%	Based on wage records for 7 States. Adjusted upward by 1 percentage point to account for differences between the 7 States and all States.	Exclude individuals employed at application to JTPA. Calculated as the percentage with positive earnings in the wage records in the quarter after termination. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Retention	78%	Based on wage records for 7 States. Adjusted downward by 0.5 percentage point to account for differences between the 7 States and all States.	Based on individuals employed (positive earnings in the wage records) in the quarter after termination. Calculated as the percentage with positive earnings in the wage records in the third quarter after termination. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Earnings change	$3,700	Based on wage records for 7 States.	Based on individuals employed (positive earnings in the wage records) in the quarter after termination. Calculated as the average of [ total earnings in the second and third quarters after termination minus total earnings in the second and third quarters before application ]. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Credential	60%	Assumes that the definition of credential will be somewhat flexible and that about 80% of those employed in the quarter after exit will obtain a credential.	Cannot be calculated from JTPA data. States should provide information on how they plan to define credentials and on the extent to which training programs are expected to lead to credentials. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
D i s l o c a t e d W o r k e r s	Entered employment	77%	Based on wage records for 7 States. Adjusted upward by 3 percentage points to account for differences between the 7 States and all States.	Based on all dislocated workers (except NRA). Calculated as the percentage with positive earnings in the wage records in the quarter after termination. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Retention	85%	Based on wage records for 7 States. Adjusted downward by 1 percentage point to account for differences between the 7 States and all States.	Based on individuals employed (positive earnings in the wage records) in the quarter after termination. Calculated as the percentage with positive earnings in the wage records in the third quarter after termination. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Earnings replacement	92%	Based on wage records for 5 States. Two of the 7 available States had extremely high replacement rates and were excluded from the average.	Based on individuals employed (positive earnings in the wage records) in the quarter after termination. A. Calculate the sum (over all applicable dislocated workers) of total earnings in the second and third quarters after termination B. Calculate the sum of total earnings in the second and third quarters before application. The measure is A divided by B. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Credential	60%	Assumes that the definition of credential will be somewhat flexible and that about 80% of those employed in the quarter after exit will obtain a credential.	Cannot be calculated from JTPA data. States should provide information on how they plan to define credentials and on the extent to which training programs are expected to lead to credentials. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
O l d e r Y o u t h 1 9 - 2 1	Entered employment	63%	Based on wage records for 7 States. Adjusted downward by 3 percentage points to account for differences between the 7 States and all States.	Exclude individuals employed at application to JTPA. Exclude individuals who entered non-Title-II training and do not have earnings in wage records for the quarter after termination. Calculated as the percentage with positive earnings in the wage records in the quarter after termination. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Retention	77%	Based on wage records for 7 States	Based on individuals employed (positive earnings in the wage records) in the quarter after termination. Exclude individuals who entered non-Title-II training and do not have earnings in wage records for the third quarter after termination. Calculated as the percentage with positive earnings in the wage records in the third quarter after termination. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Earnings change	$3,150	Based on wage records for 7 States	Based on individuals employed (positive earnings in the wage records) in the quarter after termination. Exclude individuals who entered non-Title-II training and do not have earnings in wage records for the third quarter after termination. Calculated as the average of [ total earnings in the second and third quarters after termination minus total earnings in the second and third quarters before application ]. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Credential	50%	Assumes that the definition of credential will be somewhat flexible and that about 70% of those employed in the quarter after exit will obtain a credential. Values are lower than for adults and dislocated workers because the employment rate is lower.	Cannot be calculated from JTPA data. States should provide information on how they plan to define credentials and on the extent to which training programs are expected to lead to credentials. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
Y o u n g e r Y o u t h 1 8 o r l e s s	Goal Attainment	72%	Calculated based on PY 97 SPIR data on the attainment of youth employment competencies. It is the total number of competencies (up to 3 per youth) divided by the number of types of training received (up to 3 per youth, but at least equal to competencies attained). The value was lower in PY 98 (66%).	Because the values likely to be achieved by the States and locals depend on the difficulty of achieving the goals set under WIA, it is not possible to calculate useful State values using JTPA data. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Diploma attainment	55%	Based on separate estimates for in-school youth and for dropouts. For in-school youth a rate of 75% is used. This is the percentage of youth in low-income families (bottom 20% of family incomes) nationwide who complete high school, which was derived from the Current Population Survey, October 1996. For dropouts JTPA data on completion of major level of education was used, approximately 21% for dropouts under 19 at entry but 17 or older at exit. These two numbers were combined using JTPA data on the split between in-school youth and dropouts.	Cannot be calculated from JTPA data. States may provide data on: The percentage of all low income youth who complete high school in the State (the national average is about 75%). The percentage of JTPA youth who completed a major level of education among those who were school dropouts at application (the national average is about 21%). Exclude youth under 17 at termination to approximate the exclusion of youth still attending school. Expected relative levels of service to in-school youth and dropouts. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Retention	54%	The percentage of youth 18 and under at entry and 17 and older at exit who entered employment or non-Title-II training (advanced training or postsecondary education) derived from PY 97 SPIR data.	A rough approximation can be calculated from JTPA data: Include all youth under age 19 at application and age 17 or over at termination. Measure as the percentage who enter employment or enter non-Title-II training. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
C u s t o m e r S a t i s f a c t i o n	Participants	68%	Based on an analysis of data from: Enterprise members (76%) A comparison group of non-Enterprise members (74%). Early data from early implementation States (65% - 70%) that show that State level results tend to be lower than the SDA/SSA-level enterprise results. Lower results than the Enterprise surveys are expected when youth are added to the sample.	Results of similar customer satisfaction surveys conducted by the State. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.
	Employers	66%	Early data from early implementation States suggest that employer results are likely to be 1 to 4 percentage points lower than for participants.	Results of similar customer satisfaction surveys conducted by the State. See TEGL 7-99, Core and Customer Satisfaction Performance Measures for the Workforce Investment System, for instructions on how to calculate performance on the measure.

• Beyond the State average, States may provide additional rationale (such as unique economic conditions, program designs or participant characteristics) along with valid data supporting the State's position of increasing or decreasing their level of performance.

The specifications outlined in the matrix are based on the following correspondence between WIA and JTPA data items:

WIA Item	Closest Corresponding JTPA Item
Registration date	Application date
Adult	Served by Title II-A
Dislocated worker	Served by Title III
Younger youth	Age at application is under 19 and served by Title II-C
Older youth	Age at application is 19 or older and served by Title II-C
Employed at registration	Labor force status at application = 'employed'
Entered advanced training or entered postsecondary education	Entered non-Title-II training.

Attachment C

Summary of Important Terms

Performance Measures:

The 17 performance indicators required by the Workforce Investment Act of 1998, section 136; Interim Final Rule, 20 CFR part 666, published at 64 Federal Register 18862 (April 15, 1999) to be used for determining return on investment and to be used in the incentives and sanctions process.

Program Area:

A cluster of measures used in the evaluation of performance for incentive purposes. There are four program areas: Adults, Dislocated Workers, Youth (both older and younger youth), and customer satisfaction (even though it is not technically a "program").

Expected Performance Level:

The target performance level a State offers as the first step in the negotiation process.

Negotiated Performance Level:

The numeric performance target agreed to by the State and the region for each of the 17 core performance indicators.

Negotiation Factor:

Any information which may affect a State's expected performance on any of the core performance measures that can be used in the negotiation process. The relevance, validity, and source of this information will be assessed during the negotiation process.

Cumulative Program Area Score:

The aggregate amount by which States exceed or fall below the negotiated performance levels in a particular program area.

Incentive Threshold:

To be eligible for incentives, a State must achieve at least 80% of the target performance levels on each performance measure, and have cumulative program scores greater than 100% in each program area.

Sanction Threshold:

States must achieve at least 80% of the target performance level on each of the 17 core indicators to avoid sanctions. To be subject to monetary sanctions, a State's performance must fall below 80% of the target performance level on the same measure two consecutive years.

^1.In the Interim Final Rule, 20 CFR part 666, the outcome of the reaching agreement on State expected levels of performance was termed "adjusted levels of performance." In this document and in the Final Rule, the outcome of reaching agreement of State expected levels of performance will be termed "negotiated performance level."