Detailed Information on the
Workforce Investment Act - Native American Programs Assessment

Improving capability to analyze program performance by collecting individual participant records.

Increasing federal oversight of grant recipients through more frequent on-site monitoring reviews.

Implementing efficiency measures that are linked to performance outcomes, account for all costs, and facilitate comparisons across Department of Labor training and employment programs.

Implement the new common performance measures for job training programs, including this one.

Strengthen performance reporting and procedures for holding grantees accountable for the key performance outcomes.

Continue to improve reporting and review of grantees?? financial management.

Adopting efficiency measures that are linked to performance outcomes, account for all costs, and facilitate comparisons across Department of Labor training and employment programs.

Measure: Entered Employment Rate

Explanation:Percent of participants employed in the first quarter after exit. This is a federal job training program common measure, which enables the Indian and Native American Program (INAP) to describe in a similar manner the core purposes and results of the program compared to other education, employment and job training programs. For example, while the target populations for INAP are unemployed, underemployed, or low income Indians, Alaska Natives, and Native Hawaiians; the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures how many participants got a job according to the following formula: Of those who are not employed at the start of participation - The number of participants who are employed in the first quarter after the exit quarter divided by the number of participants who exited during the quarter. This definition for entered employment was adopted in PY 2006, while prior years used a slightly different methodology for calculating the result. Experience shows actual performance results for this measure tend to fluctuate from year to year.

Measure: Employment Retention Rate

Explanation:Percent of participants employed in the first quarter after program exit still employed in the second and third quarters after exit This is a federal job training program common measure, which enables the Indian and Native American Program (INAP) to describe in a similar manner the core purposes and results of the program compared to other education, employment and job training programs. For example, while the target populations for INAP are unemployed, underemployed, or low income Indians, Alaska Natives, and Native Hawaiians; the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures how many participants retained their employment once placed in a job, according to the following formula: Of those who are employed in the first quarter after the exit quarter - The number of participants who are employed in both the second and third quarters after the exit quarter divided by the number of participants who exited during the quarter. This definition for employment retention was adopted in PY 2006, and made the measure more challenging by including an additional quarter in which a participant needs to be employed. Experience shows actual performance results for this measure tend to fluctuate from year to year.

Measure: Average Earnings

Explanation:The average six-month earnings This is a federal job training program common measure, which enables the Indian and Native American Program (INAP) to describe in a similar manner the core purposes and results of the program compared to other education, employment and job training programs. For example, while the target populations for INAP are unemployed, underemployed, or low income Indians, Alaska Natives, and Native Hawaiians; the ultimate outcomes for this program are the same as for all other employment and training programs. Common measures remove a barrier to service integration among programs by ensuring that programs no longer have different definitions and methodologies for measuring performance. In this case, the performance indicator measures participants' average six-month earnings once placed in a job (note: the average earnings for a year can be obtained by doubling the performance measure result), according to the following formula: Of those participants who are employed in the first, second, and third quarters after the exit quarter - Total earnings in the second quarter plus total earnings in the third quarter after the exit quarter divided by the number of participants who exited during the quarter. The common measure definition for average earnings was adopted in PY 2006, while prior year results were for an hourly earnings gain measure.

Measure: Cost per participant

Explanation:The program calculates the average cost for each participant by dividing the total annual appropriation for the program by the number of participants.

Is the program purpose clear?

Explanation: The purpose of the Native American Programs (from here, to be referred to as "the program") authorized by the Workforce Investment Act (WIA) is "to support employment and training activities for Indian, Alaska Native, and Native Hawaiian individuals in order--(A) to develop more fully the academic, occupational, and literacy skills of such individuals; (B) to make such individuals more competitive in the workforce; and (C) to promote the economic and social development of Indian, Alaska Native, and Native Hawaiian communities in accordance with the goals and values of such communities."

Evidence: Section 166 of WIA, which can be found at www.doleta.gov/dinap/pdf/166.pdf

Does the program address a specific and existing problem, interest or need?

Explanation: The program addresses the poverty and unemployment of American Indians, Alaska Natives, and Native Hawaiians--who number approximately XX million--by awarding competitive grants to finance job training, employment assistance, vocational education, and related services to eligible Indians and Native Americans. In 2002, 18,959 adults and 9,024 youth participated in the program. The unemployment rate and poverty level for Indians and Native Americans are both about twice as high as for the U.S. population generally. Further, the education level for Indians and Native Americans is significantly lower than the general population.

Evidence: Section 166 of WIA: http://www.doleta.gov/dinap/cfml/1wiasec166.cfm;[Insert source of participant statistics.] According to the 2000 Census, the average unemployment rate for Indians and Native Americans was 12.4%, compared to 5.8% for the general population; 24.8% of American Indians and Alaska Natives lived below the poverty level, compared to 12.4% of the general population; the percentage of American Indians and Alaskan Natives ages 16 to 19 that had not received a high school diploma was 16.1%, compared to 9.8% for the general population; and 24.4% of the general population age 25 and over had a bachelor's or higher degree, compared to only 11.5% for Native American and Alaskan Natives. (2000 census data on Native Americans: http://www.census.gov/census2000/pubs/phc-5.html)BLS data: Employment status of the civilian noninstitutional population by sex and race, February 2004

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: There are several other Federal programs that finance and/or provide employment, job training, and related assistance to Indians and Native Americans. For example, WIA Adult Employment and Training Activities, which also is administered by the Department of Labor (DOL), serves about 3,000 Indians and Native Americans annually. Several other programs that finance job training and employment assistance for Indians and Native Americans are authorized separately and administered by other Federal agencies, including DOI and the Departments of Health and Human Services (HHS), Education (ED), and Agriculture (USDA). Though there is redundancy and duplication among the populations served and the services that they finance, there are some differences. Some of the other programs serve only a subset of this program's target population. Some of the other programs are available only to economically-disadvantaged people and subsidize employment, unlike this DOL program.

Evidence: CFR 668.100, Subpart A and WIA sec. 166(a)(1); Public Law 102-477; Public Law 105-220; Public Law 93-638, the Indian Self-Determination and Education Assistance ActAd Hoc Analysis for All Population from the Federal Research and Evaluation Database (FRED). FRED web site can be found at www.fred-info.org/Executive Order No. 13270: www.whitehouse.gov/news/releases/2003/09/20030917-10.htmlDOL: WIA Adult Employment and Training ActivitiesDOI: Tribal Work Experience (under BIA's Office of Tribal Services as part of the General Assistance program); Indian Employment Assistance Program (under the Division of Workforce Development in the Office of Self-Governance and Self-Determination); BIA's Higher Education Program; BIA's Adult Basic Education Program; and BIA's Johnson-O'Malley Program; Example of difference: The BIA-administered Tribal Work Experience Program (TWEP) is limited to Federally-recognized tribes and Alaska Native entities. Therefore, unlike this DOL program, TWEP does not also serve INAs residing outside the service delivery areas (off-reservation), State-recognized tribes, or Native Hawaiians. HHS: Tribal Temporary Assistance for Needy Families (Tribal TANF) and Native Employment Works; Example of difference: Tribal TANF serves only the disadvantaged and may subsidize employment. ED: Native American Vocational and Technical Education Program; Adult Indian Vocational Rehabilitation Services; and Tribally Controlled Post-Secondary Vocational and Technical InstitutionsUSDA: Food Stamps Employment and Training (FSET)

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Some major flaws extend from the program's regulations, which were designed in part to conform with Public Law 93-638, the Indian Self-Determination and Education Assistance Act. The authorization and regulations provide significant flexibility to allow the grantees, especially the Federally-recognized tribes and Alaska Native entities, to adapt their individual programs to meet the needs of their community in accordance with its goals and values, and consistent with the principles of Indian self-determination. These purposes are important in meeting Indians' and Native Americans' needs and reflecting their relationship to the Federal government. However, the program design includes some flaws due to interpretation of the guiding policy: (1) the limited nature of competition for grants (see 3.CO1), (2) the patchwork of grantee performance data (see 2.5) resulting from a "menu" approach to measurement, (3) the lack of strong accountability for achieving performance results (see 3.2), and (4) the limited relationship between grant amounts and past performance.

Evidence: P.L. 93-638, the Indian Self-Determination and Education Assistance Act CFR 668.100, Subpart A and WIA sect. 166(a)(1))Workforce Systems Quarterly Results www.doleta.gov/performance/results/Reports.cfm?#etaqr)Indian and Native American National Summary and Time Trends, December 2003; See also the explanations for the answers to questions 2.5, 3.2, and 3.CO1.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The adult portion of the program, which accounts for about 80% of its appropriations, awards competitive grants through a formula that determines amounts based on levels of Indians' and Native Americans' poverty and unemployment in statutorily designated areas. These areas include not only reservations, but also off-reservation areas--sometimes called "urban" centers--that may be served by a private non-profit Native-American-controlled organization. The youth program, which accounts for about 20% of appropriations, awards competitive grants using a formula based on the number of Indian and Native American youth in those areas. With respect to the target population, grantees provide training and services based on specific eligibility requirements detailed in the program's regulations. To be eligible, an individual must be Indian or Native American and also be Unemployed, Underemployed, or low income or a recipient of a bona fide layoff notice.

Evidence: The WIA regulations at 20 CFR 668.296(b) state ''(1) One-quarter of the funds available will be allocated on the basis of the number of unemployed Native American persons in the grantees designated INA service area(s) compared to all such persons in all such areas in the United States. '(2) Three-quarters of the funds available will be allocated on the basis of the number of Native American persons in poverty the grantees designated INA service area(s) compared to all such persons in all such areas in the United States'. A copy of the regulations can be found at: www.doleta.gov/regs/statutes/finalrule.pdf.§ 668.300 of the Federal Register, published on August 11, 2000 20 CFR 668.440(a)Urban Indian centers: www.internal.doleta.gov/onp/index.htm#Indian; Approximately 48.4% of the program's participants are long-term unemployed, while another 32.1% are public assistance recipients (e.g., participants in HHS' Tribal TANF, BIA's General Assistance [GA], or USDA's FSET).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: As part of the Job Training Common Measures initiative, the program has adopted new long-term measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, than its current measures (see below) and (2) facilitate comparisons with similar programs. DOL is implementing the common measures and will establish numerical targets in 2004. In 2003, DOL's Assistant Secretary for Employment and Training transmitted to grantees and other workforce investment professionals the guidance to implement the common measures. ETA's Division of Indian and Native American Programs (DINAP) followed that guidance with a bulletin to all of this program's grantees.

Evidence: DOL/ETA Training and Employment Guidance Letter (TEGL) 15-03: wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1535 DINAP Bulletin 03-02: www.doleta.gov/dinap/bulletins/03-02.cfmETA FY 2005 Budget Justification, Volume 1, pp. 107-111; Adjustment Model: PY 2002 WIA Performance Standards Worksheet for Indian and Native American Grantees; Workforce Systems Quarterly Results: www.doleta.gov/Performance/results/Reports.cfm?#etaqr; ; Currently, the adult portion of the program uses two long-term performance measures: 1) Entered Employment Rate (EER) and 2) Positive Termination Rate (PTR). The EER is similar to one of the new common measures, but the PTR is not and does not directly measure outcomes. Instead, the PTR demonstrates the percentage of participants who successfully complete planned program activities that are believed may increase the individual's employability. The PTR methodology excludes from 'success' only those participants who fail to complete planned program activities, e.g., dropouts.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program is implementing the Job Training Common Measures and will establish baseline data and then ambitious numerical targets in 2004. In addition to establishing ambitious targets and timeframes for the long-term measures at the Federal level, the program will use these data to establish ambitious yet achievable goals for grantees.

Evidence: ETA's Workforce Investment Development Plan, Statement of Work ETA FY 2005 Budget Justification Adjustment Model: PY 2002 WIA Performance Standards Worksheet for Indian and Native American Grantees

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: As part of the Job Training Common Measures initiative, the program has adopted new annual measures that (1) will better indicate participants' employment and earnings outcomes, as well as program efficiency, and (2) facilitate comparisons with similar programs. DOL is implementing the common measures and will establish numerical targets in 2004.

Evidence: TEGL 15-03: wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1535 DINAP Bulletin 03-02: www.doleta.gov/dinap/bulletins/03-02.cfmETA FY 2005 Budget Justification, Volume 1, pp. 107-111; Adjustment Model: PY 2002 WIA Performance Standards Worksheet for Indian and Native American Grantees; Workforce Systems Quarterly Results: www.doleta.gov/Performance/results/Reports.cfm?#etaqr; PY 2003 goals are listed in the Workforce System Results for quarter ending December 31, 2003; Currently, the adult portion of the program uses two annual performance measures: 1) EER and 2) PTR (see 2.1). As discussed in 2.1, PTR does little to demonstrate directly the annual progress toward achieving the program's long-term goals.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program is implementing the Job Training Common Measures and will establish baseline data and then ambitious numerical targets in 2004. In addition to establishing baselines and ambitious targets for the annual measures at the Federal level, the program will use these data to establish ambitious yet achievable goals for grantees.

Evidence: ETA FY 2004 and FY 2005 Budget Justifications; Adjustment Model: PY 2002 WIA Performance Standards Worksheet for Indian and Native American Grantees; Summary of Individual Grantee Performance, PY 2002; PY 2002 Workforce Systems Quarterly Results: www.doleta.gov/Performance/results/Reports.cfm?#etaqr; For the current measures for both the adult and youth portions of the program, targets are based on results achieved during 2000, which serve as the baseline. Although targets for some grantees have increased over the period 2000-2004, the program's targets have not increased because there are some grantees that are not meeting their targets, and as a result, the program has not been able to achieve its annual goals. The program uses an adjustment model developed by Social Policy Research Associates to account for economic factors in the grantee's local area as well as the characteristics of the participants the grantee serves.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Each grantee is responsible to serve the particular community that it has applied to serve. However, grantees are not required to adopt the program's current Federal measures (EER and PTR for the adults program; see 2.1 and 2.3) or commit to helping achieve the program's annual or long-term goals. So, the grantee's relationship to achieving the goals is not clear. Instead, grantees choose three measures from a group of thirteen that includes, but does not require, EER and PTR. The measures were approved by the Native American Employment and Training Council authorized under WIA. This approach to performance measurement and accountability'although intended to address the needs and characteristics of particular participants and communities'has resulted in a patchwork of measurement among the grantees and some detachment from achievement of the Federal/program goals. In adopting the common measures, the program intends to simplify and improve not only its measurement of the overall program's performance, but also that of individual grantees.

Evidence: DINAP Bulletin 99-07: Two-year Planning Guidance: www.doleta.gov/dinap/bulletins/99-07.cfm; Each grantee is to be accountable for achieving targets that it established through the following process: Each biennial plan mandates that the grantee adopt three performance goals and measures from among thirteen measures in three categories ('employment, training, and community'). Each grantee chooses one measure in each of the three categories, subject to a requirement that one of the chosen measures applies to at least 25 percent of the grantees' participants and each of the other two measures applies to at least 10 percent of the participants. Grantees set targets working with DINAP, which uses the adjustment methodology. Grantees may request changing a measure [or target] before the [start of the second year]. Achievement of two of the three annual goals is considered successful. The Federal targets and performance are based on data from all grantees even though not all grantees choose the EER and PTR for themselves. [List the thirteen measures in three categories and statistics on the percentage of grantees using each of the measures, highlighting the percentage who choose the EER and PTR measures.]

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The most recent evaluation of this program was conducted under contract and issued in January 1999. That study evaluated the predecessor program that was authorized under the Job Training Partnership Act (JTPA), including the Summer Youth program. The study used surveys and site visits to gather information. Although it was useful to DOL in determining the strategies for implementing the subsequent (and current) WIA program, it did not include a rigorous evaluation of the JTPA program's effectiveness or attempt to quantify its impact on the target population. The current program is scheduled for an evaluation of its effectiveness [and impact on participants] in 2006 by an independent contractor.

Evidence: "An Evaluation of the JTPA Section 401 Indian and Native American Program, U.S. Department of Labor, Employment and Training Administration, Series 99-A, 1999"

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests are not tied explicitly to achievement of the program's goals, but that is expected to improve after implementation of the new common measures (see 2.1 and 2.3). However, resource needs are presented in a clear and transparent manner in the annual budget justification. WIA authorizes annual program resources of at least $55 million for the adult portion of the program. For the youth program, the annual allocation is 1.5% of the total appropriation for WIA Youth Activities, the program that provides formula funding to States and localities for youth job training and employment assistance. The 2005 Budget request supports ETA's strategy to create a "demand-driven" workforce investment system by responding to businesses' needs and encouraging grantees to merge workforce development activities with community economic development. The program also is participating in a DOL-wide initiative to develop a cost accounting system that will make it easier and faster to determine the amount of time, money, and human resources spent on program management.

Evidence: ETA FY 2005 Budget Justification WIA section 174(a)(2)(A) and WIA section 127(b)(1)(C)(i)(I)

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Within the last year, ETA has taken steps to integrate service delivery among its programs, including this one, by beginning to implement the new common measures for Federal job training and employment programs (see 2.1 and 2.3). For this program, the common measures will improve information on participants' outcomes after they exit the program and grantees' performance on outcomes of great importance for participants: whether they find a job; whether they keep a job; and how much they earn in the job. The measures will help the program identify areas of strength and weakness in its policy, strategy, and management.

Evidence: DINAP Bulletin 00-18: www.doleta.gov/dinap/bulletins/00-18.cfm TEGL 15-03: wdr.doleta.gov/directives/corr_doc.cfm?DOCN=1535DINAP Bulletin 03-02: www.doleta.gov/dinap/bulletins/03-02.cfm); [The following probably belongs later in the assessment: In addition, in 1995, the program was structured in three multi-regions of the East, Midwest and West with two Federal Project Officers (FPOs) serving each multi-region. FPOs working with the West Region are outstationed in the Seattle Regional Office and one FPO is working with the Midwest Region in the Dallas Regional Office. The East FPO is located in the National Office in Washington, DC. This structure allows for increased access to grantees and facilitates communication between the communities and regional offices via One-Stop Career Centers. Training and technical assistance (TAT) is organized both regionally through regional conferences, peer-to-peer onsite, and on a national level through the national conferences. FPOs provide one-on-one TAT sessions and workshops at the conferences.]

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: DINAP collects performance information semi-annually [but will move to quarterly collection as it implements the common measures] and expenditure information for all active grantee subaccounts quarterly. Performance goals and data affect decisions regarding designation and/or defunding decisions, including emergency terminations. When performance warrants, grantees are requested to provide corrective action plans and/or are provided options to change their individual performance measures for planning purposes based on unique circumstances or conditions within the community/reservations that may affect their ability to meet a measure. Severe remedies are used in exceptional circumstances, for major findings of fraud, abuse, or gross mismanagement. Federal Project Officers use quarterly financial reports and semi-annual program reports in their risk assessment rating of grants. DINAP uses a database to monitor grantee performance and, currently, a contractor to generate annual performance reports. ETA has developed the Grants Electronic Management System (GEMS) which will also be used by the DINAP office to manage grantee performance.

Evidence: FY 2004/ PY 2003 ETA Goals and Indicators for Office of National Programs Indian and Native American National Summary and Time Trends, December 2003 Program Year 2002 Performance Measures: Indian and Native American Programs Risk Analysis Report as of May 15, 2003 [Insert references to examples of management action, if possible.]

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Accountability for performance results among grantees is insufficient. All grantees are assessed annually on their performance against targets approved in their plan and grant. To be "successful," a grantee needs to achieve the target for two of the three measures that it chose. Grantees who exhibit 'substantial and persistent' failure to meet targets are asked to provide corrective action plans and provided with counseling and technical assistance. However, termination or noncontinuation of a grant based solely or primarily on performance has rarely occurred. The Grant Officer also may terminate a grantee's designation for cause if there is substantial or persistent violation of the statutory or regulatory requirements, or if the grantee could not demonstrate that it had program management personnel necessary to effectively deliver program services. Three grantees were terminated in the most recent designation process for the reasons indicated above. ETA managers and staff are accountable through incorporation of relevant elements and performance standards in the appraisal process.

Evidence: WIA regulations pursuant to 20 CFR 668.290 (b)WIA regulations at § 668.720DINAP Bulletin 99-07: Two-year Planning Guidance: www.doleta.gov/dinap/bulletins/99-07.cfmPersistent poor performers may not receive a continuation grant at the two-year mark, but in the first continuation grant mark for the WIA program in 2002, only [#] grantees did not receive a continuation, despite the fact that [#] grantees failed to achieve at least two of the three performance targets in the preceding year. A few grantees have had their grants terminated for failures including, but not limited to, failure to achieve adequate performance. In [year/years], however, only [one] grantee was terminated in large part because of performance results.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: This program runs on a "Program Year" (PY) that, for the adult portion, begins nine months after the beginning of the fiscal year (FY). So, PY 2004 begins on July 1, 2004, and provides funding from the FY 2004 appropriations. Funds are obligated to grantees soon afterward. Funds for the youth portion are available April 1, but the program issues those funds the middle of May. By law, funds awarded to Federally-recognized tribes may be transferred to BIA for inclusion in a demonstration account that allows tribes to pool certain related resources from DOI, HHS, and DOL. These '477' grants or contracts--so called because the demonstration is authorized under Public Law 102-477--are usually transferred approximately the same time that the non-477 grantees receive their funds. The grantees' accounts are regularly reviewed for drawdown consistency, comparability with financial expenditures reports, and allowable program purposes through project officers' desk reviews, risk analysis, on-site reviews, single audits, or in some instances by audits conducted by the Inspector General (IG).

Evidence: Illustrative examples of award document signed on first day of program year, action taken to correct single audit findings, letters to grantees informing of reporting status (reference) National Summary of Expenditures- Financial Status Report, ETA 9080 (reference) DINAP Planning Instructions Memorandum for PY 2002-2003 (reference); [Insert here or under financial mgt.: All grantees that receive combined grant funding of $300,000 or more are subject to the Single Audit Act as required by OMB A-133. There have been no systematic problems identified through audits that indicate that program funds are not being spent for the intended purpose. The DINAP office continues to make concerted efforts to ensure that federal funds are expended in a timely manner. Since the enactment of JTPA, DINAP has included a provision in the regulations that allows the program to recapture funds in excess of 20%. In fact, the DINAP office enforced this regulation in PY 1998 by de-obligating funds in excess of 20% from several grantees. The 20% excess funds provision was also included in the WIA regulations at § 668.296 (d) which states, '' Funds may also be reallocated if a grantee has carry-in in the excess of 20 percent of the total funds available to it.'. To help grantees monitor expenditures, a 'Financial Status Report' is sent out to grantees each quarter that provides a summary of the grantees expenditures for each sub account.]

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: DOL may reserve for technical and training assistance (TAT) up to one percent of the adult program's appropriation. DINAP conducted an internal cost analysis study that assessed the costs of competing TAT contracts among Indian-controlled contractors versus a 'TAT grant' to a non-profit Indian organization. It showed the potential for significant savings, and the program chose the less expensive option. In addition, although grantees are no longer held to any cost-per-participant planning level, as under the predecessor JTPA program, project officers work with grantees to help ensure that each service plan contains reasonable efficiency level(s) given the individual circumstances faced by each grantee. The program will continue use of an efficiency measure of cost per participant as part of the new common measures. DINAP also invested in IT improvements that virtually eliminated the time project officers spent correcting and filing hard copy program and financial reports. DINAP is also currently participating in the implementation of ETA's new data validation system.

Evidence: Average Cost Per Participant ('Efficiency') listed by Grantee, PY 2002 Grantees report to the Indian and Native American Program via the Internet: www.etareports.doleta.gov/CFDOCS/grantee_prod/reporting/index.cfmNative American Workforce Investment Act (NAWIA) software; TAT example: Our previous contract experience with both ACKCO, Inc., and Falmouth Institute indicated that approximately $.32 of every available TAT dollar was going towards contractor's fees, either through indirect costs, direct salaries, or profit margin. A TAT grant is now awarded to the California Indian Manpower Consortium (CIMC). Under the CIMC 'TAT grant', only $.17 of every dollar goes to pay grantee salaries and expenses, including indirect costs. The other $.83 goes into providing direct support and/or services in support of approved TAT objectives, including the multi-regional and national conferences, peer-to-peer TA, Advisory Council and work groups meetings, TAT support for the 477 program, and the Tribal Colleges initiative.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Despite the significant amount of redundancy and duplication among this program and others (see 1.3), the program collaborates and coordinates fairly effectively with other DOL and Federal programs. For example, WIA requires that this program's grantees be a partner in the WIA One-Stop Career Center system. Most grantees in urban areas, in particular, have meaningful working relationships with their local One-Stops. However, coordination with the One-Stops is a greater challenge for tribal grantees that are located in remote and isolated reservation communities. Therefore, grantees also are encouraged to link with other Federal, State, and local programs. For grantees on reservations, collaboration and coordination is often achieved through a tribal administration office that serves much like the WIA One-Stop Career Centers for WIA programs. Finally, the "477" demonstration authority can enhance coordination because it allows grantees to combine funds from different Federal programs and 'consolidate' their employment and training services under one comprehensive program.

Evidence: Additional information on P.L. 102-477: www.doleta.gov/dinap/cfml/477glnc1.cfmDINAP Bulletin 99-07: Two-year Planning Guidance: www.doleta.gov/dinap/bulletins/99-07.cfmExamples of Memorandum of Understanding between Workforce Investment Boards and Native American grantees (reference); WIA One-Stop Career Center MOUs: Grantees are to negotiate Memoranda of Understanding (MOU's) with their Local Workforce Investment Boards (LWIB's). In [Year], the first year of WIA implementation, X% of grantees signed such MOUs. Coordination by tribes of multiple Federal programs: During on-site visits, this program's grantees are required to demonstrate linkages with employment, training, welfare, social service, education, vocation, correctional, general assistance and other related services at all levels. Tribal grantees also coordinate with other Federal employment and training programs such as the Tribal Work Experience Program (TWEP) within the BIA, the Native American Vocational Education program within the Department of Education, and various other tribal employment and training programs.

Does the program use strong financial management practices?

Explanation: DINAP periodically reviews quarterly Grantee Financial Status Reports (ETA-9080) against the grantees' fund drawdowns to ensure that the grantee (1) is not drawing down too much, resulting in excess cash on hand; but (2) is drawing down sufficient funds to cover reported expenditures. DINAP also tracks grantees' single audit compliance through the Census Web site and issues conditional designations and/or conditional funding when warranted. With more than 300 individual grants to administer and manage, the program's six Federal project officers use technology, peer-to-peer reviews, the regional and national forums and DOL sponsored contract support such as financial management seminars to conduct oversight. DINAP also reviews financial management through independent audits and IG audits. DINAP worked with the Office of Technology (OT) to develop customized financial management reports that are used to analyze the grantees' financial performance based on quarterly financial reports. Under the "477" demonstration, BIA is responsible for grantee accountability, including financial.

Evidence: Grantee Financial Status Reports (ETA 9080); Tracking through census web site: http://harvester.census.gov/sac/; OIG reports: http://www.oig.dol.gov/cgi-bin/oa_rpts.cgi?s=DINAP&y=all&a=03; Each year DINAP provides the Office of the IG (OIG) with a list of grantees it determines to be at high risk for program deficiencies. In [Year], the OIG conducted four audits of the program. Although the OIG questioned some allowable costs, the 'final' disallowed costs were nominal. Copies of these reports can be found at http://www.oig.dol.gov/cgi-bin/oa_rpts.cgi?s=DINAP&y=all&a=03.; DINAP and OT developed reports to identify grantees who have financial issues such as excess cash on hand, excessive amounts of unexpended funds, administrative costs that exceed the allowable limit, and unexpended funds in past subaccounts. DINAP FPO's must review each quarterly financial report before the report can be certified in the EIMS system and accepted by the ETA. DINAP managers monitor the certification status of quarterly financial reports to ensure that FPO's review and certify reports in a timely manner.; BIA provides DINAP with some of the information that it tracks.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: DINAP staff regularly review program practices and procedures, such as the designation and planning and funding processes. DINAP uses a Microsoft shared database that allows financial and programmatic tracking of all grants and provides for data downloads through DOL's Enterprise Information Management System (EIMS), Grants Tracking System (GTS), and current Core Accounting System (DOLAR$) and HHS's Payment Management System. DINAP's internal database was implemented two years ago and provides for customized reports containing aggregate data. It allows easy access to information needed to manage the large number of grants. Such IT improvements as Web-based reporting, grantee reporting software and the creation of a DINAP database allow increased oversight. Finally, DINAP is working with ETA's Performance and Results Office (PRO) to implement data validation and a new 'Standard Individualized Record' (SIR) reporting system that will standardize definitions for all the ETA programs to facilitate the integration of service delivery and break down barriers among ETA programs.

Evidence: DINAP database; [Insert data validation and SIR references.]

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The framework for awarding grants provides for a limited type of competition. The competition for grants is limited due to guiding Federal policy, program authorization, and regulations related to this program and Indians and Native Americans. As a result, there is effectively no competition for service on reservations, and the amount of each grant is determined by a formula taking into account poverty and unemployment levels in the geographic area being served. There are usually two to three new awardees per award cycle. A Federally-recognized tribe serving its own reservation has the highest priority of designation for those areas over which it exercises sovereignty. Tribal grantees account for 117 of the total 186 grantees, or 63%. However, in those instances where a tribe wishes to serve off-reservation areas, competition is possible with another tribe, private non-profit Indian-controlled organization, or a consortium thereof in a 'full competition' year. For the latest grant cycle, there were 11 competitions. In 2000, there were 28 competitions.

Evidence: SGA announced December 31, 2003: www.doleta.gov/DINAP/pdf/SGA.pdf20 CFR 668.210WIA Section 166(c)(2) calls for a 'full and open' competition once every four years. Competition may be held on the 'off two years' for those areas served by an entity that does not perform satisfactorily enough to receive a waiver of competition. Federally recognized tribes, Alaska Native controlled organizations, Native Hawaiian controlled entities, Native American controlled organizations serving Indians or a consortium of eligible entities may apply for funds. The ETA requests applications from eligible entities through a Solicitation for Grant Application (SGA) published in the Federal Register. The competitive selection procedures require that the Grant Officer convene a panel to rate competing entities.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: [Insert summary from responses to 3.3, 3.6, and 3.7. Attempt not to duplicate information, though questions and answers are similar.]

Evidence: Grantee Financial Status Reports (ETA 9080); Tracking through census web site: http://harvester.census.gov/sac/; DINAP and OT developed reports to identify grantees who have financial issues such as excess cash on hand, excessive amounts of unexpended funds, administrative costs that exceed the allowable limit, and unexpended funds in past subaccounts. DINAP FPO's must review each quarterly financial report before the report can be certified in the EIMS system and accepted by the ETA.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects performance data on grantees quarterly and semi-annually and reports grantee performance information annually. (It reports program-level performance data quarterly and annually.) Performance information is made available to the public in a transparent and meaningful manner, including posting it on the ETA Website. DINAP collects performance data through the EIMS that is downloaded in DINAP's reporting database designed to receive such information on a quarterly and semi-annual basis. The DINAP Bulletins, which relay program specific information, policies, and procedures, also relay performance data. The program collects and analyzes information on a quarterly basis and results are published by the ETA's PRO for distribution.

Evidence: Annual grantee and program performance results: www.doleta.gov/performance/results/native_americans_performance.cfm;ETA's quarterly performance results, including for this program: www.doleta.gov/performance/results/Reports.cfm?#etaqr;DINAP data and bulletins: www.doleta.gov/dinap/

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has adopted new long-term goals as part of the Job Training Common Measures initiative (see 2.1 and "Measures" sheet). DOL will establish numerical annual and long-term targets in 2004. The program could receive partial credit here if its progress over time on its current measures were adequate to support long-term improvement. However, the program failed to achieve either of the current measures (EER and PTR) for the adult program (the most significant portion of funding'about 80%--and participants'about 68%) in the two most-recent program years for which full-year data are available. The program did achieve one of its 2001 targets and both of its 2002 targets for the youth program, which accounts for 20 percent of appropriations and 32 percent of participants.

Evidence: PY 2002 GPRA performance results for the Native American: www.doleta.gov/performance/results/Edition_6_9-30-03.pdf; ETA established the following long-term goals (by 2004) for the adult program: 1) 56% of participants will be employed and; 2) 86% of participants will have a "positive termination" at program exit. The annual targets for 2001 and 2002 were the same as the long-term goals, and the results were 53% and 82% for 2001, and 52% and 83% for 2002, respectively. ETA also established the following long-term goals for the youth program: 1) 61% of participants would attain at least two goals relating to basic skills, work readiness, skill attainment, entered employment, or skill training and 2) 66% of participants would attain a high school diploma, GED, or an improvement of basic skills by at least two grade levels. The results for the youth program were 62% and 63% for 2001 and 69% and 83% for 2002, respectively.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has adopted new measures and goals as part of the Job Training Common Measures initiative (see 2.3 and "Measures" sheet). DOL will establish annual and long-term numerical targets in 2004. However, the program can receive partial credit here based on results against its current annual performance goals. As indicated in 4.1, the program failed to achieve either of its goals for adults in 2001 and 2002. The program achieved one of two annual goals for youth in 2001 and both of the goals for youth in 2002. As indicated in 4.1, resources for youth services account for about 20% of the program's appropriations.

Evidence: PY 2002 GPRA performance results for the Native American: www.doleta.gov/performance/results/Edition_6_9-30-03.pdf

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program's average cost per participant decreased from $3,089 in 2000 to $2,592 in 2001 and $2,489 in 2002. For 2002, the cost per participant for the adult portion of the program was $2,507 and the cost per participant for the youth program was $1,401. Important program management changes that contributed to increased efficiency include the following: In 2001, DINAP and the Native American Employment and Training Council developed an automated management information system (Native American Workforce Investment Act or "NAWIA" software) for grantees. The software increased efficiency in participant tracking and reporting, which allowed front line staff to spend more time assisting participants and less time completing participant information.

Evidence: WIA Section 166 Time Trends Indian and Native American National Summary and Time Trends, December 2003 Native American Workforce Investment Act (NAWIA) software; The total cost per participant measure for both portions of the program is now calculated by dividing total annual appropriations (minus the funds transferred to BIA for the '477 grantees') by the total number of participants in each program. Unlike earlier efficiency measures for the program, Federal administrative costs are included, and the actual amount(s) expended are not taken into account. Therefore, it is a measure of program efficiency at the Federal level rather than an expression of grantees' efficiency alone. By comparison, the WIA Adult Employment and Training Activities formula grant program had a cost per participant in 2002 that was approximately $1,900.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: In comparing this program's performance with the results for Indian and Native American participants in the WIA Adult Employment and Training Activities formula grants program, the WIA Adult program compares more favorably. There were almost 3,000 Indian and Native American participants who exited the WIA Adult program in 2002, compared to almost 19,000 participants in this program. For the only measure that the program share currently, the Entered Employment Rate, the results for 2002 were as follows: WIA Adult 69%; this program 52%. Indian and Native American participants in the two programs share other similarities: The percentage of this program's participants who are dropouts was about the same as the percentage in the WIA Adult program. However, participants differ in some ways, including that the WIA Adult participants self-identify as Indian and Native American, while this program's participants must meet statutory eligibility criteria. Also, the percentage of public assistance recipients that exited this program (32.1%) was twice as high in the WIA Adult program (15.6%).

Evidence: National Summary Report - WIA PY 2001 ExitersIndian and Native American National Summary and Time Trends, December 2003Evaluation of JTPA, Section 401 INA Program, 1999PY 2002 performance results (quarter ending June 30, 2003) in the Workforce System Results Quarterly Review: www.doleta.gov/performance/results/Edition_5_6-30-03.pdfETA FY 2005 Congressional Justification; Information from other, non-DOL programs that are similar is unavailable at this time and will be added and evaluated as it becomes available.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: As indicated in response to question 2.6, an evaluation of the predecessor program under JTPA was completed in 1999. This WIA program is scheduled for an evaluation to be conducted in 2006 by an independent contractor. The study conducted in 1999 evaluated the JTPA section 401 program, including the Summer Youth program under JTPA title II-B. This study was useful in determining the directions taken by the subsequent WIA program. It examined services and strategies that grantees use to deliver them. However, it was based on surveys and site visits and addressed the effectiveness of the program and its achievement of results only very broadly, with little support beyond examples based on information from site visits to some of the grantees.

Evidence: Evaluation of JTPA, Section 401 INA Program, 1999