Detailed Information on the
Veterans' Employment and Training State Grants Assessment

Implement recommendations from follow-up evaluation to assess veteran outcomes.

Setting aggressive targets for performance outcomes to improve services to veterans.

Establishing performance targets for individual State grants using new data and refined performance measures.

Conducting an independent evaluation to assess the employment outcomes resulting from the programmatic changes made by the Jobs for Veterans Act and other changes undeway within the workforce system.

Adding the Average Earnings measure to the performance measures for All Veterans and for Disabled Veterans in developing guidance on negotiating PY 2008 One-Stop performance targets with State Workforce Agencies.

Measure: Entered Employment Rate for all veterans.

Explanation:The Entered Employment Rate for all veterans indicates the success of the employment assistance provided to veterans by One-Stop Career Centers operated by State Workforce Agencies, including the Disabled Veterans' Outreach Program (DVOP) specialists and Local Veterans' Employment Representative (LVER) staff, both funded by the Veterans Employment and Training State Grants. This assistance can include assessment, employability development, vocational training, job development and other workforce services. This performance measure is a "common measure" that is applicable to all Federal job training programs, thus enabling comparisons of results across similar programs. It is calculated by formula: Of those who are not employed at the date of program participation, the number of veteran participants who are employed in the first quarter after the exit quarter divided by the number of veteran participants who exited the program during the exit quarter.

Measure: Employment Retention Rate for all veterans.

Explanation:The Employment Retention Rate for all veterans indicates the success, in terms of lasting employment, of the employment assistance provided to veterans by One-Stop Career Centers operated by State Workforce Agencies and the Disabled Veterans' Outreach Program (DVOP) specialists and Local Veterans' Employment Representative (LVER) staff, both funded by the Veterans Employment and Training State Grants. This performance measure is intended to encourage the delivery of high quality workforce services. This measure also is a "common measure" that is applicable to all Federal job training programs, thus enabling comparisons of results across similar programs. It is calculated by formula: Of those who are employed in the first quarter after the exit quarter, the number of veteran participants who are employed in both the second and third quarters after the exit quarter divided by the number of veteran participants who exited the program during the exit quarter.

Measure: Entered Employment Rate for disabled veterans.

Explanation:The Entered Employment Rate for disabled veterans indicates the success, in terms of employment assistance provided to disabled veterans by One-Stop Career Center System operated by the State Workforce Agencies, and the Disabled Veterans' Outreach Program (DVOP) specialists and Local Veterans' Employment Representative (LVER) Program staff, funded by the Veterans Employment and Training State Grants, in providing employment assistance to disabled veterans to include special disabled veterans, as defined by USC Title 38, Chapters 41 and 42. This performance measure is a "common measure" that is applicable to all Federal job training programs, thus enabling comparisons of results across similar programs. It is calculated by formula: Of those who are not employed at the date of program participation, the number of disabled veteran participants who are employed in the first quarter after the exit quarter divided by the number of disabled veteran participants who exited the program during the exit quarter.

Measure: Employment Retention Rate for disabled veterans.

Explanation:The Employment Retention Rate for disabled veterans indicates the success, in terms of lasting employment, of the employment assistance provided to disabled veterans by One-Stop Career Centers operated by the State Workforce Agencies, and the Disabled Veterans' Outreach Program (DVOP) specialists and Local Veterans' Employment Representative (LVER) staff, funded by the Veterans Employment and Trainign State Grants. This performance measure is intended to encourage the delivery of high quality specialized workforce services that will result in lasting employment. This measure also is a "common measure" that is applicable to all Federal job training programs, thus enabling comparisons of results among similar programs. It is calculated by formula: Of those who are employed in the first quarter after the exit quarter, the number of disabled veteran participants who are employed in both the second and third quarters after the exit quarter divided by the number of disabled veteran participants who exited the program during the exit quarter.

Measure: Cost per veteran entering employment

Explanation:The cost per veteran entering employment gauges the cost-effectiveness of the Veterans Employment and Training State Grants, which are formula grants provided annually to State agencies to support the staffing costs for Disabled Veterans' Outreach Program (DVOP) specialists and Local Veterans' Employment Representative (LVER) staff members. This measure attributes the cost of the services provided by DVOP specialists and/or LVER staff to the veteran participants who receive those services and subsequently enter employment. The efficiency measure is a unit cost that is calculated by dividing the funding provided to the States to operate the DVOP and LVER programs during a given time period by the number of veteran participants who enter employment during that same period. The relatively low unit costs that appear in the table below as the targeted amounts for PY 2005 and PY 2006 reflect a less rigorous approach to measuring efficiency, based on the total number of participants. Starting with PY 2007, the targeted amounts appearing below reflect a more rigorous approach to measuring efficiency, based on the number of participants who successfully enter employment following the receipt of services.

Is the program purpose clear?

Explanation: The purpose of this State grant program is to help veterans and other qualified job seekers secure quality employment. Administered by the Department of Labor's Veterans' Employment and Training Service (VETS), the program provides formula grants to States to finance Disabled Veterans' Outreach Program (DVOP) specialists and Local Veterans' Employment Representative (LVER) staff. As required by law, these employment specialists provide job placement assistance, labor market information, and related services to veterans who are most disadvantaged in the labor force. These veterans include the disabled, homeless, or recently separated from military service, and persons transitioning from the military to civilian employment. The LVER staff also conduct outreach to employers and facilitate the provision of services to veterans within the workforce development system.

Evidence: 38 USC (sections 4100, 4102 - identifying the need for and purpose of the program; sections 4103A - 4104 establishing the infrastructure for delivery of employment services to veterans); (see www.nvti.cudenver.edu/Implementation/38usc41.pdf) as amended by PL 107-288.(see frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=107_cong_public_laws&docid=f:publ288.107)

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses the employment and training needs of unemployed and underemployed veterans by providing job search, placement, and related assistance that lead to quality employment. In program year (PY) 2003, over 1.2 million veterans were served by the One-Stop Career Center system, of which 863,000 were served by the DVOP/LVER staff. The program targets services on veterans who have the most difficulty in securing civilian employment, such as those who: 1) have a service-connected disability, 2) served on active duty during a war or in a campaign, or 3) are active-duty service members transitioning from military service to civilian careers. Data from the Bureau of Labor Statistics show that, on average, the 2004 unemployment rate for veterans 20-24 years of age was 13.6%, compared to 9.4% for non-veterans in the same age group. The unemployment rate for veterans aged 25-29 also was higher than the non-veterans' rate, 7.2% vs. 6.1%. Among veterans age 45 to 49 years of age, the unemployment rate was 5.1%, vs. 3.8% for non-veterans. Other BLS employment data ("Employment Situation of Veterans: August 2003") indicated that the unemployment rate for recently separated veterans was 6.9%, while the general rate of unemployment at that time was 5.9%. A recent study by the Battelle Memorial Institute concluded that the existence of a service-connected disability is strongly associated with a greater likelihood of unemployment.

Evidence: 38 USC 4100 (2), 4102; Unpublished annual labor force data on veterans and non-veterans for 2004, compiled by the Bureau of Labor Statistics; "Employment Situation of Veterans: August 2003," BLS, publication date July 27, 2004: ftp://ftp.bls.gov/pub/news.release/History/vet.07272004.news ; Battelle Memorial Institute final report, "Assessment of Unemployed Veterans' Needs for the DOL's Veterans' Employment and training Service, November 20, 2003."

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The DVOP/LVER State Grants Program duplicates services provided by three DOL State formula grants, including the Wagner-Peyser Employment Service (ES) grants to States, and formula grants for Adult Activities and Dislocated Worker Assistance authorized by the Workforce Investment Act of 1998 (WIA). These other DOL programs (in DOL's Employment and Training Administration, ETA) are also required by law to provide priority of service to veterans. In 1933, Congress created the ES as a public "labor exchange" under the Wagner-Peyser Act, authorizing block grants to States to match employers and job seekers and requiring the ES to serve veterans. In 1980, Congress separated the DVOP/LVER grant program from the Wagner-Peyser grant program in response to the perceived failure of the ES to serve unemployed veterans effectively. In 2002, Congress passed the Jobs for Veterans Act (JVA), adding a veterans' priority to DOL's WIA programs. Because of significant statutory duplication, VETS strives to operate the DVOP/LVER program as a service enhancement to the other programs available at the One-Stop Career Centers. DVOP/LVER staff serve as experts on the range of programs available to veterans and transitioning service members. Job-ready veterans with no severe employment barriers may be provided basic labor exchange services through the ES or WIA programs, while DVOP/LVER program staff target veterans and transitioning military service members who require more intensive services.

Evidence: Summary of Provisions of PL 96-466: http://thomas.loc.gov/cgi-bin/bdquery/z?d096:HR05288:@@@L

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The DVOP/LVER grant program enables State workforce agencies to have staff who are veterans' program specialists and to deploy them where the needs for intensive services are greatest. DVOPs and LVER specialists exist because veterans have unique employment needs as a result of their military service. Veterans' specialists employed by the DVOP/LVER program are typically veterans themselves, making them well-qualified to assist other veterans. Like their clients, the majority of DVOP staff are service-connected disabled veterans. The DVOP/LVER program's service delivery strategy is efficient because it leverages existing resources. The "two-tiered" approach to service delivery calls for job-ready veterans to be served by ES-funded staff, while veterans with employability barriers are served by DVOP/LVER staff. This approach allows more veterans to secure employment at lower cost. Efficiency indicators recently developed by SRA International for VETS show the cost per entered employment for veterans served by DVOP/LVER staff was $345, while the corresponding cost for veterans served by ES State grants staff was $119. The higher cost for those served by DVOP/LVER staff is attributable to the extra time needed to serve those veterans with employability barriers. In addition to the efficiencies derived from the program's approach to service delivery, the program's operational structure as a State grant program takes advantage of opportunities for cost savings and cost sharing with other One-Stop partners.

Evidence: "Interim Efficiency Measures Report", Department of Labor, Veterans Employment and Training Service--. Developing Efficiency and Earnings Gain Measures, conducted by SRA International pursuant to contract DCN: IEMR2005-001-003-1.0 NOTE: The final report is due in September, 2005. It will be posted on DOL website after receipt. Veterans benefits and the Wagner-Peyser Act: www.nvti.cudenver.edu/vetsresource/benefits2.htm. Brief narrative on the history of veterans benefits, referencing the Wagner-Peyser Act. Also, www.uses.doleta.gov/wp.asp describes the Wagner-Peyser Act and special employment services for veterans.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: By law, program funds can support services only to veterans or other qualified individuals. However, to provide flexibility to One-Stop operators, the Jobs for Veterans Act of 2002 (JVA) permits DVOPs/LVERs to serve non-veterans on a part-time basis, provided they charge their time to other funding sources. Under the JVA, State grant allotments are determined using a formula that is based on each State's relative share of the national total number of veterans who are seeking employment. Each State prepares a spending plan that includes the number of FTEs of DVOP specialists and LVER staff available for assignment throughout the State. Within that FTE total, State grantees allocate the available staff to different geographic regions of the State, to DVOP or LVER positions, and to full-time or half-time positions, in response to projected workload. VETS has authority to identify unneeded State funds and reallocate them to other States to improve targeting on areas where greater need exists. At the local level, DVOP and LVER services are targeted on veterans with moderate to high needs for employment services. By contrast, job-ready veterans are expected to access self-help resources or other services that are available to all from the ES program.

Evidence: 20 CFR 1001 Subpart F; Solicitation for Grant Applications, FYs 2005 - 2009; DM 9-04.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The JVA requires the program to establish outcome-based employment performance measures consistent with the statutory requirements of WIA. Consistent with the JVA, the DVOP/LVER program has established two long-term performance measures: 1) Entered Employment Rate, and 2) Employment Retention Rate. Each measure is applied to the grantees' total outcomes for all veterans and disabled veterans, not just to the veterans served by the DVOP/LVER staff alone, because the grants are expected to boost the overall performance of the labor exchange system in serving veterans. VETS has adopted the Administration's common measures for outcomes of Federal job training and employment programs. VETS is changing the definitions of its current long-term measures as necessary and is instituting an earnings measure.

Evidence: Excerpt from the FY 2005 DOL Performance Budget. www.dol.gov/_sec/Budget2005/_Toc63239903 Performance measure requirements are contained in: Section 4102.A(f)(2)(A) of the Jobs for Veterans Act, and Section 136(b) of WIA 1998

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: VETS adopted ambitious targets for its long-term measures in its last five-year strategic plan covering PY 2003 through PY 2008 (see Performance Measures section). The measures adopted were Entered Employment and Retention in Employment and applied to all veterans and to disabled veterans. Target performance outcomes for 2008 were at least 4.3 percent higher than the 2003 actuals, and more ambitious increases were set for disabled veterans, who face more employment barriers. The program monitors progress annually. Initially, VETS estimated the performance targets for employment and retention or based them on the preliminary performance data for the prior year. As a result, some of the initial performance goals were not ambitious. Following this start-up period, the new targets have been based on actual performance and set at levels to drive annual improvements in performance outcomes. As part of the Job Training Common Measures initiative, DOL is implementing the Common Measures for employment and training programs in calendar year 2005. Since the Common Measures substantially revise the current definitions, baseline values will be established under the new definitions during PY 2005, which may require the revision of the long-term and annual targets for PY 2006 through PY 2008.

Evidence: Supplementary Evidence Table showing VETS' Long-term Targets; excerpt from the FY 2005 DOL Performance Budget. www.dol.gov/_sec/Budget2005/_Toc63239903

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: As described in 2.1, this program is part of the Job Training Common Measures initiative. The same measures are used to assess annual and long-term performance. The program has adopted new annual measures on participants' employment and earnings outcomes which will facilitate comparisons across similar programs. Currently, VETS outcome measures are: Entered Employment Rate, and Employment Retention Rate. These two measures are calculated for all veterans and separately for disabled veterans. In addition to the Common Measures, for PY 2004 VETS established measures specifically applicable to the outcomes for veterans who were served by DVOP/LVER staff. VETS also established procedures for negotiating first-year performance targets with each State. Targets for each DVOP/LVER measure will be negotiated annually by VETS with each State. These "grant-based measures" are an important program management tool because they focus exclusively on outcomes associated with services funded by the DVOP/LVER grants.

Evidence: VPL 04-04: Negotiating State Workforce Agency and Grant-based Performance Measures: www.nvti.cudenver.edu/forVets/VPL2004.htm

Does the program have baselines and ambitious targets for its annual measures?

Explanation: VETS's initial annual measures include its four long-term measures described in the response to Question 2.2. Baselines for these core four performance measures (Entered Employment and Retention in Employment for all veterans and for disabled veterans) have been established, and the targets for those measures reflect ambitious increases "across the board" intended to stimulate continuous improvement and increase effectiveness. The Job Training Common Measures use the same annual and long-term performance measures. DOL will establish baseline data in PY 2005 based on revised definitions and then set ambitious numerical targets for PY 2006 and beyond. VETS also plans to collect data on Earnings Change during PY 2005 so that targets for this new Common Measure can be established for all veterans and for disabled veterans for PY 2006. VETS' performance measurement strategies are ambitious for two reasons. First, within the framework of the existing performance measures, VETS' annual targets from FY 2005 forward are designed to foster continuous improvement. Second, VETS will add a new measure to focus States' attention on earnings outcomes.

Evidence: VPL 04-04.Supplementary Evidence Table showing VETS annual targets 1. Supplementary Evidence Table will be posted to the VETS web page with other PART Evidence. (The table included as Supplementary Evidence for the response to Question 2.2 also will be posted.) 2. VPL 04-04: Negotiating State Workforce Agency and Grant-based Performance Measures: www.nvti.cudenver.edu/forVets/VPL2004.htm

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: State Workforce Agencies initially commit to the program's goals by developing comprehensive State Plans each fiscal year. Subsequently, award of funding is contingent upon negotiation with, and approval by, VETS staff regarding the extent to which each State's planned allocation of staff is expected to contribute to meeting VETS' annual performance targets. Finally, each grantee is required to negotiate annually two sets of performance targets with the VETS representatives responsible for oversight. One set relates to the performance of each State's labor exchange on behalf of veterans, while the other set relates to the performance of each State's cadre of DVOP specialists and LVER staff. State grantees submit comprehensive reports on a quarterly basis that detail the services and outcomes attained by all job seekers served by the public labor exchange. The ETA 9002 reports track performance toward the labor exchange targets for veterans, while the VETS 200 reports track performance toward the targets for the VETS-funded staff . VETS representatives review these reports quarterly and negotiate service delivery and funding refinements, as required.

Evidence: VPL 04-04; Special Grant Provisions for Jobs for Veterans Grants FY 2005 - 2009 (Section V, paragraph G). 1. VPL 04-04: Negotiating State Workforce Agency and Grant-based Performance Measures: www.nvti.cudenver.edu/forVets/VPL2004.htm 2. Special Grant Provisions will be posted to the VETS web page with other PART Evidence.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Several independent evaluations of the DVOP/LVER program have been conducted in the past decade by various public entities, including studies by the Government Accountability Office (GAO) and the 1999 report of the Congressional Commission on Servicemembers and Veterans Transition Assistance. Evaluations have been of sufficient scope and quality to assist DOL and Congress in identifying program strengths and weaknesses and to evaluate its effectiveness in achieving employment goals. These evaluations, conducted prior to passage of the Jobs for Veterans Act in 2002, helped VETS to focus remedial efforts on program weaknesses, such as improving performance accountability. They also influenced the new legislation. In addition, DOL engaged the Battelle Memorial Institute to conduct a study of the relationship among the demographic characteristics of veterans served by the public labor exchange, the services received, and earnings outcomes. Because that study was completed just as the Jobs for Veterans Act was enacted, DOL intends to launch another independent evaluation in 2007 to analyze the results of programmatic changes implemented under the new legislation.

Evidence: 1. Battelle Memorial Institute Final Report, Assessment of Unemployed Veterans' Needs for the Department of Labor's Veterans' Employment and Training Service, November 30, 2003. 2.. Proposed Performance Measurement System Improved, But Further Changes Needed; GAO-01-580, May 2001. www.gao.gov/new/items/d01580.pdf 3. Flexibility and Accountability Needed to Improve Service to Veterans; GAO-01-928, September 2001. www.gao.gov/new/items/d01928.pdf 4. Report of the Congressional Commission on Servicemembers and Veterans Transition Assistance; January 1999. (Not available in electronic form.)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: VETS presents the resource needs for the DVOP/LVER program completely and transparently in its annual budget requests. Budget request documents explicitly relate dollars to performance outcomes. VETS' 2006 budget request of $162 million will support 2,000 DVOP and LVER positions. These resources will result in approximately 400,000 veterans who have been served by DVOP specialists or LVER staff entering employment. VETS uses empirically-based formulas that reflect past experience to estimate the relationships among veteran outcomes, DVOP/LVER staff, and budgetary resources.

Evidence: Supplementary Evidence Document - The VETS FY 2006 Performance Budget. VETS FY 2006 Performance Budget will be posted to the VETS web page with other PART Evidence.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Jobs for Veterans Act of 2002 modernized several statutory provisions governing the DVOP/LVER program, which enabled VETS to address long-standing deficiencies in those areas. The program's greatest strategic planning deficiency was that it was not focused on participant outcomes and how to measure them; instead, it had been focused on system outputs and service delivery process rules. The legislative changes enabled VETS to better integrate the program's grant planning process with State Workforce Agency service delivery planning process, and to fully participate in the Common Measures initiative. Beginning with PY 2002, VETS implemented three outcome measures, including entry to employment, retention in employment, and entry to employment following staff-assisted services. These measures applied to all veterans and to disabled veterans served by the public labor exchange. During PY 2003 VETS issued guidance regarding the application of a parallel set of grant-based measures applicable specifically to the services provided under the DVOP/LVER program. For PY 2004, VETS issued guidance that clarified both sets of measures, improved the coordination of their application, and required implementation of the grant-based measures for the first time.

Evidence: 1. VPL 10-02: State Workforce Agency (SWA) Measures of Service for Program Year (PY) 2002: www.nvti.cudenver.edu/forVets/VPL2002.htm 2. VPL 10-03: Grant Based Performance Measures and Weighting System: www.nvti.cudenver.edu/forVets/VPL2003.htm 3. VPL 04-04: Negotiating State Workforce Agency and Grant-based Performance Measures: www.nvti.cudenver.edu/forVets/VPL2004.htm

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Performance data collected by VETS is both timely and credible, and it is used to manage and improve performance. VETS Federal staff in each State - most of whom are co-located with the respective State program operator's administrative staff - are the designated Grant Officer's Technical Representatives (GOTRs). They routinely receive from the grantee the required performance reports within 45 days following the end of each quarter. The performance data (e.g., the ETA 9002 and VETS 200 reports) collected by VETS covers the full range of services provided to program registrants. This enables Federal monitoring of each grantee's progress towards attaining negotiated outcome targets, frequent assessment of service delivery strategies, and timely program management actions. To ensure data quality, the programming for the federally required reports contains internal controls to prevent certain input errors, and VETS' monitoring helps the grantees to identify other possible data errors. For example, VETS' monitoring in Indiana led to the finding and fixing of a faulty software link between that State's Unemployment Insurance Wage Records system and the program that produces the VETS 200 report, a flaw that affected key program outcome data.

Evidence: 1. SGA Planning Instructions: VPL 03-04: Special Grant Provisions for JVA Grants (Section V) - Enclosure 8: http://www.nvti.cudenver.edu/forVets/VPL2004.htm 2. VPL 1-04: Reporting Requirements for the DVOP/LVER Grants for FY 2004: http://www.nvti.cudenver.edu/forVets/VPL2004.htm 3. VPL 09-03: DVOP/LVER Staffing and Reporting Requirements: http://www.nvti.cudenver.edu/forVets/VPL2003.htm

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: VETS' managers at the national, regional, and State levels have performance standards directly related to their contribution to meeting performance goals and targets. Similarly, the State grantees are required by the Special Grant Provisions to impose performance standards on DVOP/LVER staff related specifically to these same goals and targets. VETS' State and regional representatives conduct quarterly reviews of the performance of each State's labor exchange network, as well as each State's DVOP/LVER cadre, in response to the targets negotiated. In addition, VETS staff conduct a comprehensive assessment of grantee performance after each program year. VETS initially addresses deficient performance by providing technical assistance and, if that is not successful, by imposing a Corrective Action Plan. Regarding costs and schedules, VETS exercises its authority to reallocate grant funds whenever the results of grantee action or inaction are found to be unacceptable.

Evidence: 1. VETS Elements and Standards will be posted to the VETS web page with other PART Evidence. 2. SGA Planning Instructions: VPL 03-04: Special Grant Provisions for JVA Grants - Enclosure 8: www.nvti.cudenver.edu/forVets/VPL2004.htm 3. VPL 7-05: Staffing, Reporting Requirements and Roles and Responsibilities of the DVOP and LVER under the Jobs for Veterans State Grants: www.nvti.cudenver.edu/forVets/VPL2005.htm 4. DM 09-04 will be posted to the VETS web page with other PART Evidence.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Grant funds are obligated in a timely manner. Less than 48 hours generally passes from the time the allocation data are developed (based on the projected quarterly budget plans submitted annually by each State) until the States can draw their allocation from the HHS Payment Management System (PMS). In that time, VETS and the Department's Administrative Finance Office sign a funding agreement; Administrative Finance keys the allotments to HHS; VETS transmits the funding targets to VETS' Regional Administrators, who prepare the Notices of Obligational Authority (NOAs) for each State in their respective region using the DVOP/LVER Information Management System (DLIMS), and fax advance copies to VETS' Directors and SWA Administrators in each State. In accordance with the Special Grant Provisions, each State is restricted to the amounts and purposes indicated by program on the quarterly NOAs. Within 45 days of the end of each quarter, States account for the funds they drew down and expended using SF 272 and SF 269A reports submitted with their Technical Performance and budget narratives. Financial reports are reviewed, analyzed and posted to the Internet-based DLIMS. VETS monitors fund use to ensure States' spending is consistent with their approved plans and identifies the need for any adjustments or realignment of funds through grant modification. As a result of rigorous pre-planning and quarterly funding adjustments, just $342,000 (0.21%) of the 2004 appropriation of $161,408,000 remained unobligated at the end of the fiscal year. Nearly three-fourths of that amount was unobligated because: (a) a few States were prohibited from expending newly-authorized incentive award funds, in some cases by existing provisions of collective bargaining agreements that could not be re-negotiated in time to implement the incentive awards program, and in other cases by prohibitions in State statutes; and (b) VETS was restricted by the applicable statutory provisions of the Jobs for Veterans Act from reallocating those funds for other purposes. The DVOP/LVER grants are administered in accordance with OMB Circulars A-102 and A-87. VETS complies fully with key financial management statutes. Auditors have reported no findings of material weaknesses under the Federal Managers' Financial Integrity Act (FMFIA). Management controls by Administrative Finance ensure that DOL does not obligate more funds than available, and that no violations of the Anti-Deficiency Act occur in this program.

Evidence: 1. 38 USC 4102 A - State Plans: www.nvti.cudenver.edu/Implementation/38usc41.pdf 2. SGA Planning Instructions: VPL 03-04: Issuance of FY 2005-2009 Solicitation for Grant Applications, Jobs for Veterans State Grants for DVOP Specialists and LVER Staff: www.nvti.cudenver.edu/forVets/VPL2004.htm 3. DVOP/LVER Information Management System (DLIMS) entry/edit and report screens will be posted to the VETS web page with other PART Evidence. 4. Special Grant Provisions/Final agency allocation spreadsheets FY 2003/2004 will be posted to the VETS web page with other PART Evidence. 5. VETS annual budget requests will be posted to the VETS web page with other PART Evidence.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In 2004, VETS developed cost data to support an efficiency measure for all veterans served by the labor exchange, measured as cost per veteran participant. The baseline has been established and targets set for 2005 and 2006. Cost data also was developed for an efficiency measure for veteran labor exchange participants who enter employment. The cost per veteran participant measure can be implemented immediately because implementation of the Common Measures will not affect the data currently available on participants. The key challenge in developing program efficiency measures arises from the fact that a single target group (veterans) is served by dual funding streams (Wagner-Peyser and DVOP/LVER), requiring appropriate allocation of costs. The efficiency measures include cost per veteran participant and cost per veteran entered employment. The first of these efficiency measures is consistent with the definition established by ETA for the Common Measures, while the second focuses on entry to employment as the key outcome. The Entered Employment results reported for 2003 represent the first full year of data under the new VETS reporting system that applies UI wage records to determine outcomes, so 2003 was selected as the baseline year for both efficiency measures. State results have been calculated based on the 2003 baselines. VETS will publish State results on its web page to encourage States to improve program efficiency. In 2003, about 1.2 million veteran participants received services from the public labor exchange at a cost of $156 per veteran participant. During that same period, about 700,000 veterans entered employment at a cost of $268 per veteran entering employment. Based on the Cost per Veteran Participant baseline, targets have been established through 2007. Because the definition of Entered Employment will change under the Common Measures, targets for Cost per Veteran Entered Employment will be established for 2006 and 2007 on the basis of a revised baseline to be established in 2005.

Evidence: Supplementary evidence tables showing baselines, targets, and State level results; Supplementary Evidence Document: Interim Efficiency Measures Report; SRA International, Inc., June 17, 2005. 1. Supplementary Evidence Tables will be posted to the VETS web page with other PART Evidence. 2. Supplementary Evidence Document (Interim Efficiency Measures Report; SRA International, Inc., June 17, 2005) will be posted to the VETS web page with other PART Evidence.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The DVOP/LVER program is closely coordinated with four related programs: 1) Employment Service (ES). Coordination occurs at all levels with the ES State grants program, the closest partner program. At the national level, VETS' Director of Operations meets monthly with the Director of ETA's Office of Workforce Investment to ensure general coordination of these programs. Since the reporting of performance results for the DVOP/LVER program is one aspect of the One-Stop infrastructure VETS and ETA share, VETS maintains ongoing liaison with ETA's counterpart staff. At the State level, since VETS and ETA rely on the same network of State Workforce Agencies, each State grantee's designation of a Veterans' Programs Coordinator promotes effective coordination. At local One-Stop Career Centers, DVOP specialists and LVER staff, ES staff, and staff supported by other programs all collaborate to ensure that veteran job-seekers receive the services that best meet their needs. 2) Transition Assistance Program (TAP). TAP reflects a working partnership among the DVOP/LVER program, the transition units of the respective military services, and outreach units of the Department of Veterans Affairs (VA). 3) VA's Vocational Rehabilitation and Employment (VR&E) Program. VR&E relies upon the DVOP/LVER program to provide placement services. Coordination and collaboration is ensured through a Memorandum of Understanding among these agencies. 4) National Veterans Employment and Training Institute (NVTI). NVTI delivers training and distance learning to DVOP/LVER staff. The NVTI Director regularly attends meetings of VETS' field managers to ensure close alignment of staff development objectives.

Evidence: Veterans Employment and Training Service State Plan Narrative Instructions; VA-VETS Memorandum of Understanding; TAP Inter-agency Agreements. 1. SGA Planning Instructions: VPL 03-04: State Plan Narrative Instructions - Enclosure 4: www.nvti.cudenver.edu/forVets/VPL2004.htm 2. VA-VETS Memorandum of Understanding will be posted to the VETS web page with other PART Evidence. 3. TAP Inter-agency Agreements will be posted to the VETS web page with other PART Evidence.

Does the program use strong financial management practices?

Explanation: VETS management practices comply fully with key financial management statutes. There never has been a finding of a material weakness in the financial management of the program under the Federal Managers' Financial Integrity Act (FMFIA). Any audit exceptions identified through independent auditors under the Single Audit Act (29 CFR, Part 96) are reported through VETS and reconciled by the Grant Officer prior to or during grant close-out. DOL's Office of the Assistant Secretary for Administration and Management provides the independent Grant Officer function, ensuring strong controls over grant awards and modifications, and its Administrative Finance Office controls the obligation of funds. VETS uses the HHS Payment Management System (HHS/PMS) for tracking funds provided to States through a secure system that ensures only authorized grantee staff have access to program funds. Quarterly allotments, based upon approved State plans, are keyed into the HHS accounts and drawn down by States, which submit SF 272 reports. Quarterly Notices of Obligation Authority are made through VETS' Regional Administrators to States, based upon Finance Office certified-approved amounts. VETS maintains information on amounts obligated to States and their expenditures in an Internet-based Information Management System and tracks direct and indirect charges to the grants based upon States' SF 269A quarterly reports. VETS field staff review and validate States' financial reports and fund ledgers against the State's approved plan as part of the State assessment process.

Evidence: FY 2005-2009 Solicitation for Grant Applications (SGA) Special and General Grant Provisions; Notices of Grant Awards; Chapter 4, DVOP/LVER Operations Manual; DLIMS; VETS/HHS agreement. 1. SGA Planning Instructions: VPL 03-04: Special Provisions - Enclosure 8; General Provisions - Enclosure 9: www.nvti.cudenver.edu/forVets/VPL2004.htm 2. The base grant amounts for each States for FY 2006, as calculated by the funding formula, will be posted to the VETS web page with other PART Evidence. 3. Chapter 4, DVOP/LVER Operations Manual will be posted to the VETS web page with other PART Evidence. 4. "Snapshot" report screens from DLIMS will be posted to the VETS web page with other PART Evidence. 5. DOL/HHS Interagency Agreement on DOL participation in the Payment Management System (and its correct title) will be posted to the VETS web page with other PART Evidence.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: VETS' Regional Administrators identify program weaknesses as part of their annual "State of the Region" assessments of program and administrative operations and certifications of the region's integrity of financial management to the Assistant Secretary for Veterans Employment and Training. In 2004, VETS modified the process to include new program assessment instruments derived from the PART instrument, and an internal control evaluation tool derived from GAO's model. An example of one such deficiency noted prior to 2004 was lag time in posting of grantees' financial status reports that hampered the VETS' ability to make timely and appropriate decisions regarding the next quarter's grant fund allocations to each State. To solve the problem, VETS began development of a real-time, web-based information management system early in 2004. This system, called DLIMS (DVOP/LVER Information Management System) makes available to VETS program management staff timely, quarterly State, Regional and National reports that are updated daily as newly-received State reports are uploaded into the system. Another program management deficiency identified since the passage of the JVA of 2002 was the need to retrain field staff that perform the front-line oversight of the program operators. VETS arranged for a formal training needs assessment of its program staff and recently received the report of staff training needs in the area of grants management and analytical techniques. On-site and distance learning modules will be used to eliminate this training gap.

Evidence: Sample State of the Region Report - State of Region II and III - Executive Summary; FY 2004 and 2005 DLEC Annual work plan; "snapshot" report screens from DLIMS; training needs assessment report. 1. Sample State of the Region Report - State of Region II and II - Executive Summary will be posted to the VETS web page with other PART Evidence. 2. FY 2004 and 2005 DLEC Annual work plan will be posted to the VETS web page with other PART Evidence. 3. "Snapshot" report screens from DLIMS will be posted to the VETS web page with other PART Evidence. 4. Training Needs Assessment Report (GMAT Survey Summary Report) will be posted to the VETS web page with other PART Evidence.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Program oversight is accomplished through a combination of desk monitoring (e.g., quarterly reviews of program activity and fiscal reports prepared by the grantees), and on-site monitoring. On-site monitoring is conducted to identify best practices and technical assistance needs, and to ensure effective implementation of the approved State Plan. Following the passage of the JVA, VETS developed an extensive Technical Assistance Guide for monitoring and oversight. VETS has at least one professional staff member stationed in each State, typically occupying office space in the grantees' facilities. As a result, VETS field staff are knowledgeable about program operations and context.

Evidence: Special Grant Provisions for Jobs for Veterans Grants (Section V, Reporting Requirements); DM 13 -04, Technical Assistance Guide (TAG) on the Assessment of the State Workforce Agency 1. SGA Planning Instructions: VPL 03-04: Special Grant Provisions for JVA Grants (Section V, Reporting Instructions) - Enclosure 8: www.nvti.cudenver.edu/forVets/VPL2004.htm 2. DM 13-04 will be posted to the VETS web page with other PART Evidence. 3. TAG on the Assessment of the State Workforce Agency will be posted to the VETS web page with other PART Evidence.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: VETS produces an annual "Report on Services to Veterans" that includes comprehensive information on program operations, service delivery levels, and outcomes achieved. This report is published in print and electronic formats. VETS also publishes on its website quarterly performance outcomes for each State's labor exchange on behalf of veterans. The core four outcome measures published for each State include the Entered Employment Rate and the Employment Retention Rate for all veterans and the same two outcome measures for disabled veterans. The Performance Results for the most recent quarter are posted on VETS' website, under "Programs/Services."

Evidence: Draft FY 2004 Report to Congress on Services to Veterans (Report currently in DOL clearance; expected to be posted on VETS web site late summer 2005). www.dol.gov/vets/programs/State_Data_Intro.pdf is the location of the Performance Results data

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has adopted new long-term goals as part of the Job Training Common Measures initiative. Similar employment outcomes were measured in prior years, and for 2000, 2001, and 2002, the performance of the program substantially exceeded the targets established. Beginning in 2002, VETS changed the methodology for measuring employment outcomes to rely on unemployment insurance wage records instead of staff follow-up of veterans served. Because of the lag time inherent in the use of wage records, no Entered Employment outcomes were reported for 2002 and Entered Employment results for four full quarters first were reported at the end of 2003. For 2003, the 58% Entered Employment Rate result for all veterans met the established target and the 79% Employment Retention Rate result for all veterans exceeded the 72% target. The rating of "Small Extent" is based on the limited availability of performance data that are comparable across years.

Evidence: Supplementary Evidence Table showing VETS Progress Achieving Long-Term Goals; Jobs for Veterans State Grants Solicitation for Grant Applications (SGA); VPL 03-04, dated April 16, 2004; VPL 04-04, dated May 5, 2005; FY 2004 Performance and Accountability Report. 1. Supplementary Evidence Table will be posted to the VETS web page with other PART Evidence. 2. SGA Planning Instructions: VPL 03-04: Issuance of FY 2005-2009 Solicitation for Grant Applications, Jobs for Veterans State Grants for DVOP Specialists and LVER Staff: http://www.nvti.cudenver.edu/forVets/VPL2004.htm 3. VPL 04-04: Negotiating State Workforce Agency and Grant-based Performance Measures: http://www.nvti.cudenver.edu/forVets/VPL2004.htm 4. FY 2004 Performance and Accountability Report: http://www.dol.gov/_sec/media/reports/annual2004/main.htm

Does the program (including program partners) achieve its annual performance goals?

Explanation: 2003 and 2004 are the first two performance periods for which outcomes have been reported according to the UI wage record based measurement approach. Those two performance periods also are the last two for which outcomes will be reported prior to implementation of the Common Measures. As noted in 4.1, for 2003, the 58% Entered Employment Rate result for all veterans met the established target, and the 79% Employment Retention Rate result for all veterans exceeded the 72% target. There were no targets for disabled veterans for 2003. For 2004, targets were established for all of the core four performance measures. As of the end of the third quarter, performance results met or exceeded the targets for all four measures. These results for 2003 and the first three quarters of 2004 indicate that program performance corresponds closely to established targets. The rating of "Small Extent" is based on the limited availability of performance data that are comparable across years. In addition to outcome data on the public labor exchange on behalf of veterans, VETS now has a full year of data on the performance outcomes specifically for the DVOP/LVER program. VETS will use the results of this first year's application of grant-based performance measures to review and refine the measures applied. Once new baseline values have been established according to the Common Measures definitions during 2005, VETS will combine the refined measurement approaches with the new baselines to project targets for the DVOP/LVER program for 2006 and 2007.

Evidence: Supplementary Evidence Table showing performance targets and results for PY 2003 and PY 2004 (to date); Draft FY 2004 Report to Congress on Services to Veterans (Report currently in DOL clearance; expected to be posted on VETS web site late summer 2005); SWA Quarterly Reports. 1. Supplementary Evidence Table will be posted to the VETS web page with other PART Evidence. 2. SWA Quarterly Reports consist of data submissions to the ETA 9002/VETS 200 Reporting System. VETS maintains these results in DLIMS and makes them available upon request. There are no narrative quarterly reports submitted by State Workforce Agencies.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: VETS initially developed an efficiency measure for the overall labor exchange services provided in the One-Stop Career Centers. This measure of cost per participant was based on the efficiency measures used for the other job training and employment programs subject to the Common Measures. As discussed in the response to Question 3.4, a key challenge for establishing an efficiency measure at the overall labor exchange level is allocating two funding streams - the ES grants and the DVOP/LVER grants - to one target group: the veterans who seek assistance at the local One-stop Career Center. With the assistance of its contractor, VETS established a 2003 baseline of $156 per veteran participant. VETS also went beyond the Common Measure efficiency measure of cost per participant to establish an outcome efficiency measure of cost per entered employment. The PY 2003 baseline for this efficiency measure is $268 per veteran entering employment. In addition to the labor exchange efficiency measure, VETS has extended its efficiency measure focus to the DVOP/LVER State grants. To establish an efficiency measure for the DVOP/LVER program, a single funding source needs to be allocated to two distinct populations served by the DVOP/LVER staff in different locations: those veterans served at the One-Stop Career Centers and the transitioning service members served in Transition Assistance Program (TAP) employment workshops offered at military installations. Omitting the TAP participants results in a misleading efficiency measure with higher costs per person served -- $185 versus $160 in 2003. For this reason, VETS and ETA have agreed to add TAP participants to the ETA 9002/VETS 200 reporting system in conjunction with the revisions currently being undertaken to accommodate Common Measures.

Evidence: Supplementary Evidence Table of baselines and time frame for establishment of targets; Interim Efficiency Measures Report; SRA International, Inc., June 17, 2005. 1. Supplementary Evidence Table will be posted to the VETS web page with other PART Evidence. 2. Supplementary Evidence Document (Interim Efficiency Measures Report; SRA International, Inc., June 17, 2005) will be posted to the VETS web page with other PART Evidence.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The performance of the DVOP/LVER State Grants Program compares favorably with another Federal grant program with a similar purpose, the ES State grants program. Both programs help comprise the public labor exchange. The ES State grants serves all job seekers, whereas the DVOP/LVER State Grants Program serves veteran jobseekers only. An ideal comparison of the performance of these two programs would compare the outcomes experienced by non-veteran users of public labor exchange services with the outcomes experienced by those veteran users of the public labor exchange served only by the DVOP/LVER State grants program. However, the detailed data are not available to make this comparison. Based on program data for 2003, adjusted for demographic differences between the population of all job seekers and job seekers who are veterans, results in a weighted entry-into- employment rate for veterans of 61.6%, compared with the corresponding rate of 61.2% for non-veterans.

Evidence: Supplementary Evidence Tables showing method applied by VETS to calculate age-adjusted Entered Employment Rate. 1. Supplementary Evidence Tables will be posted to the VETS web page with other PART Evidence. 2. Source of the Evidence Tables: ETA 9002 C and D Reports for the periods July 1, 2003 through June 30, 2004. Routine procedures for publishing the extensive data files derived from the quarterly submissions to the ETA 9002/VETS 200 Reporting System do not exist. VETS maintains these results in DLIMS and makes them available upon request.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Department of Labor recently sponsored an evaluation conducted by Battelle Memorial Institute to assess the extent to which the DVOP/LVER State Grants Program, operating within the public labor exchange: (a) reaches the target population of unemployed veterans; (b) identifies the needs of unemployed veterans who could benefit from employment services; and (c) delivers employment services intended to improve the subsequent labor market experiences of unemployed veterans. The study included macro and micro analyses. The macro analysis applied data from the Current Population Survey, along with national level labor exchange data. The comparison of data from these two sources indicated that: (a) the vast majority of unemployed veterans access the labor exchange; and (b) the demographic characteristics of unemployed veterans who access the labor exchange closely parallel the characteristics of unemployed veterans nationally. The micro analysis examined data records for about 320,000 veterans served through the public labor exchange in eight States between July 1998 and June 2001. A multivariate analysis was conducted to explore the relationships among various "menus" of services and the veterans' subsequent levels of earnings, while adjusting the results to account for differences in the measured characteristics of the individual veterans. Key findings were: (a) 94 percent of the veterans who accessed the public labor exchange received at least one service; (b) no single service or combination of services is associated with higher subsequent earnings for all veterans; (c) referral to employment, in combination with other staff-assisted services such as career guidance, is associated with higher subsequent earnings, except for recently separated veterans; (d) referral to employment with or without other services is associated with higher subsequent earnings for young recently separated veterans; and (e) case management is associated with higher subsequent earnings for older recently separated veterans. Limitations inherent in the design of the micro analysis mean that this was not a definitive study of the impact of labor exchange services on employment outcomes. First, the study had no comparison group of veterans who received reemployment services and those who did not. Those veterans who registered but received no services effectively comprised a comparison group for the micro analysis, but the extent of selection bias cannot be determined. The second limitation is that the findings cannot be generalized to the nation because the sample of eight States was not randomly selected. However, based on a comparison of the characteristics of the veterans included in the micro analysis with the two national level characteristics profiles, the study concluded that its findings provide a reasonable representation of the experiences of all veteran registrants with the public labor exchange. In addition, the findings from this study were consistent with findings from other evaluations of labor exchange services.

Evidence: Final Report, Assessment of Unemployed Veterans' Needs for the Department of Labor's Veterans' Employment and Training Service, November 30, 2003; and Executive Briefing, December 3, 2003. For a review of other evaluations of labor exchange services, see Chapter 5, "Evaluating the Effectiveness of Labor Exchange Services," by Christopher J. O'Leary, in Labor Exchange Policy in the United States. David E. Balducci, Randall W. Eberts, and Christopher J. O'Leary, editors. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 2004