Detailed Information on the
Vocational Rehabilitation and Employment Program Assessment

Collect data on the number of rehabilitated veterans placed in suitable employment and independent living status and using these results to improve program performance.

Collect data on both established and newly developed measures to evaluate performance and using these results to improve program performance.

Work with the Department of Labor and Department of Defense to assess results of collaboration and using these results to enhance future efforts to coordinate services for veterans with disabilities

Cooperate with General Accounting Office (GAO) on an evaluation of the program to assess the effectiveness of recent program changes, including the implementation of the Five Tracks to Employment model.

Measure: Rehabilitation Rate

Explanation:Percent of disabled veterans exiting the program with a successful outcome ("rehabilitated" versus "discontinued").(Strategic Goal = 80%).

Measure: Serious Employment Handicap Rehabilitation Rate

Explanation:Percent of disabled veterans with a Serious Employment Handicap exiting the program with a successful outcome ("rehabilitated" versus "discontinued"). (Strategic goal = 81%).

Measure: Percent of participants employed first quarter after program exit.

Explanation:Percent of program participants entering "Job Ready Status" who were suitably employed within 90 days of assignment to that status. (Job Ready Status is reached when no further program services are needed to become employable).(Strategic Goal = 80%).

Measure: Percent of participants still employed three quarters after program exit.

Explanation:Percent of those program participants who were suitably employed within 90 days of entering Job Ready Status who are still suitably employed 270 days after entering Job Ready Status. (Strategic Goal = 85%).

Measure: Speed of Entitlement Decisions.

Explanation:Based on historical results, VR&E's targets are both realistic and aggressive. Although the targets have been lowered, based on 2007 actuals VR&E will need to show a two day and three day improvement to meet its 2008 and 2009 targets respectively. The targets also take into consideration VR&E's current and projected FTE levels.

Measure: Accuracy of Entitlement Decisions.

Explanation:VR&E is in the process of reviewing its Quality Assurance process and has increased the number of cases reviewed. The increases in the number of cases reviewed combined with the review of the Quality Assurance process may reveal errors that previously went undetected and thereby provide a better, more rigorous basis from which to determine target levels.

Measure: Accuracy of Evaluation/Planning and Rehabilitation Services.

Explanation:Percent of those cases undergoing quality review in which services were provided to program participants in an accurate and timely manner. (Strategic Goal = 96%).

Measure: Accuracy of Vocational Rehabilitation Program Completion Decisions.

Explanation:For the last few years, targets for this measure have been overly ambitious. As a result, the program has not reached its target since 2005. Based on this, VR&E adjusted its targets to be more realistic. VR&E is in the process of reviewing its Quality Assurance process and has increased the number of cases reviewed. The increases in the number of cases reviewed combined with the review of the Quality Assurance process may reveal errors previously undetected, thereby providing a better, more rigorous basis to determine target levels.

Measure: Average Cost of Rehabilitating Participants

Explanation:The calculation will take the total number of professional staff (counselors) salary, plus total General Operating Expenses, plus counseling services contracting dollars divided by the total number of veterans participating in a vocational rehabilitation program. The new calculation will use constant dollars instead of appropriated dollars which will provide a better indicator of program efficiency. Results data will be collected quarterly. The data used to support the calculation of the performance measure will be manually audited by Vocational Rehabilitation and Employment Service (28) and Office of Resource Management (24) analysts.

Measure: Customer Satisfaction (Survey)

Explanation:Percent of respondents to Customer Satisfaction Survey who indicated satisfaction with VR&E services. (Strategic Goal = 92%).

Measure: Percent change in earnings from pre-application to post-program.

Explanation:Percent change in average earnings from pre-application to post-program of participants who obtain suitable employment following program completion. (Strategic Goal = TBD).

Is the program purpose clear?

Explanation: The mission of the Vocational Rehabilitation & Employment Program (VR&E), as defined in law, is to provide the services and assistance necessary to enable veterans with service-connected disabilities to achieve maximum independence in daily living and, to the maximum extent feasible, become employable and obtain and maintain suitable employment.

Evidence: The VR&E purpose is defined in 38 U.S.C. 3100, 3104, 3109 and 3120; and also articulated in the VR&E Task Force Report to the Secretary dated March 2004. 2006-2011 VA Strategic Plan.

Does the program address a specific and existing problem, interest, or need?

Explanation: The VR&E program addresses the unique and specific needs of veterans with service-connected disabilities in order to help them transition to civilian life, overcome the effects of disabilities, become employable, obtain and maintain suitable employment, and maximize their independence in daily living. Existing need for VR&E services is reflected in the increasing number of applications received annually, as well as the number of rehabilitation plans developed by VR&E, and the number of participants enrolled in the program; these numbers have increased by 22%, 16%, and 12%, respectively. Additionally, Operation Iraqi Freedom and Operation Enduring Freedom (OIF/OEF) underscore the ongoing need for VR&E services. Data from the Department of Defense indicates that on an annual basis about 25,000 active duty service members are determined "not fit for duty" due to medical conditions that may qualify them for a VA disability rating and VR&E services. The Army Reserve and Army National Guard have also indicated that about 29,000 of their members have been placed in a medical hold or medical extension status since 2003. These servicemembers, as well as Reserve and National Guard components of other military branches, may also qualify for VR&E services.

Evidence: 38 U.S.C. 3100; VR&E Task Force Report to the Secretary dated March 2004; VBA Letter s20-03-36; and 20-05-14; DOD PEB database; MODS database; VA 2006 Congressional Justification; FY 2003 Annual Benefits Report; COIN TAR 1991 - 2005

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Multiple alternative programs exist to address vocational rehabilitation and employment issues for veterans. The Department of Education's Rehabilitation Services Administration (RSA) provides grants to state vocational rehabilitation agencies across the country. In Fiscal Year 2004, RSA provided services to 29,135 veterans. Additionally, the Department of Labor's Veterans' Employment and Training Service (VETS) provides services that target the same beneficiaries as VR&E, and the Small Business Administration and Office of Personnel Management administer programs designed to promote employment opportunities for veterans. Taken as a whole, there exists significant overlap between VR&E and other federal and state programs.

Evidence: 38 U.S.C. 3100; VA's 2006 Budget, p. 5A-5; VR&E Task Force Report to the Secretary dated March 2004, p. 28; 2003-2008 Strategic Plan, p. 26-27; FY 2004 Annual Performance and Accountability Report, p. 65-66; RSA Longitudinal Study of the Vocational Rehabilitation Services Program: Implications for Practice; RSA website (http://www.ed.gov/about/offices/list/osers/rsa/index.html?src=mr); Code of Federal Regulations 21.40 through 21.59.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: In FY 2003 an independent task force, known as the VR&E Task Force, was chartered to review VR&E services and make recommendations for improvement of the program. The only major design flaw that the task force identified was a need to change the established service delivery model which had directed program resources more toward re-training and education than to the major program purpose of rehabilitation through suitable employment.. To address this issue, the Task Force recommended a new "Five Tracks to Employment" process (the 5-Track process) which would increase the program's focus on employment. VR&E piloted the 5-Track process at four regional offices in FY 2005 and reported very positive impact on important performance measures, including: a reduction in the client's Days in Evaluation and Planning from 98 to 89 days; a reduction in Days to Entitlement Notification from 51 to 47 days; and an increase in the Rehabilitation Planning Rate from 71% to 76%. The Task Force report has prompted VR&E to make the changes necessary to improve program design and operation, including the implementation of the 5-Track process in FY 2006. Additionally, there is no evidence that a different approach or mechanism would be more effective or efficient than VR&E's current processes.

Evidence: VR&E Task Force Report to the Secretary dated March 2004; Task Force Recommendations Matrix; IL Circular; Guidelines on Evaluation and Planning, Self-Employment, Re-Employment, Rapid Access to Employment; Job Lab Operators Guide; Veterans and VR&E Working Together Standardized Orientation Video; VR&E Five-Tracks to Employment Pilot Project Results; VR&E Service: 5 Tracks to Employment Model; Concept Paper on National Acquisition Strategy; MOU with DOL/VETS; Congressional Testimony of Director, VR&E 3/9/06

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The legal design of the program focuses resource allocation according to basic eligibility and entitlement standards and provides assurance that intended beneficiaries are reached. Specifically, applications are automatically provided to only those veterans who receive a service-connected disability rating from the VA Disability Compensation program. VR&E conducts outreach to the most seriously disabled veterans to further focus resources on this specific population. Additionally, program procedures require that once an application is received, each veteran meets with a counselor to complete an initial evaluation and entitlement determination. This one-on-one evaluation with a trained professional further ensures that program resources are tailored to the beneficiary. In addition, VR&E employs multiple performance measures to guide and promote accountability in accurate decision-making and program management (e.g., Entitlement Determination Accuracy - rated as 97% accurate in 2005). Once the need for program services and the level of severity are established, a veteran is assigned to one of the five tracks in the 5-Track process. Each track represents a specific focus of services appropriate to the identified vocational rehabilitation needs of the veteran and delivers sufficient resources to achieve the vocational rehabilitation goals outlined in the veteran's individualized plan. Only the most seriously disabled veterans, those with a Serious Employment Handicap designation, are entitled to special rehabilitation services and an extended period of service delivery.

Evidence: VR&E manual M28, Part I, Chapter 3 (Motivation and Outreach); VR&E Monthly Operating Report (MOR); Booz Allen Hamilton study dated March 2002 (Review of Quality Instrument System Design and Quality Assurance Program Enhancements and Process); 38 U.S.C. 3102; 38 U.S.C. 3103; FY 2004 and FY 2005 Annual Performance and Accountability Report, p. 65-66; VBA Purchase Card Handbook 4080; Management Quality Assurance Service (MQAS) Report on Field Station Purchase Card Transactions.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has four long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program. The first two??Rehabilitation Rate (RR) and Serious Employment Handicap Rehabilitation Rate (SEH RR)??have been in use for several years and have sufficient baseline data. Two new measures will begin reporting in 2007??Percent of Participants Employed First Quarter After Program Exit (PPE1Q) and Percent of Participants Still Employed Three Quarters After Program Exit (PPE3Q). These measures track the percentage of participants who are suitably employed within one quarter and who are still employed three quarters, respectively, after being declared "Job Ready" by a VR&E case manager. Job Ready status is reached when the veteran has been provided all of the services necessary to become employable or needs no further services to be employable. The program's focus on suitable employment ensures that job duties match the veteran's interests, aptitudes and abilities, and does not aggravate their disability/disabilities. This standard is much higher than mere employment and reflects VR&E's greater emphasis on well-paying career solutions.

Evidence: VA's 2006 Budget, p. 5A-13; VA's 2006-2011 Strategic Plan in development, Objective 1.3; Title 38, U.S.C. 3100; `VR&E Intranet Reports; Monthly Operating Report (MOR).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: VR&E sets ambitious targets and timeframes for its long-term outcome measures of Rehabilitation Rate (RR) and Serious Employment Handicap Rehabilitation Rate (SEH RR) and has several years of clear baseline data to measure changes in performance. These measures directly relate to the program's purpose of enabling service-connected disabled veterans, especially those with serious employment handicaps, to maximize independence in daily living, to become employable, and to obtain and maintain suitable employment. During the five-year period FY 2000 to FY 2005, VR&E's Rehabilitation Rate averaged 62.5% and its SEH Rehabilitation Rate averaged 62%. VR&E proposes to improve the RR and SEH RR measures in FY 2006 to 69% and 65%, respectively, and 73% and 74% in FY 2007. Strategic targets are aggressive but achievable at 80% and 81% for RR and SEH RR, respectively. To place these improvements in perspective, state vocational rehabilitation (VR) programs administered by the Department of Education's Rehabilitation Services Administration consistently averaged a lower rehabilitation rate of 61.5% for the 13-year period 1991 to 2004. VR&E's ability to match or to exceed the performance of state VR agencies is especially notable given the more stringent criteria used by VR&E for defining rehabilitation as obtaining and maintaining "suitable employment," in contrast to state VR agencies, which set their goals as "achieving an employment outcome" or "obtaining competitive employment." In 2007, baselines will be established for VR&E's two new measures??Percent of Participants Employed First Quarter After Program Exit (PPE1Q) and Percent of Participants Still Employed Three Quarters After Program Exit (PPE3Q).

Evidence: RSA 911 Data Tapes at www.statedata.info/download/download_1.php ; RSA's Longitudinal Study of the Vocational Rehabilitation Services Program: Implications for Practice; VA's 2005 Budget Submission, p. 5-15, VA's 2006 Budget, p. 5A-13; VA's 2007 Budget Submission, p. 5E-6, Regional Office Director's National Performance Standards; VR&E Officer's National Performance Standards; Vocational Rehabilitation Counselor's / Counseling Psychologist's National Performance Standards.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has annual performance measures that track progress toward desired long-term program outcomes. Measures are designed to be sub-components of the long-term measures and to support their attainment. For example, Speed of Entitlement Decisions measures how quickly disabled veterans are evaluated and informed of their entitlement to VR&E services. The more quickly they receive entitlement decisions, the more quickly those disabled veterans who are entitled may receive rehabilitation services, and ultimately enter the work force. As a result, this measure supports the long-term Rehabilitation Rate and Serious Employment Handicap Rehabilitation Rate. Similarly, the Accuracy of Entitlement Determinations ensures that the appropriate population of veterans is identified and that resources are focused only on the eligible group of beneficiaries. The measure allows VR&E to track and assess how accurately and efficiently the program manages the entitlement determination process and improves the long-term RR and SEH RR through data feedback and adjustment. On a monthly basis a number of other annual measures are updated, including: Accuracy of Evaluation/Planning Rehab Services; Rehabilitation Planning Rate; and Accuracy of Vocational Rehabilitation Program Completion Decisions. These measures are collected and reviewed by VR&E centrally, as well as by VR&E Officers at each Regional Office, enabling program staff to track progress and make adjustments to improve the long-term Rehabilitation Rate and SEH Rehabilitation Rate.

Evidence: VA 2006 Congressional Justification; Monthly Operating Report (MOR); VR&E Intranet Reports; Case Management Index.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: VR&E has baselines and ambitious annual and strategic targets for its annual measures. For all measures, data has been collected for multiple years, baselines have been established, and the ambitiousness of annual/strategic targets can be judged. For Speed of Entitlement Decisions, FY 2004 actual was 57 days, the FY 2007 target is 53 days, and the strategic target is 40 days. For Rehab Planning Rate, FY 2003 actual was 74%, FY 2007 target is 81%, and the strategic target is 85%. Historically, about 75% of applicants found entitled to VR&E services sign rehabilitation plans with the program, while the remainder decline services and rely on other VA benefits. Increasing the target for this measure by 5% in 2007, and 10% in the long-term, is ambitious given the alternatives available to veterans. For Customer Satisfaction, FY 2002 actual was 77%, FY 2007 target is 82%, and the strategic target is 92%. Surveys have indicated that individuals dislike being denied services and that some individuals have chosen program services for either "further education" or "career counseling" rather than for the express purpose of the program??to achieve suitable employment. Therefore, given the legislative mandates governing entitlement decisions and the increased focus on employment versus education, the increases in annual targets for customer satisfaction are ambitious. The Average Cost of Placing a Participant in Employment will establish official baselines in 2006. For Accuracy of Entitlement Decisions, Accuracy of Vocational Rehabilitation Program Completion Decisions, and Accuracy of Evaluation/Planning Rehab Services, the actual measures reported have steadily increased and are presently at/over 90%. In each of these areas, VR&E is striving for a strategic target approaching 100%.

Evidence: VA's 2006 Budget, p. 5A-13; Monthly Operating Report (MOR); Regional Office Director's National Performance Standards; VR&E Officer's National Performance Standards; Vocational Rehabilitation Counselor's / Counseling Psychologist's National Performance Standards; Case Management Index.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: VR&E primary partners are the Department of Labor's VETS program and the various contractors employed at VA's regional offices. Regardless of public or private status, all partners commit to and work toward the annual and/or long-term goals of the program. VR&E and DOL VETS have established a Memorandum of Understanding which outlines data sharing between the two departments, as well as specific responsibilities supporting VR&E's annual and long-term performance goals. VR&E contractors sign performance-based contracts (developed under the program's National Acquisition Strategy) and these documents are designed to reinforce the performance goals of the program. Performance measures are tracked through data provided by contractors to VR&E and include: Rehabilitation Rate, SEH Rehabilitation Rate, Accuracy of Program Completion Decisions, Days to Entitlement Notification, Average Days in Evaluation & Planning Status, and Percentage of Cases in Interrupted Status.

Evidence: Quality Assurance Review Intranet Reports; VACOR Charter; Concept Paper on National Acquisition Strategy; MOU with DOL/VETS; VA Regional Office Directors National Performance Standards

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations have been conducted on the VR&E program by the Government Accountability Office (GAO), VA's Inspector General (IG), and public accounting/consulting firms. Evaluations have included numerous studies on program management and quality assurance, most expansively, the VA Vocational Rehabilitation and Employment Task Force Report which was issued in 2004. This comprehensive report looked at all aspects of the VR&E program, including eligibility determination, evaluation and planning services, program outcomes, staffing and organization, and account maintenance. VR&E Service has re-organized management and staffing as recommended by the Task Force to reflect a greater emphasis on program outcomes such as employment and independent living rather than on the process of providing education or training. VR&E policy circulars and guidance letters have been developed to address standards of performance and areas of needed improvement, and are available to all staff electronically at the VR&E Intranet site. Training of at least two VR&E staff per station to receive contracting officer warrants has been completed. The M28 manual which provides operational guidance under the laws and regulations governing VR&E and reflects updated policy and procedure is being re-written and published electronically. A longitudinal Program Outcome Study completed by an independent contractor recently demonstrated that veterans who had exited VR&E services and had obtained employment had increased earnings compared with earnings prior to completing VR&E. A Veterans Research and Employability Survey is also underway and projected to be completed this fiscal year. GAO and IG evaluate aspects of the VR&E program periodically, and VBA schedules an evaluation of each business line approximately every 5 or 6 years. The independent accounting firm of Deloitte and Touche also performs an annual audit of each VBA business line's accounting practices.

Evidence: VR&E Task Force Report to the Secretary; Booz Allen Hamilton Study; SRA International - Proof of Concept Knowledge Discovery; Program Outcome Study; IG Report: "Evaluation of Veterans Benefits Administration Vocational Rehabilitation and Employment (VR&E) Contracts"; GAO Report: "VA Disability Benefits and Health Care Providing Certain Services to the Seriously Injured Poses Challenges"; GAO Report: "Veterans Affairs - Status of Achieving Key Outcomes and Addressing Major Management Challenges"; GAO Report: "VA Vocational Rehabilitation and Employment Program - GAO Comments on Key Task Force Findings and Recommendations".

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: VR&E Service reports direct and indirect costs in its annual budget submissions. The program's budget is not directly tied to accomplishments or annual and long-term performance goals/measures.

Evidence: VA's 2006 budget pages 5A13 to 5A20; FY 2004 and FY 2005 Performance and Accountability Report. Pages 65-66 and 158 in the '04 report; 88-90 and 194 in the '05 report

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The VA Vocational Rehabilitation and Employment Task Force Report, which was issued in 2004, made 100 specific recommendations for improving areas of deficiency in program management. VR&E has implemented 72 of the recommendations from the Task Force report and continues to focus on completing the remaining issues. Regarding major areas of improvement, the following steps have been taken to correct deficiencies in strategic planning by VR&E: the 5-Tracks to Employment process has been designed to improve the program's focus on outcomes and is now in effect following a successful pilot program; VR&E has also re-organized management and staffing as recommended by the Task Force to reflect a greater emphasis on program outcomes; additionally, the use of performance data by leadership has been reinforced at central, regional, and local offices to determine strategic direction and to take corrective action on an on-going basis. In addition, VR&E has also reassessed its relationships with contractors and has negotiated significant cost reductions, as well as mandating training for key personnel involved with contractor selection and management. The cumulative impact of these changes has had significant impact on correcting strategic planning deficiencies at VR&E. In addition, while the program does not currently link budget requests with performance goals, the Department of Veterans Affairs has developed a methodology for estimating the marginal costs of changing performance targets. This methodology will be applied to a subset of measures and included in the Department's FY 2008 submission to OMB.

Evidence: VR&E Task Force Report to the Secretary; Task Force Recommendations Planning Matrix; VA 2003-2008 Strategic Plan; VBA's Workforce and Succession Plan, March 2003; Systematic Analysis of Operations(M28-3, Part I, Chapter 2, Change 6) ; Reassignment of ES to EC guidance and position description; Performance Standards for RO Directors; VR&E Officer's National Performance Standards; Vocational Rehabilitation Counselor's / Counseling Psychologist's National Performance Standards: Monthly Operating Report (MOR); VR&E Intranet Reports; VR&E Letter 28-05-01; Circular 28-05-01, VR&E Guidelines for the Administration of the Independent Living Program; VR&E Letter 28-05-03, VR&E Guidelines for Evaluation and Planning, Self-Employment, Re-Employment, Rapid Access to Employment; Revised Checklist to Include Utilization of MRG Codes; VR&E Letter 28-04-06, Case Management Index and Scoring; Case Management Index; 5 Tracks to Employment Model; Job Lab Operators Guide; Concept Paper on National Acquisition Strategy.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: VR&E employs various management reporting systems and vehicles to collect performance information which is used to manage the program and to improve its performance. For instance, Corporate WINRS (CWINRS) and the Benefits Delivery Network (BDN) track program data using long-term outcome and annual performance targets as the critical element, including veterans' movement through the case status system and their exit from the program. Importantly, the reliability (consistency within the database) and performance (availability of the system to users in the field and at Central Office) of the CWINRS database is 93 and 99 percent, respectively. In addition, VBA's Office of Performance Analysis and Integrity also publishes the raw data (all data elements on all records) in the Monthly Operations Report (MOR), which is available each month to all VR&E regional office employees. In turn, the Central Office and field staff closely monitor this data to adjust priorities and to achieve optimum performance with available resources. Field offices also sort raw data by case manager and adjust workloads or provide training and direction to staff as is needed. Currently, the program uses a Case Management Index (CMI) to measure twelve performance indicators which provides the collection of baseline performance data. CMI captures the raw data that reflect progress toward meeting the program's annual and long-term performance measures. VR&E has collected baseline performance data on these measures; examples include Rehabilitation Rate, SEH Rehabilitation Rate, and Accuracy of Vocational Rehabilitation Completion Decisions, which have been tracked since 1996, 2001, and 2002, respectively. Moreover, each VR&E office conducts a Systematic Analysis of Operations (SAO) annually or more often as required; from the information derived during the SAO, corrective action plans are developed as necessary to improve performance deficiencies. VR&E also collects high-quality performance data from its key program partners and considers their performance when evaluating the program's progress. For instance, through a Memorandum of Understanding, the Department of Labor currently shares data with VR&E on four measures regarding the program's participants. This data is subsequently used to identify areas of needed improvement in the referral and case management processes between agencies.

Evidence: PA&I Monthly Reports (Monthly Operating Report, Case Management Index); VR&E Intranet Reports. Performance Standards for RO Directors, VR&E Officers, and VRC/CPs; Monthly Performance Reviews; MOU VR&E and DOL/VETS; DOD PEB database; MODS database; Systematic Analysis of Operations.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: VR&E identifies Regional Office Directors as being responsible for achieving key program results. They are held accountable for meeting the goals of the program as evidenced by the six program performance measures incorporated into the national standards used in their performance appraisals. Additionally, VR&E officers are held accountable for program performance on ten of the performance measures as published in the VR&E Officer's National Performance Standards. The program's contractors are also held responsible and measured for the quality of their work performed by standards set in the National Acquisition Strategy (NAS) which was used to award performance-based contracts to achieve the statement of work within the specified costs. Furthermore, the VR&E Case Manager monitors the quality and timeliness of contractors' work through direct review of each case assigned. Work performed by contractors must be reviewed by a VA case manager and be approved prior to certification for payment of invoices by a VA manager. Regarding contract management, VR&E has addressed four of the six recommendations from the 2005 OIG Report regarding contract administration. One recommendation has been partially addressed??price reductions have been negotiated with contracts??however; VR&E felt that service to veterans would be disrupted if the next option year of the NAS contract were not exercised. The final recommendation regarding internal controls of payment procedures is being addressed by VBA's Finance/Support Services Divisions and VA Office of Resource Management.

Evidence: VR&E Officers National Performance Standards; Vocational Rehabilitation Counselor / Counseling Psychologist's National Performance Standards; Regional Office Directors National Performance Standards; Concept Paper on National Acquisition Strategy (NAS) Performance Based Contracts.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Funds for the program are spent on their intended purposes. VR&E does not have excessive unobligated balances, has no Anti-Deficiency Act violations, and a minimal percentage of erroneous payments. VR&E staff is responsible for following established policy and procedure including data entry, as required to support purchases, providing justification and documentation, and tracking data to ensure levels of authorization and funding are not exceeded. No monetary obligations exist between VA's VR&E program and DOL's VETS program, therefore there is no schedule for obligation of funds. For contracted services, VR&E Officers obligate funds as the need for particular services are identified and contracts are executed. The Office of Resource Management (ORM) independently validates VR&E activity. ORM conducts monthly reviews of obligations and expenditures then compares them to VR&E's annual operating plan. Variances against planned obligations are analyzed and corrective actions are taken where and when necessary.

Evidence: Monthly Performance Review (March 2006); SF 133 (Statement of Budget Execution).

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: VR&E defines a program matrix for every project proposed or undertaken including a cost/benefit analysis of expected gains, and subsequently measures actual results against the matrix. For proposed IT improvements, the process is: a) Submit request with costs/schedule and cost/benefit analysis b) Approval by VBA IT Investment Board c) Approval by VBA Senior Management d) Approval by VA Enterprise Investment Board e) Funding Approval (OMB Exhibit 300 process). For proposed non-IT improvements: the process is: a) Submit request with costs/schedule and cost/benefit analysis b) Approval by VBA Senior Management c) Funding Approval. The program has efficiency measures in place to help identify areas of needed improvement or to target re-allocation of resources. Overall program efficiency is measured by the Average Cost of Placing Participant in Employment. In addition, through its Speed of Entitlement Decisions and Accuracy of Entitlement Determinations measures, VR&E assesses the efficiency of its claims processing activities and pinpoints areas where improvement opportunities exist. Moreover, to ensure that participants receive services that help them move through the program and reach their desired outcome, VR&E also measures Percent of Cases in Interrupted Status and Accuracy of Vocational Rehabilitation Program Completion Decisions.

Evidence: IT Initiatives (CWINRS) in VA's Budget; Exhibit 300's in VA's Budget; Monthly Performance Review (MPR), February 2005.

Does the program collaborate and coordinate effectively with related programs?

Explanation: VR&E has updated Memoranda of Understanding (MOU) with the Department of Labor (DOL), Council of State Administrators of Vocational Rehabilitation, and Rehabilitation Services Administration. These MOUs are designed to improve coordination and collaboration with these agencies in providing services to disabled veterans and servicemembers. For example, VR&E and DOL's Veterans' Employment & Training Service (VETS) program are sharing data which will help to track job placement services that are provided to veterans and to document employment outcomes. In addition, Joint Workgroups between VR&E and DOL have met and established timelines to develop performance measures to further assess partnership program results, to develop procedures for joint data collection, analysis and reporting, and to design curriculum for staff training. This initiative will result in increased effectiveness of services provided by both agencies to the disabled veteran population whom they serve. Also, VR&E and Department of Defense (DOD) have collaborated to develop and to employ a standardized presentation during Disabled Transition Assistance Program (DTAP) outbriefings. Each DTAP site has a point of contact assigned from each agency to coordinate services to transitioning servicemembers.

Evidence: Interim Issue 28-89-1; M28-1, Part I, Appendix 2G, Change 13 (Memorandum of Agreement with VFW). MOU VR&E and DOL/VETS.

Does the program use strong financial management practices?

Explanation: The program is free of any material internal control weaknesses in this area. VR&E has procedures in place to ensure that expenditures and payments are made properly and for the intended purpose. VR&E coordinates with the VA Education Program to terminate benefits payments under one program when a veteran elects payments under another program.

Evidence: FY 2004 and FY 2005 Performance and Accountability Reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The VR&E Task Force made 100 recommendations for improving areas of deficiency; the program has implemented 72 of these. VR&E continues its focus on completing the remaining 28 recommendations of the Task Force; however, these remaining items either require additional resources which are not currently available, or are issues that must be addressed through a longer-term approach. Regarding the major areas needing improvement, the program has taken various steps to correct these management deficiencies. For example, the "Five Tracks to Employment Process" (a primary recommendation from the Task Force), which was designed to improve the program's focus on outcomes, is now in effect following a successful pilot program. In addition, VR&E Service has re-organized management and staffing as recommended by the Task Force to reflect a greater emphasis on program outcomes such as employment and independent living rather than on the process of providing education or training. VR&E Service sets policy, interprets regulations, and provides training and guidance to field staff on issues identified through monthly review of performance measures, through review of cases, and through legislative, executive, or administrative initiatives. This arm of the program also develops national training to address areas of needed improvement and issued guidance on Systematic Analyses of Operations in 2003 to be used on a routine basis in analyzing the most critical and vulnerable elements of VR&E to assist in local and strategic planning. Moreover, performance data is used by leadership to determine strategic direction and to take corrective actions. For instance, the program has adopted a number of long-term as well as annual measures which reflect progress towards achieving its purpose, and it is working to implement additional long-term measures as well. In January 2006, the Case Management Index (CMI) was updated to include seven additional indices, and this system provides an overall score which can be used by VR&E Service and by each field office to focus resources and to take corrective actions. With respect to contract management, the program's contracts were reviewed, market research was conducted, and voluntary price reductions were subsequently obtained resulting in a projected savings of $450,000 annually.

Evidence: VR&E Task Force Report to the Secretary; Task Force Recommendations Planning Matrix; VA 2003-2008 Strategic Plan; VBA's Workforce and Succession Plan, March 2003; Systematic Analysis of Operations(M28-3, Part I, Chapter 2, Change 6) ; Reassignment of ES to EC guidance and position description; Performance Standards for RO Directors; VR&E Officer's National Performance Standards; Vocational Rehabilitation Counselor's / Counseling Psychologist's National Performance Standards: Monthly Operating Report (MOR); VR&E Intranet Reports; VR&E Letter 28-05-01; Circular 28-05-01, VR&E Guidelines for the Administration of the Independent Living Program; VR&E Letter 28-05-03, VR&E Guidelines for Evaluation and Planning, Self-Employment, Re-Employment, Rapid Access to Employment; Revised Checklist to Include Utilization of MRG Codes; VR&E Letter 28-04-06, Case Management Index and Scoring; Case Management Index; 5 Tracks to Employment Model; Job Lab Operators Guide; Concept Paper on National Acquisition Strategy.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: VR&E is making notable progress toward achieving the long-term performance goals of the program's Rehabilitation Rate and Serious Employment Handicap Rehabilitation Rate. However, the program's ability to meet the latest and most ambitious targets have not been established, and two new measures are presently developing baselines by which VR&E can judge performance??Percent of Participants Employed First Quarter After Program Exit and Percent of Participants Still Employed Three Quarters After Program Exit.

Evidence: VA's 2006 Budget; VA's 2003 Budget; FY 2004 and FY 2005 Annual Performance and Accountability Reports.

Does the program (including program partners) achieve its annual performance goals?

Explanation: VR&E is making notable progress toward achieving its annual performance goals and has established a track record of achieving many of these targets, but not all. The program also has three new annual measures which are developing baseline data in 2006 but have not yet established a record of performance. Additionally, VR&E is conducting a data-sharing project with DOL's VETS program??one of the program's significant partners??but joint workgroups which will identify further opportunities to share and analyze data have not yet reported results or made recommendations.

Evidence: VA's 2006 Budget; FY 2004 and FY 2005 Performance and Accountability Reports; Monthly Operating Report.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: VR&E has used strategic sourcing and improved contracting practices to negotiate price reductions ranging from 9% to 82% for contractor services (e.g., testing, job placement, initial assessments, case management, and career counseling). These cost reductions constitute approximately 5% of the total contract costs, or a reduction of $450,000 per year within the $9,000,000 annual expenditure. VR&E also has a pilot project to consolidate administrative functions and achieve cost effectiveness, as well as joint initiatives with VA's Education Service and the Department of Labor to identify other areas for savings and efficiency. In 2007, VR&E will add Average Cost of Placing a Participant in Employment to further advance their efficiency measurement and management.

Evidence: VR&E Circular 28-05-03 VR&E Guidelines for Evaluation and Planning; OIG Report 04-01271-74 Status Update; MOUs in Tab 1; VA's 2006 and 2007 Budget; VA's 2003 Budget; Memorandum of Understandings Tab 1.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: When compared to state vocational rehabilitation agencies administered by the Rehabilitation Services Administration (RSA), the VR&E rehabilitation rate is higher and VR&E is able to place disabled veterans in positions that provide a higher income. The average hourly rate of income for an employed disabled veteran who received services through the VA's VR&E Program is $13.67 versus the average hourly rate of $7.30 for individuals who received services through state vocational rehabilitation agencies. In addition, an RSA study indicated that 45% of the population served by state vocational rehabilitation agencies achieved an "employment outcome", versus VR&E's rate of 62%. This last measure is especially notable, given VR&E's focus on finding "suitable employment". The phrase suitable employment, as used by VR&E, ensures that job duties match the veteran's interests, aptitudes and abilities, and does not aggravate their disability/disabilities. This is a greater standard than employment outcome and reflects VR&E's emphasis on well-paying career solutions.

Evidence: RSA Longitudinal Study of the Vocational Rehabilitation Services Program: Implications for Practice; VA's 2006 Budget.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The comprehensive VR&E Task Force report issued in March 2004 commended VR&E's advances in recent years; however, they also cited 100 areas where the program needed to improve performance. While VR&E has implemented 72 of the Task Force's recommendations, an independent evaluation of sufficient scope and quality has not yet assessed the program changes and reported them effective and achieving results.

Evidence: VR&E Task Force Report to the Secretary dated March 2004, pgs. 2, 41, A-30; Task Force Recommendation Matrix; Program Outcome Study conducted by Dr. David Dean from September 2002 to September 2004.