Detailed Information on the
Vocational Rehabilitation Training Assessment

Determine whether the Training program should be restructured in order to address emerging needs.

Use existing outcome data to improve program management and direct priorities.

Take significant steps to improve the accuracy of information on payback and numbers of counselors who met their CSPD standard.

Make data on payback and numbers of counselors who met their CSPD standard available to the public.

Calculate and analyze the efficiency measure data (cost per graduate) at the grantee level in order to establish targets and identify potential candidates for technical assistance.

Measure: The percentage of Masters-level counseling graduates fulfilling their payback requirements through acceptable employment.

Explanation:This measure examines the number of scholars who fulfill their payback requirement through employment in the public rehabilitation system or non-profit settings. Such employment is a goal of the program

Measure: The percentage of currently employed VR State agency counselors who meet their State's Comprehensive System of Personnel Development (CSPD) standard.

Explanation:Each State VR agency sets a personnel standard for their counselors and are expected to hire at that standard or train their staff to that standard. This measure examines their performance. Due to internal performance reporting deadlines, the 2006 target could not be adjusted. Targets from 2007-2011 have been adjusted to reflect current program performance.

Measure: The number of students supported by RSA scholarships.

Explanation:This measure counts the total number of undergraduate and graduate students supported by RSA scholarships. Due to internal performance reporting deadlines, the 2006 target could not be adjusted. Targets from 2007-2011 have been adjusted to reflect current program performance.

Measure: The number of RSA-supported students graduating.

Explanation:This measure counts the number of undergraduate and graduate students who get a degree. Due to internal performance reporting deadlines, the 2006 target could not be adjusted. Targets from 2007-2011 have been adjusted to reflect current program performance.

Measure: The percentage of public VR trainees who report an improvement in their knowledge and skills acquisition.

Explanation:

Measure: The federal cost per RSA supported rehabilitation counseling graduate at the Masters-level

Explanation:Established in 2006, this measure uses data from the Payback Reporting system to determine the total cost per Masters-level rehabilitation counseling graduate. VR Training staff will use this data to provide technical assistance to programs with unusually high (or low) costs.

Measure: The percentage of graduates of accredited masters-level rehabilitation counseling programs whom work in state Vocational Rehabilitation agencies employment upon graduation.

Explanation:Using data from the Council on Rehabilitation Education's (CORE) annual survey of member institutions, this measure tracks the national number of graduate students who graduate from CORE accredited rehabilitation counseling programs who go to work for State VR agencies instead of non-profits, the Veterans' Administration system, or the private sector. This measure tracks the national impact of the federal Training program on the State VR system.

Is the program purpose clear?

Explanation: The Vocational Rehabilitation (VR) Training program helps ensure the availability of skilled personnel to meet the rehabilitation needs of individuals with disabilities assisted through the VR, Supported Employment, and Independent Living programs. The program has three components - In-Service Training, Continuing Education, and Long-Term Training. The In-Service training program and Continuing Education programs train existing personnel. The Long-Term Training program supports student scholarships in fields with identified shortages. These programs pay for training, traineeships/scholarships, and other activities that increase the quantity and skills of qualified personnel, (especially rehabilitation counselors) to provide vocational, medical, social, and psychological rehabilitation services.

Evidence: Statutory authority for the Training program is contained in the Rehabilitation Act of 1973, section 302; implementing regulations for the Training program are as follows: 34 CFR 385 Rehabilitation training (general provisions); 34 CFR 396 Rehabilitation Long-Term Training; 34 CFR 387 Experimental and Innovative Training; 34 CFR 388 State VR Unit In-Service Training; 34 CFR 389 Rehabilitation Continuing Education Programs; 34 CFR 390 Rehabilitation Short-Term Training; and 34 CFR 396 Training of Interpreters for Individuals who are Deaf and Individuals who are Deaf-Blind.

Does the program address a specific and existing problem, interest, or need?

Explanation: The demand for new VR counselors far exceeds the supply; 43 percent of State VR counselors are expected to retire by 2007. The number of students currently graduating from rehabilitation counseling programs who enter VR employment (approximately 31% of all new graduates) will meet only about 30 percent of the need for new VR counselors. In addition, 30 percent of existing VR staff do not meet their State's personnel standard under their Comprehensive System for Personnel Development (CSPD) and require retraining. Eighty-eight percent of those who meet their State's personnel standard will require continuing education to maintain their credentials.

Evidence: A Profile of the Demand for and Supply of Qualified Vocational Rehabilitation Counselors, American Institutes for Research, January 2006; Annual State Plan Comprehensive System of Personnel Development attachments, FY2006; and Annual Grantee In-service Progress Reports.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: This is the only Federal program designed to address rehabilitation personnel shortages through a payback requirement. Payback requires students receiving scholarships maintain employment in a rehabilitation agency or provider organization for the equivalent of two years for each year of assistance. If the scholar does not pay back through appropriate employment, the individual must repay all or part of any scholarship received plus interest. In addition, while States can use VR State grant program funds to train and upgrade the skills of personnel, the In-Service Training program is the only program specifically targeted to support special training for staff in the VR agencies. The Rehabilitation Services Administration (RSA) within the Department of Education (ED) houses this program. RSA is the primary source of training funds for the State VR agencies and there is no overlap with funding for Training from other Federal or non-Federal agencies.

Evidence: Rehabilitation Act section 302 (b)(5)(A) for student payback provisions. Section 302(g)(3)(A) requires fifteen percent of the funding for the Training Program to be targeted for In-Service.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: VR Training has no major design flaws. RSA has the flexibility to direct most of the funds to areas of personnel shortages. In addition, the payback provisions help ensure scholarship recipients work in the public VR sector. Furthermore, no evidence exists that other approaches would increase the numbers of qualified VR counselors. However, program grantees (typically institutions of higher education) do not have the expertise nor resources to accurately follow students and to satisfactorily comply with their reporting requirements. A recent survey conducted by the American Institutes for Research (AIR) for RSA found that about half of the addresses and contact information provided by the grantees is inaccurate, and that the accuracy and the overall quality of the data decline significantly over time. As a result, RSA now uses a more accurate tracking system developed by AIR. In addition, the requirement that RSA set aside 15 percent of the Training program appropriation for the In-Service Training program limits RSA's authority to allocate training funds to meet the pressing needs.

Evidence: Rehab Act section 302(b)(5)(A) and 34 CFR 386. A Profile of the Demand for and Supply of Qualified Vocational Rehabilitation Counselors, American Institutes for Research, January 2006.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: VR Training is well targeted to students interested in becoming VR counselors and VR counselors in need of additional training. RSA has used its flexibility under the statute to reduce the number of fields it funds under the long-term training program from twenty to ten and has established ten new community program Rehabilitation Continuing Education Programs (RCEPs). RCEPs provide regional training and focus on meeting recurrent and common training needs of employed rehabilitation personnel. This targeting directs limited resources to the biggest needs in VR agencies-- qualified vocational rehabilitation counselors and training needs of existing State VR personnel.

Evidence: Rehabilitation Services Administration Annual reports; Federal register notices announcing Training program competitions; and creation of community program RCEPs.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program aims to provide the public VR sector with well-trained staff and to maintain and upgrade the skills of current staff through continuing education. There are two objectives and corresponding measures for each objective. Objective 1) To train graduates so they can work in the VR system and help individuals with disabilities achieve their goals. The first objective has four measures - (i) the number of scholarships given, (ii) the number of scholars graduating, (iii) the percentage of graduates fulfilling their payback requirements through acceptable employment, and (iv) the percentage of graduates working in state VR agencies (a new measure). Objective 2) To maintain and upgrade the knowledge and skills of personnel employed in the public VR system. The second objective has two measures - (i) the percentage of currently employed VR State agency counselors who meet their States' CSPD standard and (ii) the percentage of counselors reporting an improvement in their knowledge and skills as a result of training (a new measure).

Evidence: Department of Education's PPMD system; RSA payback reporting form; in-service annual reports; internal grant files.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: In 2006, based on annual trend data through 2005, ambitious targets and timeframes were established through 2011. Most targets were set using a formula of averaging the past three years of actual performance. The program seeks to improve performance by three to six percent in the next five years. These long-term goals are ambitious in light of the AIR study, which found 43 percent of current counselors expect to retire in the next few years. With rising tuition rates and looming retirements, the Training program will need to find innovative ways to reach these long-term goals.

Evidence: A Profile of the Demand for and Supply of Qualified Vocational Rehabilitation Counselors, American Institutes for Research, January 2006.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The annual measures are used to assess progress toward the long-term goals. Measures established to assess pre-service activities include the percentage of graduates fulfilling their payback requirements through acceptable employment including state VR employment, and two output measures for the number of students supported by RSA scholarships and the number graduating. Measures established to assess the training of existing rehabilitation personnel (through in-service training and continuing education programs) include the percentage of currently employed VR State agency counselors who meet their States' CSPD standard, and the percentage of public VR trainees who report an improvement in their knowledge and skills acquisition.

Evidence: Department of Education's PPMD system; RSA payback reporting form; in-service annual reports; internal grant files.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Using data from grantee performance reports, the Training program established baselines and targets for incremental improvement for its annual measures.

Evidence: Department of Education's PPMD system; RSA payback reporting form; in-service annual reports; and internal grant files.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: RSA Training program staff takes specific steps to ensure that all partners commit to and work towards the existing annual goals of the Long-Term Training scholarships, Rehabilitation Continuing Education Programs (RCEPs), and In-Service Training Programs. Program application packages explicitly specify program goals, and RSA staff review grant application and progress reports to assess performance in relation to these goals. Grantees understand that data requested in mandatory data forms such as the Payback Reporting Form are used to assess whether scholars are fulfilling the obligations of their agreements with the grantees.

Evidence: Program Priority Packages (includes Federal Register notices); Payback Reporting Form for LTT grantees; Monitoring of Projects (on-site and off-site); National Educators Conference (NCRE); National CSAVR Conference; Training Project Directors' Orientation Meeting; Annual Performance Reports; Cooperative Agreements for RCEP programs; State Plan data on CSPD; and RCEP Project Directors' Annual meeting with RSA representatives.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: RSA has had two major independent evaluations to determine the training program's effectiveness. (1) In 2000, RSA commissioned the American Institutes of Research (AIR) to study the overall effectiveness of the Training program. AIR looked at: issues of demand and supply of qualified personnel, the types of agencies or organizations that employ RSA scholars upon graduation, the variables graduates consider when choosing to enter careers in state vocational rehabilitation agencies and other acceptable employment settings, the current status of each state's CSPD standards, the number of counselors that meet those standards and the projected demand for qualified rehabilitation counselors in each state over the next ten years. AIR used six surveys to gather information. Response rates ranged from 85.9% to 34.2% with a mode of 77.5%. AIR recently issued two reports from this study: (i) An Evaluation of the Rehabilitation Services Administration Training Program's Responsiveness to State VR Agency Needs for Qualified Personnel (August 2005) and (ii) A Profile of the Demand for and Supply of Qualified Vocational Rehabilitation Counselors (January 2006). RSA used the results of the study to redistribute funds to areas of greatest need, pursue accountability systems (such as the payback reporting system), and became involved in new initiatives, such as recruitment into the rehabilitation professions. (2) In 1992, Pelavin Associates issued a "Survey of Personnel Shortages and Training Needs in Vocational Rehabilitation." The survey was conducted in FYs 1987 and 1989; the results were used to establish funding priorities.

Evidence: An Evaluation of the Rehabilitation Services Administration Training Program's Responsiveness to State VR Agency Needs for Qualified Personnel, American Institutes for Research, August 2005; A Profile of the Demand for and Supply of Qualified Vocational Rehabilitation Counselors, American Institutes for Research, January 2006. Pelavin Associates, Inc., Recruitment and Retention of Qualified Field Service Delivery Personnel in Vocational Rehabilitation, U.S. Department of Education, Contract Number HR90045001, Washington, DC, 1993.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests provide information on activities supported with program funds and include a discussion on the data reported by grantees related to the performance indicators. However, RSA does not have an overall comprehensive plan (other than the required spending plan) that details what specific projects would be funded or not funded with budget changes and RSA has not systematically evaluated how its budget structure reflects program goals.

Evidence: RSA's spending plan for the Training program; Office of Special Education and Rehabilitation Services' (OSERS) Budget Requests; and Congressional Justifications.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: RSA recently developed long-term measures and established targets for them. The 2005 AIR report stated that RSA needed to develop a better system for tracking scholarship recipients. As a result, RSA has developed an internal tracking system. This system will improve the quality of data that RSA collects on the percentage of graduates fulfilling their payback requirements through acceptable employment. In addition, the training program has biennial retreats where the staff discusses administrative and programmatic issues. The retreats help staff examine the program's responsiveness to outside events in rehabilitation and education. In these meetings, the program updates its internal mission statement and objectives which they use to guide program efforts. While this is not a formal department statement, it is a functional statement used by staff to guide program efforts.

Evidence: Department of Education's PPMD system; RSA payback reporting form; in-service annual reports; MIS system; and internal grant files.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: RSA uses data from three main sources to manage the program - the annual performance report, the "payback reporting system" and the annual State plan from State agencies. (1) ANNUAL PERFORMANCE REPORT: The annual performance plans collect information on continuing education and in-service training grants. Grantees must report on the measures contained in the Department of Education's Program Performance Monitoring Document. This provides information on number of trainings and participants and usefulness of the training. RSA staff review these data regularly, share them with partners, and revise plans accordingly. RSA requires the annual performance report before continuing a grant. Recently, RSA required Continuing Education grantees to report on the percentage of public vocational rehabilitation participants who report an improvement in their knowledge and skills acquisition, one of the key administrative measures for those programs. RSA uses the data to: revise project activities, update grantee's cooperative agreement, and issue annual grantee priorities. The program meets with grantees at least annually to discuss progress and revisions. (2) PAYBACK REPORTING SYSTEM: Since 2002, RSA has used this web-based system to collect annual data on the: number of scholars supported, number of scholars graduating, payback location, and key indicators of performance. Grantees report on each scholar during training and after training until the scholar completes her payback obligation. RSA uses this data to manage the program effectively by re-directing funds to fields with the highest personnel shortages and changing grant amounts as necessary. For example, in FY 1997 RSA funded 18 fields in long-term training, but will fund only 11 in FY 2007. In addition, RSA from FY 1998 to FY 2006 RSA doubled the amount going to rehabilitation counseling since grantee data showed a serious shortage in this field. (3) ANNUAL STATE PLANS: These plans include the "CSPD attachment" which provides annual data on the number of rehabilitation counselors who meet the State CSPD standard -- a key measure of program performance. RSA uses data in the annual plans to fund "CSPD grants" based on the progress States make in meeting their CSPD requirements.

Evidence: All Training program annual performance reports; ED's PPMD system; the Annual Rehabilitation Education Meeting, and the bi-annual CSAVR meetings.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ED's managers are subject to the Education Performance Appraisal System (EDPAS) which links employee performance to success in meeting the goals of the Department's Strategic Plan. Specifically, the Chief of the unit that administers this program has EDPAS criteria holding him accountable for the timely and accurate collection of performance information and appropriate future program decisions based on these outcomes; and the conduct of comprehensive teleconferences with all grantees to discuss accomplishments and assess whether grantees are "on target" with stated goals and objectives and accomplishments and, if not, what appropriate action is needed taken to address these deficiencies.

Evidence: Internal records; EDPAS

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: RSA successfully obligates funds for this program by the end of each fiscal year. All discretionary continuation awards are made at least two months in advance of the expiring project period, and before the (continuation award) deadline date of July 31st. New awards under the Long-Term training program are typically awarded in January or February well ahead of most new awards in ED, and eight months before the grantees' academic school year. This allows the grantee time for recruitment of scholars. Staff review and monitor unobligated balances on all discretionary grants early in the year to determine if funding is spent for the intended purposes. Staff may request a formal response from the project director and financial officer and an explanation for any high unobligated balance. Staff review the ED 524 single audit reports and Payback Reporting System on RSA scholars to determine how RSA scholarship monies are spent.

Evidence: EDCAPS financial systems, GAPS, Federal Register competition notices, and RSA's internal files.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: This program adopted an efficiency measure in 2006, but does not have targets yet. The measure focuses on the cost per graduating scholar or trainee who completes the program. However, RSA has taken steps to run the program more efficiently. For example, the program converted it scholarship tracking from paper to a web-based system in 2002.

Evidence:

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program meets with partners throughout the year. These partners include State VR agencies, rehabilitation educators, accrediting bodies, and certification commissions. Results of these collaborations include: (1) Coordinated feedback to certification body -- Staff of the training program meet twice a year with the human resource development (HRD) committee of the Council of State Directors of Vocational Rehabilitation (CSAVR) to discuss training and other issues. Recently, Training staff met with the committee to discuss recent changes in counselor certification standards, and their implications for State agencies. It included feedback to the certification body as to implications for the agencies and issues that the commission may wish to consider. A similar set of meetings brought together various partners to discuss proposed changes in accreditation of counselor training programs. (2) CSPD implementation guidelines -- From 1998 to 2002, several educational partners began meeting with RSA to discuss mutual concerns regarding CSPD, recruitment issues and the direction of rehabilitation counseling. As a result of the efforts of this group, the community established coordination between the accrediting bodies for counselor training programs, certification entities for rehabilitation counselors, rehabilitation educators, state agencies and RSA. Several tangible products emerged - the national rehabilitation recruitment center, clearly stated guidance on CSPD, a "Common CSPD Questions and Answers" document and a general effort across the field of rehabilitation in support of CSPD. Since this effort began, the number of counselors meeting required CSPD levels has increased from about 30 percent to about 70%.

Evidence: Agenda and proceedings of recent national rehabilitation education conferences; CSAVR HRD committee agenda; and CSPD agreements.

Does the program use strong financial management practices?

Explanation: (1) The program follows Departmental financial management guidelines set forth in the Education Department General Administrative Regulations (EDGAR) . Program staff monitor projects for excessive drawdown activity using the Department's Grant Administration and Payment System (GAPS) at six and nine month intervals, and prior to the obligation of a continuation award. (2) The program reviews the financial status and spending patterns of grantees each year during the review of annual performance reports, and conducts monitoring reviews of projects, including financial monitoring, as resources allow. An example of these efforts is RSA's annual review of high unobligated balances to ensure grantees are spending in the best interest of the project. Large unjustified high unobligated balances are deducted from their continuation awards. Over the past several years (from 2000), the amount of such deductions have declined from approximately $1.4 million to approximately $450,000 per year, demonstrating improved grantee performance. (3) However, in 2006 funds were improperly obligated. In response, RSA reviewed the existing financial procedures and made recommendations with staff (both Training program and RSA financial) to improve the procedures and practices used for the commitment and allotment of funds. RSA's Chief of the Training programs will oversee requests of funds from multiple Training program, record and track funds accordingly, and coordinate more closely with financial program staff by providing program specific grant information.

Evidence: Office of the Chief Financial Officer (OCFO)/Grants Policy notifies program staff of excessive draw downs by grantees; EDGAR.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: RSA might restructure the Training program. The program has struggled because of rapidly raising tuitions and a higher demand for qualified rehabilitation counselors. RSA will draft a notice to request public comment on whether or not to change its programs, and if so, how. RSA will publish this notice in the Federal Register by December 2006, decide whether to restructure the Training program, and implement any changes during fiscal year 2008. In addition, RSA has engaged in several program changes as part of the continuous improvement effort begun in the mid-1990s. To follow are a few examples: (1) RSA has made intentional shifts in Training program funding patterns to fund more projects in the area of greatest need - rehabilitation counseling and related areas (i.e., specialty programs within a masters program of rehabilitation counseling). (2) RSA coordinates the annual rehabilitation education conference which brings together state VR agency directors, human resource specialists, and rehabilitation educators. These constituents represent pre-service and continuing education programs. RSA uses these meetings to discuss current issues and share information. Benefits include: allaying concerns about the time and cost associated with meeting the CSPD standard by sharing information on distance learning technology; establishing internships for scholars at state agencies; and, forging changes in training curricula responsive to state agency need). (3) Around 2002, the training program established a national recruitment initiative with its education and state agency partners. The initiative aimed to recruit more students into the field and recruit graduates of counseling programs into state agency employment. It resulted in: a web-based recruitment page listing job openings in state VR agencies, recruitment materials, and regional workgroups to improve recruitment at the regional level. The initiative still exists and is coordinated through a national center associated with the Clearinghouse of Rehabilitation Training Material.

Evidence: RSA internal document, "RSA Program Improvements 1995 to Present".

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All new long-term training and continuing education awards are awarded competitively on the basis of announcements published in the Federal Register. Applications are reviewed by a panel of external peer reviewers. States that apply for In-Service training grants may apply for two awards - a basic award and quality award. Basic award amounts are determined by the number of personnel employed by the State agency, but States seeking funding for an additional quality award must compete for funding. They must submit separate applications that are rated by a panel of external peer reviewers.

Evidence: Federal Register competitions annually for all Training programs and the various LTT academic fields.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: RSA has developed a monitoring plan with grant procedures that ensures timelines for monitoring and oversight and the strengthening of the financial grant review process. Grantees with excessive draw-downs must resolve the issue within two weeks after being notified. Unspent balances, and any incomplete activities associated with those balances, are addressed with the grantee for appropriate resolution. Grantees who have unjustified high balances have their continuation awards reduced, which frees up money for other efforts. While the program has no funds for site visits, it has completed limited, but important monitoring of its grants. RSA leveraged these few visits by alerting all grantees that a few of them would be visited, and that RSA would look carefully at issues such as payback, project spending, project goals met and carryover balances.

Evidence: EDCAPS financial systems; GAPS; Federal Register competition notices; RSA's monitoring plan; and RSA's internal files.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects grantee performance data on an annual basis, but does not make it public. The program will publish information about RSA scholars' employment, payback patterns, and other data soon.

Evidence:

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Generally, program performance has improved from 1997 to 2004. Moreover, the percentage of graduates working in state VR agencies increased by twenty percent from 22 percent in 1997 to 43 percent in 2005. While RSA has not met its targets for the measure relating to employees meeting CSPD standards, performance in 2005 was better than in 2001.

Evidence: Department of Education's PPMD system; RSA Payback Reporting Form; and In-Service annual reports.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Since 2002, the program met its outcome targets for graduates fulfilling their payback requirements through acceptable employment. But has not met its outcome targets for current personnel meeting state standards since 2002. However, the Training program exceeded its output targets for RSA scholarships given and RSA scholars graduating.

Evidence: Department of Education's PPMD system; and RSA Payback Reporting Form.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Training program established an efficiency measure in 2006. The measure tracks the cost per RSA-supported rehabilitation counseling graduate at the Masters-level. From 2001 - 2005, the costs ranged from $10,030 to $11,150 with a low of $9,475. RSA will establish a strategy for improving efficiency based on sharing data and best practices.

Evidence: RSA Payback Reporting system.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no data that can be used to make comparisons.

Evidence:

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The AIR study found that grants awarded under the long-term training program have "a strong presence in the training of rehabilitation counselors. About 70 percent of the training programs received RSA funds, and over one-third of all enrolled students in targeted areas received RSA scholarships." In addition, grantee reports and exited scholar survey data show RSA scholars entered State VR agencies after graduation at a higher rate than non-RSA students; approximately 60 percent of masters-level RSA scholars entered after graduation compared to 39 percent of non-scholars. A related issue is whether the scholarship assistance and other funding provided under the scholarship program affects the supply of qualified personnel for State VR agencies by affecting the size or viability of training programs, including enrollment in such programs. The data would suggest that the scholarships may have such an effect. The study identified 98 masters programs, 70 percent of which received one grant in 2001-2002 from one of 7 "targeted programs" (aimed at training VR agency personnel). RSA scholars accounted for 38 percent of the total enrollment and 43 percent of the students graduating with a masters' degree in rehabilitation counseling.

Evidence: An Evaluation of the Rehabilitation Services Administration Training Program's Responsiveness to State VR Agency Needs for Qualified Personnel, American Institutes for Research, August 2005; A Profile of the Demand for and Supply of Qualified Vocational Rehabilitation Counselors, American Institutes for Research, January 2006; and Pelavin Associates, Inc., Recruitment and Retention of Qualified Field Service Delivery Personnel in Vocational Rehabilitation, U.S. Department of Education, Contract Number HR90045001, Washington, DC, 1993.