Evaluation of Disability Employment Policy Programs *
The Office of Disability Employment Policy (ODEP) of the US Department of Labor (DOL) has funded pilot demonstration programs across the United States to build the capacity and change the workforce development system to better serve adults and youths with disabilities. In order to evaluate the effectiveness of these demonstration programs, ODEP contracted with the private research firm Westat to conduct an independent evaluation of these programs. This document summarizes the findings, conclusions, and recommendations to date of Westat's evaluation of ODEP's demonstration programs.
This Research Brief summarizes Westat's synthesis report Interim Report on ODEP Demonstration Programs available as a Word document.
The independent evaluation has three objectives:
- To provide ODEP with reliable and valid indicators of program effectiveness.
- To determine the extent to which each program priority area is effective in building workforce development system capacity; and
- To document local, regional, and/or state systems change that supports program effectiveness.
Westat collected both quantitative and qualitative data to generate their conclusions and recommendations.
- Quantitative Data - Westat obtained data from a sample of the customers who had obtained competitive employment as a result of receiving services from the adult demonstration projects.
- Qualitative Data - Westat conducted site visits at all ODEP demonstration locations and interviewed the directors of these projects, employees and partners of the projects, customers and their families, and community representatives. To capture changes over time and collect longitudinal data, the Westat researchers will return to locations for multiple visits.
Overview and Definition of Terms
(For definitions of the terms in the paragraph below, click on the italicized term.)
ODEP funds three types of demonstration programs that are expected to impact systems change within agencies, organizations, and processes under the Workforce Investment Act (WIA): 1) Youth programs to provide services to help youth with disabilities reach their educational and occupational goals; 2) Programs for adults with disabilities to utilize the concepts of Customized Employment; and 3) Technical Assistance to the youth and adult demonstration programs as well as other elements of the workforce investment system.
Westat's Findings and Conclusions
Employment Outcomes
Westat has twice collected information on employment placements among randomly selected program participants of the demonstration projects. In the first round, information was obtained on 345 customers from 25 ODEP Customized Employment demonstration projects who had been placed in competitive employment. It was found that the customized employment process can be effective in helping some of the most hard-to-serve individuals with disabilities find better employment and allow them to begin to move off income support programs by obtaining better employment.
Approximately 43 percent of those who self-disclosed their disability said that they had psychiatric or emotional disabilities, and 37 percent had never worked or had only worked in a non-competitive employment environment. The majority of these respondents found high quality employment after participating in the ODEP demonstration programs: 95 percent were now earning the minimum wage or higher in jobs they obtained through the program. More than one-third of these respondents were earning more than $8.15 an hour while more than 17 percent were receiving benefits (such as health insurance or paid vacations) from their employer. More than half (54 percent) of program customers who had been placed in competitive employment obtained jobs with the potential for career advancement.
Compared to their status at the beginning of their involvement with the project, the percentage of customers of the ODEP demonstration programs who were on government benefits (such as Food Stamps, Social Security Income, Social Security Disability Income, and Temporary Aid to Needy Families) had declined in every category except subsidized housing (which increased as customers of the Chronic Homelessness were placed in housing as part of their participation in that program).
The second round of data collection on competitive employment placements yielded similar findings. Westat collected data on 536 randomly selected individuals who were program customers in 30 ODEP demonstration projects. Approximately 42 percent self-disclosed a psychiatric or emotional disability, and 1 in 5 reported a cognitive disability. Every program customer in this sample received at least one type of customized employment service, and each customer received an average of 5.6 customized employment services.
Westat found that 99 percent of all the respondents obtaining employment earned above minimum wage. One-third now have full-time jobs (more than 35 hours a week), and 27 percent have jobs in which their employer offers at least one fringe benefit. It was also reported that 63 percent of the sampled program customers have obtained employment with future career potential and that 14 percent went off at least one public benefit after receiving customized employment services.
Findings from the Westat Site Visits
Based on extensive interviews conducted during site visits to all of the ODEP projects and TA centers, Westat reported the following:
- Customization of Services - Historically, most One-Stop Career Centers have been oriented towards a self-service model that offers a "one size fits all" approach to customers. This resulted in people with disabilities being referred elsewhere because the One-Stop Career Centers were physically or programmatically inaccessible or the staff believed that other programs such as Vocational Rehabilitation could better serve people with disabilities. The ODEP projects have introduced the person-centered customized approach as an alternative to this model. This customized approach has been implemented in all demonstration projects. The goal is to improve and increase the use of individualized services for people with disabilities and test them in the local environment before further dissemination and replication to other One Stop Career Centers across the country.
- WIA Performance Measures - The ODEP projects have almost universally reported that the current indicators of performance for workforce investment activities collected under the WIA are a disincentive for serving customers with disabilities. Not only are One Stop Career Centers reluctant to enroll participants with disabilities because of these measures, but some members of the WIA system are reluctant to participate in these projects because they fear that they will be penalized as their staff members will be required to take on time-consuming responsibilities that will negatively affect their WIA defined performance metrics.
- Characteristics of Program Participants - According to the quarterly reports filed by the demonstration projects, 4,090 youth and 4,865 adults have been served by the projects (as of June 2005). Among the participants in the youth programs, boys outnumbered the girls, more whites than blacks or Hispanics were in the programs, and when it was self-disclosed, the most frequent disability type was a learning disability. Among the participants in the adult programs, more participants were male, white, had completed some type of high school education (diploma, GED, or certificate of completion) and the most frequent self-reported disability was psychiatric or emotional.
- Resistance to Full-Time Employment - Virtually every project location could report customers that did not want to obtain full-time employment - and in some cases turned down an offer of full-time employment - because they were afraid of losing their government-supplied benefits (especially their financial and medical benefits) if they were hired full-time. Several sites have instituted benefits planning mechanisms and training for One Stop Career Center staff and project partners on benefits, work incentive programs, and other financial issues.
- Strategic Planning - Although the programs were required to engage in strategic planning activities, many were still using a "plan as you go" approach to planning strategies. Conversely, a number had reinforced their understanding of local needs by gathering additional information from key stakeholders and bringing stakeholders on as part of advisory committees. (However, some people with disabilities had difficulty serving on advisory committees primarily because of timing or lack of transportation.)
- Physical and Programmatic Accessibility - Although ODEP's project funds could not be used for physical modifications, the projects served as an impetus for the sites to modify buildings or purchase equipment for physical and communication accessibility. To increase the accessibility of their programs, some One Stop Career Centers have modified their orientation sessions to make them more "disability-friendly." Disability awareness training has also been offered to their staff. Some have connected with the local Disability Service Navigator run by the Social Security Administration and the Department of Labor's Employment and Training Administration. Westat postulated that accessibility changes were easier to make if the project worked with the One Stop Career Center than if the project were implemented by an organization outside the purview of the One Stop Career Center.
- Building Capacity Through Training - The demonstration programs provided training (including formal training and technical assistance) on the following topics for their staff, partners, and collaborators: disability awareness, working with people with disabilities, availability of resources and services for people with disabilities, the available government programs, and advising customers on how to obtain financial work incentives. However, as Westat noted, the quality of this training varied from location to location - ranging from basic and informational to more thorough and problem-solving. Westat also noted that because of staff turnover at the One Stop Career Centers, it was necessary to repeat staff training to maintain sustainability.
- Training on Financial Issues - Adults and youth (and their families) are also receiving training on financial issues (such as the impact of employment on benefits, work incentive programs, and individual budgets).
- Coordination of Services and Resources - In all of the demonstration programs, coordination of services for people with disabilities (such as job accommodations, coaches, personal assistants, funds for training and education, and transportation) has begun with extensive collaboration among partners and collaborators. In many locations this coordination has transitioned to leveraging of resources among partners and collaborators, and in others it has resulted in the establishment of formal arrangements among the partners so that these leveraged resources can be sustained after the funding to the program has ended. However, the range and types of partners varies among programs as do the success of these collaborations.
- Promising Practices, Program Evaluation, and Dissemination - A number of promising practices have been documented and disseminated by the projects, by the Technical Assistance centers, and by ODEP. One example of the programs' effectiveness is the way that some customers labeled as "unemployable" by Vocational Rehabilitation have successfully found employment or have started their own businesses after experiencing the ODEP programs.
- Micro-Enterprises - A number of customers of ODEP programs have turned to micro-enterprise as a form of employment by starting their own small business (with themselves as the only employee).
- Definition of Homelessness - Although the Chronic Homelessness projects are still in their infancy, they report that the differing definitions of disability among government agencies have made it difficult for customers to obtain services.
- Sustainability of Services - A requirement for all of ODEP's demonstration projects is to ensure that the benefits gained will not disappear when ODEP funding ends. Sustainability will be achieved when customized employment becomes part of the menu of services available at the One Stop Career Centers (for the adult programs) and when providers recognize the complex needs of youth with disabilities and make changes to planning, policies and programs to address these needs (for the youth programs). To date, the project sites have reported that it could be difficult to sustain their projects without additional funding. Nevertheless, some sustainability has already been seen: Capacity-building activities at all sites (including training, improvements in accessibility, and increased collaboration among agencies and organizations) serve as a means of sustaining specific elements of the projects. Staff has been trained and will continue to receive training (primarily due to new competency requirements and formal training programs instituted at some sites). Collaborative partnerships will likely continue. Disability expertise has been built into many one Stop Career Centers. Finally, access to core services and One Stop Career Center programs has improved.
Technical Assistance
The primary goal of the Technical Assistance program is supporting the demonstration projects (adult, youth, and selected TA sites) in achieving their goals and objectives. Each TA project has developed its own approach to the provision of technical assistance based on the nature of the programs it is supporting, the characteristics and needs of the demonstration projects being supported, and the specific strengths and experience of the partners implementing each TA project.
- National Center on Workforce and Disability for Adults" (NCWD/A) and the National Collaborative on Workforce and Disability for Youth" (NCWD/Y). Since their inception in Fiscal Year 2001, the NCWD/A and NCWD/Y have seen changes in focus, structure, partnerships, and methods of operations. Now that both are approaching their last year of funding, they have evolved into useful vehicles for demonstration project sites, as well as the wider workforce development system. The NCWD/A has developed a systematic approach to providing TA to demonstration projects, including the development and implementation of a TA plan for each demonstration site. The NCWD/Y has produced a number of useful mechanisms for providing TA (e.g., the web site, provision of one-on-one TA, annual conferences, site visits, and sharing information with other demonstration projects).
- Training and Technical Assistance for Providers (T-TAP). With a target audience of approximately 3,500 organizations throughout the United States, the T-TAP has been creative in developing methods for reaching its audience. In addition to the provision of regional seminars, the primary mechanism for training in T-TAP is electronic (e.g., with online seminars, webcasts, and online courses). Another creative way of accommodating its large and disparate audience is to collaborate with ODEP to identify community rehabilitation programs (CRPs) to receive the technical assistance. These organizations serve as demonstration projects for other programs that want to make the transition to promote integrated employment in non-stereotypical jobs for people with disabilities. The T-TAP also has developed a CRP. Leadership Network consisting of individuals affiliated with exemplary programs that have already converted or made significant progress in conversion to community-based employment and can serve as mentors for TA sites.
- The Chronic Homelessness Employment Technical Assistance Center (CHETA). Despite the disadvantage of beginning operations 13 months after the Chronic Homelessness projects were funded, CHETA has managed to use its collective partner expertise in homelessness, supported housing, and employment to provide useful technical assistance to Chronic Homelessness demonstration projects. During its first year of operation it has visited all of the Chronic Homelessness sites to conduct a needs assessment and conducted a week-long training session based on the findings from this needs assessment. The training session was geared toward Chronic Homelessness project staff and included panel discussions, workshops, role plays, and discovery sessions focused on housing and employment issues. The CHETA project consists of a partnership between two companies, one with expertise in supportive housing, and the other with expertise in employment issues.
Conclusions
Summary
Westat's independent evaluation continues to meet ODEP's three key evaluation objectives of gauging program effectiveness, measuring effectiveness to build workforce development system capacity, and documenting systems change. During their external evaluation, Westat developed key indicators for each of the systems change areas. Westat then examined the inputs and processes developed by all demonstration programs, and found that demonstration programs have segued from building capacity to implementing a wide range of activities aimed at making systems change.
The evaluation has also found that the programs heading towards the end of their lifecycle are directing their efforts toward improving processes to achieve improved outputs (e.g., more staff trained to implement high quality customized employment services), as well as engaging in activities to sustain their efforts. With assistance from Technical Assistance Collaboratives, the older generation of programs (i.e. programs funded in Fiscal Year 2001 and Fiscal Year 2002) is paving the way for the younger generation (Fiscal Year 2003 and Fiscal Year 2004) to move more quickly toward improving capacity of a workforce development system to meet the employment and service needs of people with disabilities.
Next Steps
The next stage of the evaluation will examine the self-evaluation efforts of project sites to identify effective local, regional, and/or state systems change efforts to build a sustained capacity to better serve youths and adults with disabilities in achieving improved employment and education outcomes.
In the past year, Westat has continued to conduct initial and followup site visits, collected employment placement and retention data from adult demonstration projects, and concentrated on a number of issues that have arisen during the evaluation process (e.g., employer involvement in demonstration programs; the role of SSA benefits in program customers' employment decisions; and the role of intermediaries in youth demonstration programs). In the next year of the evaluation, Westat will focus on conducting final site visits to a number of demonstration projects and follow-up visits to several others. These visits will allow Westat the ability to better identify additional promising practices and systems changes to the workforce development system.
In addition to site visits, Westat will also obtain quantitative data from project sites on competitive employment placements and retention in order to provide ODEP with information on program outcomes.
Footnotes
* This Research Brief was developed from the interim report, Evaluation of Disability Employment Policy Demonstration Programs: Task 10: Interim Report on ODEP Demonstration Programs: Accomplishments and Issues Identified by the Independent Evaluation (Westat, June 2005). This project is supported by Contract No. GS23F8144H, with funding from the U.S. Department of Labor, Office of Disability Employment Policy. The findings and recommendations of the report are those of the authors and should not in any way be considered or construed as official policy of the U.S. Department of Labor.
