ACF OPRE: Evaluation Data Coordination Project (EDCP)

RURAL WELFARE TO WORK STRATEGIES DEMONSTRATION EVALUATION PROJECT

Purpose

The Rural Welfare to Work Strategies Demonstration Evaluation Project is a multiyear national evaluation designed to learn how best to help TANF and other low-income rural families move from welfare to work and to understand the challenges that rural residents face in achieving economic well-being. The evaluation will increase information on well-conceived rural welfare-to-work strategies (effective approaches for working with rural populations) to determine which rural strategies work best for different groups of welfare recipients and other low-income families to help them move from welfare to work. It will also provide lessons about the operational challenges, and the methods to address them, that state and local TANF agencies and others can use. For the initial phase of this initiative, 10 states have received planning grants and assistance to develop strategies for serving rural TANF populations. The evaluation seeks to answer the following questions:

What types and packages of services are provided under the Rural Welfare-to-Work project? How do they compare with services already available under TANF or other funding?
What are the issues and challenges associated with implementing and operating the service packages and policy approaches studied?
What are the net impacts of selected approaches under the project on employment and on families’ well-being?
What are the net costs of the programs? Do the program’s net benefits outweigh the costs?
What strategies should policymakers and program managers consider in designing approaches to improve the efficacy of welfare-to-work strategies for families in rural areas?

For more information:

http://www.mathematica-mpr.com

http://www.acf.hhs.gov/programs/opre/index.html

Agencies/Institutions

The Rural Welfare to Work Strategies Demonstration Evaluation Project is funded by the Administration for Children and Families in the U.S. Department of Health and Human Services. Mathematica Policy Research, Inc. conducts the evaluation.

Research/Survey Design

The evaluation plan has three primary components: an in-depth process and implementation study, an impact study, and a cost-benefit study. The process analysis will primarily use data from two rounds of site visits. The study will identify implementation issues and challenges and provide details on how programs have achieved observed results. Follow-up data collected through surveys and administrative data will be used to analyze participants’ activities in the programs and their employment outcomes. The rigorous impact evaluation will use random-assignment designs to determine what difference these interventions make in employment and family well-being outcomes. The impact study will draw on data from program sponsors, administrative data from human services programs, and baseline and comprehensive survey data collected in follow-up interviews at 18- and 30-months after random assignment into the evaluation’s research sample. The evaluation will use data from the impact study, the implementation study, and published research to calculate estimates of net program cost-effectiveness.

For the process and implementation study, researchers will hold discussions with the administrators and staff of Welfare to Work programs and related agencies, convene focus groups with participants and control group members, and observe programs on-site. Data from surveys and program records will be examined to find out about sample members’ participation in program activities and use of services. The impact study will use data from Welfare to Work program records, state welfare administrative records, and other state systems, and surveys.

For more information:

U.S. Department of Health and Human Services. (2002, October). Rural Welfare-to-Work Strategies Demonstration Evaluation: A summary of the evaluation design and demonstration programs. Washington, DC: Author. Available at http://www.acf.hhs.gov/programs/opre/welfare_employ/rural_wtw/index.html

http://www.acf.hhs.gov/programs/opre/index.html

Mathematica Policy Research, Inc. (2002, October). The Rural Welfare-to-Work Strategies Demonstration Evaluation, Supporting Statement for Request for OMB Approval of the 18-Month follow-up survey and site visit protocols. Princeton, NJ: Mathematica Policy Research, Inc.

Mathematica Policy Research, Inc. (2003, October). The Rural Welfare-to-Work Strategies Demonstration Evaluation, Revised Draft Supporting Statement for Request for OMB Approval of the 30-Month follow-up survey. Princeton, NJ: Mathematica Policy Research, Inc.

Ponza, M., Meckstroth, A., & Burwick, A. (2003, January). The Rural Welfare-to-Work Strategies Demonstration Evaluation Project Evaluation Design. Draft Report. Princeton, NJ: Mathematica Policy Research, Inc.

Date(s)/Periodicity

The Rural Welfare to Work Evaluation began in September 2000 and is scheduled to end in September 2007.

Population/Sample

Two programs have been selected for the in-depth process and implementation study, the impact study, and the cost-benefit analysis: the Illinois Future Steps Program and the Building Nebraska Families program. A third program, the Tennessee First Wheels Program, has participated in only the evaluation’s process study. All three programs serve both current and past TANF recipients and other low-income clients who receive government benefits (e.g., food stamps and Medicaid). The Illinois program began implementation in 2001; the other two programs began implementation in 2002. For the impact analyses, targeted sample sizes are 630 for Illinois (315 program, 315 controls) and 600 for Nebraska (330 program, 270 controls).

Each program offers individual, creative, and diverse services that promote employment and economic independence among the rural poor population. The states were chosen because they offer “substantial, intensive program services on a scale large enough to support a rigorous experimental study of impacts. …The Illinois Future Steps program offers intensive, employment-focused case management to prepare participants for work and help them get and keep good jobs. …The Building Nebraska Families program offers individualized, home-based education and mentoring to help participants develop life skills and overcome barriers, thus indirectly enhancing their employability. … The Tennessee First Wheels program provides no-interest car loans and offers individualized support to help participants maintain their vehicles and stay current in their loan payments” (DHHS, 2002, p. 2).

For more information:

Content Covered

The Rural Welfare to Work Strategies Demonstration Evaluation Project covers a number of employment-related areas for rural residents, including attitudes toward rural places and perceptions of rural changes; education and training; receipt of services; current housing arrangements; employment history; unearned income and income from household; total household income; child care arrangements; barriers to employment; confidence, control, and attitudes toward parenting, material hardship, support networks, and family well-being; and nonprogram car financing.

For more information:

Availability of Data for Public Use

Documentation and data files will be made available to the public at the end of the project, in September 2007.

Reference List for Users’ Guide, Codebooks, Methodology Report(s)

Burwick, A., & Meckstroth, A. (2002, October). Rural Welfare-to-Work Strategies Demonstration Evaluation, a summary of the Evaluation Design and Demonstration Programs. Final report. Princeton, NJ: Mathematica Policy Research, Inc.

Markesich, J., Marsh, S., & Ponza, M. (2003, February). The Rural Welfare-to-Work Strategies Demonstration Evaluation Project Implementation plan for the 18 and 30 month follow-up surveys. Final report. Princeton, NJ: Mathematica Policy Research, Inc.

Meckstroth, A., & Burwick, A. (2002, February). Rural Welfare-to-Work Strategies Demonstration Evaluation Process and Implementation Study: Objectives, Data Collection Methods, and Site Visit Protocols. Final report. Princeton, NJ: Mathematica Policy Research, Inc.

The data collected through the two rounds of follow-up surveys and from state and program administrative records and from the site visits will be incorporated in the series of evaluation reports. The impact analysis will be conducted in three rounds, each followed by some form of reporting. A short-term cost-benefit report will be prepared, and then a final impact and cost-benefit report will be prepared at the end of the project. Special topical papers, briefings, and public use data files are also part of the analysis and reporting phase.

Short-Term Impact Memoranda and Reports. We will conduct two rounds of interim, short-run analyses, with each drawing on increasing shares of the evaluation sample as it is enrolled over time, and on increasing periods of follow-up data. These findings will be reported in stages, separately by site. The first impact analysis “memoranda,” which will report on short-term impacts based on analysis of administrative data, will be submitted in 2004 - 2005. These will be followed with site-specific reports on short-term impacts based on the 18-month survey data and additional quarters of administrative data, during 2005 - 2006.

Interim and Final Process Study Reports. We will follow a similar sequence of producing interim memoranda and report for the process study. An interim cross-site report on program implementation and operations will be submitted in fall 2003. Site-specific reports on program participation and experiences will be issued during 2005 - 2006.

Reports on Costs and Short-Term Cost-Benefits. Separately for each site, we will prepare a report on program costs, and short-term benefit-costs, submitting them in 2005 - 2006.

Final Reports. We will prepare the final reports as final data on the evaluation sample become available from the 30-month follow-up survey and analysis of additional administrative records data on sample members. Analysis of longer-term impacts and costs will be conducted in late 2006 and early 2007, drawing on the 30-month follow-up survey data. A draft overall final report on impacts and cost-benefits and implementation findings will be submitted in summer 2007, with revisions completed within two months of submission. In conjunction with the final report, we will prepare a brief synthesis report.

Table of Contents | Previous | Next