6. |
AFI Program Evaluation |
||
The Assets for Independence Act requires the Office of Community Services (OCS) to establish and support a national evaluation of the overall AFI Program, with particular attention to program impacts and outcomes. OCS has contracted with a social science research organization to implement the multiyear evaluation. Below is an overview of the evaluation's overall purpose and the evaluation design. This national evaluation is a process distinct from both grantees' regular narrative and fiscal reporting and from the data collection for the annual congressional report.
Purpose
The AFI Act stipulates that the evaluation is to develop information about several important factors. They include the following:
- The effects of incentives and organizational support on savings behavior among AFI Project participants
- The savings rates of AFI Project participants based on demographic characteristics
- The economic, civic, psychological, and social effects of asset accumulation, and how such effects vary among different populations or communities
- The effects of IDAs on savings rates, homeownership, level of postsecondary education, and self-employment, and how such effects vary among different populations and communities
- The potential long-term financial returns to the Federal Government and to investors in IDAs
- Lessons to be learned from the AFI Program
Evaluation Design
The design of the national evaluation consists of two parts: an impact study and a process study. Both components are being carried out simultaneously.
Impact Study
The impact study is examining what effects the AFI Program is having on project clients. The design centers on information gathered through an ongoing three-year longitudinal survey of 600 AFI participants nationwide who opened their IDAs in calendar year 2001. The survey subjects are providing a range of information about their economic situation and related matters through periodic phone interviews and other data collection. Evaluators are using information collected through the surveys to consider the effects of incentives and institutional supports on savings behavior, homeownership, postsecondary education, and self-employment. They are also examining how these effects vary among different populations or communities.
The impact study is a vital component of the AFI Program evaluation. It is also an important contribution to the body of knowledge about IDAs generally, for most information and evidence available from other sources about the long-term client-level effects of IDAs is based on relatively small-scale interventions. The AFI evaluation process is developing the largest systematic, quantitative body of national evidence to date on the effects of IDA participation.
Process Study
The process study is designed to describe how and why AFI Projects affect project participants. This study provides a comprehensive picture of the development, planning, startup, and ongoing operations of selected AFI Projects. It describes how the projects work and the factors influencing operations. This component of the evaluation is based on information collected from two-day visits to six selected AFI Projects in the spring of each year of the study.
OCS and the evaluator select the sites to encompass diversity among the characteristics that seem to be most important for understanding project operations. The selection criteria include type of grantee organization, project size, and location (e.g., geographic region and urban or rural setting). Selected sites are not intended to be representative, but rather illustrative of the range of project models that exist among AFI Projects.
Now in the fifth year of the evaluation, the evaluator has made six sets of site visits thus far. The first year's sites were selected from AFI Projects funded in fiscal year 1999, the second year's from those funded in FY 2000, the third year's from those funded in FY 2001, the fourth year's from those funded in FY 2002, and the fifth year's from organizations that received AFI Project funding in FY 2003. Three sites selected each year receive a second visit or in-depth telephone interview the following year-so evaluators can develop knowledge about how projects evolve over time.
During the site visits and fact-finding telephone discussions, the evaluator team conduct in-depth interviews with AFI Project staff, with their financial institution partners, and with other vital organizational partners (such as providers of asset-specific training). Typically, the team also holds group interviews with project clients to discuss their experiences. Where possible, the evaluator team also observes financial education classes or other project activities.
The insights gathered from the process study site visits are useful to OCS and the overall IDA field in a number of ways. The process study is helping develop knowledge about how particular project features affect clients' experiences with IDAs. In examining how the selected AFI Projects evolved, what issues arose, and how grantees resolved them, the process study is developing an evolving portrait of the selected projects. This information is a useful complement to the snapshot project data that OCS collects and prepares for the annual report to Congress, discussed earlier in this Handbook. Finally, the process study, exploring the depth of day-to-day administrative issues that AFI Project grantees face, holds important lessons for practitioners and policy analysts alike. The site visits yield useful and interesting insights about how different AFI Projects are addressing common challenges. In enriching our understanding of actual project experiences, the process study is helping to explain the field-level implications of various policy features.
Evaluation Reports
The evaluation team produces detailed reports about the annual site visits conducted for the process study. The reports include in-depth information about how the selected AFI Projects are operating their programs. The reports also include key cross-site findings. These reports, and all related materials about the national evaluation, are posted on the AFI Asset-Building Web site as they become available.
OCS will publish a final report at the end of the evaluation that synthesizes findings from the impact study and the process study overall.