Early Head Start Research and Evaluation Project (EHSRE), 1996-2010

The Early Head Start Research and Evaluation project was funded in three waves. The Congressionally-mandated Birth to Three Phase (1996-2001) included an Implementation Study, an Impact Evaluation that investigated program impacts on children and families through their time in the program, and local research projects. In 2001, the Administration for Children and Families (ACF) funded the Pre-Kindergarten Follow-up Phase (2001-2004) to build upon the earlier research and follow the children and families who were in the original study from the time they left the Early Head Start program until they entered kindergarten. In 2005, ACF funded the Elementary School Follow-up Phase (2005-2010) to again build upon earlier research and follow the children and families from the original study while the children are in fifth grade, or attending their sixth year of formal schooling.

As the map below indicates, the programs that participated in the Early Head Start Research and Evaluation project are located in Russellville, Arkansas; Venice, California; Denver, Colorado (two programs); Marshalltown, Iowa; Kansas City, Kansas; Jackson, Michigan; New York City; Kansas City, Missouri; Pittsburgh, Pennsylvania; Sumter, South Carolina; McKenzie, Tennessee; Logan, Utah; Alexandria, Virginia; Kent, Washington; Sunnyside, Washington; and Brattleboro, Vermont.

The point of contact is Amy Madigan.

Figure 1.0 Programs that Participated in the Early Head Start Research and Evaluation Project

The map indicates the programs that participated in the Early Head Start evaluation. The locations are Russellville, Arkansas; Venice, California; Denver, Colorado (two programs); Marshalltown, Iowa; Kansas City, Kansas; Jackson, Michigan; New York City; Kansas City, Missouri; Pittsburgh, Pennsylvania; Sumter, South Carolina; McKenzie, Tennessee; Logan, Utah; Alexandria, Virginia; Kent, Washington; Sunnyside, Washington; and Brattleboro, Vermont.

Evaluation Design and Components

The Early Head Start Research and Evaluation project consisted of three phases: the Congressionally-mandated Birth to Three Phase, the Pre-Kindergarten Follow-up Phase, and the Elementary School Follow-up Phase. These phases are described below.

Birth to Three Phase (1996-2001)

In an effort to measure program effectiveness, processes, and efficacy of the program, the Congressionally-mandated Birth to Three Phase (1996-2001) included a cross-site national study that encompassed an Impact Evaluation and Implementation Study as well as site-specific research conducted by local research projects. The cross-site work was conducted through a contract to Mathematica Policy Research, Inc., and Columbia University’s Center for Children and Families, while the local work was funded through grants to each university based team.

• Impact Evaluation

A rigorous evaluation was designed to examine the impacts of Early Head Start on key child and family outcomes. The evaluation was conducted in 17 sites where Early Head Start research programs were located. The sites were selected purposively based on the following criteria: (1) in aggregate, programs had to provide a national geographic distribution that represented the major programmatic approaches and settings and reflected diverse family characteristics thought to be typical of Early Head Start families nationally; (2) programs had to have a viable research partner; and (3) programs had to be able to recruit twice as many families as they could serve. The evaluation randomly assigned 3,001 families to participate in either Early Head Start or to be in a control group. Comparison group families were not eligible for Early Head Start services but were free to avail themselves of other services in the community.

Comprehensive data from multiple sources were used to examine the effects of participation in Early Head Start. Direct child assessments, observations of the parent-child relationships, and the home environment as well as interviews with parents about child and family functioning were conducted when children were 14, 24, and 36 months of age. Information on family service use was collected at 6, 15, 26 months after enrollment and at the time of exit from the program.

• Implementation Study

As part of the national evaluation, a comprehensive implementation study was conducted. The study was designed to provide critical information on the early development of these very first Early Head Start programs; program implementation and context during the evaluation period; identify and explore variations across programs; illuminate pathways to service quality; examine service needs and use by low-income families with infants and toddlers; and investigate program contributions to community change.

The implementation study measured the extent to which programs implemented the Head Start Program Performance Standards by 1997 and 1999. Data for the implementation study came from many sources, including three rounds of site visits to the research programs, program documents, self-administered staff surveys, Head Start Family Information System (HSFIS) application and enrollment forms, and other documents and databases. The first round of site visits was conducted in summer and early fall 1996, about a year after programs were funded. The second round of site visits was conducted in fall 1997, approximately one year after they began serving families. The third round of site visits was conducted in late summer 1999.

The information from these various sources was synthesized using established qualitative analysis methods and systematic procedures established in advance. Using a consensus-based approach, a panel of site visitors, national evaluation representatives, and outside experts assessed 24 key elements of the program guidelines and the revised Head Start Program Performance Standards, including the degree of implementation both overall and separately for the children and family development areas, as well as staff development, community partnerships, and some aspects of program management.

• Local Research Projects

The local research projects, conducted by university-based researchers partnered with Early Head Start programs, were designed to address specific outcomes and program functions that reflected the uniqueness of each Early Head Start program. The major focus for these local studies was the identification of what mediates and moderates positive child and family development within the context of the specific Early Head Start programs and local communities. These local research studies identified site-specific outcomes and examined intra-site differential impacts and their reasons for them. Local researchers also assisted in the collection of cross-site data collection for the national evaluation. For a description of each research initiative, see Research Partners.

Pre-Kindergarten Follow-up Phase (2001-2005)

In order to address important policy questions related to childhood experiences after Early Head Start, ACF funded the Pre-Kindergarten Follow-up Phase and awarded cooperative agreements to the same local universities funded during the Birth to Three Phase. These universities conducted cross-site and site-specific research, building upon earlier research and following the original children and families from the time they left the Early Head Start program until they entered kindergarten. For a description of these longitudinal research initiatives, see Research Partners. Mathematica Policy Research, Inc. served as the Data Coordination and Analysis Center for this phase of the study.

Elementary School Follow-up (2005-2010)

In the Elementary School Follow-Up phase, children and families were assessed when the children were fifth graders or attending their sixth year of formal schooling. The study included direct assessments of children's cognitive, socioemotional, and physical development; parent interviews; teacher questionnaires; and videotaping of maternal-child interactions. Mathematica Policy Research, Inc. along with the National Center for Children and Families at Columbia University, Educational Testing Service, and the Early Head Start Research Consortium designed and carried out the study. Xtria and ICF provided support for consortium activities.

Tracking Activities (2010-2011)

ACF awarded a contract to the RAND Corporation to contact families in order to maintain up-to-date contact information for the participants.

Evaluation Team

Birth to Three Phase (1996-2001)

Following a competitive procurement process, ACF contracted with Mathematica Policy Research, Inc. (MPR) of Princeton, New Jersey, and its subcontractor, the Center for Children and Families at Columbia University, Teachers College. Dr. John M. Love and Dr. Ellen Eliason Kisker of MPR and Dr. Jeanne Brooks-Gunn of Columbia University led the national evaluation team.

ACF also funded 15 local research teams to work with the Early Head Start research programs to conduct their own research on issues central to the local programs and to participate in many national evaluation activities (including providing field support for the national data collection).

The Early Head Start Research Consortium—composed of federal staff, national evaluation contractor staff, 15 local research teams, and directors of the 17 Early Head Start programs—was created to facilitate collaboration on issues related to policy, assessment, and the use of research and evaluation data.

Pre-Kindergarten Follow-up Phase (2001-2005)

In order to answer policy relevant questions related to child experiences after Early Head Start, ACF funded a Pre-Kindergarten follow-up of the children in the Early Head Start Research and Evaluation sample. In this phase of the study, 15 local research teams were funded to develop cross-site measures and collect data, while MPR was funded in the role of a Data Coordinating and Analysis Center in order to facilitate training, certification, and data consolidation.

Elementary School Follow-up (2005-2010)

ACF funded a fifth grade (G5) follow-up study of the children in the Early Head Start Research and Evaluation Sample. MPR, along with the National Center for Children and Families at Columbia University and Educational Testing Service, directed the fifth grade round of data collection. The 15 local research teams advised MPR on the study design, planning and assisted with oversight of the data collection. Xtria and ICF provided support to the EHS Consortium and workgroup activities.

On the Early Head Start program, contact:

Amanda Bryans
Administration for Children and Families, DHHS
Office of Head Start
1250 Maryland Avenue, SW, 8 th Floor
Washington DC 20024
Phone: (202) 205-9380
Email: Amanda.Bryans@acf.hhs.gov

On the Early Head Start Research and Evaluation project, contact:

Amy Madigan
Office of Planning, Research and Evaluation
Administration for Children and Families, DHHS
370 L'Enfant Promenade SW, 7th Floor W
Washington, DC 20447
Phone: (202) 205-8810
Email: amy.madigan@acf.hhs.gov

Early Head Start Research Consortium

The Early Head Start Research Consortium consisted of representatives from 17 programs participating in the evaluation, 15 local research teams, the evaluation contractors, and ACF. Research institutions in the Consortium and their principal researchers included the following:

• ACF (Birth to Three: R. Chazan Cohen, J. Jerald, E. Kresh, H. Raikes L. Tarullo; TPK: R. Chazan Cohen, J. Jerald, E. Kresh, H. Raikes; G5: R. Chazan Cohen, H. Raikes, A. Bryans);
• Catholic University of America (M. Farber, L. Milgram Mayer, H. Liebow, C. Sabatino, N. Taylor, E. Timberlake, and S. Wall);
• Columbia University (L. Berlin, C. Brady-Smith, J. Brooks-Gunn, and A. Sidle Fuligni);
• Harvard University (C. Ayoub, B. Alexander Pan, and C. Snow);
• Iowa State University (D. Draper, G. Luze, S. McBride, C. Peterson);
• Mathematica Policy Research (K. Boller, E. Eliason Kisker, J. Love, D. Paulsell, C. Ross, P. Schochet, C. Vogel, and W. van Kammen [TPK, L. Tarullo]);
• Medical University of South Carolina (R. Faldowski, G. Hong, and S. Pickrel);
• Michigan State University (H. Fitzgerald, T. Reischl, and R. Schiffman);
• New York University (M. Spellmann and C. Tamis LeMonda);
• University of Arkansas (R. Bradley, M. Swanson, and L. Whiteside-Mansell);
• University of California, Los Angeles (C. Howes and C. Hamilton);
• University of Colorado Health Sciences Center (R. Emde, J. Korfmacher, J. Robinson, P. Spicer, and N. Watt);
• University of Kansas (J. Atwater, J. Carta; and J. Summers);
• University of Missouri-Columbia (M. Fine, J. Ispa, and K. Thornburg);
• University of Pittsburgh (C. McAllister, B. Green, and R. McCall);
• University of Washington School of Education (E. Armijo and J. Stowitschek);
• University of Washington School of Nursing (K. Barnard and S. Spieker),
• Utah State University (L. Boyce and L. Roggman)

Fatherhood Research

The Early Head Start Research and Evaluation Project conducted research relating to the role low-income fathers play in the lives of their infants and toddlers, in their families, and in the Early Head Start programs in which they participate. The Early Head Start father studies were among the first to investigate involvement of low-income fathers in children's lives, together with mother involvement, in the context of both an intervention program for infants and toddlers and a longitudinal study. The Early Head Start father studies focused on biological fathers and father figures (sometimes referred to as "social fathers").

Fatherhood studies within Early Head Start were a coordinated effort by a number of governmental and nongovernmental groups, working together in the spirit of the Fatherhood Initiative, begun in 1995. Early Head Start father studies were funded by the National Institute of Child Health and Human Development (NICHD), Office of the Assistant Secretary for Policy and Evaluation (ASPE), the Administration for Children and Families (ACF); and the Ford Foundation.

Study Objectives

The father studies increased our understanding of how fathers and mothers, in the context of the family, influence infant and toddler development. The Early Head Start Research and Evaluation Project assessed how programs worked with low-income families to enhance children's development and well-being, but data collection originally centered on mothers as the primary source of information. The father research enabled us to learn more about how programs support fathers' relationships with their children and with the children's mothers. Quantitative and qualitative studies of fathers were carried out in Early Head Start research sites for better understanding of:

• The characteristics of the social and biological fathers in Early Head Start communities;
• How father involvement affects children's developmental trajectories; and
• How Early Head Start programs involve fathers and mothers in program activities and as the primary educators of their children.

Fatherhood Research: Four Study Strands

The father studies were added to the evaluation research design in response to efforts within Early Head Start to involve fathers in program activities and in response to the federal Fatherhood Initiative. Father studies within Early Head Start were coordinated by the Early Head Start Research Consortium's Father Studies Workgroup. The workgroup conducted four father study strands:

1. Interviews with Fathers. Approximately 750 fathers at 12 of the 17 research sites were interviewed when the children were 2 and 3 years old and again in the spring before their kindergarten year. Workgroup members investigated how fathers influence child and family outcomes in Early Head Start and beyond; the personal and contextual factors in the lives of fathers that contribute to their involvement in and enjoyment of fathering; whether services for fathers affect their involvement in their children's lives; program and individual characteristics that are associated with program participation; and the cultural, demographic, and regional variations in father involvement in programs and in children's development. The interview protocol paralleled many of the questions asked of mothers in their interviews. Because of the Early Head Start evaluation's experimental design, this research strand was able to measure program impacts on father involvement. In addition, the study included a core set of qualitative questions that will generate new hypotheses about fathers and their involvement. This study also collected data on father-child interactions in a subsample of families in 7 sites, using videotaping procedures comparable to those used to assess mother-child interactions at 2 and 3 years and in the spring before kindergarten as part of the Early Head Start national evaluation. Interaction data provide information on the quality of the father-child relationship that is difficult to glean from self-report or maternal interviews. The Early Head Start father-child videotaping procedures add greatly to the methods used in previous research on father-child interactions in low-income families.
2. Study of Mothers and Fathers of Newborns. The study of newborns, supported by the Ford Foundation, followed 200 fathers and mothers of newborns. Fathers and mothers were interviewed when their children were 1, 3, 6, 14, and 24 months of age to provide an in-depth look at the evolving nature of fatherhood and father involvement in children's lives. A subset of families were interviewed when the children were 3 years old and in the spring before kindergarten. This study also included interviews that contain qualitative questions and videotaped observations. The research with fathers of newborns allowed us to learn about the early experiences of fatherhood.
3. Practitioner Study. Components of the practitioner study were funded by ACF, the Ford Foundation, and NICHD. This study focused on understanding:
   1. The strategies that Early Head Start programs use to engage fathers and father figures in the program;
   2. The goodness of fit between these strategies and the perceived needs and preferences of fathers;
   3. The successes programs achieve and the barriers they encounter;
   4. How programs change over time in response to their experiences with low-income fathers and their children and families;
   5. How Early Head Start programs influence fathers and their relationships with their children and families; and
   6. The roles of fathers and fathers' influences on their children and families.

The practitioner study used survey and qualitative methods to study father involvement in program activities as part of an iterative process that builds on the depth and complexity of research questions from one phase to the next. Specific practitioners study components include:

• Focus groups with fathers, mothers, and Early Head Start staff members (1997);
• An in-depth study of father involvement in one program (1998-1999);
• A survey of father involvement in the 17 Early Head Start research sites (1997);
• In-depth interviews with directors and father involvement coordinators in the 17 research sites (1999-2000);
• Qualitative questions about support needs as part of interviews with about 750 fathers (1998-2001);
• Focus groups with fathers and mothers in Head Start and Early Head Start (1999);
• A survey of father involvement in 261 Wave I to Wave 4 programs (1999-2000)
• Focus groups with Early Head Start staff members (2000); and
• Continuing analyses of program-related data (2002-2004).

1. Local Research Studies. The university researchers engaged in the Early Head Start father studies conducted a variety of special studies focused on fatherhood issues of particular significance to their program partners and the populations served in their communities. Examples include studies with a special focus on father-child interaction in the context of play, a study of father-child teaching with biological fathers, and intensive case studies with small numbers of fathers and families.

Fatherhood Research: Workgroup Members

Fatherhood Research: Dissemination

Many scholarly papers have been produced from the Early Head Start father studies work group. These presentations and papers can be viewed under the Early Head Start Reports section below.

Reports from the father studies also include a special issue of the journal Fathering: A Journal of Theory, Research, and Practice About Men as Fathers, edited by Natasha Cabrera (Winter 2004, 2-1), which is devoted to findings from the Early Head Start study and a report on fathers of newborns, which can be viewed at http://www.mathematica-mpr.com/publications/PDFs/ehsnewborns.pdf.

Unique Features of the Early Head Start Research and Evaluation Project for Assessing Impacts of an Infant-Toddler Intervention

Early Head Start research and evaluation has benefited from earlier generations of intervention studies, including the evaluation of the Comprehensive Child Development Program (CCDP), and has been able to incorporate many methodological lessons from those studies. Several unique features will enable the Early Head Start study to answer questions many previous evaluations have been unable to address. Seven unique features are described here:

• Identification of Effects as a Function of Program Model. Early Head Start program approaches vary in important ways. For example, some programs deliver child development services through child care and others use a home visitation approach. These are appropriate alternatives in the context of different community needs, populations, and geography. The early intervention literature suggests that these variations will result in different patterns and timings of effects.
• Assessment of Implementation. Programs were ranked according to how well they implement the Head Start Performance Standards, using a consensus-based rating system in which a trained panel of experts reviews implementation data obtained from evaluation site visits. (The Advisory Committee on Services for Families with Infants and Toddlers recommended taking level of program implementation into account.) Only by systematically assessing program implementation against the Performance Standards is it possible to determine whether the program is being carried out in each site as it was designed.
• Assessment of the Quality and Quantity of Child Development Services. This study assessed both the quality and the quantity of key services for children, including child care and home visits. In an age where child care is noted to be largely poor in quality, it is important to know the quality (as well as the quantity) of child care received by both the program and comparison group families. It is also important to measure the degree of child development emphasis in home visits and to create an understanding about the quality and quantity of home visits. Low-quality, infrequent visits that lack a child development focus would not be expected to have an impact on children, but high-quality, frequent visits oriented toward child development would be expected to have a positive impact on child development. Similarly, large doses of poor-quality care or small doses of high-quality care would not be expected to have an impact on children, but moderate to large amounts of high-quality child care would be expected to have a positive impact. In general, previous evaluations have not examined in detail the effects on children in light of different configurations of these child development service variables.
• Assessment of Services Received by Both the Comparison Group and the Program Group. The availability of services varies from community to community. For example, some communities are strong in health services but lack employment opportunities. Because different services are available to and actually used by the program and comparison groups, different patterns of outcomes would be expected in specific sites. The team assessed community services available to the comparison group to make predictions on a site-by-site basis about the availability of resources within the community. The evaluation team also directly measured the service use by both comparison and program groups, which provided family-level data on service use for the impact analysis.
• Analysis of Outcomes by Actual Program Participation. Program directors often state that they do not expect the program to affect the families who drop out. They expect impacts on those families with whom they had consistent and continuous contact over a period of several years. The evaluation team conducted a rigorous impact analyses comparing all families randomly assigned to program and comparison groups; then, the team made statistical adjustments for the different levels of program participation. For example, team members estimated the program impacts on those children and families who received the full program dosage.
• Implementation and Impact Study by the Same Research Team. In many program evaluations, the implementation and the impact studies are conducted by two different research organizations. This approach makes it difficult for the impact researchers to interpret site-level effects or to effectively use program implementation data in the impact analysis. In the Early Head Start study, these functions were carried out by the same organization; which will facilitate the use of program process data in the impact analysis.
• Partnerships with Local Researchers. Too often, local program interpretations are missing from national evaluations. A researcher with knowledge of the local program and community can add interpretations to local findings that might not be apparent at the national level and provide local researchers' further interpretation of site-specific findings in a way that is useful to the programs.

Timing of the Early Head Start Evaluation. One might argue that the Early Head Start evaluation was conducted too soon to detect strong program impacts. However, there is much to be learned about the early implementation of new programs as they face multiple challenges in moving to new approaches for serving low-income families with infants and toddlers and in adapting program models in response to welfare reform and the changing needs of families within Early Head Start communities. The dynamic state of new Early Head Start programs may be a factor in their early effectiveness, especially in the context of welfare reform. However, a strong, multifaceted evaluation with both implementation and impact analyses is the correct choice for facilitating continuous improvement from the lessons learned.