Table of Contents | Previous | Next |
Chapter A2 : Preparation for the Meta-Analysis
Preparation for a meta-analysis occurs in three stages, all three of which involve decisions that have important implications for the analysis and interpretation of findings. The first of these is Formulation of the Problem, in which the scope of the issue to be studied is defined. The second stage is Searching the Literature. This includes not only determining which studies should be included in the meta-analysis but also where to look for studies. The third stage is Coding the Studies which involves developing and implementing a coding system to describe the interventions, the research designs of the impact studies, and the impacts themselves. In the discussion below, each of these stages is described, together with the issues faced at each step along the way.
Formulating the Problem
Defining the issue to be studied is the starting point for a meta-analysis. The researcher is expected to start out with a "well-built" question. Such a question specifies: (a) the main interventions under investigation; (b) the participants or subjects of interest; (c) the outcomes of interest; (d) the comparison interventions of interest; and (e) the types of studies to be included (see Counsell, 1997; Meade & Richardson, 1997; Richardson, et al., 1995; and Oxman, Sackett, & Guyatt, 1993).
Defining Family Support
The first step in this process was to develop a working definition of family support, in order to determine which programs and interventions would be included in the analysis. Family support posed a number of challenges at this point. First and foremost was the lack of consensus on what constitutes a family support program or family support services. In addition, it was clear that, under any definition, a wide array of programs could be labeled family support, including programs that differ in their approaches to working with families, in what participating families actually do in the program, and in the length and intensity of family participation.
There are at least two distinct ways of formulating a definition of family support as a basis for deciding which interventions are of interest. The first, which focuses on the types of services provided and their goals, is embodied in Public Law 103-66. This legislation defines family support as "...community-based services to promote the well-being of children and families, designed to increase the strength and stability of families, ...to increase parents' confidence and competence in their parenting abilities, to afford children a stable and supportive family environment, and otherwise enhance child development." (GAO, 1996)
A second kind of definition, widely used by family support advocates, is more philosophical and focuses not only on the goals of services but more particularly on the ways in which programs work with families to provide these services. In this definition, family support is seen as: normative -- addressing issues faced by all families with young children; preventive -- not designed to address specific problems or replace more intensive, professional services required by families in need of mental health treatment, remedial education or job training, or substance abuse treatment; and a mutually respectful partnership between family and staff that does not rely on professional diagnosis and treatment of pathology to discern individual needs. Multiple sets of practice principles have been developed by practitioners and experts in the family support field to guide programs, and to distinguish family support from other ways of working with families.
The definition of family support contained in the federal legislation would include under the rubric of family support a wide array of services and interventions. It would include "traditional" family support programs whose primary mission is enhancing parents' capacity to support children's development, and which provide a variety of life skills workshops, parenting classes and parent support groups, parent-child groups and family activities, information and referral to other services outside the program, and advocacy for parents. In addition, the definition would include a set of more recent programs that have a primary mission other than enhancing parent capacity but which have incorporated family support into their programs as an integral part of their services. These programs may have the primary mission of serving adults through job training or other self-sufficiency skill building, or a childfocused mission such as preventing school drop-out, or a community development mission that focuses on housing or economic development. At the same time, these programs are also concerned with building families' capacity to support their children's development, and the services and opportunities offered to achieve the family support goals may be identical to those found in programs where enhancing parental capacities is the primary goal. In practice, it is not feasible to apply a practice-based definitional screen to the program descriptions contained in most research studies and evaluations since the definitions frequently vary. Often, it is not possible to apply such a screen to descriptions provided by program staff. Obviously, the definitions described above and earlier in the report were never intended to be used in this way. Rather, they were intended to serve as guides for program developers and staff, as well as, possibly, tools to evaluate the implementation of family support programs.
With these considerations in mind, we elected to use a two-stage definition of family support for the purposes of the meta-analysis. First, to select programs to be included in the study, we applied a definition of family support that encompassed all services intended to improve child outcomes by strengthening the capacity of parents to support their children's development. Under this definition, virtually all two-generation programs were included, as well as programs that provided family support services as an adjunct to adult-centered services or child-centered services. Operationally, this meant that the meta-analysis included research studies or evaluations of any program or intervention intended to improve child outcomes that provided services to enhance parenting capacity, either as its central or as a secondary mission.
Secondly, we rated all the programs included in the study, using a measure of comprehensiveness developed in cooperation with Sharon Lynn Kagan and other family support experts. This measure was used in conjunction with written information about program design and implementation to rate each program on how closely their practices in working with families match important practice principles in the field (e.g. the extent to which the program targets specific populations rather than offering universal services, whether participation is voluntary rather than mandated). The measure is described in more detail as part of our description of the coding process.
Given the breadth of the definition we have adopted for initial inclusion of programs, this meta-analysis is vulnerable to the criticism that we are aggregating results from too diverse a sampling of studies. The fact that the review included such a diverse set of programs, varying in types of services offered and the methods of delivering these services, raises the question of whether a summary of effects across all programs and intervention will, in fact, be meaningful. One of the criticisms of meta-analysis is that it "mixes apples and oranges." However, as one writer has noted:
To some degree any synthesis of information from multiple research studies involves the aggregation of studies that are dissimilar. The same is true at the level of repeated observations of the same object. Some degree of mixing apples and oranges must occur in the tidiest of studies. Even when studies are intended to be direct replications, exact replication probably cannot occur.
Hall et al., 1996, p.19
The same writer notes a little later:
..if the phenomenon [of interest] is conceptually broad and therefore should be demonstrated over a wide variety of contexts, then studies that vary extensively in subjects, situations and procedures may be appropriate for inclusion. Hall et al., 1996, p.20
We believe that family support meets this test i.e., it is "conceptually broad and should be demonstrated over a wide variety of contexts", and that our approach is defensible. In addition, and as another way of addressing concerns about the heterogeneity of programs, we developed a coding system that allowed us to distinguish among types of programs (where "type"was defined and redefined in different ways) and to conduct separate meta-analyses within more narrow categories.
Participant Groups of Interest
In defining the participants of interest for the meta-analysis, the only limitation we imposed is that the program or intervention should be concerned primarily with the development of children between birth and age 12 years. Once this criterion was met, the review included interventions with all other participant groups. This meant that universal or non-targeted programs were reviewed for the meta-analysis, as well as programs for at-risk families and children. "At-risk" children included children at environmental risk, children at biological risk (low birth weight, premature, etc.), and children who were identified as having behavioral or emotional problems.
Outcomes of Interest
Family support programs typically have a comprehensive set of goals, both in terms of the intended targets for change -- children, adults, families and, sometimes, communities as well-- and in terms of the breadth of the goals for each . This means that the research examines a wide range of outcomes for each group. It is not unusual for a single study to measure an extensive set of outcomes. A further complication is that outcomes are measured at different time points in the life of the intervention (during the intervention, at the end of the intervention, at various follow-up points) and at different points in the lives of the children involved in the intervention.
Despite the difficulty of reviewing and summarizing data for a wide variety of outcomes, the research question of interest for the meta-analysis asks broadly about the impacts of family support, and therefore we were interested in all the outcomes reported in the research.
Comparisons of Interest
Two types of comparisons are potentially of interest for the meta-analysis. The first type of comparison tests whether outcomes for families and children who received the intervention are different at the end of the intervention from outcomes for families and children who did not receive that intervention. This comparison allows us to estimate the effect or "value added" of the intervention, and is at the heart of the meta-analysis.
The second type compares the outcomes of different intervention approaches. For instance, a study might compare the effects of a home visit intervention that uses biweekly versus weekly visits, or professional versus paraprofessional staff. Studies that compare different interventions answer a different question from the one discussed above. They ask whether, for a given population, one form of intervention has larger impacts than another form of intervention. This is an interesting question, but to address it as part of a meta-analysis requires that a sufficiently large number of studies compare the same set of program variations.
Our decision, therefore, was to include only the first type of comparison in our analyses. The main meta-analysis will exclude treatment/treatment contrasts. At the same time, as part of the coding, we described the treatment/treatment contrasts in the research literature. If there are sufficient numbers of contrasts that test the same question, e.g., the relative effectiveness of home visiting versus center-based programming, of para-professional versus professional parent education staff, of two years versus one year of intervention, it would be possible to summarize the data on specific contrasts in a secondary meta-analysis.
Types of Studies to be Included
The question we faced here was first, whether to limit the scope of the meta-analysis to randomized studies or to include other study designs and secondly, whether any types of studies should be excluded from the review. Since it is generally agreed that randomized studies provide the strongest evidence about the effect of an intervention, many meta-analyses in other fields confine their search to experimental studies. While we recognized the analytic benefits of limiting the review to studies that used experimental designs, we were concerned that, in the field of family support, many of the bestknown programs and certainly many of those seen as exemplifying the principles of family support have been evaluated with one or another type of quasi-experimental design. Limiting the review to randomized studies could mean that we systematically excluded studies of several different kinds of family support programs.
After discussion of the issue with a group of experts in meta-analysis, including Dr. Thomas Cook, Dr. William Shadish, Dr Larry Hedges and Dr. Robert Fisher, we elected to include in the review both randomized and non-randomized studies but to record methodological variables that would allow us to examine the relationship between methodology and effects and to determine which specific categories of design we would exclude from the final analyses.
Searching the Literature
This step involved identifying strategies for collecting studies, conducting the searches and selecting studies.
Finding Research Studies
The task of identifying all relevant research studies on family support services was a daunting one, since so many different fields of research fall under the omnibus definition of family support and because the research literature exists in so many forms and forums. It was crucial that we conduct a search that accomplished an unbiased if not complete identification of relevant studies. As one researcher notes, the data collection methods used (i.e., how relevant studies are identified) are "of primary importance to the results obtained in a systematic review or meta-analysis...[The validity of the results of statistical analyses depends on the validity of the underlying data." (Dickersin, Scherer, Lebebvre, 1994: p. 1286) Exhibit A2-1 lists four ways of searching for studies that have been recommended for achieving "high" recall of documents. The goal, as described by White (1994), is to consider "all empirical studies on a subject--not only the published but the unpublished ones--so as to capture in the syntheses the full range of reported statistical effects." (p. 42)
All of the methods listed in Exhibit A2-1 were employed for this meta-analysis. We searched for both published research and unpublished manuscripts. Although some researchers believe that unpublished studies should be excluded because they have not been peer-reviewed, an unmeasured influence on publication status is not assessment of scientific rigor but the nature of the results themselves. If fewer studies with negative or null results are published than studies with larger, more positive results, reviews that exclude unpublished works are likely to overestimate the relationship between an intervention and its outcomes (Dickersin & Yi, 1993; Dickersin, et al., 1987; Simes, 1987; Egger & Smith, 1995). This publication bias appears to hold true for small, nonrandomized studies in particular ( Dickersin & Yi, 1993; Easterbrook, et al., 1991; Newcombe, 1987). Most researchers conducting systematic reviews believe that unpublished studies should be included, and, if necessary, the results can be analyzed with and without the unpublished data (Cook, et al., 1993).
As Exhibit A2-1 suggests, our search began, as an exhaustive literature search should, with research studies in published journal literature; these studies are the "most utilized by research synthesists" and the "most heavily represented in reference databases" (Reed and Baxter, 1994). The less well-represented research literature formats include books and book chapters, research and technical reports, and conference papers. Even more "fugitive" are the results of studies which are not yet in any formal written report but exist instead as research memos or notes.
These were found to a limited extent in some printed and computer databases, including:
- ERIC, which lists published works but also some unpublished papers and papers presented at meetings and conferences;
- The National Technical Information Service (NTIS), a database of summaries of completed research sponsored by 600+ federal agencies;
- Social and Behavioral Science Documents (SBSD), published by the American Psychological Association, which contains abstracts of technical papers, reports and bibliographies;
- Social Sciences Literature Information System (SOLIS), listing monographs, reports and conference proceedings in the social sciences and humanities (since 1976); and
- Federal Research in Progress (FEDRIP), which provides access to information about current and ongoing federally funded research.
Unpublished reports were also unearthed through the methods listed in Exhibit A2-1 under "Consultation." Another rich source of literature was dissertations, which were searched through databases such as Dissertations Abstracts International, Master Abstracts, and Research Abstracts. In conducting the searches, the only criteria we imposed were: (1) the research was conducted after 1965; and (2) the research was conducted in the United States, Canada or Great Britain.
Selecting Research Studies and Programs for the Meta-Analysis Database
Out of the more than 900 research reports that were collected and reviewed,1 we selected for coding 665 studies, representing 260 programs. In selecting which of the 900 studies to code, we took the approach that every relevant research study, where relevance was defined as involving an intervention, outcomes, and participants of interest, would be coded regardless of its design. All of the research studies in the data base compare two groups of participants who receive different kinds or levels of family support services. This includes: (a) studies that compare one group of families who receive the treatment (i.e., a specific set of family support services) with another group of families who do not receive this treatment; and (b) studies that compare a group of families who receive one set of family support services with another group of families who receive a different set of family support services.
a Defined by White
(1994) as "the adroit use of other authors' references to the prior
literature on a topic." |
In addition, we coded information on 167 family support programs for which there was descriptive literature but on which no outcome studies had been conducted (or which only provided data on participant satisfaction at the end of the program). We coded information on these programs to ensure that our descriptions of family support programs encompassed a broad spectrum of programs.
The result of these decisions about which studies to code was a comprehensive database of studies that would allow us to describe a wide variety of programs. A list of the programs in the database and the research studies associated with them is provided in Appendix A.
Coding the Studies
This project has two purposes: to describe family support programs, in order to characterize the field as a whole; and to summarize the research on the effects of family support programs using meta-analytic techniques. The coding system developed for the meta-analysis accommodates both of these needs by means of a hierarchical coding system. Exhibit A2-2 shows the five levels of the coding system and describes the information that is coded at each level. At the "lowest" level of the coding system is the individual finding (contrast), which is the datum on which the meta-analysis is conducted. Higher levels of the coding system capture the study characteristics and the characteristics of the intervention, which may be used to explain differences in effects or to characterize the field of family support programs and research.
At the base of this coding system is a specific contrast: numerical data describing the level of performance of two conditions (groups) on a specific measure at a specific time point. For example, a contrast might be the posttest means for a specific treatment and control group in a particular study on a child measure administered when the children were 12 months of age. The hierarchical coding system means that each contrast is linked to a large amount of information on (1) the measure itself, (2) the two groups being compared, (3) the study and its methodological characteristics, and (4) the program model being evaluated.
The actual coding scheme was developed by researchers familiar with the research domain of family support. In developing the coding categories, we also consulted with experts in the area of research synthesis methods, who advised us about which study characteristics should be coded. A copy of the coding form and instructions can be found in Appendix B.
|
Computing Effect Sizes
In the research studies that we coded, the numerical data on which a contrast is based are presented in many different ways--as means, proportions, t-statistics, probability levels, etc. In our coding system, the data were recorded in the form that they were presented and were then converted into an effect size using specialized software developed expressly for this purpose by Dr. William Shadish. This software takes data expressed in 40 different forms and computes a standardized effect size using the appropriate statistical method. Appendix C briefly describes the forty methods handled by the software.
The Coders and the Coding Process
There were two separate coding activities. The first was the coding of the research articles in terms of the intervention itself, the study methodology, and the findings. The second activity was to rate each program in terms of its adherence to the family support practice principles. The research articles were coded by a small group of senior staff. Each coder was trained individually or in a small group by the senior researcher who developed coding protocol. The training took approximately six hours, and, at the end of the training, coders were required to complete a practice protocol that was checked for accuracy against a master coding. After the training, all articles were double-coded. Two coders completed a coding protocol independently, and then the two protocols were compared. Differences were discussed and, if necessary, given to the trainer for adjudication. An agreed-upon protocol was then entered into an Access database, to be converted into a SAS database for analysis. The exception is the outcome data (i.e., the actual numerical means, etc), which were entered separately into the program that computed the effect sizes.
Rating Adherence to Family Support Practice Principles
Each program or intervention studied was rated in terms of its adherence to family support principles. This rating was based on all available written information about the program. The rating system itself was developed in collaboration with Dr. Sharon Lynn Kagan of Yale University. The rating scale is discussed in Chapter A3. The actual rating of the programs in the meta-analysis was done by a small number of senior staff who are knowledgeable about family support programs and principles.
1 In addition, we obtained over 60 research reviews that summarize multiple studies on early childhood education (25 reviews), family support and parenting education (9 reviews), parent involvement (3 reviews), two generation programs (2 reviews), home visiting programs (4 reviews), Head Start (12 reviews), and child mental health programs (6 reviews). (back)
| Table of Contents | Previous | Next |