Evaluation of Family Preservation and Reunification Programs:

Chapter 8:
The Outcomes

[ Main Page of Report | Contents of Report ]

Contents

• 8.1 Substitute Care Placement Following Random Assignment
• 8.2 Hazard Analyses of Placement
• 8.3 Allegations of Maltreatment Following Random Assignment
• 8.4 Sub-group analysis
• 8.5 Case Closing Subsequent to Random Assignment
• 8.6 Family and Child Functioning–Caretaker Interviews
• 8.6.1 Life Events
• 8.6.2 Problems
• 8.6.3 Economic Functioning
• 8.6.4 Household Condition
• 8.6.5 Child Care Practices
• 8.6.6 Caretaker Depression
• 8.6.7 Child Behavior
• 8.7 Overall Assessment of Improvement by Caretakers
• 8.8 Information from Caseworkers on Family and Child Functioning
• 8.8.1 Caretaker Functioning
• 8.8.2 Household Condition
• 8.8.3 Caretaker Problems
• 8.8.4 Child Problems
• 8.9 Predictors of Outcomes
• 8.10 Summary of Outcome Data
• Endnotes

The outcomes we examined were the placement of children in substitute care following random assignment to the experimental or control group, subsequent reports of maltreatment and a number of measures of child and family functioning. The focus is on comparisons between the experimental and control groups. Analyses we have designated as "primary" were conducted on all randomly assigned cases except those that were determined to be inappropriate referrals. This includes cases in which the assignment was violated (cases assigned to the control group that were given family preservation services) and cases assigned to family preservation that received no or little such service. Insofar as family preservation services have effects, inclusion of these cases in the analysis will tend to reduce the observed differences between the groups. However, the most rigorous approach to analysis requires that we retain these cases in the group to which they were assigned in order to maintain the statistical equivalence of the groups at the outset of the experiment, which is the reason for random assignment in the first place.

It is likely that violations and minimal service cases differ in systematic ways from other cases (perhaps not detected in the measurements of the study), hence, switching them to the other group would result in groups that were not equivalent at the beginning. It can be argued that inclusion of minimal service cases in the experimental group is quite proper on other grounds: the implementation of any program will involve some cases that do not receive the service, and estimates of impact ought to take that into account. We did conduct analyses ("secondary" analyses) in which the violations and minimal service cases were dropped, so as to examine differences between cases that actually received the intended treatment (family preservation or regular services). This analysis must be viewed as only suggestive, since it does not preserve the initial statistical equivalence of the groups created by random assignment. In fact, the results of the secondary analyses were usually similar to those of the primary analysis. We note in footnotes when the secondary analysis differed substantially from the primary analysis. The secondary analysis tables are presented in Appendix I.

Some analyses were also conducted on a more "refined" sample in which we attempted to focus on cases that approached a conception of "ideal" family preservation cases. Family preservation services are designed for families in crisis, presumably the crisis surrounding a recent allegation of maltreatment, the investigation of that allegation, and the threat of removal of a child. Theoretically, this state of crisis makes families more willing to seek and respond to help. As indicated in Chapter 6, many of the families did not appear to conform to this specification of the target group. In Kentucky and New Jersey we looked at two subgroups of cases, those with a recent substantiated allegation recorded in the administrative data (within three months prior to referral to family preservation services) and those in which an investigative worker was involved. In Tennessee, nearly all of the cases came from investigating workers, so we looked at those cases with an allegation within 30 days prior to referral.

[ Go to Contents ]

8.1 Substitute Care Placement Following Random Assignment

A principal goal of family preservation services is the prevention of placement into substitute care, so that must be the first (though not the last) outcome examined. Placement included foster care, institutions and residential treatment programs, group homes, and adoptive placements.⁽¹⁾ We are initially concerned with the character and timing of the first placement of a child following random assignment. We collected data on placement prior to August 31, 1998 in Kentucky and New Jersey and before August 31, 1999 in Tennessee.⁽²⁾ Although data were provided at the individual level, most of the analyses are presented at the family level.⁽³⁾ In Kentucky, the administrative files contained data on 942 children in 306 families, 154 in the experimental group and 152 families in the control group. Eighty-six children in 40 families (26%) in the experimental group experienced placement subsequent to random assignment compared to 60 children in 41 families (27%) in the control group. In New Jersey, administrative data were available on 1230 children in 434 families, 269 in the experimental group and 165 in the control group. One hundred sixteen children in 81 families (30%) in the experimental group were placed compared to 44 children in 39 families (24%) in the control group. In Tennessee, multiple sources of data were used to calculate the rate of subsequent placement. A statewide management information system (CORS) provided information on formal paid placements. Additionally, case record reviews provided information on unpaid relative placements. In Tennessee, placement data were available on 468 children in 140 families, 93 in the experimental group and 47 in the control group. In the analysis of CORS data, forty-six children in 23 families (25%) in the experimental group experienced placement subsequent to random assignment compared to 25 children in 10 families (21%) in the control group. Including unpaid relative placements, 60 children in 29 families (31%) in the experimental group experienced placement subsequent to random assignment compared to 31 children in 13 families (28%) in the control group. These differences were not statistically significant at the family level in Kentucky, New Jersey or Tennessee (see Table 8-1 for types of placements after random assignment).

A comparison of these percents is, however, misleading, because of varying periods of risk of placement. The proper approach to the analysis of such data is survival analysis, in which the proportions of cases placed at each point in time following random assignment in each group are compared, accounting for the numbers of cases that "survive" to that point. We examined survival curves for each group and determined whether these curves were statistically different. Family level analyses were based on the first date of placement of any child in the family if a placement occurred.

Kentucky. The family level analysis of subsequent placement is displayed in Figure 8-1.⁽⁴⁾ These survival curves show the proportion of families remaining intact (without placement of a child) at each point in time following random assignment. The curves begin at 1, indicating that at the time of random assignment, all children were at home. The curves then decline as children enter care. The higher curve at any point represents the group with fewer placed children at that point. The curves are adjusted for cases that are "right censored." For example, cases that were not observed for a full year following random assignment are dropped in the calculation of the percentage remaining intact ("surviving") at one year. The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. At the one-year interval, 23 percent of experimental group families and 24 percent of control group families experienced substitute care placement.

Figure 8-1 First Placement after random assignment (families)

"Refined" groups analyses were also conducted, limiting the sample to cases referred by investigative workers and to those families with substantiated allegations within the three months prior to random assignment. No statistically significant differences emerged. In the primary analysis of those families coming from an investigative worker, 26 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year after the random assignment date. For those with recent substantiated allegations, 29 percent of the experimental group and 16 percent of the control group experienced subsequent placement within one year.

An additional "refined" group was available for analysis in Kentucky. Prior to random assignment, workers submitted petitions to the court for placement or some other court ordered intervention on 67 families. Administrative data were available for 60 of these 67 families (29 in the experimental group, 31 in the control group). Survival analyses were conducted to explore the relationship between family preservation services and subsequent placement. At one year after random assignment, 18 percent of the experimental group and 33 percent of the control group experienced placement; a nonsignificant difference.

In addition to the administrative data on placement, in Kentucky the Westat site coordinator attempted to document all placements subsequent to random assignment, based on her contacts with caseworkers. The administrative data file contained placements not recorded by the site coordinator, and vice versa. The only systematic difference between these data sources was the documentation of relative placement. Relatives are generally not paid for placements in Kentucky, so these data were not recorded in the administrative files. Survival analyses were conducted with a combination of caseworker and administrative placement records. If either data source recorded a placement event, that family was coded as experiencing subsequent placement. The first documented date of placement, taken from either source, was selected for analysis. The patterns of placement in these analyses are similar to those reported above. At one year, 27 percent of the experimental group and 32 percent of the control group families experienced placement, a nonsignificant difference.

In addition to survival analyses, placement can be examined in terms of the proportion of time in substitute care subsequent to random assignment. If family preservation services are effective in preventing placements, we would expect them to result in lower numbers of days in foster care. Family preservation might also result in shorter stays in care, once children are placed. Comparison of days in care provides a beginning look at the question of whether family preservation results in lower costs of foster care (of course, a complete cost-effectiveness analysis must also factor in the differential costs of family preservation and regular services).

The proportion of time in care is calculated by dividing the number of days in care by the number of days of possible care (number of days between random assignment and the date of administrative data collection). As the proportions are calculated at the family level, the number of days in care represents the total number of care days summed across all children within a particular family. Similarly, the number of possible care days represents the total number of possible care days summed across all children within a particular family. The number of possible care days is adjusted for a child's eighteenth birthday and for births since random assignment. For both primary and secondary analyses, in both the experimental and control groups children spent an average of 6 percent of the days subsequent to random assignment in care.

New Jersey. The family level analysis of placements is shown in Figure 8-1.⁽⁵⁾ More families in the experimental group experienced placement of a child than in the control group (at one year, 28% of the experimental group vs. 22% of the control group) although the differences are not significant. It might be noted that in the analyses the survival curves for the two groups tend to begin to diverge at about 2-3 months, that is, at about that time more children in the experimental group are being placed. We do not have a ready explanation for this divergence.

Refined groups analyses in New Jersey revealed no statistically significant differences. In the primary analysis of those families coming from an investigative worker, 25 percent of the experimental group and 16 percent of the control group experienced subsequent placement within one year of the random assignment date. For those with recent substantiated allegations, 27 percent of the experimental group and 15 percent of the control group experienced subsequent placement.

As to the proportion of time that children spent in care in New Jersey, experimental group children spent an average of 6 percent of that time in placement, compared to 4 percent for the control group children (not a significant difference).

Tennessee. Survival rates at the family level were first calculated using only the CORS and then including relative placement (the "any evidence" analysis) data. The family level analyses of subsequent placement is displayed in Figure 8-1.⁽⁶⁾ The Wilcoxon statistic indicates that the survival rates for the experimental and control groups are not statistically different. In the analysis of CORS data, 23 percent of experimental group families and 19 percent of control group families experienced substitute care placement within one year subsequent to random assignment. In the "any evidence" analysis, 28 percent of the experimental group families and 23 percent of control group families experienced placement within one year subsequent to random assignment.

As in Kentucky and New Jersey, a "refined" group was available for analysis in Tennessee. Ninety-three families had an allegation within 30 days prior to random assignment. The Wilcoxon statistic for the survival analysis of placement in these families indicates that the survival rates of the two groups are not statistically different. In the analysis of CORS data, 17 percent of the experimental group and 15 percent of the control group experienced subsequent placement within one year of random assignment. In the "any evidence" analysis, 22 percent of the experimental group and 21 percent of the control group experienced subsequent placement within one year of random assignment.

As to the proportion of time that children spent in care in Tennessee, experimental group children spent an average of 10 percent of that time in placement, compared to 5 percent for the control group children. This difference is nonsignificant.

In a number of analyses of subsequent placement in these states, more experimental group families experienced placement than did control group families. In a few analyses, fewer experimental group families experienced placement. However, none of these analyses were statistically significant; in none of these states can the data be taken as firm evidence that family preservation resulted in more placements. Nor is there evidence that it resulted in fewer.

Imminent risk of placement. The family preservation programs in these states are designed to prevent the unnecessary removal of children by serving families with children who are at imminent risk of out-of-home placement.⁽⁷⁾ One way to explore the accuracy of the "imminent risk" designation is to examine the proportion of control group families that experienced placement within a short time after random assignment. Since the control and experimental groups were randomly assigned and are expected to be statistically equivalent before services are begun, the proportion of families experiencing placement in the control group indicates the proportion of referred families that would have experienced placement in the absence of receiving family preservation services. We looked at control group placement rates 30 days after random assignment, believing that time period provided a liberal interpretation of "imminent risk." If a significant proportion of the control group experienced placement within 30 days of random assignment, one could argue that the program was appropriately targeted. At the time of random assignment, referring workers were asked to designate those children who were considered "at risk."

In Kentucky, in the first 30 days following random assignment, in the primary analysis 4 percent of at risk children in the experimental group were placed compared to 3 percent of control group at risk children. At the family level, 6 percent of the experimental group families and 5 percent of the control group families experienced placement within the first 30 days subsequent to random assignment. The percentages were similar in the investigative group (8 percent of the experimental compared with 5 percent of the control group), and among those with recent substantiated allegations (6 percent of the experimental group compared with 3 percent of the control group).

In New Jersey, of those children judged to be at risk, 4 percent of the control group and 3 percent of the experimental group were placed in 30 days. At the family level, 4 percent percent of the families in the experimental group experienced placement of at least one child within one month of random assignment, compared to 6 percent of the control group. Rates of imminent placement were similar in the "refined" group analyses. Of those families coming from an investigative worker, 3 percent of the experimental group and 5 percent of the control group experienced placement within 30 days of random assignment. For those families with a substantiated allegation within three months prior to random assignment, 8 percent of the experimental group and 5 percent of the control group experienced placement within 30 days.

In Tennessee, rates of placement within one month were somewhat higher than in Kentucky and New Jersey. Of those children judged to be at risk, 13 percent of the control group and 11 percent of the experimental group were placed in 30 days. There were no relative placements within the first 30 days subsequent to random assignment. Thus, there are no differences between the CORS and "any evidence" analysis. At the family level in Tennessee, the CORS administrative data indicates that 11 percent of both the experimental and control groups experience placement within 30 days subsequent to random assignment. Rates of imminent placement were similar in the "refined" group analyses. Of those families with a recent allegation (within 30 days prior to random assignment), 7 percent of the experimental group and 12 percent of the control group experienced a CORS placement within 30 days of random assignment.

Although the percentages of placement within one month were somewhat higher in Tennessee, in all three states, these percentages were quite low. The numbers of interest here are those for the control group, indicating the targeting efficiency of the program in these three sites is very low.

[ Go to Contents ]

8.2 Hazard Analyses of Placement

Hazard analyses permit the examination of the effects of multiple independent variables (in addition to experimental group membership) on rates of placement. They also provide somewhat more precise estimates of the effect of experimental group membership, since they control for the effects of the other variables in examining experimental-control group differences. In addition, they allow for the examination of "interactions" between other variables and experimental-control group membership, to see if the effects of experimental group membership differ for subgroups of the sample. We conducted Cox regression analyses of placement hazards using as predictor variables case characteristics available in the administrative data. Case characteristics in the administrative data are quite limited. Unlike the survival analyses which were conducted at the family level, hazard analyses were done at the child level because we wanted to include in them characteristics of children.

Kentucky. We examined the effects of the child's age, prior placement, and prior allegation of maltreatment, together with experimental group membership, on rates of placement. Only prior allegation was significant, increasing the hazard rate by 68 percent. Interactions of the three variables with experimental group membership were not significant.

New Jersey. New Jersey hazard analyses indicate that older age and prior placement increase the hazard rate significantly (p < .05; prior placement by 77% and each year of age by 8%). Minority status is nearly significant (p = .08), increasing the hazard rate by 37 percent. In an equation with these three variables, experimental group status shows a significant effect in increasing the hazard rate of placement (since this is a child level analysis, significance levels are downwardly biased due to clustering effects). We also examined the interaction of age, prior placement, and minority status with experimental group. The interactions for age and minority status were not significant (that is, the effect of experimental group did not differ by age or by minority status). The interaction with prior placement was close to significant in some of the equations we tried. Examining the differences in placement rates between the experimental and control groups by whether or not the child had had a prior placement indicates that among those with a prior placement, there is little difference in placement rates (23% for the family preservation group and 22% for the control group) while there is a significant difference for those without prior placement (13% for the experimental group and 7% for the control group). (It should be noted that this simple three-way cross tabulation does not account for varying lengths of the observation period, which is controlled in the hazard analysis.) This suggests that children without prior placement who receive family preservation are more likely to be placed than if they did not receive family preservation.

The results do not, however, suggest that family preservation is better for cases with more prior exposure to the child welfare system.⁽⁸⁾

Hazard analyses were also performed to examine the effect of county on placement. These analyses were conducted at the family level. Burlington county was chosen as the reference category, as it had the highest rate of placement. Thus, rates of placement in the other New Jersey counties are compared to the placement rates of Burlington. In addition to the county variables, experimental group and interactions of county with experimental group were entered into the regression equation. The hazard of placement for families was decreased by 42 percent for Camden county, 71 percent for Ocean county, 78 percent for Monmouth county, 61 percent for Essex county, 71 percent for Bergen county, and 75 percent for Passaic county. There were no significant effects of experimental group or of county-experimental group interactions. This indicates that even after removing county variation, there are no significant differences between the experimental and control groups, nor does the effect of experimental group vary by county.

Tennessee. We examined the effects of the child's age, race, prior placement, prior allegation within 30 days of random assignment, prior substantiated allegation within 30 days of random assignment, and experimental group membership on rates of placement. Similar to Kentucky and New Jersey, we also explored interactions between experimental group membership and child characteristics. No significant interactions emerged. Only prior substantiated allegation had a significant effect on the likelihood of placement subsequent to random assignment. In the analysis of the CORS administrative data, a substantiated allegation within the last 30 days prior to random assignment increased the hazard rate by 209 percent. When unpaid relative placements were included ("any evidence") prior substantiation increased the hazard by 173 percent.

[ Go to Contents ]

8.3 Allegations of Maltreatment Following Random Assignment

Subsequent maltreatment of children is a second important outcome to be examined. Family preservation programs are intended to lower the risk of harm to children while keeping them at home, and subsequent maltreatment is an indicator of such risk. Furthermore, the justification for family preservation programs rests on the belief that the safety of children is not compromised when their families are referred to these programs, so examination of subsequent maltreatment rates is important to determine whether children, in fact, are safe in these programs.

As with placement, data on subsequent maltreatment come from the administrative data files of the states. As is almost always the case in studies like this, our data do not record actual maltreatment, but only investigated reports of maltreatment. Some abuse and neglect goes unreported, and, because not every report is investigated, there are cases of harm that are reported but not investigated.

As with the analyses of subsequent placement, survival graphs were developed to compare the timing of subsequent, substantiated allegations of maltreatment.⁽⁹⁾ Again, survival analyses were conducted at the family level for both the primary and secondary analysis groups as well as for the "refinement" groups.

Kentucky. One hundred thirteen children in 51 families (33%) in the experimental group were the subjects of investigated allegations of maltreatment following random assignment, compared with 103 children in 47 families (31%) in the control group. The differences were not statistically significant at the family level. The distribution of the various types of allegations is as follows: 4 percent dependency, 8 percent emotional, 49 percent neglect, 43 percent physical abuse, and 16 percent sexual maltreatment. As families can be the subjects of multiple allegations on any given day, these percentages do not sum to 100. Not all investigations result in substantiated allegations. Sixty-six children in 30 families (20%) in the experimental group were the subjects of substantiated allegations of maltreatment compared with 68 children in 28 families (18%) in the control group. These differences were not statistically significant at the family level. The distribution of substantiated allegations is as follows: 6 percent dependency, 2 percent emotional, 55 percent neglect, 38 percent physical abuse, and 10 percent sexual maltreatment.

Figure 8-2 displays the survival curves for substantiated allegations in the primary analysis. At one year subsequent to random assignment, 19 percent of the experimental group and 18 percent of the control group families experienced substantiated reports of maltreatment. Although a higher percentage of families in the "refined" analyses experienced substantiated allegations of maltreatment, similar patterns emerged. For the investigative group, 23 percent of both the experimental and control groups experienced substantiated allegations of maltreatment in the one-year interval. For those families with a substantiated allegation within the three months prior to random assignment, 22 percent of the experimental and 24 percent of the control group experienced substantiated allegations of maltreatment within a year subsequent to random assignment. For the group on which petitions had been submitted to court for placement or other orders, 19 percent of the experimental group and 32 percent of the control group experienced a substantiated allegation within one year subsequent to random assignment, a nonsignificant difference.

The survival analyses indicate that experimental and control group families had a very similar likelihood of substantiated reports of maltreatment subsequent to random assignment.

New Jersey. One hundred fifty-four children in 81 families (30%) in the experimental group were the subjects of investigated allegations of maltreatment following random assignment, compared to 80 children in 48 families (29%) in the control group. In none of the survival analyses conducted were there significant differences between the experimental and control groups. Within one year, about 29 percent of the families had an allegation of maltreatment concerning at least one child in the family. Figure 8-2 shows substantiated allegations at the family level. About 12 percent of families in both groups have substantiated allegations within one year.

Patterns of substantiated allegations were similar for the "refined" group analyses, none of which showed significant differences between groups. Of those families coming from an investigative worker, 7 percent of the experimental group and 10 percent of the control group had a substantiated allegation within one year subsequent to random assignment. For those families with a substantiated allegation within three months prior to random assignment, 10 percent of the experimental group and 17 percent of the control group had a substantiated allegation within one year subsequent to random assignment.

Tennessee. Allegation data were available for 482 children in 144 families. Sixty-four children in 36 families (38%) in the experimental group were the subjects of investigated allegations of maltreatment following random assignment, compared with 61 children in 26 families (54%) in the control group. The differences were not statistically significant at the family level. The distribution of the various types of allegations is: 66 percent physical abuse, 20 percent supervision/neglect, 2 percent sexual abuse/medical, and 12 percent other (includes allegations such as failure to thrive, truancy, and unruly child). Forty-four children in 25 families (26%) in the experimental group were the subjects of substantiated allegations of maltreatment compared with 42 children in 18 families (38%) in the control group. These differences were not statistically significant at the family level. The distribution of the various types of substantiated allegations is: 66 percent physical abuse, 20 percent supervision/neglect, 1 percent sexual abuse/medical, and 13 percent other.

Figure 8-2 displays the survival curves for substantiated allegations in the primary analysis. At one year subsequent to random assignment, 24 percent of the experimental group and 25 percent of the control group families experienced substantiated reports of maltreatment.

Figure 8-2. First substantiated allegation after random assignment (families)

Survival rates were also calculated for those families with an allegation within 30 days prior to random assignment. Significant differences emerged for subsequent allegations and near significant differences emerged for subsequent substantiated allegations. Of those families with a recent allegation, 28 percent of the experimental group and 52 percent of the control group experienced an allegation within one year subsequent to random assignment. Similarly, 18 percent of the experiment group and 30 percent of the control group experienced a substantiated allegation within one year subsequent to random assignment. These differences suggest that in Tennessee family preservation reduced the likelihood of subsequent maltreatment for those families with recent allegations.

[ Go to Contents ]

8.4 Sub-group analysis

In Kentucky and New Jersey, we examined a number of subgroups of cases to determine whether we could detect differences between experimental and control groups on placement and substantiated allegations subsequent to random assignment within each subgroup. The number of cases in Tennessee was not sufficient to support subgroup analysis. The results are shown in Table 8-2. Most of the subgroups were defined in terms of problems existing at the time of the initial interview. For both placement and substantiated allegations the table shows the number of cases in each subgroup, the percentage of cases in the subgroup experiencing the event within 6 months, the significance of the difference between the experimental and control groups in the occurrence of the event within six months, and the p value for the analysis of differences in survival curves. The first row of the table shows the results for the Kentucky and New Jersey samples as a whole. Except for substance abuse, the definitions of the subgroups were taken from the initial caretaker interview. Very few caretakers acknowledged substance use in the first interview, so that subgroup was determined from information in both the caretaker and caseworker initial interviews.

As can be seen, nearly all of the experimental-control group comparisons shown are not significant. Of the 36 comparisons in the table, only one is significant at .05, that for substantiated allegations in the single mother group in New Jersey.⁽¹⁰⁾ Among single mothers, those in the experimental group were less likely to have a substantiated allegation than those in the control group.

In the analysis so far, efforts to find subgroups for which family preservation service was related to reduced placement have been unsuccessful.

[ Go to Contents ]

8.5 Case Closing Subsequent to Random Assignment

Family preservation services are sometimes thought to lead to quicker case closings in the public agency and less frequent subsequent involvement with the child welfare agency. Administrative data on case closings and subsequent case openings were examined to determine the effects of these services on case closings and subsequent reopenings.

Kentucky. Of the 183 cases that were open in the public agency at the time of the referral to family preservation services, 84 (46%) were closed some time after the referral and 99 (54%) remained open as of August 31, 1998 (the last date of observation for these analyses). Survival analyses were performed to examine the lengths of time between the referral to family preservation services and the first closing of the case. As shown in Figure 8-3, no significant differences were found between the experimental and control groups.

Figure 8-3. First Case Closing After Random Assignment

Of the 84 cases that were open at the time of the referral to family preservation services and closed some time after that referral, 3 cases were re-opened again before August 31, 1998. Two of these 3 cases were in the experimental group, one having been closed for 6 days and the other for over a year before the case was re-opened. The third case was a control group case and was closed for slightly over 3 months before being re-opened.

New Jersey. Of the 434 cases with administrative data, 219 (51%) were closed subsequent to the referral to family preservation services. Figure 8-3 shows the results of a survival analysis in which we examined the time to case closing for the 421 cases that were open at the time of the referral to family preservation services. There was no significant difference between the experimental and control groups in the rate or timing of case closings after the referral date.

Of the 219 cases that were closed after the referral to family preservation services, 48 cases (22%) were reopened. There was no significant difference between the experimental and control groups in the proportion of cases that were reopened (21% in the experimental group, 23% in the control group).

Tennessee. Of the 147 families with case opening and closing data, 111 were open in the public agency at the time of the referral to family preservation services. Of these 111 families, 96 (87%) were closed some time after the referral and 15 (14%) remained open as of August 31, 1999 (the last date of observation for these analyses). Survival analyses were performed to examine the lengths of time between the referral to family preservation services and the first closing of the case. As indicated by the survival curves in Figure 8-3, no significant differences were found between the experimental and control groups. Of the 96 cases that were open at the time of the referral to family preservation services and closed some time after that referral, 17 cases were re-opened again before August 31, 1999. There was a significant difference in the rate of reopening. Eight (12%) of the 66 experimental group cases that were closed subsequently reopened, compared with 9 (30%) of the 30 control group cases (p < .05).

[ Go to Contents ]

8.6 Family and Child Functioning-Caretaker Interviews

Family preservation services are intended to result in improved functioning of children and families. This goal is sought both for its own intrinsic value as well as an intermediate objective in the prevention of subsequent maltreatment and placement; parents who are functioning better and better parent-child relationships should result in lower risk of abuse or neglect.

In our interviews with caretakers and caseworkers we asked a number of questions tapping various aspects of functioning. We asked most of these questions in all three interviews with caretakers (at the beginning of service, four to six weeks after service began, and one year after the beginning of service) and in the two interviews with caseworkers (at the beginning of service and four to six weeks later). In the initial interview, we usually asked respondents to answer in terms of circumstances in the last three months. In the post-treatment and follow-up interviews, we asked in terms of "since we last talked to you [at the time of the initial interview or the post-treatment interview]." To indicate the effects of family preservation services, we can compare the experimental and control groups on the responses to these questions in the second and third interviews and on change between interviews. We report on the responses to a number of individual items in our interviews. In addition, we combined the responses to many questions into summated scales.⁽¹¹⁾We examined differences between experimental and control groups in each state in the average levels of these scales at post-treatment and at follow-up and we examined changes over time in these averages using multivariate repeated measures analysis.⁽¹²⁾ The results of the analyses of the scales are shown in Tables 8-3, 8-4, and 8-5 and in Figure 8-4.

We report on the differences between the experimental and control groups as they were initially formed (the "primary analysis"). Results of the secondary analyses (dropping violations and minimal service cases) were usually similar. They are reported in footnotes when they were materially different.

We looked at a large number of differences between groups on functioning variables and as will be seen, some of these comparisons revealed statistically significant differences, usually favoring the experimental group. However, as will also be seen, the differences are not consistent across states or across time.

8.6.1 Life Events

In the caretaker interviews, there was a 15 item "life events" inventory asking about the occurrence of both positive and negative events in the last 3 months in the initial interview or since the previous interview in the post-treatment and follow-up interviews (Appendix K, Initial Caretaker Interview, p. 7). We formed three scales from this inventory, positive life events, negative life events, and a scale of those life events that might reflect depression in the caretaker (we had a more formal depression measure as well, described below). In none of the three states were there statistically significant differences between the experimental and control groups in the scores on these measures at the post-treatment or follow-up interviews nor were there differences between the groups in changes over time (see Tables 8-3, 8-4, and 8-5). It will be noted that generally the levels of these scales dropped in the second interview compared to the first and then increased again in the third interview. This is likely due to the differences in the time periods referenced in the questions asked at the three points in time (the last three months in the initial interview, last 4 to 6 weeks in the post-treatment interview, and last 10 to 11 months in the follow-up interview).

Figure 8-4. Child and Family Functioning over Time

[ Go to Contents ]

8.6.2 Problems

In the second and third interviews, caretakers were again asked questions about problems in the family. These questions paralleled those asked in the first interview, except this time caretakers were asked to respond to questions with regard to the time "since we last spoke to you" (see Tables 8-6 and 8-7). We analyze each of these problem items separately. Kentucky. At the post-treatment interview, on 8 of the 9 problem questions, there were no differences between the experimental and control groups in responses. On the question of whether the caretaker felt overwhelmed with work or family responsibilities, a greater proportion of caretakers in the experimental group responded affirmatively at post-treatment than did caretakers in the control group (47% vs. 37%, p = .08).⁽¹³⁾ In addition to the items about problems, caretakers were asked three questions about positive aspects of their lives: "gotten together with anyone to have fun or relax," "felt happy," and "felt that considering everything you're doing a pretty good job raising your kids." The primary analysis revealed that 89% of caretakers in the experimental group responded that they "felt happy" since the last interview as compared to 82% of caretakers in the control group (p = .09). For both the experimental and control groups, 64% responded affirmatively to the question of "getting together with anyone to have fun or relax," and over 90% responded affirmatively that they were "doing a pretty good job raising [their] kids." There were no significant differences between experimental and control groups on these last two items in either the primary or secondary analyses.

At the follow-up interview, there were no significant differences between experimental and control groups on any of these items.

New Jersey. On 8 of the 9 problem questions, there were no significant differences between the experimental and control groups in responses at post-treatment. On the overall question about the economic condition of family, "have you felt you just didn't have enough money for food, rent, or clothing?" 52% of the control group said yes, compared to 40% of the experimental group significantly different at p = .03, although in the secondary analysis the difference was not significant.⁽¹⁴⁾ On none of the positive questions were there significant differences between groups in either the primary or secondary analyses.

At follow-up differences between groups approached significance for only one item, "felt you had few or no friends." More experimental group respondents replied affirmatively to this item (24% vs. 15% p = .07).

Tennessee. There was a significant difference between groups on only one of the nine problem questions at post-treatment. Fewer experimental group respondents reported they felt they had few or no friends (19% vs. 38%, p = .03). As to positive items, experimental group respondents far more often reported they "had gotten together with someone to have fun or relax" (75% vs. 38%, p < .001). At follow-up there were no differences between groups on any of these items.

[ Go to Contents ]

8.6.3 Economic Functioning

In addition to the general item in the problem inventory on not having enough money for food, rent, or clothing, we asked four specific questions about difficulties in paying for the essentials of living (rent, electric service and heating, food, and clothes) (see Tables 8-6 and 8-7). These items were examined individually and were combined into an overall economic functioning scale. (See Tables 8-3, 8-4, and 8-5 for analyses of the scale).

Kentucky. On the scale, primary and secondary analyses revealed no significant differences in the average proportion of affirmative responses to the four items either at post-treatment or at follow-up.^(15)(16) There also were no differences between groups in changes over time.

New Jersey. On the economic functioning scale, the experimental group had a lower average proportion of affirmative responses to these items at post-treatment (.25 vs. .34, p = .02) although the difference was not significant in the secondary analysis. The difference was not significant at follow-up nor were there significant differences in change over time. There were, however, significant differences on two of the specific items at post-treatment. Control group respondents more often reported difficulties paying rent (29% vs. 18%, p = .02) and also more often reported difficulties in buying clothes (47% vs. 33%, p = .01). These differences were not significant in the secondary analysis. At follow-up, there continued to be a difference in regard to buying food, although it was not significant (26% of the experimental group vs. 35% of the control group p = .12). At follow-up, the groups were similar on the other three items.

Tennessee. Control group respondents more often reported problems on the economic functioning scale at both post-treatment and follow-up, although the differences were not significant. There were also no significant differences between the groups in change over time. On individual items, there were no significant differences between groups at the post-treatment interview. At follow-up, significantly fewer experimental group respondents reported difficulties paying rent (20% vs. 39%, p = .04). There were no significant differences on the other three items at follow-up.

[ Go to Contents ]

8.6.4 Household Condition

Caretakers were asked 10 questions about problematic conditions in the home (e.g., nonfunctioning heating, plumbing, or electrical systems; peeling paint; broken windows or doors).

Kentucky. In Kentucky, the experimental and control groups did not differ on the average proportions of the presence of such conditions at post-treatment or follow-up, nor did they differ on change in these proportions over time (see Table 8-3). On only one of the specific items were there any differences at post-treatment. Five percent of caretakers in the control group reported having broken windows or doors that were not fixed as compared to 1 percent in the experimental group (Fisher's exact p-value = .034).⁽¹⁷⁾ There were no significant differences on any of the individual items at follow-up.

New Jersey. In New Jersey the experimental and control groups did not differ on the average proportions of the presence of such conditions at post-treatment or follow-up, nor did they differ on change in these proportions between interviews (see Table 8-4). There were no significant differences between the groups on any of the ten individual items at post-treatment. At follow-up, 8 percent of the experimental group respondents reported "a lot of peeling paint" compared to 2 percent of the control group respondents (p = .04). Also, 4 percent of the experimental group and none of the control group respondents reported that cooking appliances did not work (Fisher's exact p-value = .09).

Tennessee. There were no significant differences between experimental and control groups on the overall scales of household condition at either post-treatment or follow-up, nor were there differences in change over time (see Table 8-5). At post-treatment, more control group respondents reported bare electrical wires (8% vs. 0%, p = .08; Fisher's exact p-value = .03) while more experimental group respondents reported living in an "unsafe building because of illegal acts (5% vs. 0%, p = .05). There were no significant differences on any individual items at follow-up.

[ Go to Contents ]

8.6.5 Child Care Practices

Caretakers were asked a series of yes-no questions about child care practices at the end of treatment and in the last three months at follow-up (both positive and negative). The results from these questions are shown in Table 8-8 and 8-9. In addition, three scales were formed using these items: positive child care practices (5 items), negative child care practices (10 items), and punishment (5 items, all of which are also included in the negative child care practices scale). Results from the scales are shown in Tables 8-3, 8-4 and 8-5.

Kentucky. In the post-treatment interview, "punishment for not finishing food" was the only item for which there were significant differences between the experimental and control groups, with a greater proportion of the control group responding affirmatively. There were no items on which there were significant differences at follow-up.

There were no significant differences between the experimental and control groups with regard to the positive and negative child care practice scales at the time of the second or third interviews. Both groups responded affirmatively to over 80 percent of the positive items and less than 15 percent of the negative items at both points in time. There were also no significant differences in change over time, in both groups there was a decline in the number of negative child care practices over time. With regard to the 5 items that pertain to punishment, caretakers in the experimental group responded affirmatively to a greater average proportion of punishment items than did the caretakers from the control group (.25 vs .20, p = .067) at the initial interview. At post-treatment, the average proportion of punishment items answered affirmatively were nearly the same (.17 for experimental group vs. .16 for the control group). Thus, from the first interview to the second, the reduction in the average proportion of punishment items endorsed was greater for the experimental group than for the control group (.09 fewer vs. .04 fewer, p = .054).⁽¹⁸⁾ However, across the three points in time, there were no significant differences in change.

New Jersey. On two of the items there were significant differences between the experimental and control groups at post-treatment: "have things sometimes gotten out of control when you punished your children?" happened more often in the control group and "have you encouraged your child to read a book?" which was done more often by experimental group respondents. At follow-up, there were two other items with differences: more control group respondents said they hit their child harder than they meant to (6% vs. 1%, p = .06) while more experimental group respondents said they had the children handle household chores (83% vs. 70%, p = .02).

On the scales, there were no significant differences between the experimental and control groups with respect to the positive child care practice items at either post-treatment or follow-up. At both points in time, both groups responded affirmatively to over 75 percent of the items. There were significant differences between the experimental and control groups with respect to the negative child care practice items at post-treatment. Caretakers in the experimental group responded affirmatively to 14 percent of the items whereas caretakers in the control group responded affirmatively to 18 percent of the items (p = .02). The difference disappeared at follow-up. At post-treatment, the experimental group significantly less often used punishment (.20 vs. .25, p = .04), but the difference was not statistically significant in the secondary analysis (p = .08) or in the primary analysis at follow-up (p = .15). There were no significant differences between groups in the change in changes over time in proportion of negative child care practice or punishment items.

Tennessee. At post-treatment, on one item there was a significant difference between experimental and control groups; more experimental group respondents indicated they had gone to an amusement park, pool, or picnic (68% vs. 43%, p = .01). There were no items with significant differences at follow-up. There were no significant differences in any of the three scales at either post-treatment or follow-up, nor in changes over time. The experimental group had a higher negative child care practices score at the first interview, and declined more than the control group, resulting in a nearly significant multivariate time-group interaction (p = .09) and a significant univariate comparison of the first interview score with the average of the scores from the later two interviews (p = .03).

[ Go to Contents ]

8.6.6 Caretaker Depression

In both the initial and post-treatment interviews we administered the SCL-90⁽¹⁹⁾ depression scale to measure the level of depression of the caretaker.⁽²⁰⁾ In none of the three states were there significant differences between the groups in scores on this scale at the post-treatment or follow-up interviews or in changes over time (see Tables 8-3, 8-4, and 8-5). Scores at post-treatment were, on average, less than those in the initial interview for both groups and the reduction was greater for the experimental group (Kentucky: .23 less for the experimental group and .14 less for the control group; New Jersey: .19 less for the experimental group, .09 less for the control group; Tennessee: .32 less for the experimental group, .19 less for the control group), though the differences were not statistically significant.

8.6.7 Child Behavior

In all three interviews, we asked 35 questions about specific child behaviors, both positive and negative. Questions were phrased in terms of "any of the children" and some questions were age specific. Responses to these questions were used to form various scales: aggression (3 items), school problems (5 items), positive child behaviors (10 items), and negative child behaviors (21 items, including the aggression and school problems items). Analyses of these scales are shown in Tables 8-3, 8-4, and 8-5.

Kentucky. Neither the primary nor the secondary analyses revealed any significant differences between the groups in scores on any of these scales at post-treatment or at follow-up or in the change over time. Specific items on whether the child was withdrawn, had stolen things or been arrested, or had engaged in substance abuse did not reveal significant differences between groups at post-treatment or follow-up. For the scale of having stolen things or been arrested, the experimental group scored higher at the initial interview and declined between the first and second interview, while the control group increased between the first and second interviews and again at the follow-up interview. As a result, the multivariate interaction between time and group was nearly significant (p = .07) and the univariate difference between the groups in the difference between the first interview and the average of the later two interviews was significant at p = .03.

New Jersey. In the primary analysis there were no significant differences between groups on these scales at post-treatment or at follow-up, except for the overall negative child behaviors scale at post-treatment, on which the experimental group was lower (an average of 28% of the items vs. 33%, p = .04).⁽²¹⁾ For none of these scales was there a significant difference between groups on change over time.

Specific items on whether the child was withdrawn, had stolen things or been arrested, or had engaged in substance abuse did not reveal significant differences between groups at post-treatment or follow-up or in change over time.

Tennessee. There were no significant differences between the experimental and control groups in the average scores on these scales at post-treatment or at follow-up, nor were there significant differences in change over time. The difference for positive child behaviors at follow-up was nearly significant, with the control group scoring higher (p = .07). The specific items on whether the child was withdrawn, had stolen things or been arrested, or had engaged in substance abuse did not reveal significant differences between groups at post-treatment or follow-up or in change over time.

[ Go to Contents ]

8.7 Overall Assessment of Improvement by Caretakers

In the post-treatment interview, caretakers were asked about general changes in their families' lives since entering the study. Results are shown in Table 8-10 and 8-11. At post-treatment, in Kentucky and New Jersey, relative to control group caretakers, a significantly larger proportion of experimental group caretakers generally thought there was "great improvement" in their lives. This difference was significant in both the primary and secondary analyses. In the Tennessee secondary analysis, results tended in the same direction, though not significantly (p = .09). At follow-up, differences between the groups in Kentucky and New Jersey had nearly disappeared. In Tennessee, control group respondents more often thought there was "great improvement," although it was not a significant difference.

[ Go to Contents ]

8.8 Information from Caseworkers on Family and Child Functioning

The caseworker interviews also contained questions regarding child and family functioning, in an effort to provide another perspective on these issues. In interpreting caseworker reports, it should be noted that experimental group caseworkers were Homebuilders workers, while control group respondents were the public agency workers responsible for the cases at the time of the interview. It is likely that there are differences between these groups of caseworkers in the knowledge they have of the cases, since Homebuilders workers had much more intensive involvement and that involvement began before the first research interview. In addition, it may be that there are systematic differences in these groups of workers in the approaches they take to the assessment of family problems. Hence, interpretations of comparisons between responses of workers serving each of the groups must be made with caution.

8.8.1 Caretaker Functioning

Caseworkers were asked nine questions tapping various aspects of caretaker functioning on a five-point scale from 0 for not adequate to 4 for very adequate. Table 8-12 provides a list of these nine questions and a summary of the results from the initial and post-treatment interviews.

Kentucky. At the initial interview, significant or nearly significant differences were found on three items, with the experimental group scoring more adequate on average: ability to provide food (p = .02), responding patiently to child's questions (p = .06), and attending to children's health needs (p = .08). On a scale averaging the nine ratings for each case, the difference between means of the experimental and control groups approached significance, with the experimental group having a higher mean (p = .06). At post-treatment there were no significant differences in the primary analysis. However, in the secondary analysis, caretakers from the experimental group were rated higher (more adequate) than those from the control group with respect to whether they attended to the children's health needs (p = .04). As for the scale averaging the nine ratings, no differences were found between the experimental and control groups at post-treatment. Looking at change over time, on one item, respecting child's opinions, the ratings for the control group increased over time (.19 change), whereas the ratings for the experimental group decreased slightly over time (-.06 change), a difference that is significant (p = .05). The differences between groups in change on the overall scale averaging the nine ratings was not significant.

New Jersey. At the initial interview, on two items there were significant differences between the experimental and control groups, the control group scoring more adequate on average: caretaker's ability in giving affection (p = .03) and the caretaker's ability to provide learning opportunities (p = .005). On the scale averaging the nine ratings for each case, there was a significant difference between means of the experimental and control groups, the control group having a higher mean (p = .02). At post-treatment, the control group scored higher (more adequate functioning) on the same two items as before. On the scale of nine items the control group scored slightly higher, although the difference was nonsignificant. As to change over time, on one item ("respecting child's opinions"), the control group had, on average, more positive change than the experimental group. The difference in degree of change was significant at .05 (this result also held in the secondary analysis, p = .05). Differences between groups in change on the overall scale were not significant.

Tennessee. At the initial interview there were four items on which the groups were significantly different, the experimental group scoring higher on all four: caretaker respect for child's opinions (p = .01), response to child's emotional needs (p = .04), adequate supervision (p = .005), and attending to the child's health needs (p = .03). At post-treatment, five items had differences between groups significant at .06 or lower, all favoring the experimental group: caretaker ability to provide food, respect for child's opinions, response to child's emotional needs, adequate supervision, and respond patiently to child's questions. The average of all nine items was also significantly different for the groups. On one item, setting firm and consistent limits, there was a significant difference in the amount of change over time, the experimental group increased by an average of .31, while the control group declined by an average of .29 (p = .01). On the scale of nine items there was no significant difference between the groups in change over time.

[ Go to Contents ]

8.8.2 Household Condition

As in the caretaker interview, we asked caseworkers about conditions in the home. Caseworkers were asked 13 yes-no questions, some positive and some negative. These items were combined in a scale which indicated that in Kentucky and New Jersey at post-treatment, control group families had, on average, a significantly better household condition than did experimental group families (Kentucky: p = .014; New Jersey: p = .02). In both states, for both groups the analysis of change over time indicated a slight improvement in the condition of the household. The difference between the experimental and control groups in change over time was not significant in either state. In Tennessee, there was no difference between groups at post-treatment or in change over time (both groups declined by .01).

[ Go to Contents ]

8.8.3 Caretaker Problems

Caseworkers were asked a number of questions about problems experienced by children, caretakers, or other adult household members (question 19 on the initial caseworker interview, question 17 on the post-treatment caseworker interview). Twenty-one of these problems concerned the caretakers.

Kentucky. At post-treatment, in the primary analysis, caseworkers reported that the experimental group caretakers had, on average, 31 percent of the problems compared to 25 percent for the control group, a difference significant at p = .0005.⁽²²⁾ There were no significant differences in change in caretaker problems between the interviews in either the primary or secondary analyses.

New Jersey. At the post-treatment interview, on average, in the primary analysis caseworkers reported that experimental group caretakers had 23 percent of the problems compared to 21 percent of the control group, a nonsignificant difference.⁽²³⁾ There were no significant differences in change in caretaker problems between the interviews in either the primary or secondary analyses.

Tennessee. At post-treatment, caseworkers reported that experimental group caretakers had 18 percent of the problems compared to 21 percent of the control group, a nonsignificant difference. There was a significant difference between the groups in change over time, the experimental group improving more than the control group (-.08 vs. -.03, p = .05).

8.8.4 Child Problems

Twelve of the items on the caseworker problem inventory concerned the children. In Kentucky at post-treatment, the percentage of child problems for the experimental group was, on average, 27 percent compared to an average of 25 percent for the control group, a nonsignificant difference.⁽²⁴⁾ There were no significant differences in change in child problems between interviews in either the primary or secondary analyses.

In New Jersey at post-treatment, the average of the percentages of child problems was 25 percent for the experimental group and 27 percent for the control group, a nonsignificant difference.⁽²⁵⁾ There were no significant differences in change in child problems between interviews in either the primary or secondary analyses.

In Tennessee, the average percentages of child problems in the two groups at post-treatment were very close (18% for the control group, 19% for the experimental group). The difference between the groups in change over time was not significant.

[ Go to Contents ]

8.9 Predictors of Outcomes

We performed regression analyses on a number of family functioning outcomes measured at the post-treatment interview and at follow-up. The analyses were intended to control for the effects of a number of variables, thereby providing more sensitive tests of the effects of family preservation, and to examine the effects of the variables on the outcomes. The dependent variables in these analyses were some of the scales of functioning discussed above: caretaker depression, child aggression, punishment, child school problems, difficulty paying bills, positive life events, negative life events, positive child behaviors, negative child behaviors, household condition, positive child care practices, and negative child care practices. Independent variables in these analyses were assignment group (experimental or control), caretaker's age, caretaker's race, family composition, caretaker's educational attainment, caretaker's employment status, residential stability, use of income support programs, caretaker's history of abuse and/or neglect, regular access to an automobile, and time to interview (days between random assignment and post-treatment/follow-up interview). The analyses also included the initial scores for the dependent variable, thereby controlling the level at post-treatment or follow-up for the initial value. Interactions between control variables and experimental group were also examined, only a few were found to be significant.⁽²⁶⁾

Caretaker's age, caretaker's race, family composition, caretaker's educational attainment, caretaker's employment status, use of income support programs, caretaker's history of abuse and/or neglect have all been examined in previous studies of outcomes in child welfare and have often been found to be predictive. Residential stability and regular access to a car have been less often examined. Since transportation and housing assistance are commonly provided in family preservation service models, the inclusion of such variables seems justifiable. Moreover, prior research does support a relationship between residential stability and major depression⁽²⁷⁾ and child adjustment.⁽²⁸⁾ Similarly, transportation (or lack there of) has been found to be related to participation in social programs⁽²⁹⁾ and family functioning.⁽³⁰⁾ We included time to interview because of the fact that that varied considerably and might have affected the degree of change that we were observing.

Regression analyses were conducted at the family level for both the post-treatment and follow-up measures. The coefficients are displayed in Table 8-13 and 8-14. All of the coefficients are shown for the initial measure of the outcome variable and for experimental group. Coefficients for other variables are shown if they were significant at p = .1 or lower. Most of the analyses are ordinary least squares regressions, logistic regressions were used for dichotomous or highly skewed variables. Generally, the initial measure was the strongest predictor. Although the size of these coefficients decreased between the post-treatment and follow-up interview, the majority of such coefficients remained significant. The positive direction of the coefficients indicates that caretakers with higher initial values also had higher post-treatment and follow-up values.

In regard to the post-treatment analyses, experimental group families generally had better outcomes, but the differences were significant in only three analyses. In New Jersey, the experimental group had lower depression scores and lower negative child care practices than the control group when controlled for the other independent variables. In the analysis without controlling for the other variables, the result for depression was in the same direction, but not significant (p = .08). The result for negative child care practices without the control variables was also in the same direction and significant (p = .02). In Tennessee the experimental group had fewer negative life events in the regression analysis. The difference between groups in the uncontrolled analysis was not significant.⁽³¹⁾ Three differences significantly in favor of the experimental group in the uncontrolled analyses were no longer significant in the regression analysis, all in New Jersey: caretaker use of punishment, negative life events, and positive child behaviors.

At the follow-up interview, the regression analysis indicates that family preservation clients had lower levels of child aggression in Tennessee, fewer school problems in Kentucky, and fewer problematic conditions in the home in Tennessee. There were no significant differences between groups in the uncontrolled comparisons.

Regarding the remaining independent variables, there was little consistency in whether or not a variable had an effect and even in the direction of the effect. The following discussion focuses on those variables significant at p = .05 or lower. At post-treatment, the variables that most often showed effects were education and the caretaker having a history of being maltreated. Caretaker education was related to three post-treatment outcomes in Kentucky. More education was associated with more punishment, more positive life events, and worse household condition. In Tennessee more education was related to fewer negative life events. In Kentucky, having a history of maltreatment was related to higher depression and more negative life events. In New Jersey, history of maltreatment was related to higher depression and children being more aggressive. Income support, ethnic minority, and caretaker employment all were predictors in 3 of the 36 post-treatment regressions. Time to interview was significant in only one of the regressions.

At follow-up, time to interview emerged as a predictor in 4 of the 36 regression equations, in all cases related to an increase (worsening). Other variables often related to outcome were caretaker age and education. In New Jersey, older caretakers had fewer positive life events and had children who were less aggressive. In Tennessee, older caretakers had children with fewer positive behaviors. In Kentucky, older caretakers had higher depression scores, less often engaged in positive child care practices, and had children with more school problems.

In New Jersey at follow-up, caretaker education was related to 3 outcomes. More education was related to lower depression, less difficulty paying bills, and more positive child behaviors. In Kentucky, more education was related to more difficulty paying bills, more positive life events, and more positive child behaviors. More education in Tennessee is related to more child school problems. In Kentucky, caretaker's employment is related to more negative child behaviors, more child school problems, and more positive child care practices. Caretaker employment in New Jersey is associated with fewer negative child care practices. Ethnic minority caretakers in New Jersey had fewer negative life events, engaged in fewer negative child care practices, and had children with fewer negative behaviors.

The 72 regression equations for post-treatment and follow-up contain a fair number of significant coefficients, but there is little consistency across states or across outcomes.

In summary, regression models were constructed to explore the relationship between caretaker demographic characteristics and experimental group and family functioning. Other than the initial value of the measures, relatively few significant relationships emerged. Moreover, these relationships were not consistent across the states. As to the effects of family preservation services, these data do not support a strong relationship between these services and better family functioning.

[ Go to Contents ]

8.10 Summary of Outcome Data

Information from the caretaker interviews, the caseworker interviews, and the administrative data were analyzed for indications of differences between the experimental and control groups subsequent to the referral to the family preservation program. Tables 8-15 and 8-16 contain a summary of those outcomes on which we found significant differences between the experimental and control groups in any state for the primary analyses (p < .05). Items in bold are those on which the experimental group had better outcomes, those in italics are those on which the control group had better outcomes.

In none of the three states were there significant differences between the experimental and control groups on family level rates of placement or case closings. Subsequent maltreatment was generally not related to experimental group membership, except for one subgroup in Tennessee. In Tennessee, in those families with an allegation within 30 days prior to random assignment, the experimental group children experienced fewer substantiated allegations than children in the control group.

In Tables 8-15 and 8-16 there are a number of child and family functioning items in which the experimental group displayed better outcomes than the control group in one of the states. It should be noted that the results have not been adjusted for the multiplicity of significance tests performed. That is, these significant items surfaced out of a large number of items and scales examined. In such a situation it is to be expected that some items will show significant differences simply by chance, so the appearance of a few significant differences should not be taken as an indication of superiority of one group over another, particularly when the results are not confirmed in more than one state. On only two items were differences found in two states: caretakers' assessment of whether goals had been accomplished and their assessment of overall change. We are inclined to believe that family preservation programs as represented in these states do result in higher assessments by clients of the extent to which goals have been accomplished and of overall change, since differences on those items were found in both states. Beyond that, we are unable to claim consistent evidence of positive effects of family preservation services.⁽³²⁾

There are a few items on which the control group had better outcomes, nearly all of them on measures provided by caseworkers. We are not inclined to read too much into these results, since experimental group caseworkers generally knew the families better and there may well have been significant differences in the ways that workers serving the two groups saw families and judged their functioning.

[ Go To Contents ]

Endnotes

(1)  The full list of New Jersey service codes that were included is: public institution, teaching family placement, para-foster care income maintenance, juvenile-family crisis shelter placement, relative placement, foster care placement, residential treatment placement, finalized adoption placement, selected adoption placement -- pending, maternity home care, group home placement, independent living, and shelter care placement. Four of these categories did not actually occur in the data: teaching family placement, para-foster care income maintenance, finalized adoption placement, and selected adoption placement -- pending. In Kentucky placement (as reflected in the variable FACTYPE), included: adoption, foster care, private institution/boarding schools, family treatment home, unmarried parent, other, children's psychiatric hospital, and foster care medically fragile. The data did not include adoption, family treatment home, and unmarried parent. In Tennessee, placements included: foster care, relative home, trial home, residential care, continuum contract, non-relative home, adoptive home, runaway, shelter, independent living, and detention.  [Back To Text]

(2)  Cases entered the study at varying points in time. In Kentucky, cases entered between May 7, 1996 and February 13, 1998; in New Jersey, cases entered between November 6, 1996 and February 26, 1998; and in Tennessee, between November 19, 1996 and May 26, 1998.  [Back To Text]

(3)  There are two reasons for focusing on family level analyses. First, we are not confident that the administrative data allow for accurate identification of children to be included in the risk pool (what would be the denominator in a rate of placement calculation). Children are identified as belonging to a family through a case number. The analysis requires that we identify children who are in the home at the time of random assignment (or who are born or return to the home subsequently). In these states, children apparently often retain a family case number even when they are not in the home, and the administrative data do not allow us to verify the location of the child at the time of random assignment (or even sometimes at the time of an event such as placement). This problem is alleviated in analyses at the family level, since we know that the family is at risk of having a child placed (as long as there are any children in the family).

As to the accuracy of the "numerator" in our analyses, we focus on the first event (e.g., placement) in the family, subsequent to random assignment. It is possible that the first event occurs with regard to a child identified with a family but not living in that family at the time of the event. We judge the likelihood of that occurring to be small (the effects of this source of error would be similar in a family and child level analysis). In addition, subsequent events involving other children identified with the family but not in the family at the time of the event would not affect the family level analysis, while they would create inaccuracies in a child level analysis.

The second reason for focusing on the family level has to do with a "clustering" effect in the child level analysis. Clustering refers to the lack of independence between children within the same family of observations of such things as placement. If one child is removed from the home, the remaining children are more likely to experience placement. The "clustering effect" leads to an underestimate of the significance levels when analyses are conducted at the child level. Conducting the analyses at the family level is one approach to resolving this dilemma.

We did conduct a few analyses at the child level, when we wanted to take into account child characteristics, but it should be remembered that significance levels in those analyses are downwardly biased. [Back To Text]

(4)  In Kentucky, the ratio of assignment to experimental and control groups was 50-50. [Back To Text]

(5)  In New Jersey, approximately 60% of the cases were assigned to the experimental group. [Back To Text]

(6)  In Tennessee, approximately two-thirds of the cases were assigned to the experimental group. [Back To Text]

(7)  Kentucky policy specifies that imminent risk includes children who are at risk of commitment as dependent, abused, or neglected; who are identified through the Regional Interagency Council, an interdepartmental unit, as severely emotionally disturbed; or whose families are in conflict such that they are unable to exercise reasonable control of the child. Both the referring worker and family members shall believe that without immediate intensive intervention, out-of-home placement is imminent. At the time of this study, New Jersey targeted family preservation services for families at imminent risk of having at least one child enter placement. The referring worker must have based the assessment of imminent risk on a face-to-face interview with the family no more than 5 days prior to the referral. The family must need services immediately and the worker must determine that other, less intensive, services have been used, are not appropriate, or are not available. In Tennessee, CPS intake workers complete a risk assessment form to identify high, intermediate, low, or no risk. High risk cases are identified as cases where "the child or children in the home are at imminent risk of serious harm if there is no intervention in the situation." A typical high risk case might involve such factors as: 1) a vulnerable child; 2) a history of previous maltreatment; 3) an active perpetrator who has continued access to the child; and 4) no available support or family strengths to offset the stated risks. [Back To Text]

(8)  When a variable representing the designation of a child at risk is introduced into the equation, it has highly significant effects in predicting placement. [Back To Text]

(9)  Analyses were also done on all allegations, whether substantiated or not. The results were very similar, although, of course, rates for all allegations were higher. [Back To Text]

(10)  The six months analyses and survival analyses are obviously not independent. [Back To Text]

(11)  Often we used average responses or proportions of positive responses rather than sums of responses to items. This was done in order to have scores for individuals when there were a few missing items on the scales. If an individual had too many missing items (usually 1/3rd or more) the score was declared missing. Rules for the calculation of all scales are given in Appendix J. [Back To Text]

(12)  In multivariate repeated measures analysis, three main hypotheses are tested, first, that the scores for the experimental group, averaged over the three points in time are equal to those of the control group, (the "group" hypothesis); second, that the averages of the groups at each point in time are the same (the "time" hypothesis); and third, that there is no interaction between time and group. It is the third hypothesis that is central, indicating whether the groups change in different ways  [Back To Text]

(13)  This difference was slightly greater and statistically significant in the secondary analysis (48% vs. 35%, p= .04). [Back To Text].

(14)  In the secondary analysis, fewer experimental group respondents reported health problems (12% vs. 21% for the control group, p = .04).  [Back To Text]

(15)  The control group had a slightly lower average proportion of affirmative responses to these items at post-treatment (.17 vs. .22, p = .16). [Back To Text]

(16)  In the primary analysis, at post-treatment, a greater proportion of the experimental group reported difficulties paying rent (20% vs 13%, p = .13) and electric or heat bills (28% vs. 20%, p = .11). In the secondary analysis, differences were smaller and p-values for both items were above .20. [Back To Text]

(17)  This difference was maintained but not significant in the secondary analysis (5% vs. 1%, Fisher's exact p-value = .077). [Back To Text]

(18)  In the secondary analysis, there was again a .09 reduction in the average proportion of punishment items endorsed by the experimental group and a .04 reduction for the control group (p = .03).[Back To Text]

(19)  Derogatis, L. R., Lipman, R. S., & Covi, L. (1973) SCL-90: An outpatient psychiatric rating scale -- preliminary report. Psychopharmacology Bulletin, 9 (1), 13 - 28 [Back To Text]

(20)  Reliability analysis yielded a Cronbach's alpha of .92 at time one and .93 at time two in Kentucky and .95 at time one and .94 at time two in New Jersey. [Back To Text]

(21)  This difference was also significant for the secondary analysis (28% vs. 33%, p = .006). [Back To Text]

(22)  In the secondary analysis, the difference was maintained and remained significant (31% vs. 24%, p = .0004). [Back To Text]

(23)  In the secondary analysis, the average percents were 24% for the experimental group and 21% for the control group (p = .06). [Back To Text]

(24)  In the secondary analysis, however, the difference increased and approached significance with 29% for the experimental group and 24% for the control group, p = .06. [Back To Text]

(25)  The difference for the secondary analysis was also not significant (25% vs. 28%, p = .12). [Back To Text]

(26)  The significant interactions with experimental group were as follows. For depression at post-treatment in New Jersey, there was an interaction of experimental group with single motherhood; for single mothers, there was no relationship between experimental group and depression, for other caretakers, the control group had higher depression scores. Also for depression at post-treatment in New Jersey, there was an interaction with employment; for those employed at the initial interview, there was no difference between the experimental and control groups, for those unemployed, the control group had higher depression scores. For negative life events at post-treatment in Tennessee, there was an interaction with income support; for those not receiving income support the control group had more negative life events, for those receiving income support, there was no difference between the experimental and control groups in negative life events. For household condition at follow-up in Tennessee, there was an interaction between age of caretaker and experimental group; in the control group there was no relationship between age and household condition while in the experimental group, older caretakers had worse household conditions. [Back To Text]

(27)  Brown, D., Ahmed, F., Gary, L., & Milburn, N. (1995) Major depression in a community sample of African Americans. American Journal of Psychiatry 152(3), March 373-378. [Back To Text]

(28)  Humke, C. & Schaefer, C. (1995) Relocation: A review of the effects of residential mobility on children and adolescents. Psychology; a quarterly journal of human behavior, 32(1), 16-24. [Back To Text]

(29)  Honig, A. & Pfannestiel, A. (1991) Difficulties in reaching low-income new fathers: Issues and cases. Early Child Development &z Care 77, 115-125. [Back To Text]

(30)  Baxter, A., & Kahn, J. (1999) Social support, needs and stress in urban families with children enrolled in an early intervention program. Infant-Toddler Intervention 9(3), September 239-257. [Back To Text]

(31)  The differing results for the uncontrolled analysis and the regression analysis may be due to the significant interaction in the regression equation of experimental group and income support. [Back To Text]

(32)  The reader is reminded of the findings reported in Chapter 7 indicating that experimental group caretakers generally had more positive views of service and of their relationships with workers than control group caretakers. [Back To Text]

Where to?

Top of Page
Contents
Main Page of Report

Home Pages:
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation (ASPE)
U.S. Department of Health and Human Services (HHS)

Updated: 03/06/01