Evaluation Methodologies

Evaluation Methodologies

As with all waiver demonstrations, the Children's Bureau requires States with flexible funding waivers to conduct process and outcome evaluations as well as a cost analysis. Beyond a determination of cost neutrality, most States have provided limited cost data and have focused primarily on their process and outcome evaluations. Table 3 summarizes the States' approaches to evaluating their flexible funding demonstrations.

Table 3
Evaluation Designs of Flexible Funding Waiver Demonstrations

State	Research Design	Sample Size/Number Served
Indiana	Matched case comparison design. Evaluators matched each child assigned to a waiver slot with a corresponding non-waiver child, creating a comparison group of non-waiver children. Matching was based on demographic, geographic, and case-related variables.	5,259 children in the experimental group matched with the same number in the matched comparison group.
North Carolina	Comparison group design. Comparison counties selected based on size, demographics, number of title IV-E eligible children, and socio-economic status of families.	Data analyzed for 103,706 children with an initial substantiated maltreatment report and 41,585 children entering out-of-home placement for the first time between State fiscal years 1994 and 2001 in experimental and comparison counties.
Ohio	Comparison group design. 14 experimental counties; counties with demographic and caseload characteristics similar to experimental counties served as the comparison group.	Data analyzed on 18,500 children who entered out-of-home placement for the first time between 1/1/1998 and 2/28/2002. ⁵
Oregon	Comparison group design. Children divided into four non-equivalent groups according to the availability of waiver and/or System of Care (SOC)⁶ funds during a one-year observation period: Waiver/System of Care (SOC), for children in branches that were Waiver and SOC active during the study period. Waiver/non-SOC, for children from branches that were Waiver but not SOC active. Non-Waiver/SOC, for children from branches that were SOC but not Waiver active. Non-Waiver/non-SOC, for children from branches that were neither Waiver nor SOC active.	Sample drawn from families active in child welfare services statewide between 1997 and 2000. Resulted in a combined sample of 6,748 children across the 4 comparison groups.

Back to Table of Contents

Challenges in Evaluating Flexible Funding Waivers

States with flexible funding waivers have encountered special challenges in evaluating the effects of their demonstrations on child welfare and fiscal outcomes. These challenges have arisen both from the underlying theories and assumptions of States' flexible funding models and from problems with the implementation of organizational, fiscal, and service innovations at the local level.

Theories of Change in Flexible Funding Waiver Demonstrations

In their evaluations, most States sought to test the benefits of flexible funding per se on child welfare outcomes. In other words, rather than testing a specific intervention, service, or treatment, the States hypothesized that the very availability of flexible funding at the local level would lead to better permanency outcomes for children at reduced or equal cost. This assumption, however, "skips a step" in the waiver demonstrations' theory of change by not establishing logical linkages between funding sources, services, and desired child welfare outcomes. Children to do not simply return home or otherwise achieve permanency because of the existence of flexible IV-E dollars; rather, the dollars must be used to provide a service or treatment that prevents placement or facilitates a timely exit from foster care. In its own experience, Ohio found that "flexible funding, by itself, is not the lone catalyst in achieving optimal program performance … Leadership is the imperative ingredient; the willingness of the agency to deploy the new practice tools afforded by financing flexibility."⁷ The fact that States' waiver evaluations did not link specific services or interventions to observed changes in child welfare outcomes seriously limits their ability to assess the effects of the flexible funding demonstrations.

Availability of Alternate Funding Sources

In many States, title IV-E was only one of many funding streams within the context of public funding for child welfare services. Consequently, many local governments without access to flexible funding dollars were able to implement innovative child welfare services similar to those used in jurisdictions with access to waiver dollars. The availability of alternative funding sources for new programs and services thereby attenuated observable differences in child welfare outcomes between waiver and non-waiver jurisdictions. The experiences of several States illustrate the diluting effects of other funding sources on program impacts attributable to flexible IV-E funding:

Local tax levies in Ohio, which account for more than half of the budgets of local child welfare agencies, could have been used flexibly for new programs and services; in fact, many comparison counties implemented some of the same managed care strategies as the experimental counties.
Funds available through the W. K. Kellogg Foundation and the State of North Carolina for the Families for Kids initiative provided another route to reform for counties not participating in the IV-E waiver demonstration. The availability of alternative funding streams in non-waiver counties blurred the distinction between changes attributable to the waiver and those attributable to non-waiver initiatives and programs.
Oregon's final evaluation reported that its waiver demonstration accounted for only a small fraction of child welfare services and spending in the State, making it difficult to distinguish the unique effects of waiver-related services on child and family outcomes. Oregon's evaluators concluded in their final report that the impact of the State's waiver could only be interpreted within the context of ongoing budgetary and policy issues and statewide reform efforts.

Inclusion of Populations "at Risk" of Placement

All four States with flexible funding demonstrations included children "at risk" of out-of-home placement in their target populations. However, the inclusion of children in the demonstrations who may have been in need of services, but who were not likely to be placed, diminished the likelihood that States could demonstrate placement avoidance. In all the demonstrations, it proved difficult to develop an operational definition of "imminent risk" that child welfare workers could apply consistently to their cases. This problem is not unique to the flexible funding waivers, but has also emerged in the evaluation of other child welfare programs that have attempted to serve cases identified as "at risk" of placement.

Incomplete or Inconsistent Waiver Implementation

Some States described problems with the incomplete, inconsistent, or delayed implementation of waiver activities and services in local jurisdictions, further attenuating their ability to measure the effects of flexible funding. North Carolina and Indiana, for example, both identified "less active" counties that appeared reluctant to use flexible IV-E funds for innovative services. Reasons cited by local jurisdictions for limited implementation of waiver activities included insufficient numbers of IV-E eligible children, increases in staff workloads, confusion regarding policy or practice issues, and concerns about the financial risks of paying for enhanced services with local resources beyond the end of the waiver. In its final evaluation report, Oregon noted that the time required to develop the infrastructure for implementing a statewide demonstration caused delays in the startup of waiver activities, thus limiting the time available to assess the impact of flexible funding on child welfare outcomes.

Lack of Random Assignment

Because they are generally systemic in nature, encompassing entire geographic regions and serving a wide range of children and families, flexible funding demonstrations are less amenable to the most rigorous research designs, including random assignment. None of the four States that implemented flexible funding waivers used random assignment research designs. Three States - North Carolina, Ohio, and Oregon - used comparison group designs that measured findings from counties or local child welfare districts with access to waiver funds against findings from counties or districts without access to the waiver. Although rigorous, high-quality evaluations are possible without random assignment, the use of county- or region-level comparison groups makes it difficult to distinguish the effects of the waiver from broader social, economic, and demographic forces operating in a wide geographic area.

Indiana achieved greater rigor in its evaluation by employing a matched case comparison design. This approach matched each child assigned to a waiver slot with a corresponding comparison group child according to a set of demographic, geographic, and case-related variables. This matching process using child-level data allowed Indiana's evaluators to create a comparison group with characteristics very similar to those of children receiving waiver services, thereby increasing the validity of comparisons in child welfare outcomes between waiver and non-waiver children.

Analysis of Aggregate-Level Data

The analysis of evaluation data on an aggregate, county-wide basis places further limits on conclusions regarding the effects of flexible funding waivers. Ohio noted in its final evaluation report that county-level analysis limited its ability to isolate the effects of other, non-waiver initiatives given a sample size of only 14 experimental and 14 comparison counties. The small size of many Ohio counties further complicated evaluation efforts because their child welfare caseloads were often too small for complex and rigorous analysis. In the fifth year of the waiver demonstration, the State's evaluation team used a different analytic method that simulated a "counterfactual" measure (i.e., an estimate of what would have happened had the waiver not been in place) for key child welfare outcomes. Ohio's evaluators then compared the counterfactual simulations with actual outcomes in experimental counties to estimate the waiver's effects.

Unlike Ohio, Oregon's evaluation involved an analysis of case-level data, with process and outcome data studied on a total sample of over 6,700 children. However, case samples from counties with access to flexible funds included both children who received waiver-funded services along with those who did not, thereby diluting observed differences in child and family outcomes across the experimental and comparison groups. Similarly, North Carolina used case-level data to identify differences between waiver and non-waiver counties in key child welfare outcomes, basing its analysis on tens of thousands of cases that entered the State's child welfare system between 1994 and 2001. Like Oregon, North Carolina could not identify the specific cases that received services paid for with flexible funds, thereby limiting its ability to attribute observed differences in child welfare outcomes to the availability of flexible funds.

Due to its different program design, Indiana was able to use case-level data analysis in which each child in the experimental group was matched with a comparison child using a set of demographic, geographic, and case-related variables. Using this approach, Indiana avoided some of the limitations common with aggregate-level data analysis and could draw more definitive conclusions regarding the effects of its waiver demonstration on targeted child welfare outcomes.

Back to Table of Contents

State Process Evaluations - Summary of Key Findings and Issues

As the rigor and scale of States' evaluations have differed, so have the conclusiveness of their findings to date. The following sections explore key process, outcome, and cost findings that have emerged from States with flexible funding waivers.

Participation in the Waiver

Some States implementing flexible funding demonstrations discovered that the mere availability of flexible funds was not always sufficient to encourage active participation in the waiver by counties or local child welfare agencies. In Indiana, evaluators identified a group of 25 "high use" counties that actively used waiver funds to augment child protection services. These counties expanded ongoing local initiatives, services, and programs to decrease out-of-home placements and increase the delivery of family preservation services, individual counseling, childcare and respite care, basic household assistance, and special education services. Many counties, however, made limited use of the waiver program; according to Indiana's final evaluation report, 20 percent of counties had not assigned any cases to the demonstration by the end of the second year and most were not utilizing their full allocation of slots. As noted earlier, reasons for waiver under-utilization identified by county administrators included insufficient numbers of IV-E eligible children, increases in staff workloads, problems or disagreements among waiver planning partners, and confusion regarding policies and practices related to the waiver.

In North Carolina, utilization of waiver dollars also got off to a slow start. Although its demonstration began operations in 1997 and experimental counties could access funds in local waiver trust accounts by 1998, many counties initially appeared reluctant to use these funds, with most trust account dollars spent only during the final year of the waiver. The State noted in its final evaluation report that this fiscal conservatism may have resulted from concerns about honoring funding commitments with local resources beyond the end of the waiver. Counties increased their use of flexible funds over time as they became more familiar with the demonstration, procedures for accessing flexible funds, and cost neutrality requirements.

Maintaining Cost Neutrality

The cost neutrality requirement of the waiver - the stipulation that States may be reimbursed for no more title IV-E funds than they would have received in the absence of the waiver - emerged as a challenge for counties in some States. Oregon child welfare agencies risked loss of funding for flexible services if they failed to maintain cost neutrality. According to Oregon's final evaluation report, many innovative services implemented by local child welfare branches early in the waiver demonstration were curtailed, discontinued completely, or shifted to other funding sources by the State because they failed to remain cost neutral. Oregon, which assumed responsibility for all costs for innovative services in excess of the child welfare branches' waiver allotment, placed more stringent limits on the startup or continuance of innovative services in an effort to limit its own financial risk.

Cost neutrality requirements also created risks for counties participating in North Carolina's demonstration. According to that State's final evaluation report, the expectation of remaining cost neutral may explain the reluctance of some counties to draw down waiver dollars. Although the State shared with counties any overruns in title IV-E expenditures to cover the costs of licensed foster care, the counties were required to pay half the difference between actual expenditures and the amount necessary to remain cost-neutral. For example, if the Federal matching rate for title IV-E dollars for licensed care were 63.09 percent, the remaining 36.91 percent of costs would be split evenly between the State and county, with each responsible for 18.45 percent of the cost of licensed care up to the State's foster care board rate.⁸ To offset risks to the counties, and to encourage maintenance of cost neutrality, North Carolina adopted a reinvestment strategy that established a trust fund for each county; through the trust funds, cost neutral counties could access their share of savings realized through the waiver demonstration after receiving State approval to amend their waiver plans to provide new or expanded services.

Contracts with Outside Service Providers

Flexible funding waivers gave local child welfare agencies more latitude to purchase contracted services for children and families through non-profit and private service providers. Evaluation findings from some States reveal both successes and challenges in establishing service contracts with outside providers:

Oregon's evaluation cited several difficulties in making available high-quality contracted services, including problems with the recruitment and retention of qualified service providers and the complexity and lengthiness of the contract process.
North Carolina successfully used its waiver to offer a range of new services through contracted providers, including family support, assessment, family reunification, post-placement and post-adoption services. By early 2001, all but three of North Carolina's waiver counties had used outside contracts to provide new or enhanced services for at-risk families.

Interagency Collaboration

All four States reported on their efforts to promote collaborative relationships among child welfare agencies, the courts, and other public and non-profit agencies that serve children and families. Overall, the waivers appeared to exert a moderate positive effect on the development of collaborative relationships among public agencies and community-based service organizations:

Ohio concluded in its final evaluation report that experimental and comparison counties had similarly strong relationships with local juvenile courts and local mental health boards; however, experimental counties were somewhat more likely to adopt joint funding and cost sharing mechanisms than comparison sites. In addition, a survey of child and family service organizations in the fifth year of Ohio's waiver revealed that demonstration partners in all 14 experimental counties viewed their collaborations with local county child welfare agencies as more successful than did partners in comparison counties.
North Carolina's final evaluation report described the positive effects of its waiver on collaboration between child welfare agencies and community service providers, with eleven county agencies reporting greater involvement in at least one collaborative effort, including shared funding and Family Group Conferencing.
In Indiana, a certain degree of cross-organizational collaboration was enforced during the waiver planning process through the use of Interagency Agreements, which were required to include the juvenile court, the child welfare agency, a local mental health center, and the school district as signatories. However, Indiana noted that the involvement of these planning partners diminished in a majority of counties following the end of the demonstration's planning phase, and hopes for revived partnerships between the child welfare agency, schools, and other local community agencies went unrealized. In the final year of the waiver, fewer than 50 percent of child welfare agency administrators reported active involvement in the demonstration by mental health services personnel, school personnel, or staff from other community-based organizations. Juvenile judges, who played an ongoing and mandated role in the demonstration due to their legal charge of assigning children to the waiver, were the major exception to this decline in involvement.
Like Indiana, Oregon built interagency collaboration into its waiver demonstration by requiring each participating county to involve community-based social service organizations, the courts, and law enforcement agencies in the development of its waiver plan. The State reported that it achieved some success in engaging community-based organizations, with local agencies representing nearly half of all organizations involved in the development and implementation of waiver plans. However, active participation by the majority of community stakeholders was somewhat limited, with representatives from most agencies typically attending only one meeting or providing indirect input on needed community services. Local school districts, mental health agencies, and adult and family service organizations were most frequently involved in waiver plan development and implementation.

Access to Services

States have used flexible IV-E funds to provide a wide variety of new or expanded services for families involved in the child welfare system. With the exception of Indiana, however, it remains unclear whether families residing in local jurisdictions with access to flexible funds received more or a greater diversity of services than families in non-waiver jurisdictions:

Indiana's evaluation found that experimental group cases received significantly more services than matched comparison cases, especially services that promoted family stability. Experimental group cases received significantly more family preservation services, individual counseling, childcare, respite care, help with basic household needs, and special education services. In addition, experimental group cases were more likely to receive these services through community-based organizations.
Ohio reported few differences between experimental and comparison counties in terms of improved service availability, the array of new services, or timely access to services. One notable exception was in the area of placement prevention, with experimental counties offering more new preventive services to eligible families.
In North Carolina, experimental group counties contracted with outside service providers to offer new combinations of family support, assessment, family reunification, post-placement, post-adoption, and legal services. The typical experimental group county contracted for three different types of services, with child welfare agencies most often using waiver funds to provide family support services. However, it was unclear from the State's evaluation whether families in experimental group counties had access to more or to a greater diversity of services than families in comparison counties.

Staff Training

In some States, the quality and quantity of staff training regarding flexible funding rules and procedures stood out as a factor affecting the success of waiver implementation. In Indiana, many child welfare agency administrators and family case managers indicated a need for additional training related to the waiver; about half of all case managers surveyed for the State's evaluation believed that insufficient training had lessened the effectiveness of the waiver in their county. Oregon's final evaluation report identified a lack of training for caseworkers regarding procedures for referring families for Family Decision Making services as an issue that limited the effectiveness of this component of its flexible funding demonstration.

Use of Managed Care Strategies

Among the States with flexible funding waivers, Ohio initially focused on the effectiveness of managed care strategies to both improve child welfare outcomes and control costs. To evaluate experimental and comparison counties in their adoption of managed care, the State developed a "managed care index" to compare counties in their use of common managed care activities, including expanded service arrays, the use of capitation and other financial risk sharing mechanisms, provider competition, utilization review, and quality management. Although a statistically significant difference in favor of experimental counties in the use of managed care strategies was noted in the second year of the State's demonstration, no significant differences emerged between experimental and comparison counties in their overall use of managed care approaches during the five years of the waiver. Over time, experimental counties placed less emphasis on managed care and concentrated instead on the provision of new child welfare services, especially placement prevention activities.

Back to Table of Contents

State Outcome Evaluations - Summary of Key Findings and Issues

Given the ultimate goal of all IV-E waivers of improving child welfare outcomes, a key question regarding flexible funding waivers relates to their effectiveness in keeping children in safe, stable home environments that provide for their physical, emotional, and developmental needs. In designing their evaluations, States posed several questions about the impact of flexible funding waivers on child welfare outcomes:

Are children who receive services through flexible funding programs more likely to avoid placement?
Do flexible funding programs increase the number of children who achieve permanency?
Do flexible funding programs reduce the length of time spent in out-of-home placement?
Are children who receive services through flexible funding programs safer from abuse and neglect?
Are children who receive services through flexible funding programs less likely to re-enter foster care?
Do flexible funding programs increase placement stability, i.e., do they reduce the number of placement episodes a child experiences over time?
Do services offered through flexible funding programs improve the physical, emotional, and developmental well-being of children?

States' flexible funding waivers have produced mixed results regarding these evaluation questions. As discussed earlier, the diffuse, systemic nature of many flexible funding demonstrations makes it difficult to attribute local or statewide changes in child welfare outcomes to waiver services and activities. Indiana, which adopted a matched case comparison design with case-level data analysis, has produced the most conclusive findings to date about the effects of flexible funding service models. The following section discusses highlights from the States' final evaluations regarding the impact of flexible funding waivers on key child welfare outcomes.

Placement Avoidance

One key outcome of interest with respect to flexible funding waivers is placement avoidance, that is, their effectiveness in keeping children home and out of foster care. Three States - Indiana North Carolina, and Oregon - studied the effects of their waiver demonstrations on the likelihood of out-of-home placement. Ohio did not specifically examine placement prevention as part of its evaluation. In all three States, a statistically significant positive association emerged between access to services through the flexible funding demonstrations and reduced entries into foster care:

In Indiana, the number of children placed in out-of-home care (including family, group, and institutional settings) declined each month during the demonstration, with 10,139 children in care one year before the waiver began in January 1997 and only 9,377 children in care by the end of the initial phase of the demonstration in December 2002. During this interval, 45.6 percent of children assigned to the experimental group never entered placement compared to 38 percent of control group children, a statistically significant difference.
North Carolina's evaluation indicated that the likelihood of entering placement for children with a substantiated report of abuse or neglect declined significantly more in experimental counties than in comparison counties or in other counties not participating in the waiver demonstration. The effects of the waiver in reducing out-of-home placement rates appeared to hold, even in the presence of other child welfare reform initiatives such as Families for Kids.
Children in child welfare branches with access to title IV-E funds or State-funded System of Care dollars in Oregon's demonstration were over three times more likely to remain home as children in branches without access to either title IV-E or System of Care funded services, a statistically significant difference.

Achievement of Permanency

States defined permanency in somewhat different ways for the purposes of their evaluations. Two States - Indiana and Oregon - focused on exits to reunification. Ohio examined exits to reunification, adoption, and relative custody following an initial placement in foster care, whereas North Carolina studied overall rates of exit from out-of-home placement. With the exception of Indiana, findings regarding the effects of the flexible funding waivers on achievement of permanency have proved largely inconclusive:

Indiana produced significant positive findings regarding the effects of the waiver on reunification rates, with nearly 77 percent of experimental group children in out-of-home placement reunified either with the original caregiver or a non-custodial parent compared with 66 percent of control group children.
Oregon uncovered no differences between experimental and comparison sites in the likelihood of reunification one year following out-of-home placement.
In Ohio, the availability of the waiver was associated with an actual drop of 11 percent in exits to reunification from first foster care placements, a decrease driven largely by one large Ohio county. The State's evaluators noted that the reasons behind this drop in reunification rates are unclear, but could have been associated with waiver-related changes in screening, risk assessment, or case management practices in experimental group counties. Although adoptions increased in one county with access to flexible funds, the availability of the waiver was not associated with significant changes in overall adoption rates. However, Ohio did find a statistically significant positive effect of the waiver demonstration on exits to relative custody from first placements, with 18 percent of children in experimental counties exiting to relative custody compared to a counterfactual projection of about 14.5 percent.
North Carolina reported that children entering placement in more recent years in which the waiver was available were more likely to exit placement than children who entered in pre-waiver years; however, this downward trend was present in all county groups, whether they were experimental counties, comparison counties, or other counties not participating in the demonstration. Thus, no strong evidence emerged that access to flexible funds had an independent positive effect on foster care exit rates in North Carolina.

Length of Foster Care Placement

Three States - Indiana, North Carolina, and Ohio - explored whether the availability of flexible funding facilitated more timely exits from out-of-home care. Oregon did not specifically examine the effects of its flexible funding waiver on placement duration. As with exits to permanency, results have varied widely across demonstrations:

In Indiana, the mean length of placement for all experimental group children removed from their homes was 290 days compared with 316 days for matched comparison children, a difference of 8.2 percent in the average number of days in placement. When the State's evaluators controlled for risk and severity of maltreatment, severity of delinquency, and the age of the child, the difference in mean placement length was even greater at 271 days for waiver children and 319 days for matched comparison children, a statistically significant difference of 15 percent in the average number of placement days.
North Carolina's evaluation highlighted a statewide, downward trend in the duration of out-of-home placements prior to the implementation of the waiver. Due to a mature reform agenda in many counties, many local child welfare agencies had made significant progress in achieving permanency more quickly as early as 1996. Although the duration of placements continued to decline statewide throughout the period of the waiver, comparison counties and other counties not involved in the waiver demonstration appeared to have a greater overall rate of reduction in placement length. To explain this trend, North Carolina theorized that because experimental counties had reduced the likelihood of placement for children with a maltreatment report more than comparison counties, those children who did need placement presented "high risk" profiles. In other words, once children who could be effectively served in their own homes and neighborhoods were diverted from placement, the evaluators hypothesized that the remaining children required longer stays in foster care. In light of this hypothesis, North Carolina found it meaningful that overall lengths of stay in out-of-home placement continued to decline in experimental counties.
Ohio's evaluation found no significant effects of the waiver on median length of stay in placement prior to reunification, adoption, or exit to relative custody.

Maltreatment Recurrence

The available evidence from three States' evaluations suggests that children served through flexible funding waivers are generally as safe from recurrences of abuse or neglect as children without access to the waiver:

Indiana discovered no differences between experimental and control cases in rates of new maltreatment reports or substantiations, nor did differences between experimental and matched comparison cases emerge when specific types of child abuse and neglect were examined.
Ohio's evaluation indicated that the safety of children in experimental counties who exited out-of-home placement was maintained at the same level as that of children in comparison counties, alleviating concerns that flexible funding services could result in premature reunification for some at-risk children.
In Oregon, access to the flexible funding waiver had no impact on the likelihood of repeat abuse or neglect of children by their original caregivers within one year of the original maltreatment incident.

North Carolina did not study the effects of its flexible funding demonstration on the probability of maltreatment recurrence.

Foster Care Re-entry

Children who experience a maltreatment recurrence or other crisis may need to return to the custody of a public child welfare agency and subsequently to foster care. Overall, access to flexible funding did not appear to affect rates of foster care re-entry:

When taking child and case characteristics into account, Ohio found no significant effects from its waiver demonstration on rates of foster care re-entry for children reunified with their families of origin.
Marginal, insignificant differences were found during Indiana's demonstration between experimental and matched comparison cases on rates of foster care re-entry, with five percent of all experimental group children having a subsequent placement episode prior to the end of the demonstration compared with 7.7 percent of matched comparison children.
North Carolina's evaluation determined that experimental group counties making active use of the waiver showed a larger decrease in foster care re-entry rates during the early years of its demonstration, although counties making less active use of the waiver surpassed active counties in lowering re-entry rates during later waiver years. Overall, the availability of flexible funds for new services and supports in experimental counties had no statistically significant effect in either direction on the likelihood of foster care re-entry. The State's evaluators cautioned that the effects of the waiver on foster care re-entry must be analyzed in the context of statewide re-entry rates that have historically remained very low.

Oregon did not examine foster care re-entry rates as part of its impact evaluation.

Placement Stability

Some States examined placement stability, defined as the degree to which children experience multiple placement settings during the period of their removal from the home. Results from the two States that tracked this outcome are largely inconclusive:

In Ohio, both experimental and comparison counties increased the percentage of children who made no moves following their first placement and decreased the percentage making five or more moves. Experimental counties in Ohio were also no more successful than comparison counties in moving children to less restrictive settings.
Oregon reported in its final evaluation that access to flexible funds was associated with an increased likelihood of change in out-of home placements within one year. However, the State's evaluators could not determine whether this finding reflected negative or positive outcomes (for example, final placement into a permanent setting) since data regarding the reason for placement changes were not collected.

Neither North Carolina nor Indiana included measures of placement stability in their projects' outcome evaluations.

Child Well-Being

Among the States, only Indiana examined the effects of flexible funding on child well-being outcomes, such as school performance and attendance, engagement in risky behaviors, access to supports and services, mental health, and overall quality of life. Indiana's evaluation studied the school performance of children involved in the waiver demonstration and found a higher percentage of school-age children assigned to the experimental group in school at case closure (91.1 percent) than children assigned to the matched comparison group (83.6 percent). This difference was most notable among children adjudicated delinquent, with 87 percent of delinquent youths in experimental cases in school at case closure compared with 71.6 percent of their matched comparison counterparts. Findings from satisfaction surveys distributed to families enrolled in Indiana's evaluation revealed additional positive effects of the waiver demonstration on child and family well-being. For example, experimental group caregivers residing in active waiver counties were significantly more likely to report that their children were better off because of the involvement of the child protection agency and that they were involved in making decisions about their case welfare case.

Back to Table of Contents

Cost Analyses of the Flexible Funding Waivers - Summary of Key Findings and Issues

The four States applied varying approaches to studying the cost implications of their flexible funding waivers. Indiana's cost study found evidence that its demonstration produced improved child welfare outcomes at the same or reduced costs as traditional child welfare services. Findings from North Carolina and Ohio suggest that their waivers generated some savings that could be used to offset the costs of additional placement or non-placement related services, whereas Oregon concluded that its waiver had little effect on statewide patterns in child welfare spending. Major findings from the States' cost analyses are summarized below.

Indiana

Indiana undertook a formal examination of the cost effectiveness of its demonstration, which it defined in terms of improvements in child welfare outcomes while maintaining costs at similar or reduced levels. Overall, average expenditures from all sources per experimental group child during the 24-month period following case opening were $12,614 compared with $11,123 per non-waiver child. However, costs were lower in active waiver counties for three of four child welfare measures: placement avoidance, length of placement, and reunification. Only for the fourth outcome measure - avoidance of out-of-state placement - did costs for experimental group children exceed those of matched comparison children. Although increases in cost-effectiveness for these three outcome measures were modest in size, Indiana's waiver demonstrated improved child welfare outcomes for similar or reduced costs.

North Carolina

Through its title IV-E reinvestment strategy, North Carolina made additional IV-E dollars available for services to both IV-E-eligible and non-IV-E-eligible children that probably would not have been provided without the waiver. Many counties used these reinvestment funds to cover the costs of care for non-IV-E eligible children. Although experimental group counties reduced the number of children in licensed care and thus lowered the Federal share of IV-E maintenance costs, they had substantially higher average costs for children in licensed care toward the end of the demonstration. This finding suggests that as the risk profiles and needs of the population of children entering placement changed, waiver counties used higher-priced and more restrictive forms of care to meet their needs.

Ohio

Overall, no significant differences in child welfare spending emerged between experimental and comparison counties in Ohio's waiver demonstration. Both experimental and comparison counties experienced growth in paid placement days and in the average daily cost of foster care, and neither group significantly changed its percentage of placement days in residential settings. Most experimental counties generated some revenues that could be used for non-placement-related services, with 10 of 14 experimental group counties spending more than their capped IV-E allocations to increase the availability of other child welfare services.

Oregon

Oregon concluded that its flexible funding waiver exerted minimal influence on statewide trends in child welfare spending. In its final evaluation report, Oregon noted that overall patterns of child welfare expenditures - including TANF, title XIX, State General Fund, and title IV-E - changed significantly in the State during the demonstration period, with total spending growing 53 percent over the five-year waiver period. However, waiver-related expenditures represented less than one percent of total child welfare spending during that time.

Back toTable of Contents

⁵Not all children on whom data were analyzed actually received services through the flexible funding demonstration, but lived in a county designated as either an experimental or a comparison county for the purposes of the evaluation.Back

⁶In its evaluation, Oregon examined the effects of SOC on child welfare outcomes in conjunction with the effects of its flexible funding waiver.Back

⁷Letter to Dr. Susan Orr, July 25, 2003.Back

⁸Jordan Institute for Families (2002). Evaluation of North Carolina's title IV-E waiver demonstration. Chapel Hill, NC: University of North Carolina at Chapel Hill.Back