Questions?   Privacy   Site Index   Contact Us
			Home   |   Services   |   Working with ACF   |   Policy/Planning   |   About ACF   |   ACF News		Search

Enter Search term hereGo Button  Advanced  Search

Home > Statistics & Research > Child Welfare Research > Progress Report to the Congress

 

What is the Status of the Development of Outcome Measures in Child Welfare?
Why is Outcome Measurement in Child Welfare Challenging?
What Data Are Available to Support Development and Use of Outcome Measures in Child Welfare?
Are There Strategies for Dealing with the Complexities in Outcome Measurement in Child Welfare?
What Does Recent Literature Advise About the Use of Performance-Based Incentives?

The information presented in this section reflects the Department's preliminary research and consultation on some key issues likely to arise in the study of performance-based incentives. Participants in focus group discussions held by the Department several months ago often acknowledged the importance of heightening attention to outcomes in child welfare, but raised concerns about the state of the art in outcome measurement and data collection. They noted the complexities involved in developing performance-based incentives, and they expressed concern that such a system might unintentionally undercut individualized decision-making for children or pit one part of the child welfare service continuum against another.

This section of the Progress Report, therefore, reviews the status of the use of outcome measures in child welfare and the development of child welfare data sources. It also highlights some of the complexities and challenges that are likely to be faced in designing a performance-based incentive system and offers some possible strategies for dealing with these. Many of these issues will require a more comprehensive review as the study moves forward.

What is the Status of the Development of Outcome Measures in Child Welfare?

The past several years have seen significant developments in the use of outcome measures in child welfare, as well as in other areas of health and human services, by States, localities and the Federal government. This expansion reflects a demand for accountability by legislators, senior government officials and the public. It also reflects an increased awareness on the part of child welfare administrators and other human services officials of the need to monitor not only the extent to which services have been used by target populations, or the extent to which established policies, procedures or protocols have been followed, but the need to try to understand the effect that these services and procedures have on the children and families they are designed to serve.

At the Federal level, a number of factors have helped to focus attention on the development of outcome measures in child welfare. The implementation of the Government Performance and Results Act (GPRA), for instance, has helped to stimulate the articulation of goals for child welfare and other government programs and the identification of measures that will help us to understand how well government programs are doing in trying to address complicated social problems.

The work of the Administration for Children and Families (ACF) to reinvent the monitoring process for child welfare programs under titles IV-B and IV-E has also been important in advancing the field of outcome measurement in child welfare. Previously, the monitoring of child welfare programs relied heavily on the review of case documentation and process, rather than on outcomes for children and families. Consequently, States that provided and documented all of the required protections were able to pass compliance reviews without necessarily having programs that resulted in satisfactory outcomes for the children and families whom they served. And, conversely, some States that might be achieving positive outcomes, but whose case record documentation did not reflect all of the required protections, were penalized through the loss of incentive funds. Reflecting this concern, the Social Security Act Amendments of 1994 (P.L. 103-432) required ACF to move forward in developing new monitoring procedures to focus more clearly on the outcomes of child welfare services.

Since that time, ACF has undertaken considerable consultation with the field and has conducted pilot tests of revised Child and Family Services Reviews and title IV-E eligibility reviews. Through its process of consultation, ACF identified three major goals or outcome areas for child welfare services -- safety, permanency, and child and family well-being. These same themes are very evident in the Adoption and Safe Families Act (ASFA) and have been embraced by a number of States that have developed outcome frameworks. The conceptualization of these three broad goals is a tremendously important first step in moving toward a more outcome-focused orientation in child welfare. It has already become evident that it makes a difference when the Federal government joins in partnership with States to look at how the goals of safety, permanency and child and family well-being are being met, rather than focusing only verifying the presence of required documentation of case activity in all of the files. Building on the foundation of these goals is likely to prove useful as the Department works with State officials and other experts both to develop the outcome measures for the annual report required under ASFA and to explore possible approaches to performance-based financial incentives.

While the identification and articulation of the goals is a significant first step, the development of useful and reliable measures and the collection of data to support those measures remain in the early stages. Therefore, it is likely that the specific indicators used to measure progress on the goals of safety, permanency and well-being will need continued refinement over time.

Why is Outcome Measurement in Child Welfare Challenging?

With the work that has already been done in building consensus in the field about the major goals or outcomes for child welfare services, and, as discussed below, with the significant investments and improvements in State information systems and data collection and reporting, important steps have been taken toward developing a performance- or outcome-focused framework for child welfare. And yet the task of identifying specific outcome measures or indicators, particularly ones that can be used to compare performance among States, remains challenging for a number of reasons.

One reason for the complexity is that variation in State policies and definitions affect the collection and interpretation of data and can make it difficult to compare performance across States. For example, States have different definitions for what constitutes abuse and neglect, different standards for substantiating or confirming reported incidents of abuse and neglect, and different ways of categorizing the results of a child protective services investigation or assessment. States that have a narrower definition of child maltreatment or that require a higher standard of evidence to confirm a report generally have lower rates of child victimization than States with broader definitions or less stringent standards of evidence. Therefore, one cannot draw conclusions about the success of a particular State in keeping children safe based only on the reported levels of child maltreatment victimization.

The interpretation of results in child welfare outcome measures as being either "good" or "bad" is, in fact, quite complex. To assure appropriate interpretation, data should be looked at in the context of the reasons for the change in a given measure. For instance: Does a decrease in the number of reports of child abuse and neglect reflect a decrease in the incidence of abuse and neglect in a given State or community? Or did the State make changes in screening criteria, mandatory reporting laws, or staff resources taking calls that might account for the change?

Does a decrease in substantiated victims reflect a true decrease in the number of children suffering abuse or neglect, or were the definitions narrowed or policies changed in some way that affected this measure? Does an increased rate of recurrence of maltreatment reflect poor services on the part of the agency in taking steps to prevent repeat abuse? Or has the agency done a better job of following up with families who have come to their attention, so that subsequent incidents of maltreatment can be addressed? Does a decrease in the average length of stay in foster care mean that children and their families are receiving more timely services by child welfare agencies, with improved outcomes of safety and permanency? Or are children either being sent home too soon, only to return to foster care again, or unnecessarily taken from their families of origin permanently without appropriate efforts being made to reunify them?

Is an increase in the number of cases in which termination of parental rights have been completed a positive sign that children are being adopted? Or are we finding that there are not enough permanent families available to adopt these children? These examples highlight the need to examine outcomes in child welfare in the context of a systemic focus, rather than isolating one or two measures. A participant in a focus groups on ASFA implementation held by the Department in January summed up the quandary by saying, "You get a specific measure like number of reports, or number of indicated reports, and if that measure goes down, the obvious question is 'Did you reduce the number of telephone operators taking calls?' ...I think that it's really important when devising goals that they not be simplistic and that they take into account that there are all different kinds of reasons why you could attain a certain number." Similarly, a recent paper written b y Charles L. Usher, et al. summarizing recent evaluation findings from several foundation-supported child welfare reform initiatives, concludes: The results of these initiatives are encouraging in that they demonstrate that improvements in performance are possible. However, they also suggest a need for caution in adopting a few simplistic performance indicators and ignoring changes in context that dictate adjustments in performance standards and expectations. The accomplishment of one set of goals (e.g. reduced admissions [into foster care]) can have repercussions for other performance indicators (e.g. length of stay). Systems for monitoring and evaluating the performance of child welfare systems must acknowledge these interdependencies.¹

What Data Are Available to Support Development and Use of Outcome Measures in Child Welfare?

One reason that the development of outcome measures for child welfare is challenging is that, traditionally, data in child welfare have not been strong. Participation in data reporting efforts of the past was inconsistent, and many State information systems lacked the functionality needed to provide data useful for outcome measurement. Over the past several years, however, the Department and the States have worked together to make progress both in data collection and reporting and in the development of more advanced State child welfare information systems.

In the last several years the Department has implemented two major data reporting activities: the Adoption and Foster Care Analysis and Reporting System (AFCARS), collecting information on children placed in foster care or adopted from the foster care system; and the National Child Abuse and Neglect Data System (NCANDS), capturing data on children and families reported to child protective services agencies.

The AFCARS is a mandatory data collection system, established under Section 479 of the Social Security Act, that collects automated case-level information on all children in foster care for whom the State child welfare agency has responsibility for placement, care or supervision. Information is also collected on children whose adoptions from the foster care system have been finalized. The AFCARS data are reported in six-month increments, twice a year. The first data collected under AFCARS were for the first half of fiscal year 1995 (October 1, 1994 - March 31, 1995). The most recent submission cover the first half of fiscal year 1998 (October 1, 1997 - March 31, 1998). Almost all of the States are now submitting AFCARS data. (Forty-eight States, the District of Columbia and Puerto Rico submitted data in the most recent period.) All States are expected to be able to submit data in the near future. The AFCARS data will allow for a wide range of complex analyses regarding the numbers and characteristic of children in foster care and children who are adopted, the circumstances associated with children's removal from home, the length of time children spend in foster care, and many other factors.

The NCANDS, authorized by the Child Abuse Prevention and Treatment Act (CAPTA), was originally designed as a voluntary data reporting system consisting of two components: the Summary Data Component (SDC) and the Detailed Case Data Component (DCDC). Under the SDC, annual aggregate data on a limited number of data elements relating to children reported to the child protective services system (e.g. number of children reported to child protective services agencies and the number of children determined to be victims of substantiated abuse or neglect) have been collected from all States for the years 1990-1996. Recently, the SDC was revised and expanded to incorporate additional data elements specified by Congress in the 1996 reauthorization of CAPTA. This expanded set of data elements will be collected beginning with the submission of calendar year 1997 data, expected to be completed in the fall of 1998. Under the provisions of the 1996 CAPTA reauthorization, States that receive a State Child Abuse Grant are now required to "work with the Secretary to provide, to the maximum extent practicable," the data elements specified in the CAPTA statute that are now incorporated into the SDC.

The DCDC annually collects more comprehensive automated case-level information on all investigated reports of child abuse and neglect. Data have been collected for 1993-1996 from a subset of States (thus far representing about a third of the national population). These data allow for more complex analyses than can be conducted with aggregate data (e.g. examining the characteristics of children suffering particular types of maltreatment, or studying recurrence of substantiated reports for children previously reported to child protective services).

Both the NCANDS and AFCARS data collection activities are designed to improve national understanding of the number and characteristics of children served by child welfare agencies, as well as to be able to see differences and similarities in the patterns of activity in the various States. The data collection activities also provide a mechanism for providing technical assistance to States in order to promote improved data quality and data collection and data utilization at the State level. The Adoption and Safe Families Act appropriately identifies AFCARS as the major source of data to be used for outcome measures and performance-based incentives relating to titles IV-E and IV-B of the Social Security Act. The NCANDS data, however, are also likely to be a useful source of information for measuring certain outcomes relating to the safety of children, including children not removed from their homes. Further information on both NCANDS and AFCARS and the status of State participation are included at the end of this report in Appendix A.

Another significant development over the past several years has been the implementation of Statewide Automated Child Welfare Information Systems (SACWIS). In order for the Federal government to collect comprehensive and accurate data on children served by State child welfare agencies, the States must have information systems capable of collecting, tracking and reporting the data. The development of SACWIS in the States will assure this capability.

The Omnibus Budget Reconciliation Act of 1993 provided enhanced Federal financial participation (FFP) at the 75 percent rate for statewide automated child welfare information systems to carry out the State's programs under titles IV-B and IV-E of the Social Security Act. This enhanced level of funding was initially authorized for the period October 1, 1993, through September 30, 1996, and was later extended through September 30, 1997. While the period of enhanced Federal financial participation has now expired, Federal funds continue to be available at the 50 percent rate. The Federal funds may be used by the States for the planning, design, development and installation of statewide systems that are able to provide the required AFCARS reports and that interface with or integrate the system used to provide data for NCANDS. The States have been given significant latitude to design systems that best meet their State needs, provided that the systems are determined to provide more efficient, economical, and effective administration of the title IV-E and IV-B programs. In addition to providing expanded data reporting capabilities, these systems will also serve as a case management tool for frontline workers and supervisors, providing them with immediate access to a much greater amount of information. Because workers will, for the first time in most places, be using computers to support their own work, data accuracy and quality are also likely to improve, since the data entered will be for their own use and not be seen as "just one more piece of paper" required for a bureaucratic purpose. Thus far, 24 States have developed SACWIS systems that are either fully or partially operational. An additional 23 States are in the process of developing SACWIS. A chart showing the status of State implementation of SACWIS is included in Appendix A of this report.

We are confident that the joint commitment made by the Federal government and State governments to strengthening information system and data collection and reporting capacities will yield positive results in terms of improved ability to manage programs and to measure program success. However, it should be emphasized that, while the groundwork has been laid, implementation is still in the early stages, which means that issues of data availability, quality and comparability are likely to remain significant factors that need to be considered in assessing the feasibility of a new performance-based incentive system in the near term.

Are There Strategies for Dealing with the Complexities in Outcome Measurement in Child Welfare?

Recognizing that design and interpretation of outcome measures in child welfare is complex, the Department has begun to explore strategies to help address these concerns. One strategy, suggested by focus group participants in our early consultation, is to pair or group measures in a way that counterbalances potentially "bad" incentives or unexpected consequences that one measure alone might create. For instance, one might seek to decrease the average length of stay in foster care for some or all of the foster care population, while at the same time holding constant or decreasing re-entries into care. Or one might seek to increase the number of children adopted from the foster care system, without seeing an increase in the proportion of adoptions that dissolve.

Another strategy was laid out by a panel appointed by the National Research Council in its report, Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. The Council's panel studying the use of performance measures in these areas of public health stated: Despite their widespread use and intuitive appeal, health outcome measures are insufficient by themselves for monitoring the efforts of a given program in reducing complex public health problems. Many measures that are recognized valid for tracking health outcomes are affected by many factors (inputs or processes), so changes in outcomes cannot be attributed only to specific program effectiveness. The panel concludes that performance monitoring must make use of process and capacity measures to complement available measures of outcomes. The panel recommends that each process and capacity measure be accompanied by reference to published clinical guidelines or other professional standards that describe the relationship between the process measure or capacity measure and the desired health outcome. ²

A similar approach might be used for child welfare. Capacity measures might be used both to help understand why a particular measure was or was not achieved and to protect against measures being met because "we reduced the number of telephone operators." The degree to which capacity and process measures can or should be used in addition to outcome measures will be one issue that the Department will include in its consultation with the field.

What Does Recent Literature Advise About the Use of Performance-Based Incentives?

The Department has begun to review a number of sources to determine what the literature suggests as the strengths and potential pitfalls of performance-based financial incentives. A number of experts suggest caution in the use of such incentives. For instance, in a 1996 report from the Finance Project, entitled A Strategy Map for Results-based Budgeting: Moving from Theory to Practice, Mark Friedman writes: Crafting money consequences to go with performance is a tricky business. Pay for performance is an appealing concept, but hard to implement when the products are changes in human conditions; when performance is often tied to the severity of client problems, not the quality of service delivery; and when there are often ready means to game the system. This means that we should not rush to implement pay for performance (or other rewards and penalty policies) before we know what good performance is. We need to build performance histories, and begin to measure and reward improvements on past performance.³ A number of sources reviewed recommended using outcome measures as a means for targeting technical assistance or supporting a system of continuous improvement. For instance, the National Research Council report cited earlier recommended that Public health performance measures [should] be considered as a central, but not the only, element of a continuous program of technical assistance. For example, if one measure or a combination of measures suggest that a given state is having unusual difficulty in making progress in meeting its ... objectives, such information should trigger an alert that some additional resources or technical assistance is needed to overcome particular circumstances.⁴ In considering various options in designing a child welfare performance measurement system, the Department and its external partners will need to consider ways to encourage use of outcome measures in a manner that promotes continuous improvement in all aspects of child welfare services.

1. Charles L. Usher, Deborah Gibbs and Judith Wildfire, "The Rebound Effect in Child Welfare Reform: Secondary Effects of Success," pp. 13-14

2. Edward B. Perrin and Jeffrey J. Koshel, editors, Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health, Panel on Performance Measures and Data for Public Health Performance Partnership Grants, Committee on National Statistics, Commission on Behavioral and Social Sciences and Education, National Research Council, Washington, D.C.: National Academy Press, 1997, p.2.

3. Mark Friedman, "A Strategy Map for Results-based Budgeting: Moving from Theory to Practice," Finance Project, September 1996.

4. Perrin and Koshel, p. 4.