Children's Bureau Website - Child Welfare Monitoring: Corrected Federal Register Announcement

CORRECTED FEDERAL REGISTER ANNOUNCEMENT

NOTE: A Federal Register Notice pertaining to the data indicators for the second round of the Child and Family Services Review was published on January 23 providing corrected information to the original Notice published on June 7, 2006. Below is a version that integrates the corrections into the original publication, so that all information pertaining to the data indicators is in one document. The corrected information is highlighted in the document.

SUMMARY: On November 7, 2005, the Administration for Children and Families (ACF) published a notice in the Federal Register soliciting comment regarding its proposal to replace the six data measures used as part of the assessment of State performance on the Federal Child and Family Services Review (CFSR) with six data composites (70 FR 67479). Based on the results of our data analyses and a review of comments from the field, ACF made the following decisions:

The CFSR will use a State's performance on two individual data measures as part of the assessment of the State's substantial conformity with CFSR Safety Outcome 1-Children are, first and foremost, protected from abuse and neglect. A national standard is established for each of these measures.
The CFSR will use a State's performance on four data composites as part of the assessment of the State's substantial conformity with CFSR Permanency Outcome 1-Children have permanency and stability in their living situations. A national standard is established for each of these data composites.

This announcement presents the following information:

The decisions made by the Children's Bureau regarding use of data composites for the Federal Child and Family Services Review (CFSR);
The composites and additional data that will be used as part of the assessment of a State's substantial conformity with the CFSR requirements; and
Descriptive statistics relevant to each composite and measure, including the score that will serve as the national standard for the second round of the CFSR.

Where relevant, the announcement addresses key comments from the field in response to the November 7, 2006 Federal Register notice.

Summary of Amendments to the June 7, 2006 Federal Register Notice.

This amended final notice replaces the final notice published in the Federal Register on June 7, 2006, Vol. 71, No. 109, pages 32969-32987. The following are the key changes in the document and the reasons for the changes:

Section III has been revised to provide a detailed specification for each of the individual measures included in each composite. After publication of the June 7, 2006 Federal Register Announcement, ACF conducted trainings on the CFSR data indicators at each of the 10 ACF Regional Offices. The trainings were provided to key administrators and staff of State child welfare agencies. During these trainings, issues were identified and questions were raised regarding the individual measures included in the composite. After reviewing these issues and questions, ACF determined that there was a need to provide more specific information about the measures. The increased specification includes using the precise terminology that is used in AFCARS and describing in greater detail the children who are in the denominator and numerator of each measure. For a few measures, ACF determined that changes were needed either in the numerator or the denominator to increase precision of the measures to better target the outcome being assessed. To increase the precision of some measures, it was necessary to make changes in the syntax used to calculate the measures.
The definition of foster parent used by the National Child Abuse and Neglect Data System (NCANDS) has been changed. This definition is provided in Section III, A. The change was made because NCANDS recently revised their definition of a foster parent.
Table 1: Ranges, Percentiles, and National Standards for the Measures and Composites to be Used in the Second Round of the Child and Family Services Review is revised to reflect new data and new national standards for all of the CFSR data indicators. This table replaces the Table 1 provided in the June 7, 2006 Federal Register notice. The data in the table and the national standards for all of the data indicators and measures are different than those presented in the June 7, 2006 Federal Register notice. This difference is due to one or both of the following reasons: (1) a change in the syntax for calculating three of the measures (as explained above), and (2) a change in the procedures for calculating the national standards. The reason for the change in the procedures for calculating the national standards is provided in the following bullet.
Attachment B: Methodology for Developing the Composites: There is a correction in the section headed "Response to concerns regarding use of PCA." In the printing of the June 7, 2006 Federal Register, a sentence was left out of the fourth bullet. This has been corrected.
Attachment B: Methodology for Developing the Composites is revised to (1) correct an error in the original Attachment B to the June 7, 2006 Federal Register notice, (2) to provide the new coefficients (weights) for the composite measures, and (3) to explain the new procedure for calculating the national standards. The error in the original Attachment B occurred in Step 10 of the process of calculating the composite scores. The process noted in the June 7, 2006 Federal Register notice was not correct, although it would result in the same calculation. The new coefficients (weights) for each of the measures included in the composites are presented in Table 1 of the attachment - Coefficients (Weights) for the Measures Included in the Permanency Related Data Composites. The coefficients/weights changed due to changes in the syntax used to calculate some of the individual measures.

The last section of Attachment B - Establishing the National Standards - has been changed. Since publication of the notice in the June 7, 2006 Federal Register, ACF changed the procedure used for establishing the national standards. In the first round of the CFSR, the procedure for establishing the national standard for each measure involved fitting the data to a normal curve based on a level of significance of .05, and adjusting for the 75th percentile by calculating the sampling error using the lower limit of a 95 percent confidence interval for estimating the population mean. This procedure was used to establish the initial national standards for the data indicators that were reported in the June 7, 2006 Federal Register notice.

Upon further analysis, ACF determined that this procedure was not appropriate for the second round of the CFSR. One reason for this is that using a level of significance of .05 resulted in eliminating States at either end of the range of each of the State permanency composite data indicators and safety-related data indicators in order to fit the distribution to a normal curve. Consequently, not all States were included in the calculation of the national standards reported in the June 7, 2006 Federal Register notice. Although this was appropriate for the first round of the CFSR because of data quality problems at the time that the standards were established, these data quality problems are no longer an explanation for the exclusion of States. Therefore, eliminating States from the distribution in the second round of the CFSR could not be justified on the basis of questionable data. Another concern was that using the 95 percent confidence interval resulted in a considerable reduction in the actual percentile represented by each of the permanency and safety standards. The new procedure changed the national standards for the safety-related as well as the permanency-related CFSR data indicators.

For Further Information Contact: John Hargrove, 202-205-8634.

SUPPLEMENTARY INFORMATION

I. Background

The CFSR is ACF's results-oriented comprehensive monitoring system designed to promote continuous improvement in the outcomes experienced by children and families who come into contact with public child welfare agencies. ACF developed the CFSR in response to a mandate in the Social Security Amendments of 1994 (see section 1123A of the Social Security Act) for the Department of Health and Human Services to promulgate regulations for reviews of State child and family services programs under titles IV-B and IV-E of the Social Security Act. ACF's final regulations on the CFSR process, issued in 2000, can be found at 45 CFR 1355.31 through 1355.37. Between fiscal year (FY) 2001 and FY 2004, ACF conducted the first round of the CFSR. A "round" is defined as a cycle of the CFSR that includes every State, the District of Columbia, and Puerto Rico.

Information for each CFSR came from the following sources: (1) the Statewide Assessment, (2) case-level reviews conducted by a team of Federal and State reviewers, (3) interviews with key stakeholders, and (4) State data from the Foster Care File of the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the Child File of the National Child Abuse and Neglect Data System (NCANDS), or an alternative data source approved by the Children's Bureau. Using this information, the first round of the CFSR assessed State performance on seven outcomes and seven systemic factors. For the most part, performance on the seven outcomes was determined through the results of the case reviews. However, in the first round of the CFSR, the assessment for two outcomes also included a State's performance on six national data measures that ACF adapted from measures developed for the Annual Report to Congress on Child Welfare Outcomes in response to the requirements of section 479A of the Social Security Act. ACF established national standards for each of the six data measures, all of which were calculated from data reported by States to NCANDS and AFCARS. ACF described these six data measures and the national standards in the preamble to the final CFSR regulation, published in the Federal Register (65 FR 4024-4025). This same regulation provides information on how ACF calculated the national standards associated with each of the six data measures. Subsequently, ACF issued information memoranda on the specific national standards that would be used in the initial CFSR implementation (see ACYF-CB-IM-00-11 and ACYF-CB-IM-01-07).

The following performance measures and national standards were used during the first round of the CFSR as part of the assessment of a State's substantial conformity with CFSR Safety Outcome 1-Children are, first and foremost, protected from abuse and neglect:

Repeat maltreatment - Of all children who were victims of substantiated or indicated child abuse and/or neglect during the first 6 months of the reporting period, 6.1 percent or less had another substantiated or indicated report within a 6-month period.
Maltreatment of children in foster care - Of all children who were in foster care during the reporting period, 0.57 percent or less were the subject of substantiated or indicated maltreatment by a foster parent or facility staff member.

The following performance measures and national standards were used as part of the assessment of a State's substantial conformity with CFSR Permanency Outcome 1-Children have permanency and stability in their living situations:

Timeliness of reunification - Of all children who were reunified with their parents or caretakers at the time of discharge from foster care, 76.2 percent or more were reunified in less than 12 months from the time of the latest removal from home.
Re-entry into foster care - Of all children who entered foster care during the reporting period, 8.6 percent or less were re-entering foster care in less than 12 months of a prior foster care episode.
Timeliness of adoption - Of all children who exited foster care to a finalized adoption, 32 percent or more exited foster care in less than 24 months from the time of the latest removal from home.
Placement stability - Of all children who have been in foster care for less than 12 months from the time of the latest removal from home, 86.7 percent or more have had no more than two placement settings.

ACF views the CFSR as a dynamic process. We made ongoing improvements after each year of the first round of reviews in response to our experiences in the field and to suggestions from State child welfare agency administrators. After completion of the first round in FY 2004, ACF contracted with a consultant to study the CFSR and make further suggestions regarding potential revisions to the process. To assist in this task, the consultant convened a CFSR workgroup including State child welfare agency administrators, child welfare specialists, and researchers. Based on input from this workgroup, the consultant presented a set of suggestions for ACF. One suggestion was to replace the existing CFSR single data measures for which national standards were established with data composites that incorporate a wider range of performance areas relevant to a particular child welfare domain. ACF determined that making this change would enhance the quality of the CFSR for the following reasons:

The recommendation is consistent with our observations during the first round of the CFSR that expanding the scope of data pertaining to a particular child welfare domain will provide a more effective assessment of State performance. For example, expanding the scope of data pertaining to the timeliness of reunification will address various performance areas relevant to this domain, including the permanency of the reunification.
Data composites will provide a more holistic view of State performance in a particular domain than a single data measure can achieve. For example, the current CFSR measure of timeliness of adoptions considers the percentage of children adopted within 24 months of entering foster care, but not children's experiences with regard to the timeframes between key points in the adoption process, such as the time from termination of parental rights (TPR) to a finalized adoption.
Data composites will ensure that the data component of a State's performance with regard to a particular domain will not depend on one measure. For example, a State's performance regarding the data composite for the domain of timeliness to adoption may be uneven, with performance higher in one area than in another. However, overall performance on the composite may be high. Thus, the data composite will account for both the strengths and weaknesses that a State exhibits within a particular domain.
Data composites are being used by the Federal government to assess other programs. For example, composite measures are being developed and used for the No Child Left Behind initiative. In addition, composite measures have been used to evaluate the performance of hospitals in various health-related domains.

II. Analysis and Decisions

ACF published a Federal Register notice presenting proposed data composites and performance areas for each composite on November 7, 2005, with a 30-day public comment period. We received 66 letters from State and local child welfare agencies, national and local advocacy groups, researchers, State and local courts, and national associations representing groups of practitioners. ACF's final decisions regarding the composites are presented below. These decisions are based on our review of comments from the field, our data analyses, and the principles and objectives of the Social Security Amendments of 1994 and the Adoption and Safe Families Act (ASFA) of 1997.

A. ACF will replace the existing six data measures used for the first round of the CFSR with four data composites and two single measures.

The majority of respondents to the Federal Register notice expressed support for our proposal to use data composites as part of the assessment of a State's substantial conformity with the requirements of the CFSR. A few respondents expressed concern about the potential burden to the States involved in revising their data systems to provide data for the composites. However, the composites will not require States to revise their basic data systems because all data necessary for the composites come from existing AFCARS or NCANDS data elements. Also, because States submit the NCANDS Child File on a voluntary basis, the CFSR regulation allows us to accept data from an alternative source from those States that do not submit the Child File to NCANDS. However, for the second round of the CFSR, the use of alternative data sources applies only to measures calculated from data reported to the NCANDS Child File. It does not apply to measures calculated from data reported to AFCARS.

A few respondents expressed concern that the composite approach would make it difficult for States to track their own performance in specific areas and to identify those areas where improvements may be needed. To assist States in tracking their performance on the composites, we will provide them with a State Data Profile that presents information on all of the individual performance areas included in the composites as well as the composite scores ¹. Attachment A itemizes the data that will be included in the State Data Profile to be provided to each State. ACF will provide States with the syntax used for establishing each of the performance areas and calculating the composite scores. In addition, we will ensure that technical assistance and appropriate tools are available to States to assist them in tracking their performance.

Although ACF initially intended to replace the six data measures with six data composites, we have decided to use two single measures that are similar to those used in the first round of the CFSR to assess State performance with regard to CFSR safety outcome 1 - Children are, first and foremost, protected from abuse and neglect. We made the decision not to develop safety composites for the following reasons:

Many respondents to the Federal Register notice expressed concern about the usefulness and appropriateness of the new measures proposed for the safety-related composites.
A review of the data for the measures revealed potential problems with consistency in State reporting, particularly with regard to how States defined certain data elements.
The results of the data analyses for the composites did not provide strong support for inclusion of some of the measures proposed for the composite.

Additional information relevant to our decision to eliminate particular measures is provided in the section of this Announcement pertaining to CFSR Safety Outcome 1.

B. ACF used principal components analysis to develop the composites.

ACF identified and implemented the methodology for establishing data composites in consultation with an internationally known expert statistician. Our goal was to increase the amount of pertinent information that would be considered in assessing a State's performance with regard to particular outcomes without increasing the number of measures that would be subject to a national performance standard. We reviewed with our expert consultant all possible statistical methodologies and determined that a principal components analysis was the most appropriate data analysis method for achieving our goal.

Principal components analysis is a commonly used statistical technique for reducing a large set of variables into a smaller set by combining highly inter-correlated variables. Use of this analysis is based on two basic psychometric principles of measurement: a) a test with more questions is more reliable; and b) combining related scores into a composite score results in a more reliable and valid score than the individual scores on which the composite is based. Each variable in the set is given a weight in accordance with its relative importance to the overall composite (See attachment B for more information on this). These sets, or principal components, usually are more stable and easier to interpret than individual variables because they incorporate several variables that are related to one another but also capture unique information.

The principal components analyses conducted to generate the composites were closely guided by our expert consultant and were systematic and conservative in nature. The analyses generated valid and meaningful results that exceed the minimum requirements of acceptability for this analytical technique. Decisions made regarding the composites were based on the empirical data resulting from the analyses. Consequently, we believe that the composites established will enhance the assessment of State performance.

A few respondents questioned whether a principal components analysis methodology was appropriate and requested an opportunity to review the details of the methodology and to provide comment on the appropriateness of the methodology. Because the methodology used is based on a sound and widely accepted statistical process, we will not be submitting it for comment from the field. Many of the concerns expressed by respondents are the result of a lack of understanding of principal components analysis. The specifics regarding these concerns are addressed in attachment B, which also provides a description of how the methodology was used in generating the composites.

ACF understands that our composite approach represents a new conceptual framework for many States. Therefore, we will conduct orientation sessions with States in each ACF region to familiarize them with the composites and the methodology prior to implementing the next round of the CFSR. In addition, the data set used for the principal components analyses and the syntax used to construct the composites will be made available to States.

C. Wherever possible and appropriate, the data composites incorporate a combination of longitudinal measures that follow a cohort of children over time, measures that capture outcomes experienced by children exiting foster care in a given year, and measures that assess the status of children in foster care within a particular timeframe.

Several respondents recommended that all measures in the data composites should be longitudinal measures that follow a cohort of children over time to establish timeliness of permanency and placement stability. These respondents suggested that such measures, particularly those that follow a cohort of children entering foster care, reflect a more accurate picture of State performance in these areas than do other types of measures. However, several other respondents expressed support for maintaining the measures used in the first round of the CFSR that capture outcomes experienced by children exiting foster care in a given year. As one of these respondents noted, "I have heard and studied much of the criticism (of the six indicators), but I find much of the criticism to be without merit. ... the six indicators have served us very well here in (State)."

To address both perspectives, we have included as many longitudinal measures as possible in the composites along with other types of measures. Some respondents expressed concern that AFCARS does not permit a longitudinal analysis that crosses over fiscal years. This is not true. We currently can and have used AFCARS data to assess children across years-i.e., children entering or exiting foster care in one year can be followed in subsequent years. However, our ability to conduct longitudinal analysis for the CFSR is restricted somewhat by the timeframes of the CFSR and, in particular, the need to have data that reflect both a recent level of performance and change in performance during the period of program improvement. For example, the data used at the time of the second round of the CFSR for a given State cannot overlap with a State's Program Improvement Plan (PIP) implementation period. Within the context of the CFSR timeframes, it is not feasible to follow children for longer than a 12-month period and no measure can incorporate more than four AFCARS reporting periods (2 years).

Given this situation, most of the final composites include a combination of types of measures. ACF believes that each type of measure contributes to an understanding of State performance from a particular perspective. We have used the principal components analyses to determine the relative contribution of each type of measure to the overall composite. (See attachment B for more information on this issue.) Specific information about decisions pertaining to the types of measures incorporated in each composite is provided in the discussion of the individual composites.

D. ACF will use the data composites for the second round of the CFSR

Many respondents to the Federal Register notice, while indicating support for the data composite approach, proposed that ACF "pilot test" this approach during the second round of the CFSR and not implement this approach for assessment purposes until a later round of the CFSR. However, because the methodology used for establishing the composites is statistical rather than theoretical, the concept of a pilot test is not applicable. For example, the process of conducting the CFSR was initially piloted in 14 States to test whether the procedures (e.g., Statewide Assessment, case reviews, and stakeholder interviews) were appropriate and yielded the desired information. Although this process is valid for testing the utility of procedures, it is not applicable to data composites, which are derived from a statistical analysis of data submitted by the States to AFCARS ². However, the quality of the data submitted by the States to these Federal systems may be an issue for some States. ACF strongly encourages States to assess the quality of the data that they report to these systems and to improve the quality if any problems are identified. In addition, ACF will continue to provide guidance to States, either directly or through ACF's resource center, the National Resource Center for Child Welfare Data and Technology, in improving the quality of the data submitted to AFCARS.

Instead of a "pilot", ACF conducted a replication of the principal components analyses on data from prior years to examine whether the resulting component structures exhibit stability over time. The composites were constructed with the focus on data from fiscal year (FY) 2004. Data from FY 2003 were incorporated for the measures involving long-term longitudinal analysis. ACF conducted two replications of the principal components analysis on data reported to AFCARS relevant to FY 2002/2003 and FY 2001/2002. The results of this replication indicate that there is a clear and stable structure in the data to support the use of the composites as a meaningful component of the CFSR assessment of State performance.

E. ACF will establish national standards for the two independent measures and for each of the four composites.

Many respondents to the Federal Register notice recommended that ACF not establish national standards for the data indicators used in the next round of the CFSR. They proposed that ACF assess performance based on continuous improvement on the data measures over time within an individual State.

After consideration of this recommendation, ACF decided to maintain the practice of establishing national standards for the CFSR and to continue to use the standards as part of the assessment of a State's substantial conformity with outcomes pertaining to safety and permanency. The reasons for this decision are the following:

ACF initially established national standards for each of the six CFSR data measures as desired national goals for the field with regard to achieving safety and permanency for children. We believe that setting national goals for the field is an important part of ensuring that Federal, State, and local agencies remain focused on achieving the highest level of results for children who come into contact with the nation's child welfare systems.
Because the national standards for the first round of the CFSR were based on the distribution of performance across States, they are relative rather than absolute. By setting the standard at the 75th percentile (as adjusted for sampling error and for normality of distribution), we believe that the goals represented by the standards are realistic and attainable and that, by establishing standards, ACF is promulgating the expectation that States make concerted efforts to achieve these goals.

The assessment of a State's performance on its individual PIP is, and will continue to be, based on change in an individual State's performance over time rather than on whether the State meets the national standard. With regard to the national data measures, ACF has not required that a State meet the national standard in order to avoid financial penalties, only that the State demonstrate an agreed-upon amount of progress in moving toward the standard.

The primary concern raised by respondents to the Federal Register notice that pertained to the issue of national standards was that the standards involve a comparison among States that is not valid because variations in State practices, statutes, and policies often impact the comparability of performance on a particular measure. ACF acknowledges that variations in policies and statutes can affect comparability and has attempted to address these variations both in the new measures proposed for the composites and in the use of composites themselves.

The standards were calculated using data pertaining to State performance in FY 2004, with data from FY 2003 included when there is a measure requiring a longitudinal analysis that spans fiscal years. When the performance of individual States is considered with regard to the national standards, we will ensure that the State data pertain to time periods that are after completion of the PIP implementation period.

F. ACF will not establish separate national standards based on variations across States with regard to the age or race/ethnicity of children in foster care, or whether the reason for entering foster care was maltreatment or the child's behavior.

Many respondents to the Federal Register notice suggested that ACF should assess performance on the composites and the measures to determine whether there are differences in performance as a result of children's age, race/ethnicity, or reasons for entering foster care and that the national standards should be adjusted accordingly. For example, respondents noted that older children are more likely to experience placement changes than younger children, and therefore, States that have a relatively high percentage of older children entering the foster care population could not be expected to perform as well on measures of placement stability as other States.

We are not establishing separate performance standards for children of different ages, races, or reasons for entering foster care. Consistent with the tenets of the Adoption and Safe Families Act and with the best interests of children, all children have the same need for safety, placement stability, and timely permanency. Rather, this type of analysis is best left to the States to further examine the characteristics of their own child welfare populations as part of their Statewide Assessment.

A few respondents to the Federal Register notice also suggested that rather than adjust the national standards, the measures for the permanency-related composites should apply only to children who enter foster care as a result of abuse or neglect. ACF decided not to exclude children from the measures who enter foster care for reasons other than child maltreatment. We believe that all children who are in the custody of the State child welfare agency and who are reported to AFCARS share the same needs for permanency and placement stability regardless of their reason for entering care.

III. DATA MEASURES AND COMPOSITES

In this section, we present the measures and composites that will be used in the next round of the CFSR. We also identify and discuss the critical features of each measure and composite and address key comments concerning the measures and composites received in response to the Federal Register notice. Table 1 provides summary information regarding all of the composites, measures, and national standards to be used in the second round of the CFSR.

A. CFSR measures that will be used as part of the assessment of substantial conformity with CFSR Safety Outcome 1 - Children are, first and foremost, protected from abuse and neglect.

Two individual measures rather than composites will be used as part of the assessment of substantial conformity with CFSR Safety Outcome 1. These measures are the following:

Recurrence of maltreatment. Of all children who were victims of substantiated or indicated abuse or neglect during the first 6 months of the reporting year, what percent did not experience another incident of substantiated or indicated abuse or neglect within a 6-month period?

Maltreatment of children in foster care. Of all children in foster care during the reporting period, what percent were not victims of a substantiated or indicated maltreatment by foster parents or facility staff members?

Key features of the measures

These measures are similar to those used in the first round of the CFSR. The only difference is that the focus has shifted from the occurrence of maltreatment to the absence of maltreatment. We made this change for the following reasons:

Respondents to the Federal Register notice and others in the field recommended that all data measures address performance from a positive perspective.
The composite measures pertaining to permanency and placement stability are all in the same direction with higher scores meaning higher levels of performance. We believe that assessing all data in the same direction will simplify the interpretation of State performance with regard to the national data.

Although there was general support from the field for the proposed measure of recurrence of maltreatment, some respondents suggested that the measure be restricted to maltreatment recurrence involving the same perpetrator and the same type of abuse. ACF decided not to make this change because children should be protected from continued maltreatment within a 6-month period even if the perpetrator is the mother in one incident, for example, and the father or grandmother in another incident, or if the perpetrator is the same but the maltreatment is neglect in one incident and physical abuse in another.

Respondents also questioned whether and, if so, how the measure of recurrence will incorporate maltreatment allegations that are referred for an "alternative response." Alternative response usually refers to the practice implemented by several States in which a maltreatment allegation that is believed to involve low risk of harm to the child is referred to an agency for an assessment to determine whether the family is in need of services. In these situations, the allegation is not referred for a formal child abuse and neglect investigation. We determined that it is not possible to include maltreatment allegations that are referred for an alternative response in the measure of maltreatment recurrence because the majority of States that implement this approach do not make a disposition as to whether the allegation is substantiated or indicated.

Although respondents to the initial Federal Register notice also expressed support for the measure of maltreatment of children in foster care by foster parents or facility staff members, some suggested that the measure include maltreatment by relative caregivers. It already does this. The maltreatment in foster care measure includes perpetrators who are relative foster parents, non-relative foster parents, and group home or residential facility staff. It does not include perpetrators who are relative caregivers taking care of children who are not in foster care. NCANDS's current definition of "foster parent" is "an individual who provides a home for orphaned, abused, neglected, delinquent, or disabled children, who are under the care and supervision of the State child welfare system." This individual may be a relative or a non-relative.

The final two measures to be associated with the assessment of CFSR Safety Outcome 1 represent those that remained after we excluded the other measures initially proposed in the Federal Register notice. ACF decided to exclude the other proposed measures based on feedback from the field and the results of our review of the data and our data analyses. The measures excluded and reasons for exclusion are described below:

Measure of multiple unsubstantiated maltreatment allegations. In the November 7th Federal Register notice, ACF proposed a safety-related measure assessing the performance area of multiple unsubstantiated maltreatment reports. This was based on the findings of several research studies indicating that many children who are the subject of multiple unsubstantiated allegations actually experience maltreatment. However, almost all respondents recommended eliminating this performance area from the CFSR assessment. They noted that the measure is problematic because of State variations in practices and procedures relevant to substantiation. A particular concern was that many States do not differentiate in their dispositions between unsubstantiated allegations and allegations that are found to be intentionally false or without merit. Consequently, there would be no way to exclude the latter types of allegations from the assessment in all States.

Measure of timeliness of initiating investigations of maltreatment allegations. In the Federal Register notice, ACF proposed a measure of timeliness of initiating investigations of maltreatment allegations, with initiation defined as establishing face-to-face contact with the child who is the subject of the allegation and with the family. The measure was designed to address the proposition that investigations that are initiated quickly are more likely to ensure the safety of children than investigations that are not initiated quickly. We decided to exclude this measure primarily because the results of our data analyses did not support its inclusion and because it was not clear from the data that States were defining either the starting point (i.e., receipt of the allegation) or the end point (i.e., initiation of investigation) of the proposed measure in a consistent manner. In addition, most respondents expressed concern that such a measure would result in the Federal government setting policy for the States with regard to timeliness of initiating an investigation ³. However, because timeliness of investigations will continue to be part of the CFSR case review assessment, we have decided to provide data relevant to State performance in this area in the State Data Profile without an associated national standard. We will require that States address their performance in this area in their Statewide Assessment.

Measure of timeliness of dispositions of maltreatment reports. ACF initially proposed this measure because of our concern that a child welfare agency may not be able to address the safety of the child fully until an investigation is completed and a disposition is made. We decided to exclude the measure from the composite analysis because the majority of respondents indicated that State child welfare agencies are able to provide the necessary services and conduct adequate safety and risk assessment prior to a formal disposition, and that often a disposition is a court decision that is made after the agency has already intervened with the family to ensure safety and address risk issues.

Measure of maltreatment of children in foster care by their parents. We proposed this measure as a result of an unanticipated finding in our initial data review that for many of the children who were reported as being victims of maltreatment when they were in foster care, the perpetrator was identified as the parent. However, almost all respondents to the Federal Register notice expressed concern that because States report to NCANDS the report date rather than the incident date of a maltreatment allegation, the measure would capture incidents of maltreatment by parents that were received while the child was in foster care but that actually occurred before the child entered foster care. We initially attempted to address this problem by excluding from the measure reports received during the first 30 days that a child was in foster care. However, respondents did not agree that this would be sufficient to resolve the problem. Although NCANDS now includes a data element that asks States to report the date of the maltreatment incident as well as the date the report was received, States are not yet using that data element on a consistent basis. ACF has decided to report data on this measure to the States in the State Data Profile. We believe that States may not be aware of the extent of this problem and that by providing these data we will encourage them to use the NCANDS data element pertaining to the date of the maltreatment incident to assess whether children are victims of maltreatment by their parents while they are in foster care.

B. CFSR composites and measures that will be used as part of the assessment of a State's substantial conformity with CFSR Permanency Outcome 1-Children have permanency and stability in their living situations

Four data composites will be used as part of the assessment of State performance in achieving CFSR Permanency Outcome 1. A composite reflects the general domain that is assessed by the data. The four composites are: Permanency Composite 1: Timeliness and permanency of reunifications; Permanency Composite 2: Timeliness of adoptions; Permanency Composite 3: Achieving permanency for children in foster care for extended periods of time; and Permanency Composite 4: Placement stability. Information pertaining to construction of the composites is provided in attachment B.

Each composite comprises one or more components, depending upon the results of the data analysis. Components are the general factors that contribute to the composite score. If a composite has two components, each one contributes 50 percent to the composite score. If a composite has three components, each one contributes 33.3 percent to the composite score.

Each component comprises one or more measures. The measures provide the actual data for the analysis. The contribution of each measure (also called the weight) to the component score is determined by the principal components analysis and is presented in attachment B. The general structure of each composite with regard to the number of components and the number of measures, a summary of the data for each measure, and the national standards are presented in table 1.

A State's performance will be assessed for a 12-month target period that is determined by the date of the State's CFSR Review. Depending upon the time of its CFSR, a State's 12-month target period may include either the combination of an AFCARS A file (the first 6 months of a fiscal year) and B file (the second 6 months of a fiscal year), or it may include the combination of an AFCARS B file (the second 6 months of a fiscal year) and A file (the first 6 months of the subsequent fiscal year). The 12-month target period used for establishing the national standards was fiscal year (FY) 2004.

1. Permanency Composite 1: Timeliness and permanency of reunifications.

The principal components analysis of the measures proposed for this composite yielded a composite comprised of two components. One component pertains to timeliness of reunifications. This component includes three measures. The other component pertains to the permanency of reunifications and includes one measure. Each component has a unique score and contributes 50 percent to the final composite score. Information regarding the contributions of individual measures to the component score is provided in attachment B. Composite scores represent the conversion of z-scores to a scale ranging from 50 to 150.

Composite 1, Component A: Timeliness of reunification

For the CFSR data measures, reunification occurs if the child is reported to AFCARS as discharged from foster care and the reason for discharge is either "reunification with parents or primary caretakers" or "living with other relatives." Children who are reported to AFCARS as discharged to a legal guardianship are not included in the count of reunifications, even if the legal guardian to whom the child is discharged is a relative. If the relative is a legal guardian, the discharge reason of "guardianship" is to be used in the AFCARS submission.

The three measures for the Timeliness of reunification component include an adjustment to account for State policies or practices in which children are reunified but the State continues to have care and supervision responsibilities for a period of time before discharging the child from foster care. This is referred to as the Trial Home Visit adjustment and is explained in more detail in the Adjustments to the Measures section of the discussion of Key Features of the Components and Measures.

The score for the timeliness of reunification component of Permanency Composite 1 is derived from State performance on the following measures.

The denominator for this measure includes children who meet all of the following criteria:

The child is in foster care for 8 days or longer;

The child's date of discharge from foster care occurs during the 12-month target period; and

The child has a reason for discharge, and the reason is either "reunification with parents or primary caretakers" or "living with other relatives."

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet one of the following criteria:

The child's date of discharge from foster care occurs less than 12 months from the date of the latest removal from home; or

Using the Trial Home Visit adjustment, the child's "length of stay" in foster care is less than 12 months from the date of the child's latest removal from home.

Individual Measure C1.2: Of all children who were discharged from foster care to reunification in the 12-month target period, and who had been in foster care for 8 days or longer, what was the median length of stay in months from the date of the latest removal from home until the date of discharge to reunification?

This measure includes children who meet all of the following criteria:

Median length of stay is calculated based on one of the following procedures:

The difference between the child's date of discharge from foster care and the child's date of latest removal from home; or

The child's "length of the stay" in foster care using the Trial Home Visit adjustment.

Individual Measure C1.3: Of all children who entered foster care for the first time in the 6-month period just prior to the target 12-month period, and who remained in foster care for 8 days or longer, what percent were discharged from foster care to reunification in less than 12 months from the date of latest removal from home?
The denominator for this measure includes children who meet all of the following criteria:

The child's date of first removal from home occurs during the 6-month period just prior to the 12-month target period, and

The child is in foster care for 8 days or longer.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and who also meet at least one of the following criteria:

The child has a date of discharge from foster care that is less than 12 months from the date of first removal from home, and the reason for discharge is either "reunification with parents or primary caretakers" or "living with other relatives;" or

Using the Trial Home Visit adjustment, the child's length of stay in foster care is less than 12 months from the date of the child's first removal from home and the reason for discharge is either "reunification with parents or primary caretakers" or "living with other relatives."

The contribution (weight) of each of these measures to the component score is determined by the coefficient resulting from the principal components analysis. The actual score is multiplied by the coefficient to achieve the actual score. This is explained further in attachment B.
Composite 1, Component B: Permanency of reunification.
The score for the permanency of reunification component of this composite was derived from State performance on the following measure:

Individual Measure C1.4: Of all children who were discharged from foster care to reunification in the 12-month period prior to the target 12-month period, what percent re-entered foster care in less than 12 months from the date of discharge?

The denominator for this measure includes children who meet all of the following criteria:

The child's date of discharge from foster care occurs during the 12-month period just prior to the 12-month target year; and

At the time of the date of discharge, the reason for discharge is either "reunification with parents or primary caretakers" or "living with other relatives."
The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet the following criterion:

The child's date of latest removal from home is less than 12 months from the date of discharge from foster care that occurred during the 12-month period just prior to the 12-month target year.

As noted above, the score for this measure contributes 50 percent to the final composite score. In the Principal Components Analysis, the actual score for this measure was adjusted to reflect performance in a positive direction so that a higher score reflects higher performance. This is explained further in attachment B.

Key features of the components and measures

Adjustments to the measures

As indicated in the information above, all measures assessing the timeliness of reunification component are adjusted to exclude children who were not in foster care for 8 days or longer. The calculation of the measures also is adjusted to include children who are placed in a trial home visit prior to discharge from foster care to reunification if the trial home visit meets specific conditions (as noted in the description of the calculation of the measures above). Most respondents to the Federal Register notice who commented on these adjustments expressed support for them.

ACF proposed that the measure of timeliness of reunification should include only children who were in foster care for 8 days or longer in order to address variation in State practices and policies concerning the placement of children in very short term foster care. We believe that for the most part, the kinds of case practices and agency efforts necessary to achieve a timely reunification for a child who has been removed from home and placed in foster care are not usually applicable for these very short-term placements. Initially, we also proposed a measure that required that a child be in foster care for 30 days or longer in order to be included in the analysis. This measure was eliminated from the composite after the principal components analysis revealed a very high correlation between the 30-day and 8-day adjustment measures, suggesting that the measures capture the same information. In addition, there was more support among respondents to the Federal Register notice for the 8-day measure than there was for the 30-day measure. To assist States in understanding how this adjustment impacts their performance, we will provide data in the State Data Profile regarding the percentage of children entering foster care in a fiscal year who are discharged from foster care in less than 8 days after the date of removal from the home.

ACF initially proposed the trial home visit adjustment to the measures of timeliness of reunification in order to address variations in State policy regarding returning children to their families (parents, relatives, or other caretakers) for a period of time before a discharge from foster care. This practice often is referred to as "physical reunification" to distinguish it from a reunification in which custody is transferred to the parents or relatives. For the most part, the purpose of this practice is to monitor and assist families in the reintegration process. This practice may be required in State statute, written into agency policy, or reflect standard case practice in a State.

Many respondents recommended that for purposes of the CFSR, ACF should consider "physical reunification" as equivalent to a discharge from foster care to reunification. We are unable to do this because the CFSR data profile considers children as reunified only if there is a discharge from foster care and if the discharge reason reported to AFCARS is "return to family" or "live with relatives." Once discharged, the child is no longer reported to AFCARS, unless the child re-enters foster care. There is no data element in AFCARS that would allow us to know specifically that a child has been physically reunified.

We believe that the trial home visit adjustment we have made to the measures of timeliness of reunification captures information about the time in foster care of most children who were physically reunified prior to an actual discharge from foster care. States that return children to their families prior to discharge usually report them as being in a "Trial Home Visit," which is one of the placement categories in AFCARS, although they may not actually consider the placement a "trial." Through a review of the data, we determined that a trial home visit placement of longer than 30 days that resulted in an eventual discharge to reunification captures the vast majority of instances that may be considered "physical reunification."

The Trial Home Visit adjustment only applies to the three measures included in the timeliness of reunification component. A child is eligible for the trial home visit adjustment if all of the following criteria are met:

The child has a date of discharge from foster care that occurs during the 12-month target period and the reason for discharge is either "reunification with parents or caretakers" or "living with other relatives;"
At the time of discharge from foster care, the child is in a "current placement setting" of "Trial Home Visit," and At the time of discharge from foster care, the child had been in the placement setting of trial home visit for longer than 30 days.

If these criteria are met, the child's calculated length of stay in foster care prior to reunification or live with relative is determined in the following way: First, the number of days between the child's latest removal from home and the date of placement in the trial home visit setting is determined. Then, 30 days are added to that number of days to provide the calculated "length of stay in foster care" prior to reunification.

Timeframe for reunification

Several respondents expressed concern that most of the measures proposed for this composite continue to focus on 12 months as the appropriate time period for assessing timeliness of reunification. These respondents suggested that a 12-month timeframe is not sufficient in many cases to achieve reunification, particularly for families in which parental substance abuse was a key reason for a child's removal from the home. They noted that 12 months is not sufficient for a parent to receive and complete substance abuse treatment services. These respondents recommended that the timeframe be extended to either 18 or 24 months to reflect the reality of many of the families whose children are in foster care.

ACF acknowledges that it is not always feasible or desirable for all children to be reunified with their families in less than 12 months and we have no expectation that this goal will be accomplished for 100 percent of the children who are eventually reunified. However, we believe that the focus of the measure on reunifications occurring in less than 12 months emphasizes the responsibility of child welfare agencies to return children to safe homes as quickly as possible. This includes working quickly and intensively with parents with difficult issues such as substance abuse to address the problems that resulted in the child's removal from home. In addition, we have incorporated a measure of median length of stay in foster care to reunification that does not specify a 12-month timeframe.

Inclusion of three measures in the timeliness of reunification component

Several respondents to the Federal Register notice suggested that the measure of reunification that follows an entry cohort of children is sufficient to capture State performance with regard to timeliness of reunification. They expressed the opinion that other measures of timeliness are not necessary, and in fact, are not valid in assessing timeliness. From the beginning of this process, ACF determined that the decision regarding the measures to be incorporated in the composite would be based primarily on the empirical results of the principal components analyses. For the timeliness of reunification component, the results of the analysis revealed that, although there is overlapping information, each of the three measures chosen for the composite makes a substantial contribution to explaining the variation in performance regarding timeliness (see attachment B for the results of the analysis). For example, the entry cohort measure only captures information about children who enter foster care in the second 6 months of the year who are reunified in less than 12 months of the time of entry into foster care. It does not provide information about what happens to the children who are not reunified in that time frame. As indicated in table 1, the median across States for the percentage of children entering foster care in the second six months of a fiscal year who are reunified in less than 12 months is 35.1 percent. This indicates that there are substantial numbers of children who are not reunified in less than 12 months of entering foster care. Although no measure is ideal, we believe that by combining all three measures in the timeliness of reunification component we are able to incorporate a broader picture of State performance with regard to reunifying children in a timely manner than we are able to capture with any single measure.

We acknowledge, however, that an entry cohort approach would be able to capture a wider range of information if each entry cohort for each year could be followed for several years. Although the timeframe for the CFSR precludes this type of analysis, it is possible for a State to use a multiple year entry cohort analysis to assess its own performance and progress. We also are aware that there are statistical procedures available to estimate the percentage of children entering foster care who are likely to be reunified within various timeframes. However, because the CFSR can result in penalties for a State, ACF determined that estimates of performance with regard to achieving particular outcomes are not appropriate. Most respondents to the Federal Register notice agreed with this determination and did not want the CFSR to use measures requiring statistical projections.

Inclusion of a measure of foster care re-entry as part of the reunification composite

As noted in the Federal Register notice, ACF proposed that State performance with regard to children re-entering foster care in less than 12 months of a prior foster care episode would be incorporated into the composite assessing the timeliness and permanency of reunification. In the first round of the CFSR, the re-entry measure was assessed separately from the timeliness of reunification measure. Although ACF believes that it is important to reunify children with their families as quickly as possible, we also believe that children should not be reunified until sufficient changes are made to prevent the child being removed from the home again. The majority of respondents supported the inclusion of a measure of foster care re-entry as part of a single composite assessing the timeliness and permanency of reunification.

In addition, the measure of foster care re-entry that was used in the first round of the CFSR has been revised to reflect a longitudinal analysis. The new measure follows children who exited foster care to reunification in the 12-month period prior to the CFSR target 12-month period to identify the percentage who re-enter in less than 12 months of the time of exit. All respondents commenting on this measure indicated support for this change.

2. Permanency composite 2: Timeliness of adoptions

The principal components analysis of the performance measures proposed for the timeliness of adoption composite yielded three components. One component pertains to the timeliness of adoptions of children exiting foster care to adoption. The second component assesses progress toward adoption of a cohort of children who have been in foster care for 17 months or longer and therefore meet the ASFA time-in-foster care requirements regarding the State filing for a termination of parental rights and pursuing adoption unless there is an exception ⁴. This may be found in section 475 (5) (E) and (F) of the Social Security Act. The third component pertains to the timeliness of adoptions of a cohort of children for who are "legally free" for adoption. Legally free means that there is a termination of parental rights for each of the child's living parents.

Each component has a unique score and each contributes 33.3 percent to the final composite score. The contribution of the individual measures to the score for each component is determined by the results of the principal components analysis, as explained further in attachment B. Data pertaining to the composite score and individual measures are presented in table 1.

Composite 2, Component A. Timeliness of adoptions of children exiting foster care

The score for the component pertaining to timeliness of adoptions of children exiting foster care was derived from performance on the following measures:

Individual Measure C2.1: Of all children who were discharged from foster care to a finalized adoption during the 12-month target period, what percent were discharged in less than 24 months from the date of the latest removal from home?

The denominator for this measure includes children who meet all of the following criteria:

The child has a date of discharge from foster care during the 12-month target period, and

The reason for discharge is adoption.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator, and also meet the following criterion:

The child's date of discharge is less than 24 months from the date of latest removal from home.

Individual Measure C2.2: Of all children who were discharged from foster care to a finalized adoption during the 12-month target period, what was the median length of stay in foster care in months from the date of latest removal from home to the date of discharge to adoption?

This measure includes children who meet all of the following criteria:

The child's date of discharge from foster care occurs during the 12-month target period; and

The reason for discharge is adoption.

The calculation for the median length of stay is the difference between the date of discharge from foster care to adoption and the date of the latest removal from home.

The contribution of each of these measures to the component score is provided in attachment B.

Composite 2, Component B. Progress toward adoption of children who have been in foster care for 17 months or longer.

The score for the component assessing progress toward adoption of a cohort of children who meet the ASFA time-in-foster-care requirements was derived from performance on the following measures:

Individual Measure C2.3: Of all children in foster care on the first day of the 12-month target period who were in foster care for 17 continuous months or longer, what percent were discharged from foster care to a finalized adoption by the last day of the 12 month target period? (The denominator for this measure excludes children who, by the last day of the 12-month target period, are discharged from foster care with a discharge reason of reunification with parents or primary caretakers, living with other relatives, or guardianship.)

The denominator for this measure includes children who meet all of the following criteria:

The child was in foster care for 17 or more continuous months or longer as of the first day of the 12-month target period, and

By the last day of the 12-month target period, the child had not discharged from foster care with a discharge reason of reunification with parents or primary caretakers, living with other relatives, or guardianship.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet all of the following criteria:

The child's date of discharge from foster care occurs during the 12-month target period (including the first day of the target period); and

The reason for discharge is adoption

Individual Measure C2.4: Of all children in foster care on the first day of the 12-month target period who were in foster care for 17 continuous months or longer, and who were not legally free for adoption prior to that day, what percent became legally free for adoption during the first 6-months of the 12-month target period?

A child is considered to be legally free for adoption if there is a parental rights termination date reported to AFCARS for both mother and father. Note that within AFCARS, if a parent is deceased, the date of death is reported as the parental rights termination date.

The denominator for this measure includes children who meet ALL of the following criteria:

The child was in foster care for 17 continuous months or longer as of the first day of the 12-month target period, and

The child was not legally free for adoption prior to the first day of the 12-month target period.

The denominator for this measure excludes any child who did not become legally free during the first 6 months of the target year, but who, during that 6-month period, is discharged from foster care with a discharge reason of reunification with parents or primary caretakers, living with other relatives, or guardianship.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet the following criterion:

The child became legally free for adoption during the first 6 months of the 12-month target period (including the first and last day of the 6-month period). (This includes children who may or may not have discharged from foster care during the 6-month period.)

The contribution of each of these measures to the component score is provided in attachment B.

Composite 2, Component C: Timeliness of adoptions of children who are legally free for adoption.

The score for the component assessing timeliness of adoptions for children who are legally free for adoption was derived from performance on the following measure:

Individual Measure C2.5: Of all children who became legally free for adoption during the 12 months prior to the target 12-month period, what percent were discharged from foster care to a finalized adoption in less than 12 months from the date of becoming legally free?

The denominator for this measure includes children who meet all of the following criteria:

The child has a parental rights termination date for both mother and father, and

The latest parental rights termination date occurs in the 12-months just prior to the 12-month target period.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet the following criteria:

The child has a date of discharge from foster care that occurs in less than 12 months of the latest date of parental rights termination, and

The reason for discharge is adoption.

Key features of components and measures

The timeliness of adoption composite does not include an entry cohort measure

Several respondents to the Federal Register Announcement expressed concern that the proposed timeliness of adoptions composite did not include an entry cohort measure-that is, a measure that follows children longitudinally from the date of entry into foster care to the date of the finalized adoption. As noted in the November 7th Announcement, in determining appropriate measures to test for the composite, our review of the data indicated that an entry cohort approach to assessing the timeliness of adoptions is not feasible within the timeframes of the CFSR. The reasons for this, which were indicated in the Federal Register notice, are the following:

An extensive timeframe is required to follow a cohort of children from entry into foster care to a finalized adoption and the timeframe is not consistent with the CFSR timeframes. For example, in following a cohort of children entering foster care in FY 2001, meaningful data pertaining to adoptions did not emerge until 3 years after the entry year.
Because not all children entering foster care will be adopted, and because the number of children waiting to be adopted changes each year, it is not possible to establish a stable denominator for an entry cohort measure pertaining to timeliness of adoptions. In following the FY 2001 cohort, for example, we found that the denominator for assessing adoptions changed on an ongoing basis as children in the original cohort were reunified or exited foster care for other reasons.
Although it is possible to apply statistical methods to historical data and estimate the "likelihood" of children who enter foster are in a given year being adopted within particular timeframes, ACF cannot use statistical projections to assess CFSR performance because of the potential for financial penalties associated with CSFR performance.

A few respondents suggested that the assessment of timeliness of adoptions used by the CFSR will not be meaningful without an entry cohort measure. However, we believe that the measures and components for this composite that resulted from the principal components analysis provide a comprehensive picture of State performance with regard to the timeliness of adoption and capture meaningful information. Furthermore, we believe that the three longitudinal measures of progress toward adoption that were incorporated into the composite follow a cohort of children but have a more stable denominator than an entry cohort measure and a timeframe that is consistent with the CFSR.

Measures of timeliness of adoption of children discharged from foster care to a finalized adoption.

The measure assessing the percent of adoptions occurring in less than 24 months of a child's entry into foster care is identical to the adoption-related data measure used in the first round of the CFSR. Support for this measure from the field was mixed. Some respondents expressed strong support for the measure, while others suggested that it be replaced by an entry cohort measure. Respondents expressed similar differences of opinion regarding the measure of the median length of stay of children discharged from foster care to adoption. In general, the measures are intended to focus on timeliness of adoption by considering children who have already experienced that outcome. One measure does this by focusing on a specific timeframe (i.e., 24 months), while the other addresses the range of possible time periods, with a focus on the median time in foster care. The results of the principal components analysis indicate that taken together, these two measures account for a large percentage of the variation in State performance with regard to the timeliness of adoptions of a cohort of children who have exited foster care to adoption.

Longitudinal measures of a cohort of children who have been in foster care for 17 months or longer.

The two measures that follow the progress toward adoption of a cohort of children who have been in foster care for 17 months or longer are intended to address the ASFA time-in-foster care requirement for States to file for a termination of parental rights and pursue adoption unless there is an exception. Several respondents to the Federal Register notice suggested that many children who have been in foster care for 17 months or longer will exit foster care to a permanency option other than adoption or will meet the exceptions noted in ASFA. They recommended, therefore, that these measures be limited to children who have a case goal of adoption.

We determined, however, that if we include in the measure only children who have a goal of adoption reported to AFCARS, we will miss those children who have other goals, but for whom adoption needs to be considered because of the length of time they have been in foster care and because they do not meet the criteria for an exception. Also, if we include in the measure only children who have a goal of adoption reported to AFCARS, we will miss those children for whom the agency is working toward adoption, but has not yet reported a goal change to AFCARS. Therefore, to address this issue in part, we incorporated an adjustment to the two measures that excludes (in retrospect) from the denominator any child who was in foster care for 17 months or longer on the first day of the target 12-month period but who discharged during the 12 month period to reunification, live with relative, or guardianship. This reduces the initial denominator to account for children exiting foster care to permanency options other than adoption.

In addition, as with all other data measures used for the CFSR, there is no expectation that a State achieve a particular goal for 100 percent of the children who are included in the denominator of a specific measure. However, ACF believes that the ASFA requirement regarding the State filing a TPR and pursuing adoption, unless there is an exception, reflects a national concern that State child welfare agencies make concerted efforts to ensure that children who cannot be reunified are legally freed for adoption and adopted as quickly as possible.

Longitudinal measure of the percent of children who become legally free for adoption in a given year who are adopted in less than 12 months of becoming legally free.

Although respondents to the initial Federal Register notice generally supported this measure, a few expressed concerns about the accuracy of information reported to AFCARS regarding termination of parental rights. Our review of the data indicated that there are a few States that do not appear to report information about termination of parental rights to the AFCARS Foster Care File, or who report this information for only a very few children. However, most States appear to be reporting this information fairly consistently, although they may not be reporting it in all instances. We believe that the problem of inconsistencies can be resolved by States improving their reporting to AFCARS on the data elements pertaining to termination of parental rights.

3. Permanency composite 3: Achieving permanency for children in foster care

The principal components analysis of the performance measures proposed for the composite addressing achieving permanency for children yielded two components. One component pertains to achieving permanency for children in foster care for long periods of time, and the other pertains to the issue of children growing up in foster care and exiting to emancipation or reaching their 18th birthday while in foster care. A State's score for each component contributes 50 percent to the State's total score for this composite. As noted for the other composites, the scores for the individual components are derived from the contribution of each of the measures to the component, as determined by the coefficient resulting from the principal components analysis.

Composite 3, Component A: Achieving permanency for children in foster care for extended periods of time

The score for the component pertaining to achieving permanency for children in foster care for long periods of time was derived from performance on the following measures:

Individual Measure C3.1: Of all children who were in foster care for 24 months or longer on the first day of the 12-month target period, what percent were discharged to a permanent home by the last day of the 12-month period and prior to their 18th birthday?

A child is considered as discharged to a permanent home if the discharge reason reported to AFCARS is reunification with parents or primary caretakers, living with other relatives, guardianship, or adoption.

The denominator for this measure includes children who meet the following criterion:

The child is in foster care for 24 continuous months or longer on the first day of the 12-month target period.

The numerator for this measure includes children who meet the criterion for the denominator and also meet all of the following criteria:

The child's date of discharge from foster care occurs during the 12-month target period;

The child's reason for discharge is reunification with parents or primary caretakers, living with other relatives, guardianship, or adoption; and

The child's date of discharge from foster care occurs before the child's 18th birthday.

Individual Measure C3.2: Of all children who were discharged from foster care during the 12-month target period, and who were legally free for adoption (i.e., there is a parental rights termination date for both parents) at the time of discharge, what percent were discharged to a permanent home prior to their 18th birthday?

The denominator for this measure includes children who meet all of the following criteria:

The child has a parental rights termination date for both mother and father at the time of discharge from foster care, and

The child has a date of discharge from foster care that occurs during the 12-month target period.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator, and also meet all of the following criteria:

The child has a discharge reason of reunification with parents or primary caretakers, living with other relatives, guardianship, or adoption; and

The child's date of discharge is prior to the child's 18th birthday.

Composite 3, Component B: Children growing up in foster care

The score for the component addressing children growing up in foster care was derived from performance on the following measure:

Individual Measure C3.3: Of all children who either (1) were, prior to age 18, discharged from foster care during the 12-month target period with a discharge reason of emancipation, or (2) reached their 18th birthday while in foster care but had not yet been discharged from foster care, what percent were in foster care for 3 years or longer?

The denominator for this measure includes children who meet one of the following criteria:

The child has a date of discharge from foster care during the 12-month target period that occurs prior to the child's 18th birthday and the reason for discharge is "emancipation;" or

The child reaches his or her 18th birthday during the 12-month target period and is in foster care at the time of the birthday.

The numerator for this measure includes children who meet one of the criteria for inclusion in the denominator and also meet one of the following criteria:

The time from the date of the latest removal from home and the date of discharge is equal to, or greater than, 3 years; or

The time from the date of the latest removal from home and the date of the child's 18th birthday is equal to, or greater than, 3 years.

In AFCARS, emancipation is defined as "the child reached majority according to State law by virtue of age, marriage, etc." The actual score for this measure was adjusted to reflect performance in a positive direction so that a higher score reflects higher performance. This is explained further in attachment B.

Key features of the composite, components, and measures

Inclusion of guardianship in the assessment of achieving permanency

A key feature of this component is that guardianship is included as one of the permanency options in two of the measures. Several respondents to the November 7th Federal Register notice expressed concern that the CFSR data measures do not assess State performance with regard to achieving guardianship as a permanency option. In response to this concern, ACF analyzed the data for guardianship and found that, nationally, only a very small percentage of children are discharged from foster care to guardianship. In several States, no children are discharged from foster care to guardianship, suggesting that guardianship is not a permanency option in these States. These small numbers did not permit a separate composite or measure focusing on timeliness of achieving guardianship. However, because we recognize that many States have made concerted efforts to achieve permanency for children through guardianship, we included guardianship as a permanency option in the two measures that assess achieving permanency for children.

Longitudinal analysis of a cohort of children in foster care for 24 months or longer

Many respondents expressed concern that most of the existing measures pertaining to adoption and reunification do not capture general permanency information for children in foster care for a relatively long period of time. In response to this concern, ACF developed a measure to assess discharges to permanency of children in foster care for 24 months or longer. The 24 month period was chosen because, nationally, about 50 percent of the children in foster care on any given day have been in foster care for about 2 years or longer. The new measure allows an assessment of what happens to these children in a 12-month time period.

Addressing concerns regarding "legal orphans"

The measure of achieving permanency for children who are discharged from foster care and who were legally free for adoption at the time of discharge addresses the concern of the field that by pursuing termination of parental rights for children who have been in foster care for 15 of the most recent 22 months, the field may be creating "legal orphans," that is, children who have no legal parents and for whom no permanent home is found. The data for this measure suggest that the vast majority of children who are discharged from foster care prior to their 18th birthday and who are legally free for adoption are discharged to a permanent home (including guardianship, adoption, and reunification). However, despite the large percentages, ACF decided to maintain the measure because it is important for States to make concerted efforts to ensure permanency for all children for whom a termination of parental rights has been granted for each living parent.

Addressing the issue of children emancipated from foster care after many years in foster care

One objective of ASFA was to ensure that child welfare agencies make concerted efforts to ensure that children do not spend many of their childhood years in foster care, only to leave foster care without having found a permanent home. Our initial measure to address this concern focused on the percentage of children emancipated from foster care or reaching their 18th birthday while in foster care who entered foster care when they were age 12 or younger. However, a few respondents noted that this measure was more likely to reflect the variation among States with regard to the ages of children at the time of entry into foster care than it was to capture the general issue of children growing up in foster care. In response to this concern, we revised the measure to focus on the length of time in foster care of children emancipated from foster care rather than the age at entry into foster care. Due to our criteria of having been in foster care for 3 years or longer, the revised measure excludes children who exit to emancipation who entered foster care at approximately age 15 or older. This addresses a large portion of the variation among States with regard to the age of children at the time of entry into foster care.

4. Permanency composite 4: Placement stability

The principal components analysis for this composite yielded one component that incorporates the following three measures:

Individual Measure C4.1: Of all children who were served in foster care during the 12-month target period, and who were in foster care for at least 8 days but less than 12 months, what percent had two or fewer placement settings?

The denominator for this measure includes children who meet all of the following criteria:

The child is in foster care at some time during the 12-month target period, and

The child's length of stay in foster care during the most recent foster care episode is at least 8 days but less than 12 months.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet the following criterion:

The child's number of placement settings during this removal episode does not exceed two (2).

Individual Measure C4.2: Of all children who were served in foster care during the 12-month target period, and who were in foster care for at least 12 months but less than 24 months, what percent had two or fewer placement settings?

The denominator for this measure includes children who meet ALL of the following criteria:

The child is in foster care at some time during the 12-month target period, and

The child's length of stay in foster care during the most recent foster care episode is at least 12 months but less than 24 months.

The numerator for this measure includes children who meet ALL of the criteria for inclusion in the denominator and also meet the following criterion:

The child's number of placement settings during this removal episode does not exceed two (2).

Individual Measure C4.3: Of all children who were served in foster care during the 12-month target period, and who were in foster care for at least 24 months, what percent had two or fewer placement settings?

The denominator for this measure includes children who meet all of the following criteria:

The child is in foster care at some time during the 12-month target period, and

The child's length of stay in foster care during the most recent foster care episode is at least 24 months.

The numerator for this measure includes children who meet all of the criteria for inclusion in the denominator and also meet the following criterion:

The child's number of placement settings during this removal episode does not exceed two (2).

Data pertaining to the composite score and individual measures are presented in table 1. The contribution of each measure to the composite score is determined by the results of the principal components analysis, as described further in attachment B.

Key features of composite and measures

This composite includes one measure that is similar to the measure of placement stability used in the first round of the CFSR-placement stability for children who have been in foster care for less than 12 months. The one revision to this measure is that it includes only children who have been in foster care for 8 days or longer. We made this revision in response to concerns expressed by respondents regarding including children in foster care for very short periods of time in the measure of placement stability. However, if a child is in care for 8 days or longer, the placement changes that occurred during the first 8 days in foster care are considered in the measure. Two additional measures were added to the composite to address the issue of placement stability for children in foster care for longer periods of time. ACF believes that placement stability is as important to the well-being of children in foster care for 2 years or longer as it is for children who have been in foster care for only a few months. Most respondents to the Federal Register notice expressed support for this composite and the measures. However, respondents raised the following concerns regarding the measures:

The measures do not define what constitutes a placement change. This issue has been raised in the past regarding reporting placement changes to AFCARS. Clarification was issued to the States in CWPM 1.2B.7.

The measures do not capture variations with regard to time in foster care. Respondents noted that the children included in the measure who were in care for less than 12 months could have been in care for only a few weeks or for several months. In response to this concern, ACF examined alternative approaches to this measure, including an entry cohort approach. However, unless a measure specified that all children were in foster care for a specified time period, all of the approaches considered had the same problem. For example, we could address this problem if we included in the measure only children who were in foster care for at least 11 of the 12 months. However, this does not capture the issue of placement stability for children who are in foster care for short periods of time. Our review of the data indicated that these children can experience multiple placements as well as those children in foster care for longer periods of time. Consequently, we have maintained the measure as proposed with the exclusion of children who were in foster care for less than 8 days.

ACF should expand the definition of placement stability from two placement settings to three placement settings for children who have been in foster care for longer than 12 months and for older children because two placement settings is not a realistic measure of placement stability for these children. In developing the outcome measures for the Annual Report to Congress on Child Welfare Outcomes, ACF engaged in a broad-based consultation process with stakeholders in the field, including representatives from State and county child welfare agencies, child advocacy organizations, and child welfare researchers. With regard to the outcome measure pertaining to placement stability, ACF, based on input from these stakeholders, established a definition of placement stability as a child experiencing two or fewer placement settings. The decision to have two placement settings in the definition instead of one was based on the following: (1) often it is difficult to determine the most appropriate placement setting at the time of the child's initial removal from home; and (2) in many States, children are placed in a shelter type placement for a short period of time in order to assess the needs of the child and determine the most appropriate placement. We have decided not to increase the number of placement settings in our definition of placement stability for any of the measures. One reason for this is that our existing definition was established in consultation with key stakeholders in the child welfare field. In addition, placement stability is a critical component of the well-being of children in foster care. States are responsible for ensuring that children who are removed from their homes by the State experience stability while they are in foster care. It is not in the best interest of a child to experience multiple placement settings regardless of the time that the child is in foster care, the child's age, or the reason for the child's entry into foster care.

The placement setting information does not capture changes in placement settings that are positive changes. AFCARS does not have information about whether a placement change reflects a positive move for the child. For example, changing a child's placement in order to move the child closer to the parents to facilitate more frequent visits. It is difficult to assess whether a placement change is positive for the child without contextual information about various factors such as the needs of the child and the existing conditions of the child's placement. For example, a child may change placements because of the death or illness of a foster parent, or because the child is in need of a specific type of treatment. The question of whether a placement change is in a positive direction is addressed in the case review component of the CFSR because more information about the child and the placements is available in that process. As noted previously, although we cannot account for these events in the data measure, we also do not expect that 100 percent of the children in any of the specified time-in-care timeframes will experience no more than two placement settings.

This announcement is intended to provide information about the national data that will be used in the next round of the CFSR as a component of the overall assessment of a State's substantial conformity with two of the seven CFSR outcomes. The attachments to this announcement provide supplementary information regarding the methodology used in developing the data composites.

Dated:

Joan E. Ohl,

Commissioner, Administration on Children, Youth and Families

TABLE 1: RANGES, MEDIANS, AND NATIONAL STANDARDS FOR THE CHILD AND FAMILY SERVICES REVIEW (CFSR) DATA INDICATORS

Attachment A: List of Data to be Included in the State Data Profile

Prior to development of the Statewide Assessment for the CFSR, each State will receive a State Data Profile. This profile will continue to include the information that was provided in the first round of the CFSR. It also will include new information regarding composite scores, the measures for the composites, and additional information relevant to the composites. This attachment provides a list of the general kinds of data that will be provided to States in the State Data Profile. Additional information may be added to the State Data Profile at a later date. Most of the data will be provided for 3 years. However, the States to be reviewed in the first year of the CFSR will have only 2 years of data for each of the composites and composite measures.

Descriptive Information

Descriptive information currently included in the State Data Profile

A. Descriptive Information from the National Child Abuse and Neglect Data System (NCANDS)

The number of reports alleging maltreatment of children that reached a disposition within the reporting year, the total number of reports, the number of unique children associated with reports alleging maltreatment, and the number of "duplicate children."

The numbers and percentages of reports that were given a disposition of Substantiated and Indicated, Unsubstantiated, or Other, and the numbers and percentages of duplicate and unique children.

The numbers and percentages of unique and duplicate child victim cases opened for services, based on the number of victims during the reporting period under review.

The numbers and percentages of duplicate and unique victims entering foster care in response to a child abuse/neglect report.

The number of child fatalities.

B. Descriptive information from the Adoption and Foster Care Analysis and Reporting System (AFCARS). (Where relevant, the descriptive data identified below will be provided for both a point-in-time analysis and for a cohort of children entering foster care in a given year.)

. Number of children in foster care on the first and last day of the fiscal year and number of children entering and exiting foster care in the fiscal year
Placement settings for children in foster care
Case plan goals for children in foster care
Number of placement settings in the current foster care episode
Number of foster care episodes of children in foster care at the end of the fiscal year
Number and percentage of children in foster care for 17 of the most recent 22 months, calculated from the number of all children in foster care on the last day of the fiscal year
The median length of stay (months) in foster care of children in care on the last day of the year
Number of children who discharged to each type of permanency goal and the length of stay in foster care (in months) for those children who discharged to each permanency goal

New descriptive information to be included in the State Data Profile

A. New descriptive information from NCANDS

The mean time from receipt of an allegation of child maltreatment to the initiation of an investigation.
The median time from receipt of an allegation of child maltreatment to the initiation of an investigation.
The percent of children in foster care who are the subject of a substantiated or indicated maltreatment where the perpetrator is a parent.

B. New descriptive information from AFCARS

The number and percent of children entering foster care in the fiscal year who were in care for 7 days or less before being discharged from foster care.
The number and percent of children exiting foster care in the fiscal year who were in foster care for 7 days or less.

Analytical Information

Analytical information currently included in the State Data Profile

A. Current analytical information from NCANDS

Maltreatment recurrence: Of all children who were victims of abuse and/or neglect during the first 6 months of the reporting year, the percent that were victims of another abuse or neglect incident within a 6-month period.
Maltreatment of children in foster care: Of all children who were in foster care during the reporting year, the percent that were victims of abuse and/or neglect by a foster parent or facility staff member.

B. Current analytical information from AFCARS

Time to Reunification: For the reporting year, of all children who were reunified with their parents or caretakers at the time of discharge from foster care, the percent that were reunified in less than 12 months from the time of the latest removal from home.
Time to Adoption: For the reporting year, of all children who exited foster care to a finalized adoption, the percent that exited foster care in less than 24 months from the time of the latest removal from home.
Placement Stability: For the reporting year, of all children served who have been in foster care less than 12 months from the time of the latest removal from home, the percent that have had no more than two placement settings.
Re-entry into foster care: Of all children who entered foster care during the reporting year, the percent that re-entered foster care within 12 months of a prior foster care episode.

New analytical information to be included in the State Data Profile

A. New analytical information from NCANDS

Maltreatment recurrence: Of all children who were victims of abuse or neglect during the first 6 months of the reporting year, the percent that were not victims of another maltreatment within a 6-month period.
Maltreatment of children in foster care: Of all children who were in foster care during the reporting year, the percent that were not victims of maltreatment by a foster parent or facility staff member.

B. New analytical information from AFCARS

The composite score for Permanency Composite 1: Timeliness and permanency of reunifications and the national standard for this composite.
Data pertaining to actual performance on the measures included in Permanency Composite 1. These are as follows:

For the reporting year, of all children discharged from foster care to reunification who had been in foster care for 8 days or longer, the percent that met either of the following criteria: (1) the child was reunified in less than 12 months from the date of the latest removal from home, or (2) the child was placed in a trial home visit within 11 months of the date of the latest removal and the child's last placement prior to discharge to reunification was the trial home visit.
For the reporting year, of all children discharged from foster care to reunification who had been in foster care for 8 days or longer, the median length of stay in months from the date of the most recent entry into foster care until either of the following: (1) the date of discharge to reunification; or (2) the date of placement in a trial home visit that exceeded 30 days and was the last placement setting prior to discharge to reunification.
For the reporting year, of all children entering foster care in the second 6 months of the year who remained in foster care for 8 days or longer, the percent who met either of the following criteria: (1) the child was reunified in less than 12 months from the date of entry into foster care, or (2) the child was placed in a trial home visit in less than 11 months from the date of entry into foster care and the trial home visit was the last placement setting prior to discharge to reunification.
Of all children exiting foster care to reunification in the year prior to the reporting year, the percent that re-entered foster care in less than 12 months from discharge from a prior episode.

The composite score for Permanency Composite 2: Timeliness of adoptions
Data pertaining to State performance on the following measures included in Permanency Composite 2.

For the reporting year, of all children who were discharged from foster care to a finalized adoption during the year, the percent that were discharged in less than 24 months from the date of the latest removal from the home.
For the reporting year, of all children who were discharged from foster care to a finalized adoption, the median length of stay in foster care (in months) from the date of removal from the home to the date of discharge to adoption.
For the reporting year, of all children in foster care on the first day of the year who were in foster care for 17 continuous months or longer, the percent that were discharged from foster care to a finalized adoption before the end of the fiscal year.
For the reporting year, of all children in foster care on the first day of the year who were in foster care for 17 continuous months or longer, the percent that became legally free for adoption within 6 months from the beginning of the fiscal year.
For the reporting year, of all children who became legally free for adoption, the percent that were discharged from foster care to a finalized adoption in less than 12 months of becoming legally free?

The composite score for Permanency Composite 3: Achieving permanency for children in foster care.
Data pertaining to State performance on the following measures included in Permanency Composite 3.

For the reporting year, of all children who were discharged from foster care who were legally free for adoption (i.e., there was a TPR for each living parent), the percent that were discharged to a permanent home prior to their 18th birthday, with a permanent home defined as having a discharge reason of adoption, reunification (including live with relative), or guardianship.
Of all children who were in foster care for 24 months or longer on the first day of the reporting year, the percent that were discharged from foster care to a permanent home prior to their 18th birthday and by the end of the fiscal year.
During the reporting year, of all children who were emancipated from foster care or reached their 18th birthday while in foster care, the percent that had been in foster care for 3 years or longer.

The composite score for Permanency Composite 4: Placement stability
Data pertaining to the following measures in Permanency Composite 4.

For the reporting year, of all children in foster care who were in foster care for 8 days or longer and less than 12 months, the percent that had two or fewer placement settings.
For the reporting year, of all children in foster care who were in foster care for at least 12 months but less than 24 months, the percent that had two or fewer placement settings.
For the reporting year, of all children in foster care during the year who were in foster care for 24 months or longer, the percent that had two or fewer placement settings.

ATTACHMENT B: METHODOLOGY FOR DEVELOPING THE COMPOSITES

After the first round of the Child and Family Services Review, the Administration for Children and Families (ACF) conducted a review of possible additional measures for assessing State performance with regard to achieving permanency and placement stability for children in foster care ⁵. The purpose of the review was to increase the pertinent data used as part of the assessment of a State's substantial conformity with CFSR outcomes. The goal was to enhance the understanding of State performance on the outcomes assessed through the CFSR.

The review of potential measures was guided by a consideration of the following key performance areas reviewed in the CSFR: (1) timeliness and permanency of reunifications; (2) timeliness of adoptions; (3) achieving permanency for children in foster care for long periods of time; and (4) placement stability. Multiple measures were developed for consideration within each performance area. ACF determined that all measures considered had to meet the following criteria:

Measures must include data currently collected through the Adoption and Foster Care Analysis and Reporting System (AFCARS). For example, although it would be useful to be able to assess such variables as adoption dissolution or the quality of a child's placement, neither type of information is collected through AFCARS. However, ACF encourages State child welfare systems to conduct their own analyses of issues such as these to further understand the outcomes experienced by the children they serve.
Measures must meet the timeframe requirements of the CFSR. Each measure must be able to be assessed consistent with the period under review and the period necessary for assessing progress in the Program Improvement Plan (PIP).
Measures must assess outcomes that are consistent with titles IV-B and IV-E of the Social Security Act and the Social Security Amendments of 1994 which authorized the reviews. While Congress granted ACF the authority to monitor the progress of State child welfare agencies, there are limits to our statutory authority with regard to the CFSR. For example, the authorization for the CFSR does not include monitoring for adherence to the requirements of the Chafee Foster Care Independence Act or to the requirements of the Indian Child Welfare Act.
Measures must incorporate an assessment of events that have actually occurred rather than be based on statistical projections of the likelihood of an event occurring sometime in the future. Although ACF is aware of the statistical procedures that can be used to estimate the likelihood of particular outcomes occurring within particular timeframes, we do not believe that it is appropriate to use these methodologies in the CFSR assessment because there are penalties associated with State performance.

The measures that we developed were presented to the public for comment in a Federal Register notice published on November 7, 2005. Based on feedback from the field and additional data analyses, several measures were eliminated from consideration or revised to more effectively capture the intended objectives.

Our initial goal was to expand the information used in the data indicators. However, ACF did not want to increase the complexity of the CFSR by having multiple measures with national standards for each measure. Instead, our goal was to implement a methodology that would allow us to create a set of composite scores, with each composite score reflecting performance on several inter-correlated measures. To assist us in achieving this goal, we hired an internationally known expert statistician as a consultant. After reviewing several possible statistical methodologies, we determined that a Principal Components Analysis (PCA) was the most appropriate approach.

A PCA is a commonly used and widely accepted statistical technique for reducing a large set of variables into a smaller set. The PCA not only combines inter-correlated variables but also identifies those that are redundant because they are very highly inter-correlated. Each variable in the set is given a weight in accordance with its relative contribution to the set as a whole. The resulting principal components are more stable and easier to interpret than individual measures because several individual variables are related to one another. The principal components that result from a PCA can be used as data for other types of statistical analyses, such as survival analysis, discriminant function analysis, and multiple regression analysis.

Although a PCA can be used to test hypotheses or theories, ACF did not use it for this purpose. Instead, we used the PCA as an exploratory tool. In an exploratory PCA, the goal is to describe and summarize data by grouping together variables that are correlated. As noted by Tabachnik and Fidell ⁶, PCA is different from factor analysis, which focuses on shared variance among variables. "In a PCA, all variance in the observed variables is analyzed, including common, unique, and error variance. The resulting components are simply aggregates of existing variables. There is no underlying theory about which variables should be associated with which factors; rather relationships emerge based solely on empirical associations. It is understood that any labels applied to derived components are merely convenient descriptions of the combination of variables associated with them. These labels are intended to describe the critical core outcomes being assessed. "

Using the PCA to develop composite scores for the CFSR

This section presents a discussion of the methodology used to implement the PCA. The definitions of the terms used and the conceptual structure are as follows:

Measure: In the discussion below, this term refers to the variables included in each PCA. Performance on each measure provides the basic data for the PCA. We have used the term measure rather than variable to clarify that it is performance on the specific measures described in this Federal Register Announcement that is considered as the focus of analysis.
Component: This term refers to the general factors that comprise a given composite. In our analysis, the number of components in a composite ranges from one to three.
Composite: This term refers to the general performance area assessed, i.e., timeliness and permanency of reunification, timeliness of adoptions, achieving permanency for children in foster care for long periods of time, and placement stability.
Results: This term refers to the output from each data analysis for each composite. That is, the analysis may be said to produce results for each composite.
Solution: This term refers to the overall pattern of results across multiple data analyses.

PCA requires a sample size of 500 or more units to achieve maximum stability in the solution. Therefore, ACF decided from the outset that the unit of analysis would be performance on the measures included in each composite domain at the county rather than at the State level. Because many counties often serve very small numbers of children in foster care, the number of children served in foster care in each of the 2,984 counties was calculated (using the FIPScodes)⁷. Small counties within a given State were combined (i.e., "rolled up") to represent a single "county" that served at least 50 children in foster care in FY 2004 ⁸. This resulted in a total of 2,119 "counties" that could possibly be included in the analysis ⁹.

Once the "counties" were established, the PCA was implemented using the steps described below.

Step 1. Rank order the counties and assign each county to one of two samples - Set A or Set B. Using matched-pair sampling, each county was randomly assigned to one of two sets-Set A or Set B. In the matched-pair sampling, counties first were ranked in descending order in terms of "size," with size defined as the number of children served in foster care in the county during the fiscal year. The counties were then paired on the basis of size, with each pair including counties of the same general size. After this matched pairing, each county in the pair was randomly assigned to either Set A or Set B. The result was that Set A and Set B were matched with respect to the size of the counties within each set. The two Sets were not matched on any other variable. We created these two sets in order to cross-validate our PCA results by comparing the solutions resulting in each set.

Step 2. Calculate the performance of each county on each measure. The performance of each county on each measure was calculated using the programming syntax developed for each measure as applied to data reported to AFCARS for FY 2003 and FY 2004 ¹⁰. The focus of analysis was on data reported for FY 2004. FY 2003 data were used when more than a 12-month time span was required to calculate the measure.

Step 3. Standardize the scores. The results were standardized by converting the actual score for each county to a z-score. The use of standardized scores rather than actual calculated results allows for variables measured in different units to be included in the analysis. For example, median length of stay in foster care is calculated in months, while reunification within 12 months is calculated in percentages. Standardized scores are helpful for two reasons: (a) all variables are converted to the same scale of measurement, and (b) scores for each variable are normally distributed. The z-scores were adjusted for the direction of the measure. For example, a positive score on one measure can indicate positive performance or negative performance, depending on the focus of the measure. To adjust for this, z-scores for some of the measures were multiplied by -1 to ensure that all scores are interpreted in the same way. That is, the higher the score the better the performance. The following measures were recoded to adjust for direction:

Median length of stay in foster care of children reunified;
Median length of stay in foster care of children discharged from foster care to a finalized adoption;
Percent of children discharged from foster care who re-entered in less than 12 months from the time of exit; and
Percent of children who emancipated from foster care or who reached their 18th birthday while in foster care who were in foster care for 3 years or longer.

Step 4. Conduct a PCA analysis on Set A and Set B independently. Using the Statistical Packages for the Social Sciences (SPSS) statistical software, we ran the PCA for Set A and Set B separately for each of the four composite areas.

Step 5. Decide what component variables to include for each composite measure. After the initial analyses, we reviewed the results and made decisions regarding the variables to be included in each composite measure in accordance with the standard procedures for conducting a PCA. All decisions were data driven and were nearly identical for both Set A and Set B. For example, when two measures correlated so highly that they appeared to be capturing the same information, we eliminated one of the measures. When one or two measures did not correlate highly with other measures but still appeared to account for a high percentage of the variance in the total composite domain, we considered those as comprising a separate principal component. The goal was to identify components that accounted for as much of the sample variance as possible. That is, we wished to select the minimum number of principal components that would enable us to reproduce the observed correlations among the variables used in the analysis. A set of principal components that explained 100 percent of the variance would reproduce the data perfectly. Generally, identifying one or two principal components that explain 50 percent of the variance is considered very good. Identifying a small set of principal components that explain 70 percent of variance or more is considered excellent.

Step 6. Compare the findings for Set A and Set B. A t-test on means from two independent samples was conducted on the county component scores comparing Set A and Set B for each of the four composites. No significant differences between the Sets were found for any of the composites. The p values exceeded 0.05 for all comparisons. This indicated that the PCA of the two independent samples produced the same results.

Step 7. Create a new data set that incorporates all counties included in Set A and Set B into one data set and replicate the PCA analysis (Steps 2 through 6 above) on the combined data set to generate the Component Score Coefficient Matrix. The PCA generates what is termed a "component score coefficient" for each measure. The data analyses may result in a number of principal components, depending on the relationships among the measures as reflected in the component score coefficients. The coefficient represents the "weight" for a given measure-that is the relative contribution of the measure to the overall component. The components that emerged from the analyses combining Set A and Set B are presented below for each composite. These components were identical to those that emerged in the separate analyses of Set A and Set B. That is, the same principal components emerge consistently and explain the same proportion of variance. We have established a "name" for each component. The name reflects the focus of the measures that have the highest loading on the component. The measure with the highest loading often is referred to as the marker variable. The coefficients (or weights) for each measure within each component are provided in table 1. The higher the coefficient, the greater the contribution a particular measure makes to the component.

Permanency Composite 1-Timeliness and Permanency of Reunification. The analyses for this composite included 1,894 counties. Two components emerged from the analysis of measures included in this composite. The two components explain 73.5 percent of the variance. We named the first component timeliness of reunification, and the second component permanency of reunification. Because these components are independent from one another, each contributes 50 percent to the total composite score.
Permanency Composite 2-Timeliness of Adoptions. The analysis for this composite included 1,453 counties. Three components emerged from the analysis of measures included in this composite. Taken together, these components explain 79.8 percent of the total variance. The first component we named timeliness of adoptions of children exiting foster care to adoption. The second component, we named progress toward adoption for children in foster care for 17 months or longer. The third component we named timeliness of adoption of children who are legally free for adoption. Because these components are independent from one another, each contributes 33.3 percent to the total composite score.
Permanency Composite 3 - Achieving permanency for children in foster care for long periods of time. The analysis for this composite included 1,682 counties. Two components emerged from the analyses of these measures. These components account for 74.9 percent of the total variance. The first component we named permanency for children in foster care for long periods of time. The second component we named children emancipated after being in foster care for long periods of time. Because the components are independent of one another, each contributes 50 percent to the total composite score.
Permanency Composite 4-Placement stability. This analysis included 2,119 counties. One component, which we have named placement stability, emerged from the analysis of the measures included in this composite. The component accounts for 67.4 percent of the variance.

Step 8. Generate the component scores for each county. For each county included in the analysis, the z-score for each measure (generated under step 3) is multiplied by the coefficient for that measure (shown in table 1), resulting in a "weighted score" for each measure within the component. The weighted scores for each measure within a component are then summed. The result is a county component score.

Step 9. Generate the composite scores for each county. The county composite score represents a combination of the component scores. If there is only one component in the composite, then the county composite score and the county component score are the same. If there is more than one component in the composite, then the county composite score is the mean of the scores for each component. For example, if there are two components in a composite, then the county component scores are summed and divided by two to generate the county composite score. If there are three components in a composite, then the county component scores are summed and divided by three to generate the county composite score.

Step 10. Generate the composite scores for each State. After the composite score for each county is calculated, that score is multiplied by the number of children served in foster care in that county during the 12-month target period. The result is a "weighted" county composite score. This "weighting" allows counties with larger foster care populations to make a greater contribution to the overall State score. The weighted county scores are then summed and divided by the total number of children served in foster care in all of the counties included in the calculation. The result is the State composite score in a standard score format.

Step 11. Conduct a consolidated variable analysis. Initially, a separate PCA was conducted for each of the composite areas. At this point, we also conducted a consolidated variable PCA in order to cross-validate the solutions that emerged from the separate PCAs. That is, the PCA was applied to all of the measures taken together. The results generated from the consolidated variable analysis were identical to those that emerged from the separate PCAs; thus, the overall four-composite solution was identical across different data analyses.

Step 12. Transform State composite scores to a scale ranging from 50 to 150. The initial composite scores were derived from of z-scores. We transformed the scores into ranked scale scores by using a transformation that assures that the maximum State Composite Score attains a value of 150 and the minimum State Composite Score attains a value of 50. The other scores fall between these two limits depending on their actual State Composite Score ¹¹.

Response to concerns regarding use of PCA.

Several individuals commenting on the notice published in the November 7, 2005 Federal Register expressed concerns about our use of PCA to generate composite scores. We believe that some of these concerns are addressed in the description of PCA and our process provided in the first section of this attachment. Additional specific concerns are presented below (and underlined), followed by our response.

The use of PCA may mask the importance of individual variables and perhaps prevent States from identifying "salient contributing variables." Although the PCA shifts the focus of interpretation to a composite score rather than individual scores that make up a composite, the relative contribution of an individual measure to the composite scores will be known to States through the county weights of the number of children served and the coefficients assigned to each measure. From a statistical perspective, the more salient a particular variable or measure, the greater the weight. In a PCA, a critical measure will have a prominent role either as the "marker variable" in a PCA (i.e., the one that makes the largest contribution to the component with regard to the amount of variance for which it accounts) or as the sole measure that loads on a particular component. With regard to actual performance on individual measures, ACF will provide these data in the State Data Profile for each of the States.

The ACF proposal seems to arbitrarily group indicators together. The methodology of putting several indicators together and forcing them to be a composite single indicator contradicts the potentially powerful intent and purpose of PCA. As noted in the first section of this attachment, the PCA combines scores based on inter-correlations among the variables used in the analysis. It does not force unrelated variables onto a single component. As indicated under step 11 above, a consolidated variables analysis produced the same results as the composite-specific analyses. That is, the same variables were inter-correlated with one another in both analyses and the same components emerged.

It would be better to use other forms of analysis such as logistic regression that might demonstrate the variables predictive of a dichotomous outcome (such as maltreatment in foster care). PCA reduces a larger set of variables into a smaller set based on observed empirical relationships. In comparison, regression uses one set of variables to predict an outcome measure. Our goal in constructing composites was to identify relationships among variables that relate to a particular performance domain. Also, the goal of the CFSR is to measure performance on given outcomes rather than to predict performance on a given outcome.

PCA does not compensate for measures that are currently misunderstood or inadequately defined; it compounds the existing weaknesses in each measure. It is incorrect to say that PCA compounds weaknesses in each measure. PCA provides a well understood empirical strategy for combining variables or measures. The Federal Register Announcement provides a response to concerns about the adequacy of the measures.

Knowledge-building and the interpretation of research is greatly limited by using component factors calculated as proposed. The current set of measures has a latent structure inherent within it. PCA analysis enables us to explore that structure and identify a variety of highly interpretable PC composite scores. We believe that the results of our analyses are very strong and lead to unambiguous interpretations of the principal components used to evaluate performance.

Even sophisticated users of this method agree that the number of factors to choose when using the method is to some extent arbitrary. We used a highly conservative, data-driven approach to identify the relationships among variables. These relationships are not arbitrary; rather they are derived empirically from the data and reflect the structure inherent within the data. It is important to note that changes in extraction and rotation would have little or no impact on the present analysis as the cross-validation analysis in Step 11 indicates. In addition, all four solutions were replicated across two different samples, suggesting a high level of stability. Although every statistical procedure includes some degree of estimation error, the present analyses are robust and do not invite arbitrary interpretation of the results.

More user-friendly approaches to creating composite outcome measures are available, but not mentioned in the ACF recommendations. We believe that the options available for constructing composites from a set of data measures are principal components/factor analysis, cluster analysis, and multidimensional scaling. Based on our discussions with our expert consultant, we believe that PCA is the most appropriate option in the present case. We began our analysis of the CFSR variables making only the assumption that the variables possess some latent structure. There was no designated criterion variable that we could use as a dependent/outcome measure. Our task was to reduce an existing set of variables to a smaller set of inter-correlated composite scores. Regression/survival methods could be used if we were to select an outcome measure as the criterion that will be predicted. However, at the outset of this effort, we determined that we would not identify or use an outcome measure to estimate the weight of each variable in relation to the designated outcome variable.

Composite scores have no intrinsic meaning or relationship to important outcomes. Composite scores are used routinely in educational testing and assessment because they are more reliable in that they represent the construct of interest better than any single variable. Two basic psychometric principles of measurement are 1) a test with more questions is more reliable; and 2) combining related scores into a composite score results in a more reliable and valid score than the individual scores on which the composite is based. This is contrary to the notion that well-planned composite scores are inferior to individual scores that are used to create the composite.

No uniformly agreed methodology exists to weight individual indicators before aggregating them into a composite indicator. A uniform methodology does exist for conducting a PCA. There are many highly respected books that lay out the steps to follow and how to make critical decisions. All of these books recommend the same general process. Our approach to using PCA was very systematic and conservative. Like all statistical procedures, the researcher must make choices that impact the outcome. For example, in regression analysis, the researcher must select variables, determine an order in which they enter the analysis, and decide whether nonlinear components are relevant. The output also will depend on sample size and what population is sampled.

Establishing the National Standard

The process for establishing the national standards on the composite scores differs from that used for the first round of the CFSR. In the first round of the CFSR, the procedure for establishing the national standard for each measure involved fitting the data to a normal curve based on a level of significance of .05, and adjusting for the 75th percentile by calculating the sampling error using the lower limit of a 95 percent confidence interval for estimating the population mean. ACF determined that this procedure was not appropriate for the second round of the CFSR. One concern was that using a level of significance of .05 would result in eliminating States at either end of the range of each of the State permanency composite data indicators and safety-related data indicators in order to fit the distribution to a normal curve. Consequently, not all States would be included in the calculation of the national standards. Although this was appropriate for the first round of the CFSR because of data quality problems at the time that the standards were established, these data quality problems are no longer as significant an issue. Another concern was that using the 95 percent confidence interval would result in a considerable reduction in the actual percentile represented by each of the permanency and safety standards.

To address the concern relevant to eliminating States from the calculation of the national standard, ACF changed the level of significance for fitting the data to a normal curve from .05 to .01. At this higher significance level, all States could be included in the fitted normal probability distribution for both the composite data indicators and the safety-related data indicators. Consequently, performance of all States was used to determine the national standards for all data indicators. To address the concern relevant to the considerable reduction in the actual percentile represented by the national standard for each of the indicators, the basis for calculating the sampling error was changed from the lower limit of 95 percent confidence interval to the lower limit of an 80 percent confidence interval. As a result of this change, the adjusted 75th percentile is close to the 69th percentile, with the percentile based on the distribution to the normal curve.

TABLE 1: COEFFICIENTS (WEIGHTS) FOR INDIVIDUAL MEASURES IN THE CFSR DATA COMPOSITES

¹ Several States requested that ACF continue to report data pertaining to the six data measures used in the first round of the CFSR. This information will be provided in the State Data Profile. back
²The composites pertain to permanency only and therefore do not involve data from NCANDS. back
³Some respondents raised concern that the proposed timeliness to investigation measures did not reflect the prioritization and classification systems based on the perceived risk of harm to the child that some States have developed for establishing timeframes for responding to maltreatment allegations. back
⁴ASFA requires State child welfare agencies to file a petition to terminate parental rights and pursue adoption for a child who has been in foster care for 15 of the most recent 22 months, unless an exception exists. A 17-month rather than a 15-month timeframe was chosen for the measure because, in accordance with ASFA, a child is considered to have "entered foster care" (for purposes of starting the clock for the 15 of 22 months) on the earlier of: (1) the first judicial finding that the child has been subjected to abuse and neglect, or (2) the date that is 60 days after the date on which the child is removed from the home. The 17 month time frame in the measure is used because AFCARS does not collect information pertaining to the date of the first judicial finding. back
⁵The same process was conducted for assessing State performance with regard to safety, but based on feedback from the field and the results of our data analyses, no additional safety-related measures were developed. back
⁶B.G. Tabachnik and L.S. Fidell (2001). Using Multivariate Statistics, Fourth Edition. Boston, Massachusetts: Allyn & Bacon. back
⁷Counties were excluded from the analyses when the State did not report a FIPS code in FY 2004. back
⁸ACF determined that the composites and national standards would be developed using data pertaining to FY 2004. This means that, for the second round of the CFSR, the data used to establish the national standards will not be the same as the data used to evaluate performance of any of the States. back
⁹The number of counties included in the PCA varies across the composites. This is because a county had to have a value for all of the measures included in a specific composite domain in order to be included in the PCA. For example, if a county did not have any children in foster care for 17 months or longer at the start of the fiscal year, then that county was not included in the PCA for the timeliness of adoption composite because there were two measures in that composite that focus on permanency for children in foster care for 17 months or longer. back
¹⁰The syntax and the aggregated database will be made available to the public. back
¹¹The formula for transforming the standard scores into ranked scaled scores was the following: [100 X ((State Composite Score - Minimum State Composite Score) / (Maximum State Composite Score - Minimum State Composite Score)) + 50].back