Skip Navigation
acfbanner  
ACF
Department of Health and Human Services 		  
		  Administration for Children and Families
          
ACF Home   |   Services   |   Working with ACF   |   Policy/Planning   |   About ACF   |   ACF News   |   HHS Home

  Questions?  |  Privacy  |  Site Index  |  Contact Us  |  Download Reader™Download Reader  |  Print Print    


Children's Bureau Safety, Permanency, Well-being  Advanced
 Search

Chapter 5
Determination of Substantial Conformity

After the completion of the onsite review phase of the Child and Family Services Reviews (CFSRs), the Children's Bureau Regional Office, in conjunction with the National Review Team (NRT) Team Leader makes a determination regarding substantial conformity for each of the seven outcomes and seven systemic factors under review. The requirements for determining substantial conformity are set forth at 45 Code of Federal Regulations (CFR) §1355.34. States may be determined to be in substantial conformity with one or more outcomes and systemic factors and not in substantial conformity with the others. These findings, along with information on the State child welfare agency's strengths and areas needing improvement in serving children and families, are submitted to the State in a Final Report prepared by the Children's Bureau Regional Office after the onsite review. (See chapter 6 for information on the Final Report.)

Program Improvement Plans (PIPs) are prepared, and financial penalties are assessed, if necessary, only for outcomes or systemic factors determined not to be in substantial conformity. (See chapter 7 for information on the preparation of the PIP.) Different methods are used to determine substantial conformity with the outcomes and the systemic factors. Appendix H, Pathway to Substantial Conformity, displays the data indicators and criteria for determining substantial conformity with the outcomes and the systemic factors.

This chapter describes the process for determining substantial conformity with the outcomes and systemic factors, including rating items and comparing data indicators with the national standards. It also provides information on resolving discrepancies between the findings of the Statewide Assessment and the onsite review, either through the provision of additional information or the review of additional cases.

A. Determination of Substantial Conformity With the Outcomes

During the Statewide Assessment and onsite review, the review team assesses seven outcomes in three domains (safety, permanency, and child and family well-being) by examining 23 items. (See Appendix B, Index of Outcomes and Systemic Factors, and Associated Items and Data Indicators.)

For two of the seven outcomes, Safety Outcome 1 and Permanency Outcome 1, decisions about substantial conformity are based on both the data indicators and the onsite case review findings. For these outcomes, the following performance indicators are used to determine substantial conformity:

For the remaining five outcomes, Safety Outcome 2, Permanency Outcome 2, and the three Child and Family Well-Being Outcomes, the percentage of cases reviewed on site in which the outcome was determined to be substantially achieved is used to determine substantial conformity (95 percent).

Following the onsite review, the Children's Bureau Regional Office, in collaboration with the NRT Team Leader for the review, uses the data gathered through the Statewide Assessment and onsite review to make determinations regarding substantial conformity with the outcomes for the State as a whole. The diagram below illustrates the process of determining substantial conformity with the outcomes.

Flowchart of steps in process of determining substantial conformity with outcomes d

Below is an overview of how case records and statewide data are examined to determine conformity.

A.1. Determination of Conformity With the Outcomes: Case Record Reviews

Pairs of reviewers, comprising one Federal Review Team member and one State Review Team member, conduct case record reviews. (See chapter 2, section C, for more information on the Onsite Review Team, and chapter 4 for more information on the onsite review.) Each pair of reviewers gathers information on a case by reviewing the case record and conducting case-related interviews. Reviewers use the automated Child and Family Services Reviews Onsite Review Instrument in conducting the case record reviews. The Instrument lists the items that reviewers examine in assessing achievement of each outcome. For each case, once the reviewers have examined the items and entered the relevant information, the automated Instrument provides a rating for each item of strength, area needing improvement, or not applicable. The system then records whether, for each case, each of the seven outcomes was substantially achieved, partially achieved, not achieved, or not applicable.

The outcomes are rated using the following guidelines (see the Onsite Review Instrument and Instructions, which is available on the Children's Bureau Web site at www.acf.hhs.gov/programs/cb):

To rate an outcome as substantially achieved, the following criteria must be met:

A.2. Determination of Substantial Conformity With the Outcomes: Data Indicators

The regulation at 45 CFR §1355.34, which sets forth the requirements for determining substantial conformity through the CFSRs, includes the establishment of national standards for certain data indicators. These standards are used in conjunction with case record reviews to determine substantial conformity under titles IV-B and IV-E of the Social Security Act. The national standards are based on information that States report to the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and Neglect Data System (NCANDS).

A.2.1. National Standards Used During the Second Round of Reviews

For the second round of reviews, the Children's Bureau will use six data indicators to determine substantial conformity with two outcomes. The data indicators include two individual data indicators for Safety Outcome 1 and four data indicators in the form of composites for Permanency Outcome 1. Each of the four composites incorporates one or more components. Components are the general factors that contribute to the composite score. Each component comprises two or more measures.

Two individual data indicators rather than composites are used as part of the assessment of substantial conformity with Safety Outcome 1. These data indicators are:

For each composite related to Permanency Outcome 1, the Children's Bureau used the distribution of county scores across States to establish six separate national standards, one for each composite. Because the primary purpose of a data composite is to capture overall performance in a particular domain, the Children's Bureau did not establish national standards for the individual measures incorporated in the permanency composites. Therefore, States are not expected to meet a standard for any individual permanency measures, but to achieve an overall performance level in the composites related to Permanency Outcome 1. However, the Children's Bureau will provide States with information regarding each individual measure within the composites, including the mean, median, and range of scores across States, to enable States to identify the individual measures within a composite where improvements are needed.

The four data indicators that will be used as part of the assessment of substantial conformity with Permanency Outcome 1 are:

A.2.2. Failure To Meet the National Standards

If the State's data fail to meet the national standards, the State is required to implement a PIP designed to improve the State's performance on each outcome for which the data indicators do not meet the standards. (The criteria for determining the amount of improvement that must be achieved through a PIP are discussed in chapter 7.)

The goal of the CFSRs is continuous quality improvement. A State, therefore, whose data remain below a national standard in subsequent reviews is required to establish new benchmarks for improvement, with the goal of eventual attainment of the standard. As long as the State reaches the level of improvement agreed to in the PIP for a data indicator, failure to reach the national standard on an indicator is not a basis for withholding Federal funds.

Below are two examples of determining substantial conformity using the case record ratings and data indicators.

Example 1

The following example illustrates how cases reviewed on site are tallied by outcome, as part of determining whether the State is in substantial conformity. In this example, assume that this was the State's second review and that the data indicators met the national standards for Safety Outcome 1 and Permanency Outcome 1. The CFSR Data Management System tallies the ratings for the cases reviewed onsite, as shown in the following table.

 

Outcome Number of Cases Substantially Achieved Number of Cases Partially Achieved Number of Cases Not Achieved

Percentage of Cases Substantially Achieved
1N=65
2N=40

Safety Outcome 1: Children are, first and foremost, protected from abuse and neglect. 54 11 0 83%1
Safety Outcome 2: Children are safely maintained in their homes whenever possible and appropriate. 62 0 3 95%1
Permanency Outcome 1: Children have permanency and stability in their living situations. 21 8 11 52%2
Permanency Outcome 2: The continuity of family relationships and connections is preserved for children. 40 0 0 100%2
Child and Family Well-Being Outcome 1: Families have enhanced capacity to provide for their children's needs. 43 12 10 66%1
Child and Family Well-Being Outcome 2: Children receive appropriate services to meet their educational needs. 65 0 0 100%1
Child and Family Well-Being Outcome 3: Children receive adequate services to meet their physical and mental health needs. 62 3 0 95%1

 

According to the case record review ratings, and the State's performance on the national standards, the State would not be in substantial conformity with: Safety Outcome 1, Permanency Outcome 1, and Child and Family Well-Being Outcome 1.

Example 2

The example below illustrates the process of determining substantial conformity for Permanency Outcome 1, "Children have permanency and stability in their living situations."

The following six items (reviewed on site) and four data indicators are used to determine substantial conformity for Permanency Outcome 1:

As discussed in sections A.1. and A.2., the items are rated on the basis of the case record reviews, using the Onsite Review Instrument, and the data indicators are obtained from the data profiles included in the Statewide Assessment.

The following must occur for the State to be determined to be in substantial conformity with Permanency Outcome 1:

In this example, assume that we determine from the case record reviews that 95 percent of the cases reviewed achieved ratings of "substantially achieved" for Permanency Outcome 1. In addition, the Statewide Assessment shows that the State meets the national standards for three of the relevant data indicators (timeliness and permanency of reunifications, achieving permanency for children in foster care, and placement stability), but not for one indicator (timeliness of adoptions).

Because the State did not meet the national standards for all four data indicators, it is not in substantial conformity with this outcome. The State therefore must prepare a PIP to improve its performance on the data indicator that did not meet the standard, "timeliness of adoptions."

B. Determination of Substantial Conformity With the Systemic Factors

The Child and Family Services Plan (CFSP) and other program requirements provide the basis for determining substantial conformity with each of the systemic factors. (Review team leaders examine 22 items under the systemic factors; these items are listed in Appendix B, Index of Outcomes and Systemic Factors, and Associated Items and Data Indicators.) During the Statewide Assessment, the Statewide Assessment Team compiles and evaluates information on the systemic factors. During the onsite review, the Team Leaders and Local Site Leaders conduct State and local stakeholder interviews to collect the information necessary to evaluate the systemic factors. Using this information, the Children's Bureau Regional Office, in collaboration with the NRT Team Leader for the review, makes the following determinations regarding each systemic factor:

Because the systemic factors are statewide issues, the NRT Team Leader collects information from the three local review sites by downloading the information collected using the automated Stakeholder Interview Guide. The Children's Bureau Regional Office, in conjunction with the NRT Team Leader, then makes the final determination of substantial conformity regarding the systemic factors following the onsite review. This information is included in the Final Report, which the Children's Bureau Regional Office releases to the State after the onsite review.

Six of the seven systemic factors are rated on the basis of multiple CFSP and other program requirements. One systemic factor, "statewide information system," is rated on the basis of only one CFSP requirement. For a State to be found in substantial conformity with a systemic factor, the information obtained from the Statewide Assessment and stakeholder interviews must indicate that the required number of CFSP and other program requirements for that factor are in place and functioning as required.

The following table describes how the CFSP and other program requirements are used to determine substantial conformity with the systemic factors, using the ratings shown. For a specific systemic factor to be determined to be in substantial conformity, the Children's Bureau Regional Office, in collaboration with the NRT Team Leader for the review, must assign it a rating of "3" or "4."

Rating the Systemic Factors
Not in Substantial Conformity Substantial Conformity
1 2 3 4
None of the CFSP or program requirements is in place. Some or all of the CFSP or program requirements are in place, but more than one of the requirements fail to function as described in each requirement. All of the CFSP or program requirements are in place, and no more than one of the requirements fails to function as described in each requirement. All of the CFSP or program requirements are in place and functioning as described in each requirement.

 

Two of the seven systemic factors are rated for substantial conformity slightly differently, as follows:

If item 31 is not in place or is not functioning at the required level, however, the systemic factor is rated either a "1" or "2" depending on the State's performance on item 30. If item 30 is in place, but not functioning, the factor is rated a "2." If item 30 is neither in place nor functioning, the factor is rated a "1."

B.1. Example of Determining Substantial Conformity With the Systemic Factors

Below is an example of how the method described in section B is used to determine substantial conformity for the systemic factor "case review system."

The systemic factor, "case review system," has five CFSP and other program requirements subject to review. For purposes of this example, the Statewide Assessment indicates that policies and procedures are in place statewide that address all five requirements:

While all five requirements are in place, we cannot determine from the Statewide Assessment whether they are functioning properly; the Team Leaders and Local Site Leaders make that determination on the basis of the stakeholder interviews that they conduct during the onsite review.

In this example, assume that those interviews indicated the following:

The information from the Statewide Assessment and the onsite review indicates that three of the five CFSP and other program requirements for this systemic factor are both in place statewide and functioning as required, while two are in place but fail to function as required. This systemic factor, therefore, would be rated a 2 and would not be in substantial conformity, according to the table above "Rating the Systemic Factors."

C. Resolution of Discrepancies Between the Statewide Assessment and the Onsite Review

In some instances, there may be discrepancies between the data indicators, information in the Statewide Assessment, and information on the corresponding items that is obtained during the onsite review.

For example, the State might acknowledge that the onsite review findings accurately reflect State practice, although they differ from the information in the Statewide Assessment or the data indicators. In these situations, the Children's Bureau can make a determination about substantial conformity or how to rate the performance indicator in which the discrepancy exists. In other circumstances in which there is no clear explanation of the discrepancies, however, they must be resolved before a determination about substantial conformity or how to rate the performance indicator in question can be made.

If the Children's Bureau Central Office determines that there is a discrepancy in the findings, the Children's Bureau Regional Office informs the State, in writing, of the discrepancy within 30 days of the onsite review, describes the options available to resolve the discrepancy, and informs the State about the timeframe for responding.

The regulation at 45 CFR §1355.33(d) provides a State with two options for resolving discrepancies:

C.1. Resolution of Discrepancies Through Submission of Additional Information

The standards applied to the review process in determining substantial conformity are the same whether or not a discrepancy exists: applicable data indicators must meet the national standards, and the required percentage of cases reviewed on site must be substantially achieved. States therefore need to determine the most effective option for resolving the discrepancy on the basis of the nature of the discrepancy.

Submitting additional information, for example, is not a reasonable option for resolving a discrepancy when the data indicators meet the national standards, but the case record reviews indicate that in less than 95 percent of the cases, the outcome was determined to be substantially achieved. In this situation, only a review of additional cases would help to determine whether the State meets the criteria for substantial conformity. On the other hand, submitting additional information is a reasonable option when, through the case record review, it is determined that in 95 percent or more of the cases, the outcome was determined to be substantially achieved, but the data indicators associated with that outcome fail to meet the national standards. In this case, submitting additional information could resolve the discrepancy by establishing that the State has met the national standards.

Typically, the additional information that the State submits is data related to the national standards:

If the State chooses to submit data from a source other than NCANDS, the Children's Bureau must approve the use of the additional data (such as data from a special study or an alternate source). In assessing the alternate data, the Children's Bureau uses the following criteria:

C.2. Resolution of Discrepancies Through Review of Additional Case

Reviewing additional cases is the best option for resolving a discrepancy in cases in which the following occur:

  1. The percentage of cases reviewed on site that are determined to be substantially achieved falls short of the percentage required to establish substantial conformity for the outcome, and

  2. The data indicators for the outcome conform to the national standards.

If the State chooses this option, the additional cases are selected from the original samples of in-home services and foster care cases that were drawn for the review. If additional cases are needed to comprise a statistically significant sample, the Children's Bureau Central Office works with the Children's Bureau Regional Office and the State to determine the methods for identifying and selecting the additional cases to be reviewed. (See chapter 4, section C, for more information on case selection and review.)

For the State to be determined to be in substantial conformity through the review of additional cases, on an outcome for which there is a discrepancy, the following conditions must occur:

Typically, about 150 cases are needed to comprise a statistically significant sample at this level. Children's Bureau statisticians are available to assist the Children's Bureau Regional Office staff in determining the exact number of additional cases to be drawn. A joint Federal and State team reviews the additional cases only for the item or outcome in question, and the cases must fall into the original review's period under review. The results of the additional case record review, combined with the results of the original case record review, are the basis for determining substantial conformity.

The Children's Bureau Regional Office determines the timing, process, and review team associated with the additional case record review, on the basis of the number and complexity of the discrepancies to be resolved. When an additional case record review is needed, it follows the onsite review as soon as possible so that a prompt and accurate determination regarding substantial conformity can be made.


Return to Table of Contents