Children's Bureau Website - Child Welfare Monitoring: Statewide Assessment

The Child and Family Services Reviews (CFSRs) are designed to support a stronger Federal-State partnership for improving the outcomes of child welfare services to children and families. The reviews seek to achieve this goal by linking an initial review of State child welfare services with a program improvement process and subsequent reviews that measure progress toward those improvements. Moreover, the CFSR process is intended to be coordinated with other Federal child welfare requirements, such as the planning and monitoring of the Child and Family Services Plan (CFSP). The reviews were authorized by the 1994 Amendments to the Social Security Act (SSA), and are administered by the Children's Bureau, Administration for Children and Families, U.S. Department of Health and Human Services.

Specifically, the CFSRs assess how well States perform in achieving positive outcomes in the following three domains for children and families engaged in child welfare services: child safety, child permanency, and child and family well-being. The CFSRs also examine seven State and local child welfare agency systemic factors that affect the achievement of positive outcomes by the children and families that agencies serve. The systemic factors assessed during the CFSRs are based on the requirements in the title IV-B and IV-E regulations, and States are rated on (1) the extent to which they have met those requirements through systems, policies, procedures, or training, (2) how those systems are operating in day-to-day practice in the field, as demonstrated through data or stakeholder input, and (3) the effectiveness of the State with regard to the systemic factors in achieving positive outcomes for children and families.

The CFSR Process

The CFSR is a two-phase process. The first phase is a Statewide Assessment conducted by a State child welfare agency in collaboration with the agency's external partners or stakeholders and the Children's Bureau Central and Regional Office staff. The Statewide Assessment Team completes the Statewide Assessment Instrument.

The second phase of the review process is an onsite review, conducted by a team of Federal representatives (including consultant reviewers) and State representatives (including external partners). The onsite process includes case record reviews, case-related interviews, and stakeholder interviews. The Onsite Review Team uses the Onsite Review Instrument and Instructions to rate cases, and the Stakeholder Interview Guide to guide and document the results of the stakeholder interviews.

Information from both the Statewide Assessment and the onsite review then is used to determine the State's conformity with the State plan requirements for child protective services, foster care, adoption, and family preservation and support services. States found out of conformity are required to develop a Program Improvement Plan (PIP) to address the identified areas needing improvement. States participate in subsequent reviews at intervals related to their achievement of conformity. (For more information about the CFSRs, see the Child and Family Services Reviews Procedures Manual at http://www.acf.hhs.gov/programs/cb/cwmonitoring/tools_guide/proce_manual.htm.)

The Statewide Assessment Instrument

The Statewide Assessment Instrument is intended as a tool for States to use in examining their capacity and performance in improving outcomes for children and families engaged in child welfare services. Each section, as outlined below, is designed to enable States to gather and document information that is critical to analyzing that capacity and performance during the Statewide Assessment phase of the CFSR process.

Section I of the Statewide Assessment Instrument requests general information about the State agency.
Section II contains data profiles for the safety and permanency outcomes. These include the data indicators, which are used, in part, to determine substantial conformity. Additional contextual data are provided that was not included in profiles during the first round. The data profiles are developed by the Children's Bureau based on the Adoption and Foster Care Analysis and Reporting System (AFCARS) and National Child Abuse and Neglect Data System (NCANDS) data submitted by the State. The Children's Bureau provides the profiles to the State upon initiation of the Statewide Assessment. (Note: If a State does not submit child-specific NCANDS data and requests to use an alternate source of safety data for the Statewide Assessment, that request must be made through the Children's Bureau Regional Office prior to initiating the Statewide Assessment. States must provide sufficient documentation of the alternate source in the request for the Children's Bureau data team to approve the use of such data; see the Child and Family Services Reviews Procedures Manual. Following approval of the alternate data source, the State also must produce the necessary data in time for inclusion in the data profiles and for timely initiation of the Statewide Assessment.)
Section III requires a narrative assessment of the seven outcome areas based on the data profiles in section II and new information on the State's performance in these areas since the previous Statewide Assessment. In the narrative, States should use evaluative language to present changes in practice and performance, as measured by the State, that resulted from the implementation of the PIP. These might include changes highlighted through the State's quality assurance system, or via other data generated by the State's Statewide Automated Child Welfare Information System (SACWIS) or another automated information system.
Section IV focuses on State child welfare agency characteristics and requires narrative responses for each of the seven systemic factors. These responses are developed by analyzing data, to the extent that the data are available to the State, and using external stakeholders' and partners' input. In the narrative, States should use evaluative language to present changes in policy, practice, and performance since the previous Statewide Assessment that resulted from the implementation of the PIP and other strategies initiated by the State.
Section V requires the State to assess its strengths and challenges and identify issues and geographic locations requiring further examination during the onsite review. The State also should list here the stakeholders that it involved in developing the Statewide Assessment.

The Statewide Assessment Instrument is available electronically on the Children's Bureau Web site at http://www.acf.hhs.gov/programs/cb/cwmonitoring/tools_guide/statewide.htm.

The Statewide Assessment Team

States must include broad representation from within and outside the child welfare agency in forming a team to conduct the Statewide Assessment. The Statewide Assessment Instrument must be completed in collaboration with State representatives who are not staff of the State child welfare agency (external partners or stakeholders), pursuant to 45 Code of Federal Regulations (CFR) 1355.33 (b). Those individuals should represent the sources of consultation required of the State in developing its title IV-B State plan and may include, for example, tribal representatives; court personnel; youth; staff of other State and social service agencies serving children and families; and birth, foster, and adoptive parents or representatives of foster/adoptive parent associations. Moreover, States should consider including on the Statewide Assessment Team individuals who have the skills to serve as case record reviewers during the onsite review and to assist in developing the PIP, as needed. States must include a list of the names and affiliations of external representatives participating in the Statewide Assessment in section V of the Statewide Assessment Instrument.

How the Statewide Assessment Is Used

Information about the State child welfare agency compiled and analyzed through the Statewide Assessment process is used to support the CFSR process in a range of ways. The Statewide Assessment is used to do the following:

Guide site selection by the Children's Bureau and the State for the onsite review
Provide an overview of the State child welfare agency's organization, capacity, and performance for the Onsite Review Team
Facilitate identification of issues that need additional clarification before or during the onsite review
Serve as a key source of information for rating the CFSR systemic factors
Provide context for the outcome ratings
Enable States and their stakeholders to identify early in the CFSR process the areas potentially needing improvement and to begin developing their PIP approach
Inform the CFSP and Annual Progress and Services Report (APSR) processes
Educate stakeholders about State strengths and needs and enlist their support in developing and making program improvements
Inform stakeholders and the public about the improvements/progress the State has made since the previous Statewide Assessment
Openly share with stakeholders and the public the areas that the State child welfare agency has identified as continuing to need improvement

Further information about conducting the Statewide Assessment and developing the Statewide Assessment document is provided in the Instructions section of this instrument.

THE PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13)

Public reporting burden for this collection of information is estimated to average 240 hours for the initial review and 120 hours for subsequent reviews. This estimate includes the time for reviewing instructions, completing the assessment, and reviewing the collection of information.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.

Instructions

While each State will adopt an approach to conducting the Statewide Assessment that best suits their unique circumstances, they should undertake the following steps in completing the Statewide Assessment process and instrument:

Develop a process for working with the designated Children's Bureau Regional Office staff member regarding the development and review of the Statewide Assessment. The Regional Office staff are one of the State's key external partners during the Child and Family Services Review (CFSR) process.
Identify key agency staff and community representatives or stakeholders to serve on the Statewide Assessment Team and/or to participate in the Statewide Assessment process (such as those who are serving on the Child and Family Services Plan [CFSP]/Annual Progress and Services Report [APSR] planning committee and those who are involved in the Court Improvement Program). States should select agency staff on the basis of their expertise in a specific area, such as quality assurance, child protection or safety, permanency planning, youth services, licensing, and foster care provider support. States should select a mix of external partners so that the Statewide Assessment process will include the representation of organizations, agencies, and individuals with (1) experience providing an array of service types and delivery mechanisms relevant to the needs o children and families, including relative and kinship care providers, (2) a range of perspectives on the State agency's practice and performance, including cross-system collaborative efforts, and (3) expertise in the specific areas that the State has identified as priorities for making improvements.
Consolidate the process for developing the CFSP and Statewide Assessment to address common goals in the most efficient way possible. The development of the CFSP and the Statewide Assessment requires extensive consultation with a wide array of representatives of State, local, tribal, and judicial agencies and organizations. This includes both public and private community-based entities with experience in administering programs for infants, children, youth, adolescents, and families (this also may include faith-based organizations). The State also should consult with children, youth, and families who have received or are receiving child welfare services.

States are encouraged to use a variety of approaches in consulting with external partners and stakeholders. The agency might gather information, for example, through the following:
- Initiating strategies for linking the Statewide Assessment with the ongoing consultation process used for CFSP/APSR development
- Holding focus groups with stakeholders or consumer groups
- Conducting surveys or interviews
- Hosting joint planning forums in the State
- Developing a Web site through which the State updates stakeholders about the status of the Statewide Assessment process and allows those stakeholders to share their experiences regarding State child welfare services and/or comment on drafts of the Statewide Assessment, as appropriate
Review the Statewide Assessment, Final Report, and Program Improvement Plan (PIP) progress reports from the previous CFSR to identify promising approaches, areas needing improvement, and progress made.
Review and use existing data sources, develop additional sources of information to inform the Statewide Assessment process, and continually assess how to further analyze the data provided through existing systems.
- Examine existing State documents that provide evaluative information about State agency performance during the period under review and that might be useful in completing the Statewide Assessment. The review team, for example, would look for evaluative data in the (1) CFSP/APSR, (2) quality assurance reports, (3) management reports, (4) studies, (5) commission reports, (6) State auditor reports, (7) task force findings, (8) National Resource Center technical assistance and training reports, and (9) descriptions of new strategies and initiatives. Through these reports, the State can identify evaluative information, for example, about new policies, training, and practices that led to improvements in the outcomes and systemic factors.
- Assess data and data collection methodologies used by the State to determine its performance on the CFSR outcomes and systemic factors. The sources of the data may include the State's quality assurance systems, for example (1) CFSR-style case reviews, (2) surveys (such as client satisfaction surveys), (3) supervisor or administrative reviews, peer reviews, and other quality assurance system components, and (4) the results of efforts to analyze quality assurance system findings.
- Use data from State child welfare management information systems (MIS), including the following:
  - Data profiles
  - Data addressing State performance on CFSR items (for example, data on timeliness of investigations, face-to-face contacts, and filing for termination of parental rights)
  - Results of efforts to further analyze information system data to more clearly pinpoint variations in performance (for example, analysis of performance based on case characteristics, such as age, type of case, or location)
- Review internal and external partner and stakeholder input/feedback documented through the following:
Analyze the data profiles provided by the Children's Bureau, using the following steps (including consulting with non-review team members, as appropriate):
- Review and analyze the data related to each safety and permanency outcome in the Onsite Review Instrument and Instructions, and identify areas of strength or the need for further examination during the onsite review.
- Compare the State's performance on the data indicators with the national standards, where applicable. States will need to address in their PIP the indicators that fall below the national standards. It therefore is important for States to identify the factors affecting these indicators.
- To the extent possible, examine other data and community or situational factors that may be affecting State data trends. These data analyses then can be used to lay the foundation for developing future PIP action strategies.
Consult with external partners or stakeholders regarding the data indicators. For example, the Statewide Assessment Team might talk with caseworkers, guardians ad litem, foster parents, youth in foster care, and group care providers to help identify the underlying causes of presenting data issues, such as the high number of placement settings experienced by children. States also will consult with stakeholders to obtain information that will be used to complete the narrative section on systemic factors in the Statewide Assessment Instrument.

States can use the exploratory issues that appear in the body of the Statewide Assessment Instrument to analyze key issues and to consult with stakeholders regarding the outcomes and systemic factors under review. Those simply are intended to provide States with guidance for managing the Statewide Assessment process; they are not intended to impose additional requirements on the States nor to limit their flexibility and creativity in managing that process or completing the Statewide Assessment Instrument. States also may access technical assistance, through the Children's Bureau Regional Offices, on planning for and implementing the Statewide Assessment process.

National/State Data Presentation and Analysis

Data Analysis: States are encouraged to explore how they can broaden their data collection analysis by identifying which data might warrant further examination. To do so, they can ask questions such as the following:
- Are there subject or functional areas in which quality assurance or MIS data could be further analyzed to provide more information on State strengths, needs, and barriers to performance?
- Are there specific CFSR outcomes or items that might warrant further attention/ analysis, for example by reviewing additional cases or specific subsets of case types?
- How could the existing data be further analyzed to provide information about local jurisdictions during site selection for the CFSR?
Data Presentation: For data referenced in the Statewide Assessment, States should provide information such as the period that the data address, and the sample size and type of cases. States might use the following language, for example, when describing the data: "A random statewide sample of 100 open foster care cases for the period from July 1, 2005, to December 31, 2005, indicated the following . . ."

In addition, for the data presented, States should fully describe each data methodology used, such as a CFSR-style case review or other performance measurement system. States also should explain how each system measures CFSR performance outcomes and any alternate data approaches used to provide additional perspective on State performance.

Finally, States should describe trends in the agency's performance, for example, by comparing the quality assurance system results for each quarter across the previous CFSR, the previous PIP, and the current Statewide Assessment.

Production of the Statewide Assessment

States should use the Statewide Assessment Instrument to compile the information collected through their assessment process. States are encouraged to summarize key points from other related documents and to avoid attaching those, whenever possible. The Statewide Assessment should be approximately 75–85 pages and contain the following:

Brief description of the agency structure and programs
Information about the relationship between the outcome item description, the available data, and the State's practices and policies
Changes since the previous Statewide Assessment, with an emphasis on changes resulting from PIP implementation
Information on the effectiveness of each systemic factor item being reviewed, including any quality assurance results, if available (See also the specific requirements by systemic factor below.)
Most recent State data profile and summary of State data provided by the Children's Bureau that are relevant to the outcomes and systemic factors during the past 3 years
Trends or notable changes in the data over the past 3 years, including possible contributing factors affecting the trends
Effectiveness measures that demonstrate the State's functioning for each item

For each systemic factor, the State should provide the following:

Overview of the system under review, including the requirements, structure, law, policy, and functions
Identification of areas in which the State agency relies on stakeholders or external partners to carry out some of these responsibilities through contracting or intergovernmental agreements, including those areas in which the State provides oversight of such activities.
Information on the effectiveness of the system, including strengths, gaps, needs, and usefulness
Information on how the State's functioning in each systemic area affects the outcomes of safety, permanency, and well-being
Changes since the previous Statewide Assessment, with an emphasis on changes resulting from PIP implementation or other strategies or initiatives implemented by the State
Information on ongoing processes or mechanisms, such as the State's quality assurance system, that routinely examine the effectiveness of the systemic factor and promote continuous improvement in that area

It is important that the completed Statewide Assessment clearly show an analysis of the relationship between State data and practice, and the quality/effectiveness of the system under review. If a State's data show that children experience frequent re-entries into care following reunification, for example, the State should use the Statewide Assessment process to explore, and then document, the possible reasons that this is occurring. To do so, the State might examine the availability, accessibility, and quality of services to support family reunification. Or if the State's data show that children wait long periods for permanent placements, the State might explore the case review system and its effectiveness in moving children to permanency in a timely manner.

Evaluative Language

States should present information in the Statewide Assessment using "evaluative language" whenever possible. Presenting information in this format requires the Statewide Assessment Team to focus on what their State's data are showing about State practice and to present information about the quality and effectiveness of policies and practices, rather than simply describing those.

Evaluative language does the following: (1) presents judgments, (2) assesses status and outcomes, and (3) gauges, ranks, and rates performance. This type of language provides the reader of the Statewide Assessment with an understanding of how well the State agency is doing; it offers an analysis of the effectiveness of the agency's policies and practices and the areas that require ongoing improvements to achieve positive outcomes. States should use evaluative language as frequently as possible throughout their Statewide Assessment. This is particularly important during the second and subsequent rounds of reviews, when the previous review findings and PIP measures provide a bar against which to evaluate the agency's progress and current effectiveness and identify underlying strengths and challenges.

While some descriptive language may be necessary, it should be used only to present a brief overview of what the State agency does (policy and practice) and to provide context for the more critical information to follow: the assessment of the agency's performance. Descriptive language does the following: (1) presents a picture, (2) shares a narrative story, and (3) outlines characteristics. In other words, States should use descriptive language sparingly to set the stage for their analysis of the child welfare agency's ability to create positive outcomes for children and families.

Illustrative Examples of Descriptive and Evaluative Language

The following examples of descriptive versus evaluative language are designed to illustrate how States can share their CFSR-related findings in an evaluative manner. These examples should not be considered exhaustive with regard to how a State should address the child welfare practice issues shown in the examples.

Example 1:	This example shows the difference between using descriptive versus evaluative language to present information about how the agency is monitoring the implementation of a new staff training program on managing caseworker face-to-face contacts with children during investigations of child abuse and neglect. The descriptive statement simply reiterates that the State is providing training to staff; the evaluative (preferred) language provides detailed information on who has been trained, how training participation is tracked, and the agency's process for institutionalizing the training.
Descriptive:	The State child welfare agency provided training last year to all of its supervisors and caseworkers on the required face-to-face contact with children during investigations of child abuse and neglect.
Evaluative:	During fiscal year (FY) 2005, 80 percent of State child welfare agency supervisors and caseworkers were trained on State policy on face-to-face contact with children during investigations of child abuse and neglect. The remaining 20 percent of agency supervisors and caseworkers were trained during the first quarter of FY 2006. In addition, during the third quarter of FY 2005, the State incorporated the new training into the initial training that is conducted for all new child protection workers. Staff training is tracked through the State Automated Child Welfare Information System (SACWIS), and supervisors receive a monthly report on the status of staff completion of agency-provided trainings.
*****
Example 2:	This example shows the difference between using descriptive versus evaluative language to present information on the effectiveness of an agency's PIP action step—training—in increasing caseworker face-to-face contact with children during investigations of child abuse and neglect. The descriptive statement does not provide supporting data about how the new training is affecting staff performance in this area; the evaluative (preferred) language does.
Descriptive:	After participating in the training on managing face-to-face contact with children during investigations of child abuse and neglect, staff increased their compliance with the requirement to establish face-to-face contact within 24 hours of a report of abuse or neglect.
Evaluative:	The Quality Assurance Team's review of 100 cases during the first 2 quarters of FY 2006 showed that State child welfare agency staff increased face-to-face contact with a child within 24 hours of a report of abuse or neglect from 53 to 75 percent following staff participation in the training on this agency-required interaction. To further assess the reasons that staff did not make face-to-face contact with a child following a report of abuse or neglect, the Quality Assurance Team reviewed the 25 percent of the cases in which this did not happen. They found that in 10 percent of the cases, the caseworker attempted to make contact but was unable to locate the family. In the other 15 percent of the cases, half were from offices located in rural areas in which staff must drive long distances to reach families and the other half were from offices with social worker vacancies.

Using the Statewide Assessment Checklist

At the end of this instrument is a Statewide Assessment Checklist that should be used by State child welfare agency staff before beginning the Statewide Assessment process; the checklist will help them to focus on the type of information that they need to collect and analyze. Both the State and the Children's Bureau Regional Office staff then can use the checklist to review and comment on Statewide Assessment drafts.

Back to Table of Contents

Child and Family Services Reviews: Statewide Assessment Instrument

December 2006

U.S. Department of Health and Human Services Administration for Children and Families Administration on Children, Youth and Families Children's Bureau

Table of Contents

Introduction

Instructions

Child and Family Services Reviews:
Statewide Assessment Instrument

U.S. Department of Health and Human Services
Administration for Children and Families
Administration on Children, Youth and Families
Children's Bureau