Child and Family Services Reviews Procedures Manual

	Questions?
	Privacy
	Site Index
	Contact Us

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Administration on Children, Youth and Families
Children's Bureau

Child and Family Services Reviews
Procedures Manual

Washington, D.C.
August 2000

Chapter 1. Framework for the Child and Family Services Reviews

A.		Purpose of the Reviews
B.		Conceptual Framework
C.		Structure of the Reviews
D.		Steps in the Review Process

Chapter 2. The Review Team

A.		Composition of the Review Team
B.		Functions of the Review Team
C.		Leadership of the Review Team

Chapter 3. Statewide Assessment

A.	Purpose of the Statewide Assessment
B.	Completing the Statewide Assessment
C.	Technical Assistance With the Statewide Assessment
D.	Reviewing the Statewide Assessment and Preparing the Preliminary Assessment
	D.1.	Making Decisions Regarding the Onsite Review
	D.2.	Preparing the Preliminary Assessment
	D.3.	Example of Preliminary Assessment
E.	Interim Statewide Assessments Between Full Reviews
	E.1.	State Responsibilities
	E.2.	Regional Office Responsibilities

Chapter 4. Onsite Reviews

A.	Purpose
B.	Onsite Review Activities
C.	Advance Preparation for the Onsite Review
	C.1.	Regional Office ACF Responsibilities
	C.2.	Central Office ACF Responsibilities
	C.3.	Peer Review Contractor Responsibilities
	C.4.	State Agency Responsibilities (Central Office)
	C.5.	Local Agency Coordinators' Responsibilities
D.	Case Selection and Review
	D.1.	Sample of Cases Reviewed
	D.2.	Location of Case Records
	D.3.	Preparation of Records for Review
	D.4.	Case Interviews
E.	State and Local Stakeholder Interviews
	E.1.	Local Stakeholder Interviews
	E.2.	State Stakeholder Interviews
F.	Team Briefings
G.	Instruments

Chapter 5. Final Report

A.	Purpose of the Final Report
B.	Format of the Final Report
C.	Preparation of the Final Report
D.	Content of the Final Report
	D.1.	Example of Final Report
E.	Distribution of the Final Report

Chapter 6. Substantial Conformity

A.	Substantial Conformity on the Outcomes
	A.1.	Rating Outcomes on an Individual Case
	A.2.	Establishing the National Standard on Statewide Aggregate Data
	A.3.	Determining Substantial Conformity for the State
	A.4.	Example of Determining Substantial Conformity on the Outcomes
B.	Substantial Conformity on the Systemic Factors
	B.1.	Example of Determining Substantial Conformity on the Systemic Factors
C.	Resolving Discrepancies Between the Statewide Assessment and the Onsite Review
	C.1.	Example of Resolving a Discrepancy

Chapter 7. Program Improvement Plans

A.		Criteria for Developing a Program Improvement Plan (PIP)
B.		Content of the PIP
C.		Preparation of the PIP
D.		Technical Assistance
E.		Role of Regional Office in Coordinating Technical Assistance
F.		Timeframes for Implementing the PIP
G.		Evaluating the Implementation of the PIP

Chapter 1
Framework for the Child and Family Services Reviews

A. Purpose of the Reviews

The 1994 Amendments to the Social Security Act authorized the U.S. Department of Health and Human Services to review State child and family service programs in order to ensure substantial conformity with the State plan requirements in titles IV-B and IV-E of the Social Security Act. Title IV-B of the Social Security Act and 45 Code of Federal Regulation (CFR) 1357.15 require States to submit child and family services plans (CFSP), i.e., State title IV-B plans, to the Administration for Children and Families (ACF) that include many of the requirements included in the child and family services reviews. The reviews cover child protective services, foster care, adoption, family preservation and family support, and independent living.

In addition to reviewing for the State's substantial conformity with applicable requirements, the reviews are designed to help States improve child welfare services and the outcomes for families and children who receive services by identifying strengths and needs within State programs, as well as areas where technical assistance can lead to program improvements. Other purposes of the reviews include the following:

Ensure that Federal funds are spent in accordance with Federal statute, regulation, and policy
Link the reviews to the joint planning, technical assistance, and program improvement processes that exist between States and Regional Offices
Assist States to become self-evaluating over time
Assemble data that will inform national policy
Provide timely and specific feedback to States that is directly related to program performance and outcomes

B. Conceptual Framework

The child and family services reviews are based on a number of central principles and concepts, including the following:

The child and family services reviews are a collaborative effort between the State and Federal governments. A review team composed of both State and Federal staff conducts the reviews and evaluates State performance.
The reviews rely on information from a variety of sources in making decisions about a State's performance. The sources of information include a statewide assessment, completed by State members of the review team; onsite reviews of a sample of children and families served by the State; statewide aggregate data; and interviews with State and community representatives.
The reviews examine State programs from two perspectives: First, the reviews cover outcomes of services provided to children and families served by the State agency. Second, they cover systemic factors that have an effect on the agency's ability to help children and families achieve positive outcomes.
The reviews are designed to capture both the strengths and needs of State programs. With a strong emphasis on using the reviews to drive program improvements, the reviews identify the strengths of State programs that can be used to make improvements in other program areas, where needed.
The reviews promote practice principles believed to support improved outcomes for children and families, such as family-centered practice, community-based services, strengthening parental capacity to protect and provide for their children, and individualizing services that respond to the unique needs of children and families.
The reviews emphasize accountability. While the review process includes opportunities for States to make program improvements before having Federal funds withheld due to nonconformity, there are significant penalties associated with the failure to make the improvements needed to attain substantial conformity.

C. Structure of the Reviews

There are two phases included in the child and family services reviews: the statewide assessment, which the State completes during the 6-month period prior to the onsite review, and the onsite review.

In examining outcomes, State child and family services programs are reviewed in both phases of the review process, with the information gathered from both phases contributing to the overall evaluation plan and plan for program improvement. The outcomes included in the reviews fall into three domains: safety, permanency, and child and family well-being. Within each domain, specific outcomes are assessed through the onsite review and statewide assessment. The outcomes are as follows:

Safety
Children are, first and foremost, protected from abuse and neglect.
Children are safely maintained in their homes whenever possible and appropriate.
Permanency
Children have permanency and stability in their living situations.
The continuity of family relationships and connections is preserved for children.
Child and Family Well-Being
Families have enhanced capacity to provide for their children's needs.
Children receive appropriate services to meet their educational needs.
Children receive adequate services to meet their physical and mental health needs.

In the first phase of the review, the State members of the review team complete a statewide assessment, using statewide aggregate data to evaluate the programs under review, and address the other requirements subject to review. In the second phase, the onsite review examines outcomes for a sample of children and families served by the State. The onsite review addresses specific outcomes through a focus on the quality of services and practice.

A set of performance indicators is used to evaluate each of the outcomes listed above (see appendix A), based on information from each case reviewed onsite and data in the statewide assessment. The performance indicators are in two forms: one includes qualitative items that are evaluated through case reviews conducted onsite in the State, and the other includes statewide aggregate data obtained from the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and Neglect Data System (NCANDS).

A combination of performance indicators is used to assess the achievement of each outcome. For example, in evaluating the outcome "children are, first and foremost, protected from abuse and neglect," the performance indicators for this outcome include the following statewide aggregate data:

The recurrence of maltreatment statewide
The incidence of child abuse and/or neglect in foster care statewide

In addition, the reviewer considers the following performance indicators gathered on each individual case during the onsite review to evaluate this same outcome:

The timeliness of initiating investigations following the receipt of a report of suspected abuse or neglect
The recurrence of maltreatment of the child

In a similar manner, the review of systemic factors occurs at two levels. First, the statewide assessment includes an evaluation by the State members of the review team of the requirements related to each systemic factor covered in the review. Second, during the onsite review, members of the review team interview selected community stakeholders to determine how well each of the systemic factors functions in the State.

The systemic factors examined in the reviews include the following:

Statewide information system
Case review system
Quality assurance system
Staff training
Service array
Agency responsiveness to community
Foster and adoptive parent licensing, recruitment, and retention

The reviews evaluate a set of State plan requirements and other program requirements for each of the systemic factors. Decisions about the State's substantial conformity with the systemic factors are based on the satisfactory implementation of those requirements in the State.

D. Steps in the Review Process

A summary of the timeframes for the major activities in the child and family services review is provided in appendix J. The following steps in the review process include both State and Federal roles and some joint responsibilities:

Determine the dates for the review. The Central Office and Regional Office determine the dates for the review in collaboration with State officials. Reviews must be conducted within the timeframes specified in the final regulation governing the frequency of State reviews.
Form the review team. This involves identifying State and Federal members of the review team, along with individuals from the State who are not staff of the State agency. This step also includes identifying peer reviewers, selected from a nationwide pool of reviewers, maintained under contract, who supplement the Federal review team.
Transmit data profiles, pulled from AFCARS and NCANDS, to the State. In order to reduce the burden on States, the Federal government compiles data submitted by the State to NCANDS and AFCARS into reports that the States use in completing their statewide assessment. If the State does not have complete AFCARS or NCANDS data, it must provide the required data from an approved alternate source.
Complete the statewide assessment. State members of the review team, including those members who are not State staff, analyze the data provided through AFCARS and NCANDS and address the outcomes and systemic factors included in the statewide assessment.
Designate locations for the onsite review activities. The Regional Office and the State jointly identify three locations in the State where the onsite review activities will occur. The State's largest metropolitan subdivision is a required location, and the other two locations are determined from information in the statewide assessment (see chapter 3, D.1).
Select the sample and types of cases to be reviewed on site. The exact number of cases to be reviewed, by location and type of case, i.e., foster care or inhome, will be determined jointly by the Regional Office and the State, based on information in the statewide assessment.
Prepare and disseminate the preliminary assessment. The Regional Office prepares an analysis of the statewide assessment on the Summary of Findings Form (see appendix F). The Regional Office provides the preliminary assessment to the peer reviewer contractor for distribution to all members of the review team.
Prepare for the onsite review. The State, in collaboration with the Regional Office, schedules local stakeholder interviews, prepares reviewer schedules, and plans logistical arrangements, e.g., hotels and transportation for State review team members and space for meetings and review activities. The State also pulls the 30-50 cases for the onsite review from the total sample of 150 foster care cases and 150 inhome cases randomly identified by the Central Office. The Regional Office, in collaboration with the Central Office, the State, and the peer review contractor as needed, assigns team members to local sites, provides orientation or training needed by review team members, and disseminates review materials to the team members before the onsite review.
Onsite review in the State. The review is completed in 1 week by a joint State and Federal review team.
Complete and issue the final report. The Regional Office prepares and distributes the final report of the review, which includes the written determination of substantial conformity, within 30 calendar days of the onsite review or within 30 calendar days of resolving any discrepancies encountered between information in the onsite review and the statewide assessment.
Develop the program improvement plan. The State, in collaboration with the Regional Office, develops a program improvement plan that covers all areas of nonconformity, as determined in the review, and submits the plan to the Regional Office for approval within 90 calendar days of receiving written notice of nonconformity.
Implement the program improvement plan. The State implements the approved program improvement plan, technical assistance is provided through the sources identified in the plan, and the Regional Office monitors implementation of the plan through quarterly reports and other methods addressed in the plan.
Withhold Federal funds for nonconformity. If the State fails to make improvements needed to bring areas of nonconformity into substantial conformity, Federal funds are withheld from the State commensurate with the level of nonconformity.
Conduct subsequent reviews. For States determined to be in substantial conformity, rereviews are conducted at 5-year intervals, with an interim statewide assessment prepared by the State 3 years following the onsite review. States determined not to be in substantial conformity are rereviewed at 2-year intervals.

Chapter 2
The Review Team

A. Composition of the Review Team

The State should begin to identify the State members of the review team approximately 8 months prior to the onsite review (see timeframes in appendix J). The purposes of starting at that point are to have the key review team members from the State agency and those State members who are not staff of the State agency in place to begin the statewide assessment 6 months prior to the onsite review and to provide initial orientation of those members of the team. Some team members will not be identified until closer to the onsite review, i.e., staff of the local sites selected for the reviews based on the completed statewide assessment.

A team of individuals, including the following members, conducts the reviews:

Administration for Children and Families (ACF) Central and Regional Office staff
State central office and local office staff
Representatives external to the State agency with whom the State was required to consult in the development of its title IV-B State plan
Peer reviewers selected from a national pool of reviewers maintained under contract to supplement the Federal review team

The team may also include cross-Regional or cross-State representatives, at the expense of the Region or State sending staff on the review. States and Regions preparing to conduct reviews may find it helpful to send staff to reviews in other States and Regions to help prepare and train them. States or Regions interested in participating in other reviews will make their requests through the Regional Office that is leading the review.

Staff of the federally funded National Resource Centers and other technical assistance providers who may be working with the State on program improvement planning are not included on the review team, in order to avoid possible conflicts of interest if the States are later in a position to purchase technical assistance from the providers. The review teams comprise approximately 25 people, more if needed, at least half of whom are State representatives (both State staff and external representatives from the State). There is a designated Federal team leader for the review team, in addition to local team leaders for each of the three local sites in the State where review activities occur. Also, there are team members designated to perform quality assurance functions during the onsite review in order to ensure consistency and objectivity in reviewing cases.

The reviews encourage cross-system coordination and consultation in States, as promoted by the title IV-B, sub-part 2 planning process, by including review team members from outside the State agency who represent the planning team that develops the State's Child and Family Services Plan (CFSP). Examples of the external representatives the State agency may consider having on the team include foster parents, university social work educators, advocates, coordinators of State child welfare initiatives or projects, tribal representatives, and staff of private and public agencies.

Through arrangements with the Federal government, a contractor will maintain a pool of prospective peer reviewers nationwide. Having access to such a pool of reviewers will help ensure that the review team is composed of individuals who have the knowledge, background, and preparation to conduct qualitative reviews of State child and family service programs. The peer reviewers are not to be confused with State members of the review team who are not State agency staff members. The peer reviewers used through a Federal contract will be used to supplement Federal members of the review team.

B. Functions of the Review Team

Membership on the review team involves a significant commitment of time and effort. State members of the team must be in a position that enables them to work closely with the State in preparing the statewide assessment and fulfill all the responsibilities noted below through the onsite review.

The State members of the review team are responsible for completing the statewide assessment within the timeframes specified in the regulation by submitting the completed statewide assessment to the Regional Office for approval no later than 60 days before the onsite review. For that reason, the State members of the team should be identified before work begins on the statewide assessment. State members of the team who are not directly involved in writing the statewide assessment may be involved in other ways, such as reviewing the data included in the statewide assessment, attending meetings to provide input, submitting information on selected items, and reviewing and commenting on drafts as requested by the State agency.

Following the completion of the statewide assessment, the functions and responsibilities of both State and Federal review team members include the following:

Review the completed statewide assessment in preparation for the onsite review
Participate in any scheduled orientation or training sessions prior to or during the review
Remain present in the review site(s) for the entire week of the onsite review, including participation in all scheduled review activities from the entrance conference through the exit conference
Conduct all assigned activities associated with review of case records and interviews, including completion of the review instruments
Participate in focus groups, open forums, or other group meetings scheduled during the onsite review
Attend daily briefings of the local review teams, and present information on cases reviewed or stakeholders interviewed
Assist in compiling a summary of the team's findings during the onsite review
Turn in completed review instruments on assigned cases and interviews prior to departing the review site at the end of the onsite review
Remain available for telephone consultation following the review, if needed, in order to clarify or supplement information recorded on the instruments

C. Leadership of the Review Team

The team leader for the review will be a Federal staff member of the Regional Office. The major responsibilities of the team leader and staff assisting the team leader include the following:

Serve as liaison with State leadership in planning review activities
Ensure that the statewide assessment and accompanying data are sent to the State on a timely basis and received back in time to plan the onsite review
Prepare an analysis (preliminary assessment) of the statewide assessment 30 days before the onsite review and record it on the Summary of Findings Form
Plan the details of the onsite review with State and Central Office representatives, including arranging conference calls as needed and transmitting review instructions and procedures to State liaisons
Work in collaboration with the peer review contractor for the reviews to ensure that all documents needed for the review, including the preliminary assessment, instruments, handouts, instructions and orientation materials are available and delivered to the State administration and members of the review team in time for the onsite review
Provide leadership for entrance and exit conferences at the beginning and conclusion of the onsite review
Ensure that local site team leaders (see below) are aware of their responsibilities during the onsite review
Coordinate the development, revision, circulation, and release of the final report of the review (which includes the written notification to the State regarding substantial conformity) with the Central Office
Work with designated State staff toward development of the program improvement plan

During the onsite review, the review team will be divided into smaller "local" teams that are assigned to different geographical areas in the State. There should be a local site coordinator, who is a State staff member, in each of the three review locations in the State who handles logistical arrangements, sets up interviews, and so forth. This person may or may not actually be on the review team. Each local team will also have a designated local team leader, who is a Federal representative, with the following responsibilities:

Provide strong positive leadership to the team through setting the pace of work for the week, assisting members to resolve problems with their cases or schedules, supporting the team in all ways to complete the work, and promoting a positive and objective approach to the review
Coordinate the review schedule with local agency staff, e.g., communicate with the local site coordinator to: (1) confirm interview arrangements, and (2) develop schedules, including for local entrance and exit conferences
Coordinate team members' responsibilities for case record reviews; interviews; and other review activities in the local sites, including reassigning members with gaps in their schedules to other tasks
Provide leadership for entrance and exit conferences with staff of the local review sites
Convene and lead daily briefings with local team members
Provide leadership for any scheduled focus groups, open forums, or discussion groups arranged as part of the local review work
Perform a quality assurance review of all onsite review instruments (see appendix C) and stakeholder interview guides (see appendix D), as they are completed, in order to identify missing information and inconsistencies in completing the instruments, and ensure that the performance indicators are used correctly to rate outcomes (if the team leader must have assistance with this function, (s)he may select another team member who is experienced in the reviews to assist)
Ensure that team members complete all assigned review functions, including completing and submitting all instruments, prior to departing the local review site
Coordinate the completion of the Summary of Findings Form for the local team at the end of the review week, including providing all local team members with opportunities for input into the document
Submit local team members' completed instruments and the team's Summary of Findings Form to the designated team leader prior to departing the State at the end of the review week
Participate in review activities by reading case records and conducting interviews

Chapter 3
Statewide Assessment

A. Purpose of the Statewide Assessment

The statewide assessment is the first phase of the review process. It provides States an opportunity to examine data relating to their programs and to consider the data in light of programmatic goals and outcomes for children and families served by the State. The statewide assessment requires State review team members to consider the State's success in helping children and families achieve positive outcomes in the areas of safety, permanency, and well-being. The statewide assessment process leads to identification of the areas where the State is performing well and those areas that need further examination through the onsite review.

The statewide assessment includes data that the Administration for Children and Families (ACF) extracts from the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and Neglect Data System (NCANDS) (Detailed Case Data Component data), or another approved source of data in the absence of these sources, and transmits to the State in report format. The data included in the statewide assessment assist States to examine service populations from two different perspectives:

First, the NCANDS and AFCARS data provide point-in-time profiles of the foster care and child protective service populations that include all children in the State who are included in the AFCARS and NCANDS databases. The profiles provide States with information on the status of the service populations as of a given date.
Second, AFCARS data are also used to provide States with information on the cohort group of children entering foster care for the first time in their lives during the period under review. Through these data, States have the opportunity to begin tracking progress in certain areas longitudinally.

In part, the value of examining data on the cohort group of children who enter foster care for the first time in the State is that the effects of the agency's practices and policies, particularly with respect to family preservation, time-limited reunification efforts, and permanency planning, are more easily discernible for a group of children recently entering foster care for the first time than for the entire foster care population. The State's total foster care population will include children who entered foster care several years ago and children who have had multiple entries into foster care. These differences among children in foster care make it difficult to identify the most typical experience of children recently entering foster care in the State. If the data are available, for each statewide assessment, we will provide data on the cohort group of children entering foster care for the first time in each of 3 years, so that the State can compare changes for first-time entries over time.

Since the cohort profiles only include data on children entering foster care for the first time during a given year, we must also use point-in-time data profiles that provide information on all children in foster care in the State. Using both profiles, States can evaluate the status of all children served by the agency, as well as gain insight into where program improvements are most needed based on recent data and practice.

The data profiles also include statewide aggregate data indicators that are used to determine substantial conformity. The Children's Bureau will establish national standards for each of the statewide aggregate data indicators used to determine substantial conformity. When a State is undergoing a child and family services review, the State and the Regional Office will compare the State's data for the period under review with the national standards and determine the State's compliance with the standards.

B. Completing the Statewide Assessment

The Regional Office transmits the statewide assessment instrument and accompanying data to the State for review and completion approximately 6 months prior to the onsite review (see appendix B). States also have ongoing access to the final version of the instrument through the Children's Bureau Web site. (State-specific data used in the statewide assessment are not available on the Internet.) The State must return the completed statewide assessment to the Regional Office no later than 60 days prior to the scheduled onsite review.

AFCARS and NCANDS submissions from the States do not correspond to the same time periods, i.e., AFCARS is submitted on a fiscal year basis and NCANDS is submitted by calendar year. Also, NCANDS submissions, which are voluntary, are a year behind AFCARS submissions. The data from these two sources will, therefore, be based on fiscal year AFCARS data and calendar year NCANDS data, leaving approximately a 3-month time difference in the data included in the permanency and safety profiles.

If a State does not submit data to NCANDS, the State and the Regional Office must agree on an alternate source of statewide data to be used in the safety profile for the statewide assessment. Also, for its first preview, if the State has incomplete AFCARS data, an alternate source of data, approved by the ACF, may be used to generate the foster care data profiles. In the absence of AFCARS or NCANDS data, the Regional Office must request that the State submit its alternate source of data to the Regional Office 8 months prior to the onsite review in order to allow time to approve the data and transmit it to the Children's Bureau to prepare the profiles. The Regional Office must either approve or disapprove the alternate data. The criteria for approving or disapproving the alternate data include:

The source of the data, i.e., the Regional Office must ensure that the data accurately represent the State's service population.
The alternate sources must provide the same data as would have been provided by NCANDS and AFCARS if the State had made complete submissions.

Since some of the data elements on the data profiles are used to determine the State's substantial conformity, failure to provide the needed data, in the absence of AFCARS and NCANDS data, could result in a determination that the State is not in substantial conformity on the outcomes to which those data elements apply.

When the Regional Office has approved the alternate source of data for the profiles, it will transmit the data to the statisticians in the Children's Bureau who will use it to prepare the profiles. The Children's Bureau will prepare the profiles and send them to the Regional Office. The Regional Office will then transmit the profiles and the statewide assessment form to the State 6 months prior to the onsite review.

In addition to statewide aggregate data, the statewide assessment includes a series of narrative-style questions. The State's review team should be in place to conduct the statewide assessment and should be the primary group that responds to the narrative questions.

The responsibilities of the State members of the review team at this point include the following:

Analyzing the data
Meeting to discuss the data and the issues behind the data
Developing responses to questions about the data, in consultation with sources outside the review team as needed

The review team must include representatives outside the State title IV-B/IV-E agency in order to provide a broad perspective on the items in the statewide assessment (see chapter 2). In completing the statewide assessment, States have the option of gathering information through focus groups with stakeholders or consumer groups, surveys, joint planning forums within the State, or otherwise connecting the statewide assessment with ongoing consultation through their title IV-B planning process.

In analyzing the data profiles, the State members engage in the following activities:

Review the statewide aggregate data related to each outcome. The team will identify areas of strength or need for further review onsite. The team members will attempt to identify the reasons for the status of certain data indicators. For example, under Permanency Outcome #1, "children have permanency and stability in their living situations," the State will examine the data on the number of placement settings experienced by children during their current episode of foster care. If the data are available through AFCARS and NCANDS, the State will have 3 years of data on this indicator and will be able to identify whether the indicator is moving in the desired direction. In order to understand the reasons behind the numbers, however, the State will have to look further. For example, the State may have other data that help explain the number of placement settings that children experience, or it may have to review a sample of cases to identify the reasons, or interview caseworkers or foster families.
Compare the State's performance on statewide aggregate data with the national standards, where applicable. For those statewide aggregate data indicators used to determine substantial conformity, the State will compare its data with the national standard and begin to determine the reasons behind the numbers as described above. The importance of understanding the factors that affect these numbers, in particular, lies in the requirement that the State implement a program improvement plan that addresses the numbers that fall below the national standard.
Use the data to identify areas of strength and areas needing further review. The State will need to examine the data in a manner that identifies the program areas that are most in need of further review onsite. For example, if the State members of the review team identify safety as a major concern, they can work with the Regional Office to structure the sample of cases for the onsite review in a way that is likely to capture the safety issues. In this same example, the State will also need to suggest those locations in the State, other than the State's largest metropolitan subdivision which is a required review location, where either the most typical or the most needed observations of the safety issues can be made, depending upon the strengths and needs identified in the statewide assessment.
Use supplemental data, other than the NCANDS and AFCARS profiles, to review other outcomes and systemic factors. Since the NCANDS and AFCARS profiles only cover the permanency and safety outcomes, any other data that the State can produce that address the remaining outcomes and the systemic factors will increase the State's ability to understand the factors that affect its performance.

C. Technical Assistance With the Statewide Assessment

The State has an opportunity to build its capacity for continuous program evaluation and improvement by using the statewide assessment to examine progress on the basis of data. Although some States have the analytical capacity in place to examine and interpret data, others will need assistance in interpreting and manipulating the data, comparing indicators, and relating indicators to outcome measures. To the extent possible, the Federal government will provide technical assistance to States in developing the ability to analyze the data. This assistance may be in the form of Federal staff helping to analyze the statewide assessment; conference calls to discuss the analysis of the information; or access to other sources of technical assistance, such as the National Resource Center for Information Technology in Child Welfare. States will be encouraged to integrate the statewide assessment process into their ongoing quality assurance and program review functions.

D. Reviewing the Statewide Assessment and Preparing the Preliminary Assessment

The Regional Office will review the entire statewide assessment to ensure that it is complete and it addresses all areas appropriately. If critical information is missing or not adequately covered in the statewide assessment, the Regional Office may ask the State to address those areas more completely.

Upon receiving the completed statewide assessment, the Regional Office will review the document and use the information in two ways.

First, in collaboration with the State, the Regional Office will make certain decisions about the onsite review, including:

Locations of review activities
Sample composition and size
Specific issues to target through interviews

Second, the Regional Office will use the information to prepare a preliminary assessment of the State's performance, as reported in the statewide assessment.

D.1. Making Decisions Regarding the Onsite Review

The onsite review activities are conducted in at least three locations in the State. The State's largest metropolitan subdivision is designated in regulation as a required location for the onsite review. This means the metropolitan subdivision in the State, whether that is a county or a city, that has the largest population in the State.

The other two locations are to be determined by the Regional Office in collaboration with the State, and the selection will depend upon issues raised by the statewide assessment. The Regional Office will consider the following criteria in making this decision:

It is not necessary to target the most troublesome geographic areas for the onsite review, although the locations selected must fairly represent practice in the State.
If the statewide assessment identifies particular geographic areas, program areas, or populations of children and families served that need the more intense review provided onsite, locations may be selected that will permit that type of review.
If there are no outstanding programmatic or systemic issues that will be the focus of onsite review activities, the criteria for selecting locations may be to identify the most typical practice in the State, including a mix of rural and urban sites, small and large sites, and so forth.

Because the circumstances in each State will vary, imposing additional requirements on site selection would impede the flexibility of the Regional Office and State to ensure that the onsite review is responsive to individual State issues and needs.

In using the statewide assessment to determine the composition of the sample of cases to be reviewed onsite, the State and the Regional Office determine whether there are program needs that warrant a weighting of the sample toward one program over another. In all State reviews, the sample must include both inhome cases and foster care cases, but the proportion of each case type need not always be equal. For example, if the statewide assessment raises significant safety concerns around children served in their own homes, but does not indicate problems in the foster care program, the sample might be structured to include a larger percentage of inhome cases than foster care cases. If there are not overriding concerns raised in the statewide assessment about one program over another, the sample should be divided evenly between inhome and foster care cases.

Finally, in using the statewide assessment to structure the onsite review, there may be issues raised around outcomes or systemic factors that warrant specific questions of stakeholders. Some examples encountered in the pilot reviews include State policies or practices around screening investigations of reports of child maltreatment that affect child safety, bifurcated systems of service delivery that affect the agency's responsiveness to the community, and contractual issues that affect service delivery. Where such issues exist and are within the scope of the Child and Family Services (CFS) reviews, the Regional Office may address this either by asking the State to schedule specific types of stakeholders for interviews or by advising the review team of specific issues that should be addressed during the interviews.

D.2. Preparing the Preliminary Assessment

The Regional Office will prepare a preliminary assessment of the State's performance on each of the outcomes and systemic factors, based on information from the statewide assessment. The preliminary assessment should be completed by the Regional Office 30 days prior to the onsite review in order to provide adequate time to disseminate it to members of the review team.

The preliminary assessment is based entirely on information from the statewide assessment and is simply a matter of recording information pertaining to the outcomes and systemic factors from the statewide assessment onto the Summary of Findings Form. Decisions about substantial conformity and the final report to the State will be based on information from both the statewide assessment and the onsite review. Recording information from the statewide assessment onto the Summary of Findings Form prior to the onsite review accomplishes the following objectives:

It provides all members of the review team, who will receive a copy of the preliminary assessment, with basic information about the State and the statewide assessment as they begin the onsite view.
It assists the team in combining information obtained onsite with the information in the statewide assessment for use in the exit conference.
It permits a quick identification of areas where there may be discrepancies between information obtained onsite and information from the statewide assessment, so that the discrepancy resolution process can begin immediately following the onsite review.
It identifies, before the onsite review, the State's performance level with regard to the statewide aggregate data and the national standards.
It provides an opportunity to begin preparing the final report, since the Summary of Findings Form is a part of the final report. Given the importance of providing timely feedback to the State following the review, this is an especially important consideration.

Completing the preliminary assessment involves the following steps:

The Regional Office records pertinent data and narrative information from the statewide assessment, describing each of the outcomes and systemic factors, on the Summary of Findings Form.
The Regional Office compares the State's aggregate data used to make determinations about substantial conformity with the national standards and records that information on the form.
The Regional Office works in collaboration with the State and the peer review contractor to see that all review team members receive a copy of the preliminary assessment prior to the onsite review (as part of the information package sent to review team members).

D.3. Example of Preliminary Assessment

The example that follows is one page from the Summary of Findings Form (see appendix F) that illustrates how information from the statewide assessment is used to prepare the preliminary assessment. This information is updated and supplemented during and after the onsite review and serves as the final report to the State for the CFS review.

Example of Preliminary Assessment

II. PERMANENCY

Outcome P1: Children have permanency and stability in their living situations.
Number of cases reviewed by the team according to degree of outcome achievement:
	Team 1	Team 2	Team 3	Total Number	Total Percentage
Substantially Achieved:	__________	__________	_________	_________	__________
Partially Achieved:	__________	__________	_________	_________	__________
Not Achieved or Addressed:	__________	__________	_________	_________	__________
Not Applicable:	__________	__________	_________	_________	__________
Conformity of Statewide data indicators with national standards:
	National Standard	State's Percentage	Meets Standard	Does Not Meet Standard
Foster care re-entries	13%	20%	_________	X
Length of time to achieve reunification	80%	87%	X	___________________
Length of time to achieve adoption	26%	34%	X	___________________
Stability of foster care placements	77%	73%	_________	X
Length of stay in foster care	12 months	9 months	N/A	N/A

Item 5: Foster Care Re-entries

_______ Strength ______ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates that more children (20%) in foster care have had multiple entries within the previous 12 months than the national standard for this indicator (13%). Percentage of children re-entering foster care has increased over the past 3 years from 16% to 20%.
According to the statewide assessment, the State has observed an increase in foster care re-entries as the length of stay in foster care has decreased. State indicates that most re-entries are by children discharged to reunification, as opposed to other discharge reasons. State is concerned about the availability of services to families following reunification.

Item 6. Stability of foster care placement

_______ Strength ______ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates fewer children in the State (73%) have no more than 2 placements in a 12-month period than the national standard (77%) for this indicator. The number of placement settings for children in foster care has not changed significantly over the past 3 years.
Statewide assessment indicates the average number of placements per child in the first episode of foster care is 3. Most children go to shelter care first and then to another placement setting. Foster care disruptions are increasing.

For those outcomes and systemic factors reviewed for which there are no data, the Regional Office will record other relevant information from the statewide assessment in the preliminary assessment. For example, under "Child and Family Involvement in Case Planning" (item 18 on the Summary of Findings Form), the Regional Office will summarize information from section IV-C of the statewide assessment related to this item. Under each of the systemic factors, the Regional Office will record relevant information addressing each factor from sections II and IV of the statewide assessment.

Some of the performance indicators used in the review are only collected on site, and are not addressed in the statewide assessment, e.g., current risk of harm to the child, current relationship of child in care with parents. Those indicators are not addressed in the preliminary assessment.

E. Interim Statewide Assessments Between Full Reviews

E.1. State Responsibilities

States determined to be in substantial conformity are required to complete interim statewide assessments between full reviews. There is no similar requirement for States determined not to be in substantial conformity, since those States are reviewed at 2-year intervals.

The State must submit to the Regional Office a completed interim statewide assessment 3 years from the date of the previous onsite review, meaning that the work on the interim assessment must begin approximately 6 months prior to that time. The process for completing the interim statewide assessment is the same as completing it at the time of a full review, including the participation of representatives external to the State agency.

E.2. Regional Office Responsibilities

The Regional Office initiates the statewide assessment process by transmitting the statewide assessment form and the data profiles to the State 6 months prior to the date the completed statewide assessment is due in the Regional Office. The Regional Office works with the Central Office of the ACF as needed to obtain the data profiles from NCANDS and AFCARS.

The Regional Office reviews the completed interim statewide assessment for indications of the State's status in relation to the outcomes and systemic factors subject to review. It is not necessary for the Regional Office to approve the interim statewide assessment, but if it is incomplete, the Regional Office will ask the State to provide additional information. In particular, the Regional Office will review the interim statewide assessment to determine whether the State is maintaining the level of achievement on the statewide aggregate data required to comply with the national standard. If the State drops below the national standard for the statewide aggregate data, or the agreed upon percentage of achievement for the State, the Regional Office will follow the procedures at 45 Code of Federal Regulation (CFR) 1355.32 (c) for reinstating reviews based on information that the State is not in substantial conformity. The Regional Office will first request that the State submit additional information. If the additional information submitted by the State continues to indicate nonconformity, the Regional Office, in consultation with the Children's Bureau, may initiate either a partial or a full review, as appropriate, to make the determination of substantial conformity.

Chapter 4
Onsite Reviews

A. Purpose

The onsite review is the second stage of the review process. The onsite review includes the examination of a sample of cases for outcome achievement and interviews with community stakeholders to evaluate the systemic factors under review.

The onsite review of cases is designed primarily to gather qualitative information. The cases reviewed on site include child-specific performance indicators that correspond to certain statewide aggregate data, such as foster care re-entries and the recurrence of maltreatment. Other performance indicators reviewed on site cannot be reported in aggregate form through databases, such as the risk of harm to children and the nature of the relationship between children in care and their parents; therefore, the onsite review is the only source of information for those indicators. Through the combination of aggregate data reported on the statewide assessment and case-specific information gathered on site, the review team is able to evaluate outcome achievement within programs and to identify areas where technical assistance is needed to make improvements.

B. Onsite Review Activities

The onsite review occurs over a period of 1 week. While the exact review schedule must be developed for each individual State, a sample agenda is provided in appendix G.

The State agency must schedule the following activities for the onsite review:

An entrance conference for Federal officials to meet with State staff and review team members to review the structure and agenda of the week's activities and provide opportunities to raise and clarify issues pertinent to the review (Note: If an entrance conference is impractical logistically, it may be forgone altogether or conducted by video or audio conference call.)
An orientation of the review team members at the start of the review, if not previously done, to the instruments used in the review and functions of the team (this should be done prior to the onsite review if at all possible)
Appointments with State-level stakeholders for interviews with the reviewers
A meeting of the entire review team at the end of the review week to conduct the final debriefing, compile the Summary of Findings Form for the State, and prepare for the exit conference
An exit conference for Federal officials to meet with State staff and review team members to provide an overview of tentative findings of the review, discuss next steps, and raise and clarify issues related to the review or the preliminary findings

Each local review site coordinator must arrange the following activities for the onsite review:

Informal entrance and exit conferences, where practical, with local officials and local team members at the local review sites within the State
Schedules for each reviewer that include time for case record reviews; stakeholder interviews; and travel time and means of transportation, if needed, to interviews
Appointments for reviewers to interview individuals in the cases being reviewed
Appointments for the reviewers to interview stakeholders
Daily team briefings at the local review sites

C. Advance Preparation for the Onsite Review

Preparation for the onsite review will be carried out by the Central Office of the Administration for Children and Families (ACF), the Regional Offices, the State central and local agencies, and the contractor handling peer reviewers and certain logistical arrangements for the reviews. The responsibilities of each of those parties are listed below:

C.1. Regional Office ACF Responsibilities

Assign an overall review team leader and other Regional Office team members
Provide Regional Office staff to serve on the review team, including leading or co-leading local teams
Consult with State and Central Office staff about the review team composition, including representatives from other States or Regional Offices if they are to be included on the review team
Notify the peer review contractor, 3 months prior to the onsite review, of the number of peer reviewers needed for the review and approve the list of peer reviewers selected for the review
Provide the peer review contractor with information needed to send out to review team members before the onsite review week
In collaboration with the State and Central Office, identify State-Specific systemic issues from the statewide assessment that need further review onsite, determine locations of the local review sites, and determine the sample size and composition
In collaboration with the State, ensure that all required State and local stakeholders are scheduled for interviews during the onsite review
Request a listing of cases that were open in the State for inhome services during the period under review, from which the sample of inhome cases will be drawn, and transmit the listing to the Children's Bureau for sample selection
Return to the State the random sample listings from the Adoption and Foster Care Analysis and Reporting System (AFCARS) and the list of inhome cases after the Children's Bureau has drawn the samples
Plan for the orientation and training of review team members
Develop the agenda for the entrance conference in collaboration with the State
Determine which individuals on each local review team will provide quality assurance functions to ensure consistency in ratings, correct application of criteria to performance indicators and outcomes, and accuracy in making review-related decisions

C.2. Central Office ACF Responsibilities

Identify Central Office review team members and make arrangements for all team members to arrive and be present for the entire review period
Arrange for the transportation and lodging of Central Office team members
Provide Central Office reviewers with training and preparation to participate as reviewers or local team leaders, including Regional Office staff by teleconference or otherwise where possible
Assign Central Office staff members to serve as local team leaders on the review, as needed
Review the statewide assessment and the preliminary assessment in order to become familiar with the State's performance before the onsite review
Consult with the State and Regional Office on sample size and composition, locations of review sites, and issues needing particular attention in the review
Identify a random sample of cases to be reviewed onsite from AFCARS data (foster care cases) and the listing of inhome service cases provided by the State, and transmit the sample listing through the Regional Office to the State
Assist, as needed, with conducting prereview orientation or training for team members

C.3. Peer Review Contractor Responsibilities

Identify peer reviewers for the review and ensure that they are trained and prepared to participate in the review
Provide the Regional Office with the names of peer reviewers 2 months prior to the onsite review
Obtain Regional Office approval of the peer reviewers selected for the review
Make logistical arrangements for the peer reviewers, e.g., transportation and lodging (Note: The contractor should coordinate these arrangements with Federal and State staff to ensure that peer reviewers are housed in the same locations as Federal and State members of the review team.)
Transmit advance packages of information to all review team members, including logistical information and other materials to prepare them for the review (A checklist of information that should be sent to review team members is included in appendix H.)
Make arrangements for copies of the review instruments and any other information needed by team members during the review week to be available in the State at the beginning of the review

C.4. State Agency Responsibilities (Central Office)

Assign a State coordinator for the review to act as liaison with the Regional Office and the contractor in making arrangements for the review
Identify State members of the review team and provide that information to the Regional Office
Assign local agency review coordinators in each of the local sites selected for the review: they may or may not be members of the review team; they are responsible for setting up interviews, making local arrangements, and ensuring that case records to be reviewed are available
Identify local review sites, including the State's largest metropolitan subdivision, in consultation with the Regional Office and on the basis of information from the statewide assessment
Ensure that all local agency review coordinators have a copy of the procedures manual and are well oriented to the review process
Prepare, and submit to the Regional Office 45-60 days prior to the onsite review, or as soon as the composition of the onsite sample has been determined, a listing of cases that were open for inhome services for any portion of the year under review, from which the sample of inhome service cases will be drawn
Transmit the total sample listings of 150 foster care cases and 150 inhome cases to local agency coordinators 30 days prior to the beginning of the onsite review; local agencies must have sufficient time to review the total sample listings, identify the 30-50 cases to be included using the criteria provided in this manual, contact the persons involved in the cases, and schedule interviews
Schedule State stakeholder interviews
Consult with Regional Office staff and local agency coordinators on logistical arrangements for the review, including:
Identification of lodging arrangements for onsite review team members
Identification of locations for entrance and exit conferences
Identification of space needed for other scheduled meetings and review activities during the week

C.5. Local Agency Coordinators' Responsibilities

Select the cases to be reviewed from the random sample pulled for the review, on the basis of criteria discussed below under "Case Selection and Review"
Orient local staff to the purposes and activities of the review
Ensure that workers assigned to the families selected for review are scheduled for interviews on their case(s)
Schedule and confirm interviews with each person involved in a case who is to be interviewed and orient those persons to the purposed of the review (see appendix E)
Schedule and confirm local stakeholder interviews and orient those persons to the purposes of the review
Schedule and confirm any planned focus groups or other meetings that will be a part of the local review activities
Prepare an agenda for each review team member that includes time to review cases assigned to that individual; the name, time and date of each scheduled interview and/or meeting; and time for the local entrance and exit conferences
Prepare maps or other written directions for each reviewer to get to the scheduled appointments
Reserve space for record review, interviews, and other planned meetings
Assemble all case records to be reviewed so that they will be accessible and ready for review at the onset of the review
Secure any releases of information or confidentiality forms needed to permit reviewers who are neither State nor Federal staff to access case records and interview the individuals associated with a case

D. Case Selection and Review

D.1. Sample of Cases Reviewed

The Regional Office collaborates with the State to determine the number and composition of the sample of case records to be pulled for the onsite review, on the basis of information in the statewide assessment. The sample of cases reviewed will include children in foster care and children receiving services in their own homes.

The sample of foster care cases reviewed on site is selected randomly from the AFCARS data submitted by the State, after the locations and composition of the sample have been determined. The sample of inhome cases is selected randomly from a listing of inhome service cases that were open to services for at least 60 days during the period under review, including cases that were subsequently closed. The State must provide that listing of cases, since the information is not currently available through the National Child Abuse and Neglect Data System (NCANDS) or other national data sources, and should submit it to the Regional Office 45-60 days prior to the onsite review. The listing of cases must include information that will permit selection of the sample by location in the State, since cases will only be drawn for those locations selected for the onsite review. The Regional Office will provide the list of inhome cases to statisticians in the Children's Bureau, who will select a random sample of 150 cases from the listing of inhome cases and 150 cases from the AFCARS data for those locations in the State where the onsite review activities will be conducted. The Children's Bureau will send the sample listings back to the Regional Office, which will then transmit them to the State.

The sample is selected from the universe of cases in the program area under review. The universe includes all children reported on the AFCARS and all families whose cases were open for services during the period under review in the locations selected for the onsite review. Cases of children in foster care will be pulled by child rather than by family, and cases involving inhome services will be pulled by family. Cases selected for the sample must have been open for services for at least 60 days during the period under review, although they may be closed at the time of the review. The Regional Office will advise the State of the specific timeframe covered by the review, which will most likely correspond to the Federal fiscal year of the AFCARS data used for the permanency profiles.

The total sample pulled will consist of 150 foster care cases and 150 inhome cases from the population of cases in the three locations to be reviewed on site. From these listings, a smaller subsample totaling 30 to 50 cases will be selected for review on site across the three locations. For example, approximately 10-20 cases will be reviewed in each of the three locations. The total of 30-50 cases reviewed across the three locations will include both foster care and inhome cases. The proportion of cases pulled from the foster care and inhome case listings will reflect the composition of the sample as decided jointly by the Regional Office and the State, on the basis of information in the statewide assessment. However, both inhome and foster care cases must be included in the sample of cases reviewed onsite. If necessary, to ensure that both types of cases are adequately represented in the sample, the size of the sample of cases reviewed may be increased.

Interviews may not be possible in all the cases in the sample. Therefore, local agency coordinators in the sites being reviewed will select the cases for the review from the random listings of 150 cases, using the criteria below:

The local review coordinators must begin with the random list of case records and select, in order, the first cases on the list for their locations that meet the criteria listed in the bullets below, bearing in mind that all key individuals may not be available for interviews in each case selected.
The local review coordinators will determine whether the individuals in the case who must be interviewed are willing to be interviewed and available and can be reached during the review week. The case may be rejected and the next case on the list considered only under the following circumstances:
If enough of the individuals to be interviewed in the case are either unavailable or completely unwilling to be interviewed that sufficient information cannot be obtained to complete the review instrument
If the case was pulled in error and had not been open for service at any point during the review period, or, for inhome cases, open at least 60 days during the review period
All cases selected for the sample must have some interviews with the relevant parties in the case. Otherwise, the instrument cannot be completed accurately.
The case must have been open for a long enough period of time, i.e., 60 days, during the year under review to have sufficient information to review the case. The listing of cases from which the sample is pulled should only include those cases open for at least 60 days during the period under review. If cases open less than 60 days appear on the list of 150 cases, they should not be considered for the sample of 30-50 cases that will be reviewed.
Local review coordinators must record the reasons for eliminating any cases from the sample of cases to be reviewed on site and make that information available to the Regional Office review team leader.

The cases in the sample of 150 cases that are not selected for review will be used:

To substitute for cases in the sample of 30-50 that cannot be reviewed for the reasons listed above

To provide a pool of additional cases to be reviewed, if needed, to resolve discrepancies between information in the statewide assessment and the findings of the onsite review

D.2. Location of Case Records

All case records being reviewed will be assembled in the local review sites so the reviewers will have access to the caseworkers assigned to the cases.

D.3. Preparation of the Records for Review

Case records must be as orderly and up-to-date as possible, including any files maintained separately, e.g., separate child protective service files or separate family and child records. If the agency uses electronic files instead of paper files, it will be necessary for the local review coordinator to either make computers and technical support available to the reviewers for viewing the electronic records or obtain hard copies of the files or the portions of the files containing information relevant to the review. The caseworkers assigned to each case must be available for interviews with the reviewers. If electronic files are used, staff of the local agency must also be available to assist reviewers in obtaining additional information from the files that might be needed in addition to the hard copy files. If necessary, the State agency will obtain confidentiality statements or releases of information required by the State agency prior to the onsite review, in order to permit reviewers to read case records and interview the relevant individuals in a case. The peer reviewer contractor will obtain signed confidentiality statements from peer reviewers participating in the review.

D.4. Case Interviews

The review team member(s) assigned a particular case is responsible for interviewing the individuals involved in the case, as well as reviewing the case record.

The following persons must be interviewed in a case unless they are unavailable or completely unwilling to be interviewed:

The child (if old enough)
The child's parent(s)
The child's foster parent(s) if the child is in foster care
The family's caseworker with the agency
Any major service providers involved with the child or family; where there are numerous service providers involved with a family, it may only be necessary to schedule interviews with those most recently involved, those most knowledgeable of the family, or those representing the primary services the family is receiving

As needed on a case-by-case basis, other individuals who have relevant information on the case may also be interviewed, such as the child's guardian ad litem, advocate, or other family members.

Only school-age children will be interviewed unless other arrangements are made with the State. Cases involving preschool-age children may be selected for the sample with no child interview scheduled. For example, a preschool-age child included in the review may be observed by the reviewer in the foster home while interviewing the foster parent(s).

If possible, interviews should be conducted where the persons to be interviewed are located, i.e., in the foster home or in the family's home. Where travel arrangements and the availability of reviewers do not permit the time needed to travel to those locations, or when persons to be interviewed prefer not to have the reviewer in their homes or offices, the local coordinators may arrange the interviews in a central location. Also, telephone interviews may be arranged for individuals who are located outside of the local review site.

The interviews with the individuals in a case should follow the time designated on the reviewer's schedule for reviewing the case record. Without first becoming familiar with the circumstances of the case through a review of the record, the reviewer will be unable to explore pertinent issues with the person being interviewed.

Local site coordinators will schedule the interviews not to exceed 1 hour per interview and allow for time between interviews for any necessary travel to the appointments. Maps or other written directions to the interview sites should be prepared in advance and provided to the reviewer. Unless there are specific concerns about a reviewer interviewing someone alone, the caseworker will not need to accompany the reviewer on the interviews. If there are concerns about safety or other issues related to the interview, the local coordinator will advise the reviewer and assist in taking whatever precautions are needed, e.g., scheduling two reviewers for a particular interview or arranging the interview in the office.

Persons to be interviewed must be prepared for the interview by the local coordinator or designee by helping them to understand the purpose of the review. They should be assured that their participation is voluntary, but that their participation is critical to the success of the review. Also, when the interviews have been scheduled, the appointments should be confirmed in writing. (See Preparation for Interviews and Sample Confirmation Letters in appendix E.)

E. State and Local Stakeholder Interviews

The onsite review includes interviews with community or State representatives, i.e., stakeholders, who are knowledgeable about the functioning of the agency in the State and community. The purpose of these interviews is to obtain information about the systemic factors under review and about how the systemic factors affect the outcomes for children and families in general, not on a case-specific basis. Information from the stakeholder interviews is used in combination with information from the statewide assessment to determine the State's conformity with State plan and program requirements for each of the systemic factors.

The review team will interview a complete set of stakeholders in each local review site for the local perspective. In addition, the team will interview stakeholders at the State level who can provide a broader, statewide perspective.

Stakeholder interviews are not to be confused with the interviews conducted on individual cases. Stakeholder interviews are not designed to elicit information on specific cases.

The perspectives and knowledge of individual stakeholders will vary and will determine which systemic issues they can and cannot address. It is unlikely that any single stakeholder will be able to cover each of the systemic factors with equal knowledge. Therefore, in each local review site, and at the State level, the review team must ensure that the combined information obtained from all the stakeholder interviews adequately addresses the three outcome areas and the seven systemic factors.

A Stakeholder Interview Guide is provided (see appendix D) to guide reviewers in interviewing stakeholders. The Regional Office team leader will complete the State-Specific Issues section of the Stakeholder Interview Guide prior to the onsite review so that reviewers will cover the same issues in the three review locations.

State or local agency coordinators may prefer to set up group meetings or focus groups with some stakeholders in place of individual interviews. If so, the meetings should generally be limited to 8-10 individuals whose interests and involvement in child and family services are similar, for example, groups of foster parents, or law enforcement or education representatives.

If possible, State and local stakeholder interviews should be scheduled during regular work hours since the review teams often meet in the evenings for team briefings. However, it may be impossible to arrange for all interviews to occur during regular work hours, and some may have to be scheduled in the evenings.

E.1. Local Stakeholder Interviews

In each local review site, prior to the onsite review, the local review coordinator will schedule a maximum of seven to ten stakeholder interviews. The interviews may be scheduled during the other review activities, depending upon the availability of individuals for interview. Stakeholder interviews should be scheduled for 1 hour each, and the schedule should allow for any necessary travel between appointments. Stakeholder interviews may be conducted either at the local agency or where the stakeholders are located. As in the case-specific interviews, the stakeholders to be interviewed should be prepared for the interviews, and the appointments confirmed in writing.

The following stakeholders must be scheduled for interviews in each local site:

Local child welfare agency administrator
Foster parents (preferably a small group meeting)
Juvenile court judge (or the judge's designated court representative)
Caseworker(s) from the local agency (preferably a small group meeting)
Guardian ad litem/legal representatives (individually or in a group)
Agency attorney(s) (individually or in a group)
Local representatives of administrative review bodies, e.g., foster care review boards, if they exist

Additional representative stakeholders may be selected from the representatives with whom the State consulted in the development of its State Plan, such as:

Tribal representatives
Law enforcement representatives
Youth services representatives
Major initiative/project representatives
Major service providers
Mental health representatives
Education representatives, including special education or early intervention coordinators
Local child and family advocates

E.2. State Stakeholder Interviews

In each State, interviews will also be scheduled with stakeholders who can address issues of concern to the State as a whole, as opposed to local issues. One or two reviewers of the review team will take responsibility for conducting the State interviews. If the location of the stakeholders relative to the review team presents a logistical problem, reviewers may conduct interviews by phone. Prior to the onsite review, the State coordinator for the review will schedule these interviews in collaboration with Federal staff. No more than five to 10 State stakeholder interviews should be scheduled.

The following State stakeholders must be scheduled for interviews:

State child welfare director
State child welfare program specialists (foster care, protective service, adoption, etc.)
State court system representative(s)
Major tribal representatives
State representative(s) of administrative review bodies, e.g., foster care review boards

Additional State stakeholder may be selected from the representatives with whom the State consulted in developing its State Plan, such as:

State education system
State youth services agency
State health department
State Medicaid program
State mental health agency
State child welfare advocates
University social work education program
Major initiative/project representatives
State foster parent association

F. Team Briefings

The local teams will meet daily during the onsite review to review the day's activities. One of the primary purposes of the briefings is for each local review team to complete, over the course of the review week, a Summary of Findings Form based on their case reviews and local stakeholder interviews.

The briefings are the designated forum for individual reviewers to discuss their cases and their rationale for assigning particular ratings. While individual reviewers assign ratings to the cases they review, the briefings provide an opportunity for the team leader and other reviewers to help ensure that all reviewers use consistent rating criteria and are able to substantiate their ratings with adequate information. The briefings must provide opportunities for case discussion within a structured agenda in order to ensure that all cases are adequately briefed and considered by the team.

The briefings should occur following the onsite review activities for the day, and should include the following activities:

Team members who have completed case reviews that day give a brief summary of the case to the local team and provide their ratings for each of the outcomes.
The local team leader records the ratings on the Summary of Findings Form.
Team members who have interviewed stakeholders give a brief summary of the interviews, addressing the systemic issues covered in the interviews.
Team members raise problems or concerns about the schedules, logistical arrangements, instruments, or other areas before the team.
The team leader determines whether all review activities are proceeding according to schedule and whether adjustments are needed.

During the daily briefings, the team leader uses the Summary of Findings Form to record the number of cases reviewed by the team, according to the degree of outcome achievement (substantially achieved, partially achieved, not achieved, or not applicable) for each of the safety, permanency, and well-being outcomes. Team members also discuss, and the team leader records on the form, the performance indicators that substantiate the ratings on the outcomes.

The local teams also summarize the information obtained from the local stakeholder interviews as it relates to the systemic factors and the outcomes. The team leader records this information in the section of the Summary of Findings Form dealing with systemic factors. By the end of the review week, all of the cases reviewed and information from all of the stakeholder interviews should be recorded on the form.

At the end of the onsite review, when the three local teams come together for the final briefing, the Regional Office review team leader will use the Summary of Findings Form as a guide for the final briefing, addressing each item with the entire review team. The information gathered by the three teams during the onsite review is consolidated with the preliminary assessment, which was prepared before the onsite review, for purposes of the exit conference. The Regional Office review team leader should be able to give the State a preliminary report on the outcomes and systemic factors. This verbal report provided at the exit conference should be presented as the team's tentative findings, since a complete analysis and compilation of the information will not be possible until after the onsite review. A determination of substantial conformity cannot be provided at the exit conference, and will be included in the written final report to the State following the onsite review.

G. Instruments

Standardized instruments and instructions are provided by the ACF for all phases of the review (see appendices B, C, D, and F). Orientation to the instruments used during the onsite review will be provided to review team members prior to or at the onset of the review. The following instruments are needed to complete the review:

Statewide assessment
Onsite Review Instrument
Stakeholder Interview Guide
Summary of Findings Form

Copies of the instruments and instructions will be provided for the review by the Regional Office through the peer review contractor. Also, copies of the instruments will be available through the Children's Bureau Web.

Chapter 5
Final Report

A. Purpose of Final Report

The final report is a compilation of the agency's strengths and needs for each of the outcomes and systemic factors reviewed. The primary purpose of the final report is to document for the State the determination of substantial conformity or nonconformity in each area reviewed. The State will use the information in the final report to develop a program improvement plan (PIP) for any area determined not to be in substantial conformity.

B. Format of the Final Report

The format for the final report is the Summary of Findings Form, which is used throughout the review process, supplemented by additional information described below. The preparation of the form begins with a preliminary assessment prior to the onsite review (described in chapter 3), and is then completed during the onsite review and immediately thereafter. The final version of the Summary of Findings Form serves as the final report of the review to the State.

C. Preparation of the Final Report

Information is gathered for the final report at three points in the process:

The preliminary assessment of the outcomes and systemic factors under review is completed by the Regional Office when the State submits the statewide assessment.
The preliminary assessment is updated and supplemented by the team leaders during the onsite review with information gathered on site from case reviews and stakeholder interviews.
The final compilation of the information from the onsite review and the statewide assessment is prepared by the Regional Office immediately following the onsite review.

Following the onsite review, the Regional Office completes the Summary of Findings Form and makes a determination about substantial conformity. Since the preliminary assessment includes information on the performance indicators from the statewide assessment, completing the final report consists of adding information from the onsite review, i.e., case reviews and stakeholder interviews.

D. Content of the Final Report

The information recorded under each item in the Summary of Findings Form must be specific to the item, clearly stated, and relevant to the applicable State plan requirements for the item. If possible, each item should be addressed by including information obtained from all three sources of information used in the reviews: the statewide assessment, the case reviews, and the stakeholder interviews. This will ensure that all of the necessary information will be used in making determinations about substantial conformity. Each item must be addressed using only that information obtained during the review process. The confidentiality of individual children, families, and representative stakeholders is protected in the report by not citing names of persons or organizations from which information was obtained.

The completed final report to the State will include:

Cover Letter. The cover letter includes a statement about substantial conformity; the amount of the penalty, if applicable; and the date by which a PIP must be submitted to the Regional Office, if applicable.
Executive Summary. This section is a summary of the major strengths and needs noted in each outcome and systemic area and the status of substantial conformity.
Introduction. This section provides an overview of the background and purposes of the review; the outcome areas reviewed: and dates and descriptions of the review activities, such as time periods involved, methods of completing the statewide assessment, review locations, review team representatives, and number and type of cases reviewed. (See appendix K for sample introductory language.)
Summary of Findings Form. This form is completed and noted above, along with the final sections of the form that address the areas of substantial conformity.

D.1. Example of Final Report

The example on the following page is one page from the Summary of Findings Form, illustrating how the preliminary assessment example in chapter 3 is updated with information from the onsite review for the final report.

Example of Final Report

II. PERMANENCY

Outcome P1: Children have permanency and stability in their living situations.
Number of cases reviewed by the team according to degree of outcome achievement:
	Team 1	Team 2	Team 3	Total Number	Total Percentage
Substantially Achieved:	7	7	12	26	81%
Partially Achieved:	3	1	1	5	16%
Not Achieved or Addressed:	0	1	0	1	3%
Not Applicable:	6	6	6	18	__________
Conformity of Statewide data indicators with national standards:
	National Standard	State's Percentage	Meets Standard	Does Not Meet Standard
Foster care re-entries	13%	20%	_________	X
Length of time to achieve reunification	80%	87%	X	___________________
Length of time to achieve adoption	26%	34%	X	___________________
Stability of foster care placements	77%	73%	_________	X
Length of stay in foster care	12 months	9 months	N/A	N/A

Item 5: Foster Care Re-entries

_______Strength ___X___ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates that more children (20%) in foster care have had multiple entries within the previous 12 months than the national standard for this indicator (13%). Percentage of children re-entering foster care has increased over the past 3 years from 16% to 20%.
According to the statewide assessment, the State has observed an increase in foster care re-entries as the length of stay in foster care has decreased. State indicates that most re-entries are by children discharged to reunification, as opposed to other discharge reasons. State is concerned about the availability of services to families following reunification.
Of 32 foster care cases reviewed onsite, ten had been discharged from foster care and re-entered within the past year. All ten had re-entered due to the same general reasons as the original entry. There was little evidence of post-reunification services to families, and no safety plans were developed for children being reunified.
Service providers interviewed in all 3 review sites indicated they receive few requests for post-reunification services. When requested, they typically provide individual or family counseling and parenting classes. There was no evidence of more specialized services to support reunification.

Item 6. Stability of foster care placement

_______ Strength ___X___ Area Needing Improvement _____ Both

Basis:

Statewide aggregate data indicates fewer children in the State (73%) have no more than 2 placements in a 12-month period than the national standard (77%) for this indicator. The number of placement settings for children in foster care has not changed significantly over the past 3 years.
Statewide assessment indicates the average number of placements per child in the first episode of foster care is 3. Most children go to shelter care first and then to another placement setting. Foster care disruptions are increasing.
Of the 32 children in foster care reviewed, 18 of the children had been in two placement settings, 9 had been in three settings, and five had been in four or more settings. Most of the children were initially placed in shelter care and later moved. Except for moving children from shelter, all but five of the other moves were unplanned disruptions.
Foster families interviewed indicated a need for more support services, such as respite care and behavior management. They ndicated that while caseworkers provide assistance when requested, they generally have infrequent contacts with their workers unless there is a problem.

E. Distribution of the Final Report

The Regional Office will provide copies of the final report to the State agency executive officer, the State review coordinator, the Central Office and Federal review team members, and the peer review contractor within 30 days of completing the onsite review. In accordance with Federal requirements that the reports be made available to the public, States will distribute copies of the report to State members of the review team and make the results of the review available to the public. The Children's Bureau will also publish information pertaining to the reviews on its Web site or through other means, as appropriate.

Chapter 6
Substantial Conformity

Determinations of substantial conformity are made separately for each of the seven outcomes and the seven systemic factors under review. A State may be determined to be in substantial conformity on one or more of the outcomes or systemic factors and not in substantial conformity on the others. Program improvement plans and penalties cover only those areas determined not to be in substantial conformity.

Appendix I contains a chart that displays the criteria for determining substantial conformity with the outcomes and the systemic factors. The chart also lists each of the performance indicators, including the statewide aggregate data, used to determine substantial conformity. There are separate methods for determining substantial conformity on the outcomes and the systemic factors, as described below.

A. Substantial Conformity on the Outcomes

Two sets of information are used to determine the State's substantial conformity on each of the seven outcomes: the percentage of cases reviewed in which the outcome was determined to be substantially achieved and the State's performance on the statewide aggregate data for which national standards have been established.

First, reviewers must determine whether the outcomes in each individual case they review are substantially achieved. Those decisions are based on the performance indicators listed in the Onsite Review Instrument, which is completed for each case.

Second, the review team must make determinations regarding substantial conformity for the State as a whole. Those decisions are based on the percentage of cases reviewed in which the outcomes are substantially achieved, and statewide aggregate data for selected outcomes. Currently, in only two of the seven outcomes (Safety Outcome #1 and Permanency Outcome #1), decisions about the State's substantial conformity are made on the basis of statewide aggregate data in addition to the findings of the onsite review. For the remaining five outcomes, the performance indicators reviewed on site provide the basis for determining the State's substantial conformity. In the future, other statewide data indicators may be included in determinations of substantial conformity.

The diagram below illustrates the process of determining substantial conformity on the outcomes.

Step 1

Step 2

Step 3

Step 4

A.1. Rating Outcomes on an Individual Case

When a reviewer has gathered all of the information on a specific case through review of the case record and interviews with parties in the case, (s)he must make a decision as to whether each applicable outcome in the case has been substantially achieved, partially achieved, or not achieved.

The performance indicators used to make determinations as to whether each outcome has been substantially achieved are listed in the Onsite Review Instrument (see appendix C). The reviewer uses the following criteria to determine whether or not the outcomes have been substantially achieved in each individual case:

Safety Outcome #1: Both onsite indicators must be rated a "strength."

Safety Outcome #2: All three indicators must be rated a "strength."

Permanency Outcome #1: Not more than one of the six onsite indicators may be rated an "needing improvement."

Permanency Outcome #2: Not more than one of the six onsite indicators may be rated an "area needing improvement."

Well-Being Outcome #1: Not more than one of the five onsite indicators may be rated an "area needing improvement."

Well-Being Outcome #2: The one onsite indicator must be rated a "strength."

Well-Being Outcome #3: Both onsite indicators must be rated a "strength."

The reviewer will determine an outcome in an individual case to be partially achieved if some of the performance indicators for that outcome have been rated as "strengths," but fewer than the number noted in the paragraph above. If none of the performance indicators are rated as "strengths" for a particular outcome, the reviewer will determine that the outcome is not achieved.

A.2. Establishing the National Standard on Statewide Aggregate Data

A national standard will be established for each of the statewide aggregate data indicators used to determine substantial conformity in the reviews. For the foster care indicators that are based on the Adoption and Foster Care Analysis and Reporting System (AFCARS) data, the national standard will be established according to the following procedure:

Each State and the District of Columbia submits data through AFCARS for selected time periods. (Note: The foster care standards will initially be set using 1998a and 1998b AFCARS submissions.)
All of the data submissions for the time periods are pooled. Assuming that each of the States and the District of Columbia submitted complete data for each of two periods, there will be 102 data elements in the pool.
All of the data submissions are then rank-ordered on a scale from highest to lowest.
The point on the scale that represents the 75^th percentile is the national standard for that statewide aggregate data indicator.

The same process is used to establish the national standards on the two statewide aggregate data indicators derived from the National Child Abuse and Neglect Data System (NCANDS) data, except that two NCANDS reporting periods are used to establish the standard. (Note: The safety standards will initially be set using 1997 and 1998 NCANDS data.)

When a State begins a child and family services review, its data for the period under review are compared with the national standards to determine the State's substantial conformity. States whose data fall below the national standard in a review will be required to implement a program improvement plan (PIP) designed to improve the States' performance on the data indicators in order to achieve substantial conformity. However, the State and the Regional Office may negotiate a percentage of improvement to be made in the statewide data indicators over the course of a PIP that is less than the national standard. In those circumstances, if the State achieves the level of improvement agreed upon in the PIP, the State will not be penalized for nonconformity on the basis of the statewide aggregate data indicator. The criteria for determining the amount of improvement that must be made through a PIP are discussed in chapter 7, Program Improvement Plans.

With a goal of continuous quality improvement, States whose data remain below the national standard in subsequent reviews will be required to establish new benchmarks of improvement to be made toward the eventual attainment of the national standard. As long as the State reaches the agreed-upon level of improvement, failure to reach the national standard will not be the basis for withholding Federal funds with respect to the outcome in question.

A.3. Determining Substantial Conformity for the State

In order for the State to be determined to be in substantial conformity on any given outcome, the outcome must be determined to be substantially achieved in 95 percent of the cases reviewed (90 percent in the first review). In addition, the State must meet the national standard that has been established for any statewide aggregate data attached to that particular outcome.

The example on the following page illustrates how the cases reviewed on site are tallied by outcome to determine, in part, if the State is in substantial conformity.

Currently, national standards will be established for statewide aggregate data on two of the outcomes, Safety Outcome #1 and Permanency Outcome #1. Those statewide aggregate data include the following:

Safety Outcome #1:

Recurrence of maltreatment (Of all children who were victims of substantiated or indicated child abuse and/or neglect during the first 6 months of the reporting period, XX percent had another substantiated or indicated report within a 6-month period.)
Incidence of child abuse and/or neglect in foster care (Of all children in foster care in the State during the period under review, XX percent were the subject of substantiated or indicated maltreatment by a foster parent or facility staff.)

Permanency Outcome #1:

Foster care re-entries (Of all children who entered care during the year under review, XX percent re-entered foster care within 12 months of a prior foster care episode.)
Stability of foster care placement (Of all children who have been in foster care less than 12 months from the time of the latest removal, XX percent had no more than two placement settings.)
Length of time to achieve adoption goal (Of all children who exited foster care during the year under review to a finalized adoption, XX percent exited care in less than 24 months from the time of the latest removal from home.)
Length of time to achieve reunification (Of all children who were reunified with their parents or caretakers at the time of discharge from foster care, XX percent were reunified in less than 12 months from the time of the latest removal from home.)

Example of Calculating Substantial Achievement of Outcomes in Cases Reviewed On Site

Outcome	Number of Cases Substantially Achieved	Number of Cases Partially Achieved	Number of Cases Not Achieved	Percentage of Cases Substantially Achieved (N=30)
Outcome S1: Children are, first and foremost, protected from abuse and neglect.	25	5	0	83%
Outcome S2: Children are safely maintained in their own homes whenever possible and appropriate.	29	0	1	96%
Outcome P1: Children have permanency and stability in their living situations.	30	0	0	100%
Outcome P2: The continuity of family relationships and connections is preserved for children.	15	5	10	50%
Outcome WB1: Families have enhanced capacity to provide for their children's needs.	29	1		96%
Outcome WB2: Children receive appropriate services to meet their educational needs.	20	5	5	66%
Outcome WB3: Children receive adequate services to meet their physical and mental health needs.	30	0	0	100%

Although it will not be used initially to determine substantial conformity, we expect a national standard to be established for the statewide aggregate data indicator, "length of stay in foster care." This indicator is defined as follows: the median length of time to discharge for children in the first-time cohort entry group in foster care for the year under review. Currently, the indicator is included in the data profiles so that States and the Federal government can evaluate this important indicator and its relationship to outcomes for children in foster care. Using it to determine substantial conformity, however, requires that each State's cohort group for the period under review achieve the median discharge rate prior to the initiation of the statewide assessment. Preliminary review of States' data indicates that, for some States, there will be insufficient time from the time the cohort group of children entered foster care to the initiation of the review for 50 percent of those children to exit care and thus achieve the median length of stay. Therefore, we are providing the indicator for contextual purposes only at this point.

For the remaining five outcomes, the determination of substantial conformity is based on whether the outcomes have been determined to be substantially achieved in 95 percent (90 percent in the initial reviews) of the cases reviewed on site.

If these findings on the case reviews were consistent with the statewide aggregate data indicators from AFCARS and NCANDS, and information in the statewide assessment, the State would not be in substantial conformity on the following outcomes: Safety Outcome #1, Permanency Outcome #2, and Well-Being Outcome #2.

A.4. Example of Determining Substantial Conformity on the Outcomes

The example below illustrates the process of determining substantial conformity on Permanency Outcome #1, "children have permanency and stability in their living situations."

In this example the outcome "children have permanency and stability in their living situations" includes seven performance indicators and statewide aggregate data indicators that are used to determine compliance:

Foster care re-entries (both onsite and statewide data)
Stability of foster care placement (both onsite and statewide data indicators)
Permanency goal for the child (onsite data only)
Independent living services (onsite data only)
Length of time to achieve adoption goal (both onsite and statewide data)
Length of time to achieve reunification (statewide data only)
Permanency goal of other planned living arrangement (onsite data only)

The onsite performance indicators in the Onsite Review Instrument are rated on information obtained from the cases reviewed. The statewide aggregate data indicators are obtained from the AFCARS data profiles included in the statewide assessment. In order for the State to be determined to be in compliance with this outcome, it must meet the national standard on each of the statewide aggregate data indicators listed above and the outcome must be rated as substantially achieved in 95 percent of the cases reviewed.

In this example, assume that we determine from the statewide assessment that the State meets the national standard for compliance with three of the four statewide aggregate data indicators attached to this outcome (length of time to achieve adoption goal, length of time to achieve reunification, and foster care re-entries), but does not meet the national standard on one statewide aggregate data indicator (stability of foster care placement). In the onsite case reviews, assume that we determine that 95 percent of the cases reviewed included ratings of substantially achieved for Permanency Outcome #1.

Because the State did not meet the national standard on all four of the statewide aggregate data indicators, it is not in compliance with this outcome. The State must enter into a PIP to improve its performance on "stability of foster care" in order to be determined to be in compliance.

B. Substantial Conformity on the Systemic Factors

Individual determinations of substantial conformity are made on each of the systemic factors, on the basis of the State plan or other program requirements for each factor reviewed. Information pertaining to the systemic factors is gathered and evaluated in the statewide assessment and the stakeholder interviews. Using this information, the review team makes the following determinations regarding each systemic factor:

Whether or not the State plan requirements and other program requirements attached to the systemic factor are actually in place in the State
Whether or not the State plan requirements and other requirements attached to the systemic factor are functioning as described in the applicable regulation or statute

At the beginning of the onsite review, the review team will have the information on systemic factors that the State has included in the statewide assessment. During the onsite review, information from the local stakeholder interviews and the State stakeholder interviews will be gathered and used to evaluate the systemic factors. As with the outcomes, local review teams do not make determinations about substantial conformity on the systemic factors since the factors are Statewide issues. A determination about substantial conformity on the systemic factors cannot be considered until all three teams come together with their information at the end of the review. The final determination of substantial conformity is made in the written report to the State following the onsite review.

Each of the seven systemic factors reviewed is rated on the basis of multiple State plan or other program requirements, with the exception of "information system capacity," which is rated on only one State plan requirement. Information obtained from both the statewide assessment and the stakeholder interviews must indicate that the State plan and other program requirements reviewed for each systemic factor are in place and functioning as required in order to make a determination of substantial conformity for the systemic factor.

The scale below describes how the State plan and program requirements are used to determine substantial conformity on the systemic factors. In order for a specific systemic factor to be determined to be in substantial conformity, the review team must assign it a rating of three or four, based on the criteria in the scale below.

Not in Substantial Conformity		Substantial Conformity
1	2	3	4
None of the State plan or program requirements is in place.	Some or all of the State plan or program requirements are in place, but more than one of the requirements fails to function at the level described in each requirement.*	All of the State plan or program requirements are in place, and no more than one of the requirements fails to function as described in each requirement.*	All of the State plan or program requirements are in place and functioning as described in each requirement.

*For the systemic factor "information system capacity," if it is determined that a system is in place but not functioning at the level described in the one State plan requirement reviewed, that factor is rated "2," rather than "3."

B.1. Example of Determining Substantial Conformity on the Systemic Factors

Using the systemic factor "case review system" as an example, a determination of substantial conformity using the method described above would occur as follows:

In this example, the systemic factor "case review system" has five State plan or program requirements subject to review. The statewide assessment indicates that there are procedures and policies in place statewide that address the following requirements for this systemic factor:

The requirement that each child have a written case plan with the required content
The requirement that the status of each child in foster care is reviewed no less frequently than once every 6 months
The requirement that permanency hearings are held as required
The requirement that termination of parental rights petitions are filed under the required circumstances
The requirement that foster parents, preadoptive parents, and relative caregivers of children in foster care are notified of reviews and hearings held with respect to the child

In summary, all five State plan or program requirements are in place according to the statewide assessment, although we cannot determine from the statewide assessment whether they are functioning properly.

The onsite review indicates the following information, based on stakeholder interviews:

The case plan requirements are met consistently statewide and in the three local review sites.
The procedures for periodic review are in place in all three local review sites, but the reviews do not occur on a timely basis in two of the sites.
The procedures for permanency hearings are in place statewide, but the hearings are not held according to the requirements in State and Federal law in any of the three local review sites.
The procedures for termination of parental rights are in place and functional in all three local review sites.
The required parties are notified of hearings in all locations.

In combination, the information from the statewide assessment and the onsite review indicates that three of the five State plan or program requirements for the systemic factor "case review system" are in place statewide and functional. The onsite review determines that two of the requirements, periodic reviews and permanency hearings, are in place but fail to function as required. Therefore, the State would be rated a "2" according to the table on the preceding page, since more than one of the requirements fails to function properly. The State would not be in substantial conformity on the systemic factor "case review system."

C. Resolving Discrepancies Between the Statewide Assessment and the Onsite Review

In some situations, the statewide aggregate data, or information in the statewide assessment, may not be consistent with the information on corresponding performance indicators obtained during the onsite review. For example, the statewide data on foster care re-entries may indicate the State is not within the national standard, although the onsite review of cases indicates satisfactory performance in that area. In those situations, the discrepancy between the two sets of information must be resolved in order to make a determination about substantial conformity.

In order to resolve the discrepancies where they exist, the Regional Office will provide the State with the option of:

Submitting additional information that explains or resolves the discrepancy, such as additional data or other evidence that explain or support a determination of substantial conformity, or
Reviewing additional cases, selected from the original samples of 150 cases pulled for the review.

If the State chooses to submit additional information, it can be in the form of additional aggregate data, special studies, quality assurance review findings, or other similar information.

If the State chooses to have additional cases reviewed on site, the number of additional cases, when combined with the original sample of 30-50 cases reviewed, will comprise a statistically significant sample with a compliance rate of 90 percent (95 percent in reviews beyond the initial review), a tolerable sampling error of 5 percent, and a confidence coefficient of 95 percent. Typically, the number of cases needed to comprise a statistically significant sample at this level is around 150 cases. Statisticians in the Central Office will be available to assist Regional Office staff in determining the exact number of additional cases to be pulled. The additional cases, reviewed by a joint State and Federal team, will only be reviewed for the performance indicator, outcome, or systemic factor in question and will cover the same time period as the original review. The conclusions made from reviewing the additional cases, in combination with the original cases reviewed, will form the basis for determining substantial conformity.

The timing, process, and review team associated with the review of additional cases will be determined by the Regional Office, depending upon the number and complexity of the discrepancies to be resolved. The review of additional cases, where needed, will follow the onsite review as quickly as possible so that a prompt and accurate determination of substantial conformity can be made.

The following is an example of resolving a discrepancy between the statewide assessment and the findings of the onsite review.

C.1 Example of Resolving a Discrepancy

This example involves the performance indicator "stability of foster care" from the earlier example of determining substantial conformity on an outcome.

For Permanency Outcome #1, "stability of foster care" is both a statewide aggregate data indicator and one of the performance indicators rated in the cases reviewed on site. In the example described in section A.4 of this chapter, the statewide aggregate data on stability of foster care indicated that the State did not meet the national standard. However, assume that the reviewers determined that the performance indicator "stability of foster care" was a strength in the cases reviewed.

In this example, the State chooses to provide additional information on the indicator to resolve the discrepancy between the information from the statewide assessment and the findings of the onsite review. We present two possible scenarios resulting from the States providing additional information:

Scenario 1: The State analyzes the aggregate data on "stability of foster care," including county-by-county breakdowns of the data. The analysis indicates that the State, as a whole, fails to meet the national standard, but that the stability of foster care in some counties in the State, including the local review sites of the onsite review, met the national standard. The State also provides detailed data showing steady improvements over time in the stability of foster care placements. However, the additional data fail to establish that the State as a whole meets the national standard on this statewide aggregate data indicator. Therefore, the discrepancy is resolved by determining that the State's performance on the indicator "stability of foster care" does not meet the national standard for purposes of determining the State's substantial conformity on Permanency Outcome #1.
Scenario 2: The State provides additional data on "stability of foster care" from an AFCARS submission that is more recent than the submissions from which the permanency data profiles for the statewide assessment were created. These data demonstrate clearly that the State has met the national standard for this statewide aggregate data indicator, which is consistent with the findings of the onsite review. An analysis of the more recent data, compared to a set of prior AFCARS submissions, indicates that the State has made steady improvements over time in the stability of foster care placements and has only recently achieved compliance with the national standard. Having met the national standard obviates the need for a PIP to bring the State's performance on the indicator up to standard. Therefore, the discrepancy is resolved by determining that the statewide aggregate data indicator "stability of foster care" meets the national standard for purposes of determining substantial conformity on Permanency Outcome #1.

Chapter 7
Program Improvement Plans

A. Criteria for Developing a Program Improvement Plan

The State must develop a program improvement plan (PIP) when:

The review indicates that the achievement level of any one of the seven outcomes falls below the threshold for substantial conformity.
The review indicates that the State falls below the threshold for substantial conformity on any one of the systemic factors subject to review.

B. Content of the PIP

The format of the PIP may vary, but the plan must include the following components:

The PIP must address each outcome that has been determined not to be in substantial conformity. For each outcome found not to be in substantial conformity, the PIP must address the onsite performance indicators or statewide aggregate data, noted in the final report, that contributed to the low achievement level of the outcome.
The PIP must address each systemic factor found not to be in substantial conformity. For each systemic factor determined not to be in substantial conformity, the PIP must address each State plan requirement noted in the final report that contributed to a determination of nonconformity on the systemic factor.
The PIP must include an action strategy to bring each outcome or systemic factor not in substantial conformity up to a level of substantial conformity, as specified in the Regulation, including the following elements:
- Priorities for correcting areas of nonconformity, beginning with areas that directly affect the safety of children
- Steps necessary to improve performance
- Timeframes for accomplishing each step
- Individuals responsible for carrying out the various steps
- The geographical areas of the State where efforts will be undertaken or targeted
- A description of how progress will be evaluated, including measurable benchmarks of progress
- A description of how the Regional Office will know that conformity has been achieved by the State

For each statewide aggregate data indicator that is determined to be out of conformity with the national standard, the PIP must include a specific percentage of improvement that will be achieved over the duration of the plan, with which the State and Regional Office have concurred.
For each outcome or systemic issue addressed, the State must describe the technical assistance resources it plans to use to improve performance. Although States are not required to use the National Resource Centers (NRCs) as sources of technical assistance, if the State plans to use the centers for technical assistance, the State and the Regional Office should jointly develop a strategy for using the centers.
In outcome areas that have particularly adverse effects on families and children served by the agency, and those requiring long-term solutions, both short-term and long-term goals and strategies should be included in the PIP in order to address immediate needs and plans for lasting reforms.
The PIP must include a description of how progress on the plan will be evaluated by the State and reported to the Regional Office, including the frequency and format of the evaluation procedures.

In determining the amount of improvement that a State must achieve through the PIP with regard to statewide aggregate data indicators that fall below the national standard, the following criteria should be used:

The amount of improvement must be stated in the PIP in terms of absolute percentage points to be achieved.
If the amount of progress to be achieved through the PIP does not reach the national standard, the amount negotiated between the State and the Regional Office must be significant enough to move the State toward compliance with the national standard in a reasonable period of time.
The amount of progress to be achieved should be consistent with the level of effort required by the State to improve its performance on the data indicators.
The amount of progress to be achieved should be determined with consideration of current or prior efforts to improve performance in the State, and it should build on any program improvement planning initiatives underway.
The amount of progress to be achieved should be determined with consideration of the length of time needed for program improvements to be reflected in the statewide data indicators.

(Note: When the national standards are established, the Children's Bureau expects to provide additional guidance on the amount of improvement that Regional Offices may negotiate with States, based on statistics pertaining to the indicators, e.g., the range of performance on indicators by States and deviation from the standard.)

C. Preparation of the PIP

Preparation of the PIP is an extension of the collaborative planning process used by the State in developing its 5-year Child and Family Services Plan (CFSP), by including members of the title IV-B planning group in developing the PIP and tying improvements to the goals and strategies of the CFSP, particularly long-range improvements. Responsibility for developing the PIP rests with the State child welfare agency in collaboration with the Regional Office. To the extent possible, State members of the review team, including those from outside the State agency, will be actively involved in developing the PIP.

The State must submit its PIP to the Regional Office for approval within 90 calendar days from the date the State receives written notice from the Administration for Children and Families (ACF) that it is not operating in substantial conformity. Regional Office staff, in consultation with the Central Office as needed, will review the PIP and notify the State in writing of approval. If the ACF determines that revisions to the PIP are needed, the State must revise and submit the revised PIP to the ACF within 30 calendar days of receiving written notice from the ACF that the PIP was not approved by the ACF.

D. Technical Assistance

To the extent possible, technical assistance needs should be coordinated with other program improvement efforts underway in the State, especially with the implementation of the State's 5-year CFSP. States have flexibility in deciding which providers of technical assistance can best address their needs. Particularly where long-range assistance is needed, emphasis must be placed on building capacity within the State to meet ongoing needs and sustain progress. For that reason, States are encouraged to develop existing community and professional relationships as sources of ongoing technical assistance. Examples of this include:

Using intra-State university social work programs for training and evaluation needs
Using community groups as consultants or advisors to improve the agency's responsiveness to the community
Coordinating the agency's improvement plans with similar plans developed through other publicly or privately funded initiatives in the State
Using existing advocacy or consumer groups as consultants on policy development and practice issues

The sources of technical assistance most readily available from the ACF are the NRCs, funded by the ACF. The NRCs currently funded by the ACF that are available to provide training or technical assistance to States include the following:

National Abandoned Infants Assistance Resource Center
National Child Welfare Resource Center for Family-Centered Practice
National Child Welfare Resource Center for Legal and Judicial Issues
National Resource Center on Child Maltreatment
National Resource Center for Community-Based Family Resource and Support Programs (FRIENDS)
National Resource Center for Foster Care and Permanency Planning
National Resource Center for Information Technology in Child Welfare
National Resource Center for Organizational Improvement
National Resource Center for Special Needs Adoption
National Resource Center for Youth Development

Examples of other sources of technical assistance that may be arranged directly by the State or with the assistance of the Regional Offices include the following:

Peer State consultation and assistance
National organizations and foundations
Private providers and agencies
National experts

E. Role of Regional Office in Coordinating Technical Assistance

States will request the assistance of the NRCs in implementing their PIPs from the designated Regional Office staff member, who will coordinate the technical assistance request with the NRC.

The amount of technical assistance that a State receives through the NRCs to implement a PIP depends, in part, upon the level of need for technical assistance identified in the plan. The work of the NRCs is currently prioritized toward providing technical assistance to States that are implementing PIPs in order to achieve substantial conformity through the child and family services reviews. Decisions about the amount of technical assistance available to States through the NRCs will be made jointly by the Regional Office and the Central Office.

Regional Offices must coordinate the provision of technical assistance through the NRCs in a manner that provides maximum benefit to the States within the timeframes specified for completion of the PIP. A coordinated technical assistance response is particularly important when States are determined not to be in substantial conformity on multiple outcomes or systemic factors and there is a need for more than one source of technical assistance.

The Regional Offices have an important role in assisting the State to develop the portion of its PIP that concerns technical assistance. That role includes the following responsibilities:

Familiarize States with the various NRCs and their functions
Assist the State to identify areas of nonconformity that can best be addressed by individual NRCs
Make referral to the NRCs for technical assistance
Involve the NRCs in the development of the PIP with the State to the extent needed and appropriate
Assist the State to develop a strategy, within the PIP, for using technical assistance from the NRCs that avoids duplication of effort, targets technical assistance to the most appropriate areas of need, and provides for most efficient use of both free and purchased technical assistance
Ensure that the technical assistance provided by the NRCs complements and is coordinated with technical assistance that is provided through other sources
Monitor the provision of technical assistance to the State by the NRCs, as part of monitoring the overall implementation of the PIP

F. Timeframes for Implementing the PIP

The time period for completing the PIPs will not exceed 2 years. Not all components of the plan will require a full 2 years to implement, and this timeframe is provided as an outside limit for those elements of the plan requiring more extensive planning and action steps.

Where the State has been determined not to be in substantial conformity due to child safety issues, those components of the PIP pertaining to child safety must be implemented in less than 2 years (45 Code of Federal Regulation [CFR] 1355.35 (d)(2)). Child safety issues must receive priority in developing and implementing the PIP.

In the event the State is required to make major improvements that are too complex or extensive to implement within 2 years, the Secretary of the U. S. Department of Health and Human Services may grant up to a 1-year extension to the 2-year timeframe for completing the PIP, extending the maximum implementation period to 3 years. The request for an extension will only be approved in highly exceptional situations. The primary criterion for approving extensions to the PIP will be the complexity of the improvements to be made, not the failure of the State to act in a timely manner in implementing the components of the PIP.

When needed, the ACF and the State may renegotiate the terms and conditions of the PIP under the following conditions:

The renegotiated plan must be designed to correct areas of the State's programs determined not to be in substantial conformity.
The total time period for implementing the plan may not exceed the maximum of 3 years.
The ACF must approve the renegotiated plan.

States requesting an extension of the PIP up to a third year must submit the request in writing to the Regional Office with supporting documentation that the extension is necessary to make the required improvements. The written request must be received by the Regional Office 60 days prior to the approved completion date of the PIP. The Regional Office will submit the request, along with the supporting information and the recommendation of the Regional Office, to the Central Office of the ACF for review and decision making by the Secretary.

G. Evaluating the Implementation of the PIP

The Regional Office, in collaboration with the State, will evaluate the State's substantial conformity with the terms and conditions of its approved PIP, including the achievement of the benchmarks included in the PIP.

The State must provide written progress reports to the Regional Office no less frequently than quarterly, unless the Regional Office and the State mutually agree that less frequent reporting is necessary. The quarterly progress reports must include sufficient detail to describe the progress made during the reporting period, including data or other measurable indicators, along with the timeframes covered by the data, that address the specific timeframes and benchmarks of progress included in the PIP.

Penalties are suspended while the State is implementing the approved PIP. However, if the ACF determines that the State has failed to meet critical benchmarks toward achieving substantial conformity on specific outcomes or systemic factors within the timeframes of the approved plan, the ACF will cease suspension of the penalties and begin immediate withholding of funds related to the outcome or systemic factor to which the penalty applies.

As individual components of the program improvement plan are determined by the ACF to be complete, to the degree that substantial conformity in a particular area has been achieved, the ACF will find the State to be in substantial conformity in that area prior to the end date of the plan. At that point, withholding of funds related to the particular outcome or systemic factor will be rescinded. In other words, individual components of the plan will be declared completed or achieved throughout the duration of the plan, not just at the end of the plan.

If the ACF cannot determine from evaluating the progress of the PIP that the State has achieved substantial conformity, a determination of substantial conformity will be made at the next scheduled review.

Download FREE Adobe Acrobat® Reader™ to view PDF files located on this site.

Last Updated: February 17, 2006

Child and Family Services Reviews Procedures Manual Washington, D.C. August 2000

TABLE OF CONTENTS

Child and Family Services Reviews
Procedures Manual

Washington, D.C.
August 2000