Chapter V: Procedures for the On-Site Review Phase: State Guide to an AFCARS Assessment Review

Chapter V: The On-Site Review Phase

A. Introduction

The on-site review occurs over a period of five consecutive working days. During this week, there is an entrance and exit conference with management staff, the entry of a case scenario, a discussion of the State's provision of child welfare services and policies, a review of the technical documentation used to extract the AFCARS data, and a review of case files. During the exit briefing a summary of the major findings is presented. Draft documents of the team's preliminary on-site findings are left with the State team in order to allow them to begin making changes, if any, to the information system and the AFCARS reporting process. This also allows the team time to identify priority areas and a work plan.

The State's information system is assessed against the AFCARS requirements in the Federal regulation, policy issuances, and the AFCARS Technical Bulletins. The AAR evaluates a State's information system's capability to collect, extract, and transmit the AFCARS data accurately to the Children's Bureau. A second focus of the AAR is to assess the accuracy of the collection and documentation of information related to the foster care and/or adoption case of a child.

There are two major areas that are assessed during an AAR. They are: 1) the AFCARS general requirements, and 2) the data elements. The general requirements include the population that is to be reported to AFCARS and the technical requirements for constructing a data file. During an AFCARS Assessment Review, each of the 103 data elements and each of the 28 AFCARS general requirements are assessed on the basis of the requirements in the regulation and other policy and technical issuances. Information that is collected from each of the components of the AAR is combined and the item is assigned a rating factor. The general requirements are evaluated and rated separately from the data elements.

Back to Table of Contents

B. On-Site Logistics

During the week of the AAR, work hours will generally be between 8:30 a.m and 5:30 p.m. The agenda in Appendix H gives a general timeframe for each day. The review team will require access to the State's information system all week (primarily on Monday, Wednesday and Friday), as well as the capability to project in the meeting room.

For the case file review, it is necessary for the reviewers to be in one room. If this is not possible, a smaller second room can be designated for the group that will review the adoption files. The information system will not be needed for the case file review.

Back to Table of Contents

C. On-Site Review Activities

There are five components of the AFCARS Assessment Review process. These are the:

system demonstration using a sample case scenario;
reports from the AFCARS utilities (Data Compliance, Data Quality and Frequency);
State policy and procedures;
paper case file review; and
system review (screens and extraction code).

The Federal AFCARS team utilizes each component to assess the State's AFCARS data. Each component is discussed with the State team in order to gain a better understanding of the State's child welfare practice and policy, how the data are collected and extracted, and State staff's understanding of the data elements.

1. Day One Entrance Conference and System Demonstration

The first day of the on-site visit begins with the entrance conference which serves to:

introduce the Federal team to State management, project staff, and State review team;
review the nature and scope of the on-site review;
respond to any questions the State staff might have related to the AAR process; and
make any final modifications to the on-site agenda.

Immediately following the entrance conference, the State presents a brief walk-through of the system that is used to collect and extract the AFCARS information. The State should ensure that appropriate technical and programmatic staff participate in the demonstration. The Federal team will provide the State review team with a sample case scenario onsite. This sample case scenario is entered as part of the system demonstration. The purpose of this activity is to observe the entire process of entering data, and learn the flow of the screens used to enter the data. The State team should also be prepared to identify the test cases that were entered during the pre-site visit phase.

2. Day Two Case File Review

The Federal team provides the forms for each record drawn in the sample. This form contains the data submitted by the State for each element in the record. There are four columns to the form (see Appendix J for an example):

column one lists the data elements;
column two lists the information submitted to AFCARS by the State;
column three is for noting information found in the case record; and
column four is marked if the information does not match.

Procedure

An introductory briefing will take place before reviewing the cases.
The entire State and Federal team will review case files and will be divided into two groups. One group will review the foster care files and the other group will review the adoption files.
Once the team is finished reviewing the files, each team member shares with the group their significant findings. Once this process is completed, the Federal review team coordinator collects the forms.
A preliminary analysis is conducted by the Federal team on site and the results are shared with the State team during days three and four. Final analysis of these forms is completed during the post-site visit review phase.

3. Days Three and Four Foster Care and Adoption Population and Elements

a. Population and Data Element Review

During days three and four the Federal team meets with the program and technical staff that are responsible for oversight of the foster care and adoption services, policy development/implementation, and for implementing the technical requirements of AFCARS. Days three and four comprise the core components of an AFCARS Assessment Review. They build on the initial analysis conducted during the pre-site phase by the Federal team and the results of the test case entry and the case file review.

At this time, the Federal team asks several questions regarding the State's organizational structure, the population of children served, and other practice-related questions. These questions are asked throughout the day. At the same time, the technical documentation is reviewed and assessed on whether it is reflecting State and Federal policy, and State practice.

Appendices B and C contain a sample of the type of questions that the Federal team asks in regard to the foster care and adoption populations and the data elements.

The frequency report generated from the State's data for the report period under review, is assessed by the Federal and State staff to determine if the data are representative of the State's child welfare population. The States have their own copy of the AFCARS Frequency Utility and can use this report on an ongoing basis to evaluate their AFCARS data. Technical Bulletin #12 describes how to use the frequency report to evaluate the data.

When analyzing the program code, the types of issues that are identified include:

Are there any AFCARS elements or values that do not have information mapped/extracted to it from the State system?
Are there any relevant values in the State system that do not map to an AFCARS element?
Are there any elements or values where the State's definition does not appear to match the AFCARS definition?
Are there any State data collection policies or procedures that may cause some data not to be reported to AFCARS, or to be reported inaccurately (i.e., not reflecting the true circumstances of the case)?
Does the program code exclude a group of children served that must be reported to AFCARS? Or, is it including children that should not be in the AFCARS reporting population?

b. Document Review Findings

A preliminary draft of the findings is completed by the Federal review team on-site. Generally, the above review activities are completed by noon of the fourth day. The afternoon of the fourth day is left open in the event more time is needed for the population and data element review, or if time is needed for some other reason. Otherwise, this time is used by the Federal team to begin consolidating its findings.

At this time, the Federal team reviews all of the on-site findings for each component and assigns a rating factor for each general requirement and data element. The findings are documented in three tables, one for adoption, one for foster care, and one for the general requirements. The tables include notes, comments and significant findings of the AAR for each data element and general requirement.

In regard to errors found during the case file review, if there are ten percent or more of the cases having errors, this is determined to be a data quality issue and is noted in the preliminary findings. This may be the sole reason for an element receiving a "3." The State should also note that the preliminary findings are based on a cursory review of the case file review findings. Based on the final review, a rating factor of "4" may get adjusted to a "3," and the State will need to address improvements in data quality.

A preliminary draft of the tables are completed by the Federal review team onsite and shared with the State review team prior to the exit conference with the State agency administrators. The Federal and State teams review the findings in order to ensure agreement on the action items.

Depending on time, this meeting may be at the end of day four, or early on day five prior to the exit conference.

4. Day Five Exit Conference

Once the on-site review process is complete, an exit briefing is held with the State review team and management officials.

The purpose of the exit conference is to:

present specific findings resulting from the AAR process;
report the preliminary recommendations and/or proposed resolutions to outstanding issues; and
answer any questions that the State agency administrators might have as a result of the AAR.

The exit conference should include all staff members that were involved with the AAR during the week and senior management staff.

A one-page summary of the AAR findings is distributed during the exit briefing.

Back to Table of Contents

Continue to Chapter VI: Procedures for the Post-Site Visit Phase

Attachments:

ACYF-CB-IM-02-05

State Guide to an AFCARS Assessment Review HTML or PDF(332 KB)