Home > Federal & State Reporting Systems > SACWIS Assessment Review Process > SACWIS Assessment Review Guide

2 SACWIS Assessment Review Process

2.1 Overview of the SACWIS Assessment Review Process

The process of conducting an on-site SACWIS Assessment Review (SAR) involves four phases, illustrated in Exhibit 2-1 below. These reviews will consist of:

• a Review Initiation Phase;

• an Assessment Review Planning Phase;

• an On-Site Review Phase; and

• a Post-Review Phase.

Depending on resource availability, reviews may be conducted off-site instead of on-site. With the obvious exception of the site visit, the process is essentially the same. For an off-site review, the depth of the Document Review Phase will be expanded and supplemented with telephone interviews with appropriate State staff.

Back to Table of Contents

2.2 Review Initiation Phase

The Review Initiation Phase involves two steps:

2.2.1 Request Review and Submit Documentation

SARs should take place after the SACWIS system becomes operational (see section 1.3 for guidance on requesting a SAR before completing statewide implementation). These reviews are to be initiated approximately six months after the approved project completion date. If the State is initiating the review process, it should submit a written request to the:

Associate Commissioner Children's Bureau
Attention - Director, Division of State Systems
Administration for Children and Families
Department of Health and Human Services
330 "C" Street, S.W.
Washington, D.C. 20201

The written request must provide assurances that all operational components of the SACWIS system are used (i.e., that staff are not using another information system or paper processes to perform the work of a SACWIS component), operates uniformly statewide (or in the converted jurisdictions if the Assessment is being conducted prior to statewide implementation) and meets all mandatory requirements.

The request should be accompanied by the following documentation in hard and, if available, soft copy (please contact ACF if these documents are not already available — States should not create these documents for the sole purpose of this review):

• System diagrams (both the technical architecture and elementary processes performed by the system include functional decomposition, data flow, and hardware configuration diagrams if available);

• An Organizational Chart (inclusive of all SACWIS personnel);

• Current user manual (for both technical and end users);

• Training manual and materials;

• Data element dictionary (with table definitions, data element data types, mandatory vs. optional elements, and table and key constraints)

• Numbered list of system screen prints;

• Numbered list of system alerts/ticklers (with a brief description of the alert/tickler including how the tickler is invoked, resolved and a general description of the tickler process);

• Numbered list of system notices(with a brief description of the notice including how the notice is invoked, resolved, and a general description of the notification process);

• Numbered list of reports (including management and financial reports and reference to whether reports are on-line, available in hardcopy, and generated in real-time or by batch);

• Planned schedule for submitting NCANDS Detailed Case Data Component (DCDC) data (If State's Child Abuse & Neglect system is included within the SACWIS);

• Copies of SACWIS reports used to complete the Federal IV-E 1 report(formerly IV-E 12); and

• Other cross-reference material the State may wish to provide.

The Guide must be submitted to ACF in an electronic format. If possible, ACF prefers to receive the files in MS Word. If that format is not available, the files should be submitted as an electronic rich text format file (*.rtf).

In addition, if the State has received approval to link a "SACWIS type system¹ existing in a large urban area" to the new SACWIS system, the "existing SACWIS type" system must be identified, the linkages described, and the following assurances made.

• The staff using either SACWIS type systems (existing or new) must be able to easily check all SACWIS related systems for prior incidents and other available information.

• Either system is able to generate management alerts if an individual is active in more than one of the SACWIS type systems.

A copy of all documents should be sent to the ACF Regional Administrator.

SARs may or may not be scheduled for the same time as AFCARS Reviews. Beyond the condition that the system must be operational statewide, scheduling depends on ACF and State staff time and resources. For additional information on AFCARS reviews, see the "State Guide to an AFCARS Review. This can be accessed from the Children's Bureau's web page.

If ACF decides to initiate a formal SAR, the State will be notified in writing of the need to conduct the review and the reasons for this decision. The failure to cooperate with an ACF-initiated SAR could result in suspension of the project's approval and the disallowance of applicable title IV-E funds.

2.2.2 Federal Response

The Federal review team and leader will be selected. The SAR team is composed of Federal (Regional and Central Office) staff that have technical and program expertise, as well as a general knowledge of the State under review.

The Federal and State teams will work together to schedule the review.

Back to Table of Contents

2.3 Assessment Review Planning Phase

The Assessment Review Planning Phase involves the following three steps:

2.3.1 Review Documents

The Federal team members will analyze the documents submitted by the State. Given schedule and resource availability, States should plan to submit the review documents well in advance of the review to ensure that all parties have sufficient time to either prepare or review the SAR documentation.

The preliminary evaluation of the State documentation is intended to prepare the Federal team, expedite the review and evaluation process, determine if there are any impediments to the review, identify those areas of the system that require additional evaluation, and document early findings. This will maximize the effectiveness of the on-site review.

The Federal staff will examine the State's APD and contract documents to establish the scope of the project.

2.3.2 Plan Review and Draft Agenda

The Federal team, in consultation with the State, will plan review activities. The State team will be responsible for developing an agenda based upon the agreed upon activities. Activities will include all proposed county and local office visits, computer center visits, and interviews/demonstrations to review all functional areas. These Federal/State discussions will focus on: who the team must interview; where the review will take place (e.g., on- or off-site); and reach agreement on the tasks, responsibilities, milestones, deliverables, and schedules. The purpose of this phase is to clarify review expectations, answer State questions, and advise the State of preliminary findings resulting from the document review.

Appendix E contains a sample on-site SAR agenda. The schedule lists days, time estimates, activities, and recommended State participants. The State will identify the participants for each activity and schedule interviews with key State, county, and local officials before the visit. Interviews with local office staff (e.g., social worker and supervisors) should not be scheduled in advance. These interviews will be conducted at workers' desks, as they are available.

2.3.3 Finalize Arrangements with State

After the agenda is defined, the Federal/State teams will conduct at least one conference call prior to the on-site review to finalize schedules, clarify expectations, organize the interviews, confirm sites for the local office visits, and the availability of key State staff. At the discretion of the Federal team leader, the State may be asked to provide meeting space, if available, for team members during the on-site review.

Once the details are finalized, the Federal team leader will conduct a conference call that will:

• provide information on how the review will be conducted, addressing entrance and exit conferences, names of Federal team members, scheduling, etc.;

• discuss the on-site agenda, including the local sites to be visited; and

• identify the review techniques that will be employed, such as interviews, examination of outputs, case sampling, review of general production outputs, viewing screens and operations, collateral verifications, etc.

Appendix A of ACF's Information Systems Review Guide (see resource list in Chapter 1 of this document) contains a generic "Sample Entrance Letter to State" on page A-5.

The State should disseminate information on the nature and scope of the review to prepare program and data-processing staff that will either participate in, or be interviewed during the on-site review.

Back to Table of Contents

2.4 On-site Review Phase

The on-site review phase is usually conducted over a four to five-day period and includes the following:

2.4.1 Entrance Conference and Detailed System Demonstration

The first day of the on-site visit begins with the entrance conference which serves to:

• introduce the Federal team to the State management and project staff;

• establish contacts for consultation later in the on-site review;

• advise the State of the nature and scope of the on-site review; and

• respond to any questions the State staff might have related to the review process.

Immediately following the entrance conference, the State presents a detailed functional walk-through of the system and an overview of actual and projected program improvements resulting from the system implementation. The demonstration focuses on how the system meets all of the SACWIS functional requirements and references the State's responses in the completed Guide. The State should ensure that appropriate State technical and programmatic staff participate in the demonstration. We would expect that State staff would lead the actual walk-through of the system, although contractor staff can be available during the walk through.

2.4.2 Local and State Office Visits and Interviews

The purpose of the local and State office visits and interviews is to confirm that the system functions in the field as it was described in the State's APD, Guide responses, and functional walk-through. In that light, it is the Federal review team's intent to interview at least one local or State office staff member in each of the different functional positions represented in those offices. Site reviews should include visits to at least two counties or local offices of varying size and complexity as well as State offices. In order to complete all the tasks identified on the agenda, the Federal team will divide into smaller groups.

The Federal team will use a variety of techniques to assess the functional conformance of the operational system. The team will:

• interview office managers and staff (including staff involved in referral/intake, investigation, assessment, resource development, out-of-home placement, family preservation, adoption, claims, contracts, etc.) about how they use the system, what works well and what needs to be improved;

• ask the office staff to share their "lessons learned";

• observe actual case information being entered into the system; and

• ask follow-up questions based on the information gathered during the review of the State documentation and on-site functional walk-through.

While State project staff should accompany the Federal team to the office site visits, they should not participate in the actual interviews. The State project staff person that accompanies the Federal team to the local and State offices should be available to answer questions that may arise during the interviews. The importance of allowing field office personnel to share their experiences will be re-confirmed at the entrance conference.

The Federal team will work with the State to identify the specific offices to be visited (no fewer than two and no more than ten). Each local office visit should include at least one hour of observing workers entering case data into the system. The Federal team will identify the types of system users that they would like to interview.

The local office visit will allow the Federal team to talk to the SACWIS system users about how the system supports the flow of work. These discussions cover the SACWIS functions supported by the system, but less formally than was done in the Guide and the supporting documentation.

2.4.3 Document Review Findings

a. Daily Meetings.

As necessary, the full Federal team will reconvene at the end of each day to review the day's findings, summarize results, complete documentation, and plan the next day's schedule. The purpose of these daily meetings is to:

• Prepare a one to two page overview of the initial findings to present to the State;

• Discuss issues; and

• Refine or refocus the on-site process because of the day's findings.

If part of the Federal team cannot return to the central site for the daily team conference, the meeting should take place by teleconference. The full team should be present the night before an exit conference to finalize findings.

b. Use of the Guide.

The State's completed Guide is used during the on-site inspection to direct the review process. ACF will not change a State response, which the State can update later. Federal team members may record notes on their own copy of the Guide.

The Federal team leader ensures that appropriate documentation is collected and recorded in the Guide. The Guide provides the primary documentation of the review and its findings. All supporting documentation collected and prepared during the preliminary evaluation, on-site inspection, and final evaluation is included in the permanent State SACWIS review file, which is retained by ACF.

The Federal team records the results of the functional review on the ACF portion of the Guide. For each mandatory functional requirement and each funded optional requirement, the team notes whether the system conforms, conditionally conforms, or does not conform. If the system has the required functionality, no further supporting documentation is required. However, if the system does not have full functionality (conditionally conforms) or lacks functionality (does not conform), the team will note the basis for the finding and record in the "comment" section of the report any actions the State must take or information the State must provide. A draft report will be provided to the State after the appropriate ACF managers have reviewed it.

The completed Guide will be initially issued to the State as a draft report. While it is anticipated that the State's responses and ACF's comments in the Guide will evolve after the draft report is issued, the final version will reflect the identified issues, planned or completed actions, and the corresponding resolution dates.

c. Finding Summary Worksheet

The Finding Summary Worksheet (see Exhibit 2-1 on page 2-1) may be completed if there are significant deficiencies that must be corrected in order for the system to meet the requirements established in the SACWIS AT or the State's approved APD. Use of the Finding Summary Worksheet will only be necessary when State and Federal staff cannot reach agreement on a specific finding. The Finding Summary Worksheet will be prepared after the on-site review is completed and the draft report has been given to the State.

The Finding Summary Worksheet is founded on the performance-based review methodology commonly used by evaluators for the examination of facts. The methodology requires that the following information be obtained and evaluated to determine whether the finding is valid and corrective action is needed:

• Criteria: What should be;

• Condition: What is;

• Gap: Difference between "what should be" and "what is;"

• Effect: Significance;

• Cause: Why; and

• Recommendations: How to make the condition equal the criteria.

2.4.4 Conduct Exit Conference

Before the Exit Conference, the Federal team will summarizes its findings, consolidate the documentation, notify Central and Regional Office management of the preliminary review results, and prepare for the last day of meetings. The purpose of the exit conference is to:

• present specific initial, but not all, findings resulting from the review process;

• report the preliminary recommendations and/or proposed resolutions to outstanding issues; and

• answer any questions that the State might have as a result of the review.

Back to Table of Contents

2.5 Post-Review Phase

The Post-Review Phase involves three steps:

2.5.1 Finalize Findings and Conclusions

Although the major portion of the evaluation of the SACWIS system will be conducted on-site, a number of steps in the assessment and documentation process may be completed off site. These steps include:

• consolidation of findings from all team members;

• final consultation among Federal team members;

• organization of the review record;

• recording ACF's findings, requirements, recommendations, and/or observations in its portion of the Guide; and

• completing any collateral verifications from other sources such as State plans, NCANDS and AFCARS reports/reviews, State Cost Allocation plans, etc.

All documentation or verifications from collateral sources are retained by ACF as a part of the archive file on the State's SACWIS project.

2.5.2 Prepare Final Report

A SAR report will include:

• an overview of the SACWIS system;

• a summary of the functional components of this system that work particularly well;

• a summary of the deficiencies and recommendations for improvements; and

• the completed Guide.

Once the Guide (and any Finding Summary Worksheets) is complete, the Federal team will draft a brief overview of the SACWIS system. The overview will describe the findings and identify any actions the State needs to complete. The overview will be included in the cover letter sent to the State with the completed Guide.

The completion of the draft SAR report is based on available resources, however, ACF will attempt to have the report drafted and approved within 120 days after the on-site review ends. The reports are unlike the "typical" audit or management review report which includes separate sections on background, purpose and scope of the review, review methodology and so forth. Instead, this Guide establishes background, purpose, scope, and methodology for all SACWIS reviews.

The purpose of the report is to support a decision by ACF that the SACWIS system has met the tests of efficient, effective, and economical operation and qualifies for the SACWIS enhanced funding and cost allocation methodology.

All Federal team members will contribute to the final draft report. The draft report requires review and approval prior to issuance by the:

Associate Commissioner, Children's Bureau

Director, Division of State Systems.

Once approved, the draft report will be sent to the State. The State has an additional 160 days to respond to the draft report. The process will continue until all issues have been resolved and/or corrective action plans approved. The final report will be issued within 120 days of receiving the State comments and ACF approval of the response.

Although the final report is prepared primarily for ACF, the State under review, and other States for transfer evaluation, it is public information and will be prepared in such a manner that it may be shared freely.

2.5.3 Post-Report Activities

As noted at 45 CFR Part 95.605, cost-benefit and other program-performance improvements must be reported through the Annual APD Update until the Department determines that the projected benefits have been achieved. More rigorous follow-up is required for systems that do not meet SACWIS functional requirements. States will be required to report the results of their approved action plan(s) as part of their Annual APD.

ACF may also take the following steps:

• For functionally deficient systems, ACF may request an APD Update that presents a workplan for any additional development necessary to meet the required functionality. In addition, periodic monitoring reports may be required if development is extended.

• To assist the State in eliminating reported deficiencies, ACF staff may regularly contact the State about its progress towards achieving compliance in each of the deficient functional components. If corrective actions are not undertaken in a timely way, ACF may limit funding or recoup previously approved funds.

• ACF may require revisions to the State's past claims for FFP or the SACWIS cost allocation methodology to reflect benefits other programs derived from the system's use, or to adjust funding for functionality that exceeds the SACWIS requirements, but was never approved by ACF in an APD.

Before initiating a negative financial action against a State, ACF will work diligently with the State to come to a mutual agreement on the findings. If necessary, ACF will work with the State on action plan(s) to resolve any outstanding issues. It is ACF's desire to support innovative State solutions and program successes that ensure that program needs are met.

¹ See ACF Action Transmittal number ACF-OISM-001 for discussion of alternative approaches to a single statewide system. back

Back to Table of Contents