MODULE 9 – Quality Assurance

http://www.cdc.gov/vaccines/programs/vfc/default.htm
http://www.cdc.gov/vaccines/programs/afix/default.htm
http://www.cdc.gov/vaccines/programs/cocasa/default.htm

Introduction

The purpose of this module is to provide immunization programs at both the state and local level with information on the quality assurance requirements. Quality assurance (QA) involves review and evaluation of VFC provider practices and is a legal requirement of the VFC program. Quality assurance is implemented through site visits to VFC providers. Outcomes of provider site visits help program staff determine how well the VFC program is being implemented at the individual provider site level, and individual evaluations also assist with improving the overall VFC program at the state level.

Quality assurance takes place during two types of site visits:
•	VFC provider site visits
•	Assessment, Feedback, Incentives, and eXchange (AFIX) visits

VFC provider site visits help to determine if VFC vaccines are being distributed, handled, and administered in accordance with the laws and policies that govern the VFC program.

AFIX (Assessment, Feedback, Incentives, and eXchange) is a Continuous Quality Improvement strategy that consists of 1) assessment of the healthcare provider's vaccination coverage levels and immunization practices, 2) feedback of the results to the provider along with recommended strategies to improve coverage levels, 3) motivating the provider through incentives to improve vaccination coverage levels, and 4) exchanging healthcare information and resources necessary to facilitate improvement.

VFC Provider Site Visits

All immunization grantees are required to conduct site visits to VFC providers in both the public and private healthcare sectors. The purpose of the site visit is to review records of children who are immunized through the VFC program and to evaluate the provider's recordkeeping, vaccine handling and storage procedures and compliance with the requirements of the VFC program. Appendix 6 contains instructions on how to conduct VFC eligibility screening by selecting a random sample of children within a practice.
Persons conducting VFC provider site visits must complete all the questions in Section I of the VFC Site Visit Questionnaire. These are the required questions that must be asked to assess the provider's compliance with the VFC requirements. Questions identified with a red exclamation point (!) are considered high priority. Corrective actions must be developed if the provider is not in compliance with any of these questions. Section II of the questionnaire includes optional questions related to the Standards of Pediatric and Adolescent Immunization Practices. Section III of the questionnaire allows grantees to create custom questions specific to their programs (Section III is only available in the CoCASA software application).

The VFC site visit questionnaire is reviewed and updated annually and is available in the Comprehensive Clinic Assessment Software Application (CoCASA). Every grantee should develop a written protocol for provider site visits that outlines procedures for selecting provider sites for QA visits, conducting site visits—including instruction on how to administer the VFC Site Visit Questionnaire—and reporting results from completed site visits. All immunization grantees are required to conduct VFC provider site visits to enrolled VFC provider sites annually. Please refer to the annual grant guidance for specific information.

VFC Site Visit Objectives
The main VFC site visit objectives are
•	Reviewing of VFC eligibility screening procedures;
•	Verifying the information in the provider profile;
•	Administering the VFC Site Visit Questionnaire;
•	Monitoring VFC vaccine administration, storage and handling;
•	Ensuring VFC program policies are being properly implemented;
•	Providing feedback and, as necessary, requesting corrective actions and follow-up of identified problems.

The ABCs of Conducting Site Visits
Grantees must develop written policies on how to schedule, prepare, implement, and document both VFC and AFIX provider site visits. The policies should be as specific as possible. A good rule to follow is to provide enough detail in the policies so that a person with experience and the appropriate skills could conduct a site visit with minimal supervision or clarification of expectations. Grantees should have a flexible training plan to educate staff about why site visits are important and how to conduct site visits. The outline below can be used to assist grantees in developing procedures for separate VFC and AFIX site visits or, preferably, combined VFC/AFIX visits.
A. Training staff and required equipment
In order to achieve the objectives, staff responsible for conducting VFC and AFIX site visits must be able to schedule, conduct and appropriately document the site visits (following the grantee-developed site visit protocol). Scheduling site visits includes notifying the provider of the time and date of the visit and what working space and materials the visitor will need.

Staff conducting the site visits must have received training on and must understand the following concepts:
•	The purpose and importance of the visit;
•	How to correctly administer the VFC Site Visit Questionnaire (if applicable); 
•	Objectives of the visit (preplanning);
•	Communication skills required to effectively schedule, conduct, report feedback findings and document the content of the visit;
•	Documentation required after completing site visit;
•	Follow-up with provider.

B. Selecting provider sites
Grantees must develop a protocol for determining how VFC provider sites are selected for a visit. The following situations necessitate that a VFC site visit be conducted more frequently:
•	High-volume vaccine usage
•	Unusual ordering patterns
•	Healthcare providers who are outliers with respect to vaccine ordering
•	Reports of vaccine abuse or fraud
•	Medicaid billing inconsistencies
•	Unacceptable storage and handling practices

Additional provider sites should be visited according to a process of random selection. All staff should be made aware of the process used to select provider sites for QA visits and follow it accordingly. Grantees that conduct separate AFIX visits must develop protocols for determining how providers are selected for AFIX visits.

C. Scheduling site visits
A scheduling policy should include the following key points on how to schedule a site visit:
•	Identify a contact person in the office to discuss site visit requirements;
•	Arrange the date and time for the visit;
•	Confirm contact name, job title and phone number;
•	Confirm office address and location;
•	Discuss with the office manager how much time you estimate the visit will take and whom you need to talk to during the visit;
•	Request to have the following materials ready for your review on the day of the visit:
		-Charts (number and any criteria necessary to select charts)
	-VFC-related materials for VFC visits (such as VIS statements used, 	-   temperature logs)
•	Send a confirmation letter or fax to contact with date, time, materials needed and summary of visit process;
•	Confirm the visit with the provider 1 to 2 working days before the scheduled appointment.

D. Reviewing previous site visit information
Before making a site visit, staff should review all available information related to that provider site. Relevant documents to review may vary by grantee and type of visit but may include doses distributed and doses administered reports, enrollment data, provider profiles, past VFC site visit questionnaires and AFIX findings. A thorough review of these documents will help staff to be more aware of the past performance of the provider site as well as provide insight into questions that should be asked during the site visit. The provider profile should be carefully examined before every site visit; staff should be prepared to discuss necessary updates to the profile. In addition, staff should be prepared to review the VFC eligibility screening procedure with the provider.
Staff should also have on hand a selection of VFC-related forms and educational brochures available to distribute to provider site staff as needed.
E. Site Visit Questionnaire
Staff conducting VFC site visits are required to ask all questions in Section I of the VFC Site Visit Questionnaire developed by CDC. The questions contained in this questionnaire are essential to evaluating a provider's compliance with the requirements of the VFC program. All grantees must include CDC's required questions (Section I) as part of their VFC provider site visit data collection instruments. Some questions are designated as "high priority" and are identified by a red exclamation point (!). If answered inappropriately, the high-priority questions require a written corrective action. The inappropriate answer must be discussed with the provider and a corrective action agreed upon. Some inappropriate findings on the high-priority questions require that corrective actions be implemented immediately upon identification. Findings that place the viability of vaccine at risk, such as no working thermometers or no temperature log on the storage units, would need immediate attention.
Additional optional and custom questions can also be included in the VFC Site Visit Questionnaire at the discretion of the grantee. If the program decides to include optional and/or custom questions, these questions must be used consistently at all VFC site visits. Without consistent data collection, data cannot be used for meaningful analysis. All grantees are sent an electronic version of the VFC questionnaire in Microsoft Word annually before the start of the new calendar year.
Please Note: For programs using the CoCASA software, the VFC Site Visit Questionnaire is programmed into the software. Staff can enter responses to questions electronically and print out a copy of the completed questionnaire to share with the provider. Questions designated as "high-priority" are marked with a red exclamation point (!) icon. If noncompliant responses are given for any of these high-priority questions, the person entering the questionnaire will be required to enter a corrective action for each noncompliant response given. This function is programmed into the software to ensure that the areas of highest importance are being appropriately addressed. Staff members are not limited to entering corrective actions for these high-priority questions; they can enter recommendations for any issue identified during the site visit. Further information on how to use CoCASA and the current version of CoCASA can be found at http://www.cdc.gov/vaccines/programs/cocasa/default.htm .
F. Preparing for the site visit
The site visit policy should provide the staff with information on what equipment and resources to bring with them to the site visit. Some essential items to bring include:
•	Laptop computer for staff that use CoCASA to enter VFC Site Visit Questionnaire responses
•	Certified thermometer to check temperature of storage units (required for completion of VFC site visit questionnaire)
•	Previous reports as applicable (i.e., previous VFC questionnaire results, provider profile or vaccine accountability reports)
•	Blank forms as necessary (provider profile, VFC questionnaire)
•	Handouts/resources, such as
	-Immunization brochures and other educational materials
	-Latest list of vaccines available through the VFC program
	-VFC Eligibility Screening Form
	-Standards for Pediatric Immunization Practice
		-Vaccine Information Statements (VIS) and instructions for their use
		    (Make sure the VISs are current.)
		-Monthly temperature log for refrigerator and freezer recordings
		-Stickers for refrigerators/freezers/electrical outlets
	-VFC labels for labeling public vaccine
	-Refrigerator plug locks and signs
	-State Children's Health Insurance Program (SCHIP) information
	-Other information about upcoming CDC satellite courses, copies of any ---         recent mailings from the grantee to providers about VFC or specific -------      vaccine-preventable diseases
	
G. Conducting the site visit
The main focus of the site visit will depend on the type of site visit being conducted. The VFC Site Visit Questionnaire guides the reviewer through a VFC provider site visit with the primary focus on adherence to VFC program requirements and correct vaccine storage and handling practices. The AFIX process requires the reviewer to identify strengths and opportunities for improvement related to administration of appropriate vaccines to eligible patients according to the recommended schedule. In a VFC/AFIX combined visit, the reviewer must focus on the entire immunization process from how the vaccine is handled and stored to ensuring that the appropriate vaccine is given to an eligible patient on schedule. Each grantee must provide written policies on how to conduct site visits, but these policies and procedures will vary depending on the type of site visit. Training staff on how to conduct site visits is an essential component of provider quality improvement activities. Certain interpersonal skills and behaviors can be helpful in making the site visit a success, and these skills should be emphasized:
•	Be organized;
	-Know where to go, what time to arrive, and whom to talk to
	-Have identification and provide business cards 
•	Expect the unexpected and try to be flexible in addressing unexpected situations;
•	Be a good observer and listener; use observations to back up findings of office's strengths and opportunities for improvement; 
•	Be an immunization resource and a partner to the provider and office staff.

H. Reviewing the findings
Upon completion of the site visit, staff should discuss the outcomes in a face-to-face meeting with appropriate staff, either at the conclusion of the site visit or soon thereafter. This discussion should include a review of the visit findings and should address any recommended corrective actions for the provider site. A follow-up plan for addressing any issues of noncompliance or opportunities for improvement should be agreed upon between the immunization program staff and the provider site staff and should be documented in writing for the both office staff and immunization program. All details of the follow-up plan should be documented electronically. Please refer to the AFIX Standards for specific content requirements regarding the AFIX feedback session. (See AFIX section below.)

I. Analysis of provider site visits
In addition to sharing site visit results with the provider, the VFC/AFIX coordinator and/or immunization program manager should regularly review summary data from completed site visits. Reviewing summary data will help to track staff activities as well as identify any trends across multiple provider sites. Identified issues should be carefully reviewed with staff, and follow-up plans should be made to address staff and/or provider needs.
Please Note: Use of CoCASA will allow users to print out past VFC site visit results (at the provider level), which would be a valuable tool for identifying  potential problems within that provider's office. Other reporting  options are available through CoCASA and could be used to assist with the review of data at the program level.

Combined VFC/AFIX visits
Every effort should be made to combine the VFC site visit with an AFIX visit to reduce disruption to provider offices.

Assessment, Feedback, Incentives and eXchange (AFIX)

One of the most effective strategies for improving immunization coverage levels and standards of practice at the provider level is use of the continuous quality improvement process known as AFIX. As described in the introduction, AFIX stands for Assessment of immunization levels, Feedback of immunization information to key staff, Incentives to motivate and/or recognize outstanding performance, and eXchange of information on best practices to improve immunization coverage levels.
The "A" or Assessment component of AFIX assists providers in identifying opportunities for improvement. Once an opportunity is identified, an improvement goal is created. A plan is then developed, implemented and evaluated to determine if the goal was achieved. This process continues with revisions to the plan until the goal is met. The "F-I-X" components of AFIX are used during this process to develop, implement and evaluate the continuous quality improvement process.

The goal of AFIX is to ensure that viable vaccines reach all children served by the provider site in accordance with the ACIP schedule. A successful AFIX process requires implementation of all four components and repeated contacts with to a provider site. It requires that the immunization staff and the provider site become partners in improving immunization coverage levels. CDC encourages grantees to develop programs that merge the content of both VFC and AFIX programs into a single site visit. By combining visits, grantee resources can be used more efficiently to reach VFC providers, particularly in the private sector where the majority of vaccinations are delivered.

CoCASA, the software tool described above, may be used to conduct the assessment component of AFIX. The assessment component is the foundation of the AFIX program. The other components build on the findings from the assessment. Grantees must be mindful that all the components of AFIX interact and must be included when working with both public and private providers.

The benefits of AFIX have been demonstrated in public clinic settings and private provider offices. CDC offers technical assistance and documentation to assist grantees with implementing their VFC and AFIX site visits, including guidance for meeting all the grant requirements for AFIX and VFC programs. The information described above is available on the AFIX and VFC websites located at http://www.cdc.gov/vaccines/programs/afix/default.htm and http://www.cdc.gov/vaccines/programs/vfc/default.htm, respectively. 

The following sections describe elements of a successful AFIX quality improvement program.

AFIX Standards
Funding for implementing AFIX activities in VFC provider sites is made available to grantees through the immunization grant. To assist grantee program staff in implementing, managing, and evaluating an AFIX program, AFIX Standards have been developed. These standards contain the essential elements of all AFIX programs and are organized into three levels, each one building on the components of the previous level. The AFIX standards are specific enough so grantees can design their programs to fulfill the CDC grant requirements, yet they offer flexibility so grantees can individualize their programs for the specific conditions in their area. 

Standards for a Level I program focus primarily on the development and implementation of written protocols and procedures and represent the basic components of the grant requirements. A Level II program builds upon Level I written protocols and procedures. Level II standards focus on improving protocols and procedures and increasing activities and objectives. A Level III program builds upon Levels I and II and focuses on developing innovative strategies for improving the AFIX process.

The key concepts for AFIX Standard Level I are summarized below. This summary does not include all Level I Standards, only the major concepts from each of the six components contained in each level. Grantees should be actively working toward achieving all Level I Standards by developing written policies and procedures. The complete AFIX Standards document is available on the AFIX website at http://www.cdc.gov/vaccines/programs/afix/stds-guide.htm#what

Level I Program Operations Component
•	Clearly defined measurable short- and long-term objectives and methods to evaluate achievement of each objective;
•	Written protocol and procedures for daily program activities, including
		-Provider selection
		-Staff development (written job descriptions, orientation process for new 			   staff, ongoing staff training, supervision, monitoring and evaluation of 			   job performance) 
•	Explore possible collaborations with external organizations.

Level I Assessment Component
•	Written assessment parameters
		-Assessment methodology
		-Age range of sample
		-Number of records to assess 
		-Active patient definition
		-Immunizations to assess
		-Demographic fields to be collected
		-Moved or gone elsewhere (MOGE) definitions
•	Method for generating the sample
•	Method for monitoring and evaluating performance of staff conducting assessment component of the AFIX process

Level I Feedback Component
•	Coordination of feedback session
		-Timing (on day of assessment or within 2 weeks of assessment)
		-Logistics (face-to-face discussion)
		-Participants (what staff are required to attend the feedback session)

•	Content of session 
		-Prioritize opportunities for improvement and discuss two
		-Identify office's strengths related to immunization delivery
•	Discussion and development of quality improvement plan
		-Agree upon follow-up plan with provider office
•	Method to evaluate feedback session, including joint visits with supervisor

Level I Incentives Component
•	Development of guidelines and options to include two informal incentives during the feedback session
•	Development of formal incentive program for providers with improved or sustained high immunization coverage
•	Documentation of process for implementing formal incentives
	
Level I eXchange Component
•	List of specific information to exchange with provider/staff during feedback session and how to use the information to improve immunization coverage levels;
•	Promoting VFC and AFIX participation through health professional meetings or conferences.

Level I Program Evaluation Component
•	Utilize an electronic database to monitor site visit activities. At a minimum, the database must be able to generate the summary information required in the annual VFC Management Survey.
•	Develop written protocol for the utilization of the database.
•	Submit VFC Management Survey in appropriate format by the due date.
•	Develop and implement a procedure for conducting a process evaluation.

Training Plan
A successful quality improvement program requires certain technical and interpersonal skills. Grantees should develop a training plan for AFIX/VFC staff that provides the following:

•	New employee orientation, on-the-job training, and formal training on conducting AFIX visits. This activity is a Level I Program Operations Standard. CDC no longer provides grantee-based training on AFIX or CoCASA. CDC staff are available for technical assistance in developing the training curriculum and training plan for grantee staff to implement, and CoCASA has a training module located under the "Help" function on the main toolbar of the software.

•	Periodic staff development meetings that include updates and refresher courses for experienced staff on the latest developments.

•	Development of state or local training resources, particularly a "CoCASA expert" to provide assistance to field staff implementing AFIX using CoCASA. The CoCASA expert will be the liaison to CDC if he/she is unable to resolve problems experienced by staff.

The Core Elements for AFIX Training and Implementation are available on the AFIX website at http://www.cdc.gov/vaccines/programs/afix/downloads/coreelements.pdf. This document can be a helpful resource when developing training materials at the grantee level.

Partnerships and Collaboration
Grantees should have a plan to involve state and local organizations to help promote, educate, and collaborate with their constituencies on VFC and AFIX programs. Please refer to Module 5 of this manual for more information on how to market and develop collaborations around the VFC and AFIX programs. Some potential partners or collaborators are listed below:

•	Local chapters of the American Academy of Pediatrics (AAP), American Academy of Family Physicians (AAFP), American Nurses Association (ANA), National Medical Association (NMA), medical societies, or similar organizations (these organizations can help market the program through meetings and in newsletters and bulletins)
•	Health insurance organizations such as Medicaid and commercial insurers in the grantee's region
•	Academic settings for allied health professionals, including but not limited to:
		-Colleges of nursing (registered nurses and advanced practice nurses)
		-Technical or vocational nursing schools (licensed practical nurses or 			   licensed vocational nurses (LVNs)
		-Medical assistant schools

Successful partnerships and collaborations can be helpful in implementing the "I" and the "X" components of AFIX by having partners publicize successful strategies and information about AFIX and VFC to their target audiences. It may be possible to work with partners to recognize and reward high achievers or outstanding accomplishments through sponsorship of an annual award ceremony. Partners can facilitate exchanges of information on AFIX and VFC at local, regional and state levels. 

Reporting Requirements
Grantees should have written plans and procedures for meeting CDC's annual reporting requirements. Detailed summary reports of provider site visits are submitted in the VFC Management Survey and should include findings from VFC-only visits, AFIX-only visits, and combined VFC/AFIX visits. (Site visit definitions are in Appendix 6.) 

Responses to the VFC Site Visit Questionnaire are reported in aggregate form. The AFIX outcome measure, immunization coverage levels, is also reported as the number of providers with coverage levels within a specific percentile. For assessment visits that employ the hybrid method, the number of providers meeting the predetermined threshold should be reported. In addition, information regarding repeat assessment visits and outcomes should be documented. The VFC Site Visit Questionnaire and the VFC and AFIX reporting templates are located in the CoCASA program.

If the VFC/AFIX Evaluation module of CoCASA is used, the software will generate the VFC/AFIX activities information to be documented in the VFC Program Management Survey. 

The VFC Activity section of the VFC Management Survey asks the user to record providers' answers to a subset of questions from the VFC Site Visit Questionnaire administered during every VFC site visit. If information is entered into the VFC/AFIX module of CoCASA, the software will produce a summary report that will aggregate the responses to the selected questions that grantees are required to report to CDC. 

The VFC/AFIX Evaluation module of CoCASA is a helpful tool for organizing the VFC/AFIX site visit information and producing aggregate results. Use of this software is not required; however, if this software is not used, grantees must create their own system for collecting the data elements required to complete the VFC Management Survey.