Sample Immunization Program Evaluation Plan
June 2008 

Grantee:__________________________
This form completed by: _____________________	Title:________________________
Evaluation Lead Name:______________________	Title:________________________
Evaluation Lead Phone #:___________________________
Evaluation Lead Email:_____________________________
Projected start date for implementing this evaluation plan:_______________________
Projected end date for this evaluation:_____________________________

A. Program Component(s) - Specify the component you will be evaluating.

___Adolescent vaccination	___Population Assessment
___Adult vaccination		_X_AFIX (Provider Quality Assurance)
___Education/training		___Surveillance
___IIS				___Vaccine management and accountability
___Perinatal hepatitis B        	___Women Infants and Children Program

___Other	If checked, please specify: ____________________________________________

B. Please BRIEFLY list which activity or activities within the above component you plan on evaluating. 

The feedback sessions of AFIX visits. (see Page 10, AFIX standards) 

C. Evaluation Goal(s) - State the goal(s) related to the component/activity you have chosen. Refer to Chapter 2, Page 11 of the Guide to Immunization Program Evaluation for help with this section. 

To improve the quality and effectiveness of AFIX feedback sessions.

This space intentionally left blank.

D. Stakeholders - Fill out the table below. Note: This table is a combination of Tables 1 and 2 in the Guide; therefore, it should include members of the evaluation team. Refer to Chapter 1, Pages 7-10 and Chapter 2, Page 12 of the Guide to Immunization Program Evaluation for help with this section. 
Tables 1 and 2: Stakeholder Assessment and Engagement

Group Name OR Person’s Name and job title:Program Manager
Interests in or perspectives about the evaluation:Ensure adequate stakeholder input; Complete evaluation plan on time; Ensure results of evaluation are useful and utilized to improve program performance
Roles in the evaluation:Evaluation team leader
Responsibilities in the evaluation:Oversee data collection and analysis;  Write evaluation plan; Utilize evaluation findings
How and when to engage:Involved in all activities throughout the evaluation 

Group Name OR Person’s Name and job title: AFIX Coordinator
Interests in or perspectives about the evaluation:Evaluation and its findings could impact routine activities; Interested in providing useful feedback for providers
Roles in the evaluation:Stakeholder liaison;  Data collection and analysis
Responsibilities in the evaluation:Coordinate stakeholder meetings; Provide input on evaluation design, methods, and interpretation and use of findings; Share findings with stakeholders; Utilize evaluation findings
How and when to engage:Meet regularly with PM to develop methods; Attend all SH meetings

Group Name OR Person’s Name and job title: AFIX field staff
Interests in or perspectives about the evaluation:Evaluation and its findings will directly impact day-to-day activities; Interested in providing useful feedback for providers
Roles in the evaluation:Provider liaisons;  Data collection
Responsibilities in the evaluation:Provide input on data collection methods; Data source and data collectors; Share findings with providers; Utilize evaluation findings
How and when to engage:Meet regularly with AFIX Coordinator; Attend initial SH meeting and subsequent ones as needed

Group Name OR Person’s Name and job title: Provider office staff (clinical and admin.)
Interests in or perspectives about the evaluation:Interested in receiving constructive feedback on ways to improve immunization delivery and providers; Interested in minimizing AFIX-related burden on staff time 
Roles in the evaluation:Represent providers’ perspectives on AFIX Feedback;  
Responsibilities in the evaluation:Provide feedback during evaluation planning; Data source; Utilize evaluation findings
How and when to engage:Invite to initial stakeholder meeting; Phone/Email communication during data collection phase; As needed, invite to other SH meetings to troubleshoot, share findings, etc.
(end of Table)

E. Describing the program component/activity.  Briefly describe the following aspects of the component/activity you are evaluating. Note: Please provide a more detailed description if the activity you are evaluating is new or not well established (e.g. activities related to adolescent vaccinations). Refer to Chapter 2, Pages 13-14 of the Guide to Immunization Program Evaluation for help with this section.

Background:
–	Program has 5 AFIX staff 
–	Staff get some instructions on AFIX feedback sessions at the time they are hired
–	None have received standardized training on AFIX Feedback standards
–	Currently, program does not track how many feedback sessions include use of written quality improvement (QI) plans (as described in the AFIX standards, Feedback, Level I – pg 10)
–	Additionally, we do not know if QI plans contain the minimum requirements 
–	Although AFIX staff work with providers to agree upon opportunities for improvement, we do not track in a standardized manner whether providers are implementing the agreed upon changes. 

Context:
–	Feedback sessions are the only opportunity to interact directly with provider staff (especially decision-makers)
–	Feedback visits are often combined with VFC visits and therefore, some of the items discussed for the AFIX portion might not get adequate time and attention
–	Want to use the visit to help providers improve immunization delivery and increase their immunization rates

Stage of Development:
We have been implementing AFIX visits (including Feedback sessions) for the past 8 years. The process is very well-established. 

Target population:
•	Clinical and administrative staff in private and public immunization provider offices
•	AFIX staff

Objectives:
•	By March 2009, 100% of AFIX field staff in the state will receive training on how to conduct feedback sessions that meet all AFIX Level I standards. 
•	By June 2009, all AFIX field staff will report an increase in knowledge about the components of a feedback session including which key items should be included in a quality improvement (QI) plan.
•	Between July and December 2009, during AFIX Feedback sessions, all AFIX field staff will develop QI plans that contain all key items. 
•	Starting in January 2010, all AFIX field staff will follow up with providers at least on a biannual basis to document their progress in implementing the improvement opportunities that were agreed upon at their last feedback session. 
•	Starting in January 2010, all providers will implement by the proposed date at least 50% of the improvement opportunities that were agreed upon at their last feedback session. 

F. Fill out the table below. You may choose to describe the entire component or the specific activity or activities within the component that you are evaluating. Note: This table should represent the current status of the component or activity NOT where you would like to be in the future. Refer to Chapter 2, Pages 15-17 of the Guide to Immunization Program Evaluation for help with this section.

Table 3: Program Component/Activity Description

Inputs:Funding
Activities:
Initial:Develop AFIX protocol
Subsequent:Review provider records & assess coverage level
Outputs: Protocols
Outcomes:
Short-/Mid-term: Increased provider knowledge
Long-term:Decreased in missed opportunities

Inputs:Staff
Activities:
Initial:Develop AFIX protocol
Subsequent:Review provider records & assess coverage level
Outputs: Trained staff
Outcomes:
Short-/Mid-term: Increased provider knowledge
Long-term:Decreased in missed opportunities

Inputs:Time
Activities:
Initial:Train AFIX staff
Subsequent:Provide feedback & recommend strategies for improvement
Outputs: Providers visited
Outcomes:
Short-/Mid-term:Increased provider motivation
Long-term:Decrease in invalid doses

Inputs:CoCASA
Activities:
Initial:Set up provider visits
Subsequent:Acknowledge and reward improved performance;
Document outcome of visit in database (CoCASA); 
Promote information exchange;
Outputs: Coverage assessment;
Feedback provided;
Rewards provided;
Data in database;
Information exchanged;

Outcomes:
Short-/Mid-term:Change in office practices and policies;
Change in physician behavior
Long-term:Increase in timely vaccine receipt for eligible population;
Increased immunization coverage;
Decreased VPD

(end table)

G.  Optional – If you have created a logic model in addition to Table 3, please submit your logic model as an attachment (instructions will be provided in PAPA).  Refer to Chapter 2, Page 17 of the Guide to Immunization Program Evaluation for help with this section.

H. Focusing the evaluation and Data Collection and Analysis - Fill out the table below. Note: The number of questions and indicators per question can vary. For each indicator, you need to develop a target and BRIEFLY list/describe the data sources and the method, timeline and person responsible for data collection and analysis. Refer to Chapter 2, Pages 18-29 of the Guide to Immunization Program Evaluation for help with this section. Table 8: Data Collection and Analysis

Evaluation Question #1: How many providers receive a quality improvement (QI) plan containing the 4 key items* (Level 1, AFIX Feedback Standards)?

INDICATOR(S): 1. Percent of providers that received a QI plan following an AFIX visit
TARGET(S):	1. 80% of providers received a QI plan following an AFIX visit
DATA SOURCE(S): 1. AFIX field staff; site visit notes
DATA COLLECTION:
1. Method:  Review site visit notes from 2007-2008; if not available or incomplete, survey AFIX field staff
Timeline: Dec 2008-Jan 2009 
Person Responsible:  AFIX coordinator and AFIX field staff
ANALYSIS: 1. Method:  Frequencies
Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator and AFIX field staff

INDICATOR(S): 2. Percent of QI plans that include all four key items
TARGET(S):	2. 100% of QI plans contain all four key items
DATA SOURCE(S): 2. QI Plans
DATA COLLECTION: 2. Method: Review a sample of QI plans from 2007-2008 developed by each AFIX field staff Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator
ANALYSIS: 2. Method: Frequencies Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator

				

Evaluation Question #2: Do AFIX staff know how to write a QI plan (including 4 key items)?

INDICATOR(S): 1. Percent of AFIX field staff trained on how to write a QI plan (including all four key items)
TARGET(S):	1. 100% of AFIX field staff received training on QI plans before conducting AFIX visits
DATA SOURCE(S): 1. Staff training records
DATA COLLECTION: 1. Method: Review staff training records Timeline: Dec 2008
Person Responsible: AFIX coordinator
ANALYSIS: 1. Method: Frequencies Timeline: Dec 2008
Person Responsible: AFIX coordinator

INDICATOR(S): 2. Percent of AFIX field staff who feel confident in their ability to write a QI plan
TARGET(S):	2. 100% of AFIX field staff feel confident in their ability to write a QI plan
DATA SOURCE(S): 2. AFIX field staff
DATA COLLECTION: 2. Method:  Survey AFIX field staff about barriers and facilitators of QI plan development
Timeline: Dec 2008-Jan 2009 
Person Responsible: AFIX coordinator
ANALYSIS: 2. Method: Frequencies; Qualitative content 
Timeline: Jan 2009
Person Responsible: AFIX coordinator

Evaluation Question #3: How many providers implement the interventions (AKA “recommendations”) that were agreed upon during their AFIX Feedback visit?

INDICATOR(S): 1. Percent of providers that implemented one or more recommendations (by the proposed date)
TARGET(S):	1. Providers implemented at least 50% of recommendations (by the proposed date)	
DATA SOURCE(S): 1. Providers; AFIX field staff;
Site visit notes and QI plans from 2007-2008
DATA COLLECTION: 1. Method:  Survey (a sample of) providers who received visits during 2007-2008 to identify which recommendations were implemented; survey (a sample of) providers and all AFIX field staff about barriers and facilitators of implementation of recommendations; Review site visit notes and QI plans from 2007-2008
Timeline: Dec 2008-Feb 2009 
Person Responsible:  AFIX coordinator and AFIX field staff	
ANALYSIS: 	1. Method:  Frequencies; Qualitative content
Timeline: Feb 2009 
Person Responsible: AFIX coordinator and AFIX field staff

INDICATOR(S): 2. Percent of feedback visits that were conducted with provider staff who have the authority to implement changes
TARGET(S):	2. 100% of AFIX feedback sessions are conducted with provider staff who have the authority to implement changes
DATA SOURCE(S): 2. Site visit notes from 2007-2008
DATA COLLECTION: 2. Method:  Review site visit notes and QI plans from 2007-2008; Timeline: Dec 2008-Jan 2009 
Person Responsible:  AFIX field staff

ANALYSIS: 2. Method:  Frequencies
Timeline: Jan 2009 
Person Responsible: AFIX field staff
(end table)

I. Ensuring use and sharing lessons learned - Fill out the table below. Note: We understand that this portion of your evaluation will occur in the distant future but please provide your best estimate. Refer to Chapter 2, Page 30-31 of the Guide to Immunization Program Evaluation for help with this section.

Table 9: Disseminating Findings 
TARGET PERSON OR GROUP’S NAME:Program manager
TARGET’S EVALUATION USE(S):Determine the current status of AFIX Feedback sessions; Utilize findings to improve training provided to field staff and maximize Feedback sessions to ensure more providers are implementing agreed upon recommendations
DISSEMINATION ITEMS AND METHODS:Provide a complete report of findings (electronic and hardcopy)
TIMELINE FOR DISSEMINATION:At the completion of the evaluation  (~  March 2009)
PERSON RESPONSIBLE:AFIX Coordinator

TARGET PERSON OR GROUP’S NAME: AFIX field staff
TARGET’S EVALUATION USE(S):Identify areas of improvement for AFIX Feedback sessions; Adjust methodology for recommendations made during feedback sessions.
DISSEMINATION ITEMS AND METHODS:Present a summary of evaluation findings during monthly staff meetings
TIMELINE FOR DISSEMINATION:At the completion of the evaluation (~ March 2009)
PERSON RESPONSIBLE:AFIX Coordinator

TARGET PERSON OR GROUP’S NAME:Immunization providers(All AFIX providers including data sources)
TARGET’S EVALUATION USE(S):Improve implementation of agreed upon interventions; Use as benchmark to see how they are performing compared to other providers with regard to implementing  recommendations

DISSEMINATION ITEMS AND METHODS:Provide a summary of relevant findings (electronic and hardcopy)
TIMELINE FOR DISSEMINATION:At the completion of the evaluation (~ March 2009)
PERSON RESPONSIBLE:AFIX Field staff 

TARGET PERSON OR GROUP’S NAME:NIC 2009 Conference
TARGET’S EVALUATION USE(S):AFIX Field staff 
DISSEMINATION ITEMS AND METHODS:Presentation
TIMELINE FOR DISSEMINATION:February 2009
PERSON RESPONSIBLE:Program Manager
 (end table)

END EVALUATION PLAN

This document can be found on the CDC website at:
http://www.cdc.gov/vaccines/programs/progeval/downloads/eval-afix-examples-2008.rtf