Purpose of this document
This document provides a list of example program evaluation topic areas, questions, and data collection methods. However, this list is not meant to be exhaustive and is meant only to provide grantees with examples of the types of questions programs may want to ask in the context of program evaluation. Grantees may choose to use one or more of the example questions, but are by no means required or expected to use any of the questions. The primary purpose of this document is to help programs think about selecting an evaluation topic area and developing appropriate questions. 

Stakeholder involvement
Programs should always collaborate with stakeholders when developing program evaluation questions.  Part of the process of focusing an evaluation is to list out all the potential questions related to a particular program component or topic that stakeholders may want to answer. These questions then must be prioritized to take into account both the importance of the question to stakeholders, their practical utility (the extent to which they will provide information that can be acted on), and their feasibility (resources needed). 

Process vs. outcome questions
Evaluations may include a combination of process and outcome questions. Process questions focus on program operations and measuring how well we did the activities we set out to do.  Outcome questions focus on measuring the changes that occurred as a result of our program’s activities.  In general, an examination of a program area should include a combination of process and outcome questions. For new activities, examination of some outcomes may be premature, so a greater emphasis on process is appropriate. Whenever an outcome evaluation is undertaken, process questions should be asked as well. If this is not done, results of the outcome evaluation may not be interpretable. For example, if an activity is found not to be effective in terms of changing outcomes, lack of process evaluation will limit the ability to determine whether the intervention itself was not effective, or whether it was not effective because it was not implemented ‘ideally’ and/or as intended.

Data collection methods
Commonly utilized data collection methods include (but are not limited to): observations, interviews, surveys, review of program records, pre- and post-tests, chart reviews, and analysis of IIS data.  Methods must be adapted to evaluation questions with program resources and capacity in mind. Ideally, evaluation of an activity is planned prior to the activity’s implementation, and thus the data collection for the evaluation can be ‘built-in.’ For example: 1) special logs can be developed for an outreach activity so that staff can collect information to see if the targeted population attended the activity; 2) surveys are developed before an outreach activity and disseminated at the start and end of the activity to determine if the targeted population’s knowledge or attitudes changed. 

NEED MORE INFORMATION?

Please email IPE@cdc.gov  and your Project Officer with any related questions.

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic	
Examples of Potential  Evaluation Questions
	
Data Collection Methods

VFC compliance site visits 	PROCESS: Are VFC staff appropriately trained? 
	Role play or observation of staff during site visits, pre-post test of staff,  interviews of practice staff (to see to what extent they are receiving correct messages)	
	PROCESS: Are practice/provider VFC contacts the right people (i.e. in a position to implement change)?
	Interviews of practice staff 	
	PROCESS: Is training of staff done in a standardized manner across grantee’s jurisdiction? 
	Review of  training protocols for consistency with program/CDC guidelines and across protocols	
	PROCESS: Are there practice barriers to implementing recommendations? 
	Interviews of practice contacts	
	PROCESS: Are practices satisfied with VFC compliance site visits?
	Interviews or anonymous survey of practice contacts	
	OUTCOME:  Are VFC compliance site visits effective in improving vaccine storage and handling practices?      
	Follow-up visit a few months after initial visit to assess the degree to which recommendations have been implemented (e.g. if temperature was monitored only once daily, is it now monitored twice daily).	

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic
	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Enrollment of new adolescent providers to VFC (for example: planned parenthood, teen clinics, ob/gyn practices, juvenile detention centers)
	PROCESS: Percent of facilities enrolled, by type and local jurisdiction?
	Review of program data	
	PROCESS: Number of facilities that refused enrollment, by type and local jurisdiction; reasons for refusal.
	Review of program data	
	PROCESS: Is approach to enrolling providers standardized across grantee’s jurisdiction?
	Observation, review of materials	
	PROCESS: Are staff who enroll providers appropriately trained?
	Pre-post test, interviews of providers	
	OUTCOME: Is there an increase in adolescent providers enrolled in VFC?
	Review of program data	
	OUTCOME:  Has there been an increase over time in ordering vaccine on a regular basis?
	Vaccine ordering data	
	OUTCOME: Has there been an increase in vaccinations in these settings?
	Chart review, IIS	
	OUTCOME: What percent of providers submitted immunization data on adolescents in the past 6 months?	IIS	

	These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Outreach activities 
NOTE: outreach efforts can be conducted in a variety of ways; this list is a generic list of questions to consider and needs to be adapted to any specific effort.
	PROCESS: Where were outreach activities conducted, was the target population reached? 
	Review of activity logs, interviews of target population members  	
	PROCESS: What problems were encountered in reaching the target population? 
	Outreach worker interview
	
	PROCESS: If peer counselors were part of the outreach effort, how well were they trained, how many contacts did they make? 
	Observation, pre-post test
logs	
	PROCESS: If partners were engaged, what activities did they undertake? 
	 Review meeting or activity records	
	PROCESS: If special materials were developed, how are they received by the target population? 
	Conduct focus groups	
	PROCESS: Satisfaction of target population with outreach activities for those that were reached
	Interviews, surveys of target population members  	
	OUTCOME:  Did the outreach effort result in a change in awareness, knowledge or attitudes in the target population?
	Interviews, surveys of target population members  	
	OUTCOME: Did the outreach effort result in an increase in vaccination uptake in the target population?	Survey, or calculate based on doses administered, if size of target population is known (e.g. residents of housing complex) 	
	OUTCOME: Did the benefits of the activity justify allocation of resources?	Compare outcome to costs (e.g. program cost per person vaccinated)	
These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Provision of hepatitis B vaccine to settings like STD clinics or prisons
	PROCESS: Do settings have protocols for administration of Hep B vaccine?
	Survey of settings/facilities; Review of protocols and policies	
	PROCESS: Are staff knowledgeable about the protocol?
	Survey, interview	
	PROCESS: For patients who don’t accept vaccination, why not?
	Ask clinic to collect data at encounter	
	PROCESS: Do the settings have needed resources to vaccinate? What are barriers? 
	Interviews of staff, administration	
	PROCESS: What is level of administration and healthcare provider support for vaccination?
	Interviews of staff, administration	
	OUTCOME: Was there a change in percent of patients/clients who were 1. Assessed for vaccine eligibility; 2. Offered vaccine?
	Chart reviews	
	OUTCOME: Was there a change in percent of patients receiving 1,2 and 3 Hep B doses?	Chart review, vaccine inventory data	

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Immunization information systems
	PROCESS: For providers not utilizing IIS or not using it regularly, what are barriers and facilitators to use?
	Survey, interview of provider staff	
	PROCESS: Are staff who train providers in use of IIS appropriately trained?
	Pre-post test, observation of trainers	
	PROCESS: Are practice staff familiar with proper IIS use and features?
	Interview, survey of provider staff	
	PROCESS: What are the barriers to timely reporting?
	Interview, survey of provider staff	
	OUTCOME: Was there an increase in percent of children and of vaccinations entered in the IIS?
	Compare IIS to chart review or other data source	
	OUTCOME: Was there an increase in completeness of reports?
	Compare IIS to chart review or other data source	

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic
	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Assessment methods for school coverage
	PROCESS: Does sampling method used to estimate coverage rate in schools affect measured rates? For example, if private schools are excluded, does this affect the overall coverage rate?
	Compare coverage rates of public and private schools	
	PROCESS: Are coverage rates from a self-reported school census accurate?	Compare self-reported rates to audit

Topic
	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

New school mandate
	PROCESS: Did schools enforce the policy? 	School staff interviews, review of  policy/protocols
	
	PROCESS: How was the school vaccination coverage calculated?
	School staff interviews, review of  policy/protocols
	
	PROCESS: What kind of educational activities were undertaken by public health and by schools, and did messages get to parents? 
	Surveys, focus groups
	
	PROCESS: What were barriers to vaccination? 
	Surveys, focus groups	
	OUTCOME: Did this mandate result in increased vaccination in target group? 
	School coverage data	

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic
	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Promoting hepatitis B prevention in birthing hospitals	PROCESS: What are barriers and facilitators to hospitals implementing protocols / standing orders for hepatitis B birth dose or testing of mothers of unknown status?
	Interview, survey of hospital infection control lead and/or administrators	
	PROCESS: Are hospital contact persons the right ones, i.e. in a position to implement or promote implementation of change?
	Interview	
	OUTCOME: Was there change in knowledge /attitudes of hospital staff regarding perinatal hepatitis B?
	Survey	
	OUTCOME: Was there an increase in protocols /standing orders pertaining to preventing transmission of perinatal hepatitis B?
	Survey, interview	
	OUTCOME: Was there increased delivery of vaccination and prophylaxis in delivery hospitals?	Chart review	

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic
	
Examples of Potential  Evaluation Questions
	
Data Collection Methods	

Influenza vaccination clinics targeting populations with low vaccination rates
	PROCESS: What percent of clinic participants had never previously received influenza vaccination?
	Collect vaccination history information at intake	
•		PROCESS: What percent of target population was aware of clinic?
	Interview, survey 	
•		PROCESS: If target population can be defined and quantified, what were the reasons for not attending clinics?
	Interview, survey those that did not attend (if possible)	
•		OUTCOME: If target population can be defined and quantified, what proportion was vaccinated?
	Generally difficult, but estimating it may be feasible if size of targeted population is known (e.g. residents of a housing complex)	

These are sample topics, questions, and methods and these lists are NOT exhaustive. Grantees may choose to use one or more of these, but are by no means required to use them. Grantees who use them should feel free to adapt them to their specific needs.

Topic
	
Examples of Potential  Evaluation Questions 
	
Data Collection Methods	

School-located influenza vaccination
	PROCESS: What were the roles assigned to various SLV staff (e.g., school staff, health department staff, volunteers, other partners)?  Were these roles appropriate and acceptable? 
	Interviews with SLV staff, program records	
•		PROCESS: What are the barriers to SLV acceptance by key stakeholders (e.g., parents, school staff, health care providers)?
	Surveys, interviews, focus groups	
•		OUTCOME: Does participation in an SLV program lead to a change in acceptability of SLV among school staff?
	Pre-post test surveys, interviews, focus groups	
•		OUTCOME: How many and what percent of enrolled students were consented to be vaccinated?
	Consent records, school records	
•		OUTCOME: How many and what percent of consented students were vaccinated?
	Consent records, school records 	
	OUTCOME:  Did SLV impact (increase) coverage for school-aged children?
	IIS and program data, including vaccine order data	

EXAMPLES FROM GRANTEE EVALUATION PLANS

Perinatal Hepatitis 

1)	Topic - Post vaccination serology testing

•	To what degree did the number of health care professionals who are aware of the value of early detection and the criticality of post-vaccination serology testing (PVST) for infants born to HBsAg positive mothers increase from March 31, 2009 to December 2009?
•	To what degree did the number of health care professionals who have been educated on the value of early detection and the criticality of PVST for infants born to HBsAg positive mothers increase between March 31, 2009 and December 2009?
•	To what degree did the number of health care professionals have written standing orders for PVST for infants born to HBsAg positive mothers increase from January 31, 2009 to December 2009?
•	To what degree did the rate for PVST for infants born to HBsAg positive mothers increase from December 2006 (baseline collected in 2007) to December 2009 (collected in December 2010)?

2)	Topic - Identification of contacts

•	Does training of community partners on the web-based case management system improve the number of contacts identified and reported?
•	What are the obstacles to susceptible contacts obtaining hepatitis B vaccination?
•	Does the training increase the community partners’ knowledge base about perinatal hepatitis B follow up and contact investigation?

Education and Training

1. Topic - Provider education

•	To what extent are providers or provider sites satisfied with communication received from the immunization program?
•	To what extent does immunization program communication raise provider or provider site knowledge or awareness of immunization related topics?
•	To what extent has immunization program communication with providers improved since the adoption of a communication plan?

2. Topic - Distance learning

•	How many provider sites have at least one person who has completed one of the 2 DOH-developed web-based lessons (“Preparing Vaccines” and “Administering Vaccines”) and passed the post-test? 
•	How many provider sites have at least one person who has completed both web-based lessons (“Preparing Vaccines” and “Administering Vaccines”) and passed both post-tests? 
•	What feedback has been received from users that could indicate potential obstacles to accessing and completing the web-based lessons? 

Population Assessment - School vaccination assessment
•	Has the knowledge of using the online system increased among school health aides?
•	How many problems have been reported from school health aides who use the modified online system?
•	Has the number of school reports submitted online by registered school users increased?
•	Are the school assessment immunization data collected more efficiently and accurately?

IIS 
•	Will intervention increase the timeliness, completeness and accuracy of IIS data submitted by providers?
•	Will intervention increase the use of IIS tools (e.g. forecasting, reminder-recall, inactive status, etc) available to providers?

WIC
•	What proportion of WIC clients ages 0 to 24 months have their shot records assessed at recertification and midpoint exams?
•	Are WIC client shot records being accurately assessed by WIC staff, and are appropriate forms being submitted to the WIC linkage program on the client’s immunization status?
•	Are the trainings provided by WIC linkage staff to WIC staff effective?
•	What can WIC and Immunization program staff do to improve coordination and teamwork to support the WIC linkage program?

This document can be found on the CDC web site at: http://www.cdc.gov/vaccines/programs/progeval/downloads/prog-eval-topic-que.rtf