Slide 1
Process evaluation

Pascale Wortley, MD, MPH
Immunization Services Division, NCIRD
April 17, 2008

(CDC Logo with bird)
 Safer – Healthier – People
Department of Health and Human Services
(CDC logo)

Slide 2
Objectives
•	To demonstrate that process evaluation is an integral part of evaluation
•	To show how process evaluation goes beyond measuring outputs

Slide 3
Definitions
•	Process evaluation: examines whether program activities been implemented as intended

•	Outcome evaluation: measures program effects in the target population

Slide 4
In what situations can process 
evaluation be useful?

•	When expected outcomes are not observed, process evaluation can suggest reasons.

•	When outcomes are positive, process evaluation may help understand which components are especially important.

•	When an activity is newly implemented and it is too early to expect changes in outcomes.

•	When an outcome evaluation is not feasible due to resource constraints.

Slide 5
Examples: Standing orders in Nursing Homes
•	Goal: increase nursing home vaccination rates in 14 participating states 
•	Intervention: QIOs promote implementation of  standing orders in nursing homes
•	Outcome: vaccination rates did not increase significantly 
•	Limited process evaluation conducted, degree of intensity of the intervention is not known 
•	Conclusion: we do not know what it takes for QIOs to work effectively with NH to promote standing orders.

Slide 6
Examples: “Gimme 5”
•	School-based intervention to increase consumption of fruits, 100% juice, and vegetables
•	Curriculum based intervention
–	Classroom teaching
–	Point-of-purchase education
–	Family participation
•	Outcome evaluation: modest effect
•	Process evaluation: 
–	Teacher satisfaction with training
–	Teacher self-report
–	Teacher observation
–	Phone interviews of parents
–	Observation at point-of-purchase education sessions

Slide 7
Examples: “Gimme 5”, cont’d
–	Teacher satisfaction with training: largely satisfied
–	Teacher self-report: reported completing 90% of curriculum tasks
–	Teacher observation: about 50% of tasks completed—selectively 
–	Phone interviews of parents: a substantial proportion did not do curriculum-related homework with their child, about 1/3 did not view any of the 3 videos sent home. Only 10% participated in point-of-purchase educations session
–	Observation at point-of-purchase education sessions: corroborated low participation reported by parents
•	Conclusion: sub-optimal implementation 

Slide 8
Basic Logic Model
Inputs = Activities = Outputs = Short-term Effects/ Outcomes = Intermediate Effects/Outcomes = Long-term Effects/Outcomes
Context
Assumptions
Stage of Development

This slide shows a logic model or flow chart to show in a graphical way how process evaluation fits in.  Its purpose is to articulate the links between what you do and the results you expect to achieve.  In a standard logic model, boxes of the flow chart are connected by arrows horizontally.  From left to right the logic model begins with inputs which are the resources you have, and then links to activities which are the things that you do; the next link is the Outputs, which are the products of your activities. Examples of outputs are trainings, materials produced, or vaccine delivered.  The last link is the Outcomes/effects.  These are the results that you expect to achieve.  They can be classified as short term, intermediate or long term outcomes

Slide 9
Inputs = Activities (What the program and its staff actually do) = Outputs = Short-term Effects/ Outcomes = Intermediate Effects/Outcomes (Results of activities who, what will change) = Long-term Effects/Outcomes
Context
Assumptions
Stage of Development

So activities are what you do in your program, and outcomes are what others do or what happens to others as a result of your actions. For example a short term outcome could be an increase in knowledge, or a change in attitude, a practice-level change, a hospital change in policy, etc… and long term outcomes would be increases in vaccination rates and decreases in disease.

Slide 10
Sphere of control (Inputs = Activities = Outputs)

Sphere of influence (Short-term Effects/ Outcomes = Intermediate Effects/Outcomes = Long-term Effects/Outcomes)

Context
Assumptions
Stage of Development
One way to think about activities versus outcomes is that activities fall into your sphere of control, and outcomes are in your sphere of influence. You do activities so that others will do their part, you pass the baton and you hope they will do certain things or change in certain ways as a result of your influence. Your ability to impact outcome rests on whether the right activities were conducted and how well they were conducted – this is of course stating the obvious, but I do think it is worth stating in the context of process evaluation.

Slide 11

Process evaluation (Inputs = Activities = Outputs)

Outcome evaluation (Short-term Effects/ Outcomes = Intermediate Effects/Outcomes = Long-term Effects/Outcomes)

Context
Assumptions
Stage of Development

Thus, process evaluation involves looking at inputs, activities, outputs and the link between these categories. The main question being asked in process evaluation is: How well are we doing what we set out to do? Are we implementing activities in the manner we intended to in order to get the results we want?

Slide 12
What are we trying to learn through process evaluation?

–	Do we have the right mix of activities?
–	Are we reaching the intended targets?
–	Are the right people involved as partners, participants, and providers?
–	Do the staff/volunteers have the necessary skills?
–	How well do our activities meet with our priorities

Slide 13

Questions?

Slide 14
Example 1: Child immunization outreach campaign—pocket of need
•	Peer-to peer counselors identified through church or CBO have been trained to approach mothers of young children, discuss immunization, provide materials, refer to clinics
•	Media campaign: radio, newspaper, flyers, posters

Slide 15
Example 1: Child immunization outreach campaign—pocket of need
I.	Outputs:
–	Number of counselors trained
–	Number of contacts made with target population
–	Number of materials created, handed out
–	Number of PSAs aired, and time
–	Number of newspaper ads

Slide 16
Example 1: Child immunization 
outreach campaign—pocket of need
II. Process evaluation questions
–	Were counselors well-trained, was training standardized?
–	Did peer counselor knowledge increase with training?
–	Do mothers find the message appropriate?
–	What percent of target population approached?
–	Did target population hear ads, read newspaper articles, see posters?
–	Were there barriers to getting to the clinic?
–	How well do counselors deliver message?

Slide 17
Example 1: Child immunization 
outreach campaign—pocket of need

III. Outcome evaluation questions
–	Did mothers knowledge or attitudes change? 
–	Did mothers bring children to clinics?
–	Did vaccination rate increase?

Slide 18
Example 2: AFIX
•	Outputs
–	Number of assessments conducted
–	Number of feedback sessions performed
–	Number of staff trainings, of staff trained

Slide 19
Example 2: AFIX
•	Process evaluation questions
–	How well are AFIX staff trained?
–	Is training standardized?
–	Do AFIX staff carry out interactions as per training?
–	Did AFIX staff make recommendations to improve vaccination practices?
–	Are AFIX staff interacting with the right people?
–	Are provider staff satisfied with interactions?
–	What key messages did they take away from feedback session?

Slide 20
Example 2: AFIX
•	Outcome evaluation questions
–	Were recommendations implemented? 
–	Did chart organization improve?
–	Did missed opportunities decrease?
–	Did vaccination rates increase?

Slide 21 
Example 3: influenza vaccination clinic
A county health department is holding new influenza vaccination clinics in a predominantly African American  neighborhood. Information to advertise the clinics has been disseminated through a local paper, flyers and posters in community centers, and radio spots. 
Slide 22
Example 3: influenza vaccination clinic
•	Outputs
–	Number of clinics held
–	Number of persons attending
–	Number of radio announcements
–	Number of flyers/posters posted

Slide 23
Example 3: influenza vaccination clinic
•	Process evaluation questions
–	Percent of clinic participants 50+ or chronic disease?
–	Percent of clinic participants who don’t usually get vaccinated?
–	What percent of target population is aware of clinics?
–	Reasons for not attending clinics?

Slide 24
Example 3: influenza vaccination clinic
•	Outcome evaluation questions
–	Has vaccination rate increased in neighborhood 
(Note: may be difficult to measure, but can be extrapolated from process evaluation information on number of “new vaccinees”)

Slide 25

Questions?

Slide 26
Some useful concepts

•	Reach
•	Quality of implementation
•	Appropriateness
•	Satisfaction
•	Barriers

Slide 27
Reach
•	Degree to which intended audience participates in intervention
–	Percent of target population that heard messages
–	Percent of persons attending influenza vaccination clinic that do not usually get vaccinated
–	Percent of mothers contacted by peer-to-peer counselors

Slide 28
Quality of implementation
•	Was activity implemented properly, according to standards or protocol
–	Was AFIX feedback session conducted as per guidelines?
–	Are AFIX staff interacting with right people?
–	Did peer counselors interaction with mothers follow training or protocol?
–	Is training of staff for a given activity standardized?
–	Was training curriculum delivered in its entirety?

Slide 29
Appropriateness
•	Interventions or messages that are delivered may only be effective if judged appropriate by target population, or if designed in manner to achieve objective 

–	Did messages “speak” to target audience?
  
   Note: short term outcomes also related to appropriateness, e.g.
	Did knowledge or skills increase as a result of training?
	Was information provided in training subsequently used?

Slide 30
Satisfaction
•	The extent to which participants are satisfied with training or interaction may influence subsequent behavior
–	Provider satisfaction with AFIX session
–	Peer counselor satisfaction with training

Slide 31
Barriers

•	This attempts to understand why something didn’t happen, and may identify key environmental variables
–	Reasons mothers didn’t bring their children to the clinic (transportation, clinic schedule, other?)
–	Reasons providers didn’t implement AFIX recommendations (resources?)
–	Reasons seniors didn’t come for influenza vaccination clinic (unaware of clinic, concern about vaccine, no perceived need?)

Slide 32

Prioritizing evaluation questions
Is the question:
•	Important to your program staff and stakeholders?
•	Does it reflect key goals and objectives of your program?
•	Does it reflect key elements of your logic model?
•	Will it provide information you can act upon to make program improvements?
•	Can it be answered using available program resources?
•	Are there any available data sources?

Slide 33
Process evaluation: Methods
•	Quantitative
•	Qualitative
•	Combination

Slide 34
Qualitative methods
•	Case studies
•	Structured or semi-structured interviews
•	Focus groups
•	Direct observation
•	Reviews of program meeting minutes, progress reports

Slide 35
Quantitative methods
•	Surveys
•	Information collected from program participants
•	IIS

Slide 36
Involving stakeholders
•	Gain broader perspective, avoid blind spots, try to ensure utilization of results
•	Key stakeholders:
–	Those served or affected by activity
–	Those involved in program operations
–	Those in a position to make decisions about the activity
•	For a manageable process, the list of stakeholders must be narrowed to primary intended users
Slide 37

Working with an external consultant
•	Scope of work can range from:
–	Conducting specific tasks
–	Designing and conducting an evaluation
•	External evaluator must work closely with you and stakeholders to ensure result is useful

Slide 38
Resources
•	www.cdc.gov/vaccines/programs/progeval
	Immunization evaluation resources

•	Contact us
–	Email via website
–	Via your public health advisor

This document can be found on the CDC website at:
http://www.cdc.gov/vaccines/programs/progeval/downloads/eval-course-3rd.rtf