Slide 1
Practical Program Evaluation—Follow Up: Working a Case Example
By:
Thomas J. Chapel, Office of Workforce and Career Development
Betty Apt, Division of STD Prevention
Centers for Disease Control and Prevention

Slide 2
Today…
•	Review main points from overview by applying to a case example
•	Choosing a component to evaluate
•	Identifying stakeholders that matter most
•	Developing simple logic models
•	Choosing a strong evaluation focus
•	Discuss November workshop

Slide 3
Why Bother? Because We Want Programs to Improve

•	Continuous Quality Improvement (CQI) cycle.
Planning—What actions will best reach our goals and objectives.
Performance measurement— How are we doing?
Evaluation—Why are we doing well or poorly?

(circle chart)
What do we do?- How do we do it?- How are we doing?- Why are we doing well or poorly?

Slide 4

What can Program Evaluation do?

•	Increase Program Knowledge 
•	Direct Program Improvement
•	Maximize Resources
•	Provide Accountability

Slide 5
Avoid Wasted Effort!
(graphic image of two sharks and bottled wasting liquids in the ocean)

Slide 6
Two Pieces of Work
1.Choose a program component to evaluate
2. Apply evaluation steps and standards

Slide 7
Choosing a Program Component to Evaluate
Evaluation can focus on any/all parts of a large program like the immunization program

Slide 8
Some Potential Criteria..

•	Where am I spending the most?
•	Where am I concerned the most?
•	Where are my big opportunities/new areas?
•	Where are my big successes?
•	Other…
•	Other…

Slide 9
Some Potential Components to Evaluate
•	Immunization information systems
•	Adolescent immunization
•	Perinatal hepatitis B prevention
•	AFIX
•	VFC

Slide 10
Today’s Case Example: AFIX—Reasons for Selecting Component
Substantial program effort
Research shows it works…
But implemented in various ways…
Want to ensure “our” approach is working

Slide 11
Next—Apply Evaluation Steps and Standards

Figure 1. Recommended framework for program evaluation 

(circle chart)

Steps –engage stakeholders- Describe the program – 
Focus the evaluation design- Gather credible evidence- -Justify conclusions-
Ensure use and share lessons learned

Standards
Utility
Feasibility
Propriety
Accuracy

Step 12
Evaluation Step—Stakeholders

1.	Identify and engage stakeholders
2.	Describe the program component
3.	Focus the evaluation design
4.	Gather credible evidence
5.	Justify conclusions
6.	Ensure use and share lessons learned

Slide 13
Which Stakeholders Matter Most?

Who is
•	Affected by the program?
•	Involved in program operations?
•	Intended users of evaluation findings?

Of these, who do we most need to:
Enhance credibility?
Implement program changes?
Advocate for changes?
Fund, authorize, expand program?

Slide 14
Some Key Stakeholders

Types-	Definitions-Examples
Decision makers-Decide and direct program operations, including how evaluation findings are used

Examples-Immunization program director, program manager, HD director, health commissioner, legislators  

Implementers-	Involved in program operations	
Examples-Immunization program director, program manager, field staff

Participants-Served by the program
Examples-Physicians, parents, community members

Partners-Support/invested in immunization program or target population
Examples-Managed care plans, health systems, AAP and AAFP chapters, CBOs

Slides 15
Payoffs for Involving   Stakeholders

•	Strong input on purpose and design of the evaluation 
•	“Reality check” on utility and feasibility of evaluation plan and methods.
•	Increase credibility of evaluation findings.
•	Increase likelihood that evaluation results will be acted upon.

Slide 16
Involve Stakeholders from the Start!
Potential stakeholder roles at all steps:
•	Input on evaluation design
•	Input on data collection methods
•	Assist with data collection/analysis
•	Outreach to the target population for the evaluation
•	Interpretation and use of results

Slide 17
Case Example: AFIX Stakeholders

•	Program staff
•	Clinical staff (clinic managers, nurses, physicians)

Slide 18
•	Evaluation Step—Description
•	Identify and engage stakeholders
•	Describe the program component
•	Focus the evaluation design
•	Gather credible evidence
•	Justify conclusions
•	Ensure use and share lessons learned

Slide 19
Seeking Consensus On…

•	The activities and outcomes depicted
•	The “roadmap”
•	Which outcomes = program “success” 
•	This will facilitate future discussions of:
How much progress on outcomes = program “success”
Choices of data collection/analysis methods

Slide 20
Describing the Program Complete Logic Model
Inputs- Activities- Outputs- Short-term Effects/Outcomes- Intermediate Effects/Outcomes- Long-term Effects/Outcomes

Context
Assumptions 
Stage of Development

Slide 21
What the program and its staff actually do
Inputs- Activities- Outputs- Short-term Effects/Outcomes- Intermediate Effects/Outcomes- Long-term Effects/Outcomes

Context
Assumptions 
Stage of Development

Slide 22
Results of activities: Who/what will change?
Inputs- Activities- Outputs- Short-term Effects/Outcomes- Intermediate Effects/Outcomes- Long-term Effects/Outcomes

Context
Assumptions 
Stage of Development

Slide 23

Simple Logic Model—AFIX
(blank chart)
Activities:
If- Early- then Later

Outcomes
Then short-term if- long-term - then

Slide 24
Case Example: AFIX Logic Model

Activities

Review records and assess coverage level

If we do this: Develop AFIX protocol
And then this: Provide feedback and recommend strategies for improvement

Outcomes
Then this will occur: Increase in provider motivation
And then this: Change in provider behavior
And then this: Decrease in missed opportunities
And then this: Increase vaccination= Decreased VPD

If we do this: Train AFIX staff
And then this: Acknowledge and reward improved performance

Outcomes
Then this will occur: Increase in provider knowledge
And then this: Change in office practice and polices
And then this: Decreased in invalid dose administration
And then this: Increased vaccination = Decreased VPD

Activities
If we do this: Set up provider visits
And then this:  Document outcomes of visit in database (eg CoCASA),
Promote information exchange

Outcome
And then this: Increase in time lines of vaccine receipt,
Increased vaccination, = Decreased VPD

Slide 25
Resource “platform” for the program
Inputs- Activities- Outputs
Tangible products of activities
- Short-term Effects/Outcomes- Intermediate Effects/Outcomes- Long-term Effects/Outcomes

Context
Assumptions 
Stage of Development

Slide 26
AFIX Full Logic Model

Inputs: Budget
Activities: Develop AFIX protocol, Review records and assess coverage level
Outputs: Protocol

Inputs: Staff
Activities: Develop AFIX protocol, Provide feedback and recommend strategies for improvement
Outputs: Training
Outcomes: Increase in provider motivation, Changes in provider behavior; Decrease in missed opportunities, Increased vaccination= Decreased VPD

Inputs: Time
Activities: Train AFIX staff, 
Provide feedback and recommend strategies for improvement
Outputs: Training
Outcomes: Increase in provider motivation, Changes in provider behavior; Decrease in missed opportunities, Increased vaccination= Decreased VPD

Inputs: Immunization information systems
Activities: Set up provider visits, Acknowledge and reward improved performance,
Outputs: Coverage assessments, Recommendations
Outcomes: Increased provider knowledge, practices and policies, Change in office practices and policies, Increase in timeliness of vaccine receipt

Inputs: CoCASA
Activities: Document outcomes of visiting database (eg CoCASA)
Promote information exchange
Outcome: Increase in timeliness of vaccine receipt

Slide 27

Inputs- Activities- Outputs- Short-term Effects/Outcomes- Intermediate Effects/Outcomes- Long-term Effects/Outcomes

Moderators:
Contextual factors that will facilitate or hinder getting our outcomes
Context
Assumptions 
Stage of Development

Slide 28
Evaluation Step—Focusing the Evaluation

1.	Identify and engage stakeholders
2.	Describe the program component
3.	Focus the evaluation design
Tailor evaluation to program and stakeholder’s needs
Determine resources and personnel available
Develop and prioritize evaluation questions
4.	Gather credible evidence
5.	Justify conclusions
6.	Ensure use and share lessons learned

Slide 29
Evaluation Standards Help Choose Focus
•	“Utility”— Which stakeholders/others want the evaluation findings? How/for what purpose  will they use the findings?  

•	“Feasibility”— How reasonable are these questions given: How long program has been in existence; program intensity; data collection logistics/constraints.

Slide 30
Generating and Prioritizing Evaluation Questions

Utility: Questions which:
Are important to stakeholders;
Are consistent with program goals and objectives
Will provide information that can be acted upon

Feasibility: Questions which:
Can be answered with available resources and within available timeframe
Are consistent with program maturity and intensity
Will be supported by program decision-makers.

Slide 31
Setting Focus: Process and Outcome Evaluation

•	Process Evaluation
Determines if program activities are implemented as intended.
Tracks who, what, when, and where program information.  
Provides feedback loop for program improvement.
Conducted throughout project life.

•	Outcome Evaluation
Measures program effects; changes in target population’s knowledge, attitudes, self-efficacy, skills, intentions, behaviors; or organizational changes (e.g., policy adoption).   
Linked to process evaluation. 
Provides feedback loop for program improvement

Slide 32
Case Example: Setting AFIX Focus
Utility
•	Purpose/User:  Program staff
•	Use: Determine if:
AFIX feedback is actually implemented by providers. 
If not, determine barriers and challenges to use of feedback
And, determine how to improve activities and followup to surmount identified barriers and challenges
Feasibility
•	Stage/Intensity: Well-established and multi-faceted program.  Should be making these impacts if working
•	Resources: [Assumption] Evaluation questions of interest have some ready data sources that would be credible for those interested in findings

Slide 33
AFIX Logic Model—Focus

Activities
Activities

Review records and assess coverage level

If we do this: Develop AFIX protocol
And then this: Provide feedback and recommend strategies for improvement

Outcomes
Then this will occur: Increase in provider motivation
And then this: Change in provider behavior
And then this: Decrease in missed opportunities
And then this: Increase vaccination= Decreased VPD

If we do this:Train AFIX staff
And then this: Acknowledge and reward improved performance

Outcomes
Then this will occur: Increase in provider knowledge
And then this: Change in office practice and polices
And then this: Decreased in invalid dose administration
And then this: Increased vaccination = Decreased VPD

Activities

If we do this: Set up provider visits
And then this:  Document outcomes of visit in database (eg CoCASA),
Promote information exchange

Outcome
And then this: Increase in time lines of vaccine receipt,
Increased vaccination, Decreased VPD

Slide 34
AFIX—Sample Evaluation Questions
Proposed focus: both process and outcomes
•	Outcome:
Are AFIX recommendations implemented by practices within 6 months of visit? (short term outcome)
•	Process
Factors that may be related to implementation/non-implementation of AFIX recommendations:
Are AFIX staff appropriately trained? 
Are practice contacts the right people?
What is the quality of the feedback? Are practices satisfied with feedback received?

Slide 35
Where Next….
Engage Stakeholders Throughout!
Indicators
Data Collection Sources and Methods
Data Collection Procedures
Data Analysis
Data Interpretation
Shared Results and Lessons Learned
Results used to improve program
Slide 36

But

These steps more likely to be done well because we did a good job on earlier steps!!!

Slide 37
Practical Program Evaluation
Coming! This Fall…

Slide 38
Fall Training Session

•	Reaffirm/reinforce today’s points
•	Work through remaining steps:
Define indicators
Identify data collection sources and methods
Consider data analysis and interpretation
Maximizing use

Slide 39
Pre-Work for November
•	Consider criteria for choosing a program component
•	Give some preliminary thought to:
•	Potential program components
•	Key stakeholders
•	Purpose/use of evaluation

Slide 40
Practical Program Evaluation
Life Post-Session

Slide 41
Helpful Publications @ www.cdc.gov/eval

(CDC MMWR page  logo )

CBPH – An Evaluation Framework for Community Health Programs

Slide 42
•	Helpful Resources

•	NEW! Intro to Program Evaluation for PH Programs—A Self-Study Guide: http://www.cdc.gov/eval/whatsnew.htm
•	Logic Model Sites
Innovation Network: 
http://www.innonet.org/
Harvard Family Research Project: http://www.gse.harvard.edu/hfrp/
University of Wisconsin-Extension: http://www.uwex.edu/ces/lmcourse/
CDC/DASH:http://www.cdc.gov/healthyyouth/evaluation/resources.htm#4 
CDC/STD: http://www.cdc.gov/std/program/progeval/TOC-PGprogeval.htm
•	Texts
Kellogg Foundation Logic Model Development Guide: www.wkkf.org
W.K. Kellogg Foundation Evaluation Resources: http://www.wkkf.org/programming/overview.aspx?CID=281
Rogers et al.  Program Theory in Evaluation.  New Directions Series: Jossey-Bass, Fall 2000
Chen, H.  Theory-Driven Evaluations.  Sage.  1990

Slide 43
•	Community Tool Box
http://ctb.ku.edu

( Community tool box graphic web page)

This document can be found on the CDC website at:
http://www.cdc.gov/vaccines/programs/progeval/downloads/Eval_Course_2nd.rtf