Program Evaluation 101
Juliana M. Blome, Ph.D. , MPH
Chief, Office of Program Analysis and Evaluation
National Institutes of Health, USDHHS
Bethesda, MD
April 18, 2008
Bridges to the Baccalaureate
Technical Assistance Workshop
National Institute of General Medical Sciences
“Program Evaluation…!?”
Evaluation: What is it?
Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working.
- #GAO/GGD-00-204 Program Evaluation
Program Evaluation Answers Questions Like….
-
Does it work?
-
How well does it work?
-
Does it do what we want it to?
-
Does it work for the reasons we think it does?
-
Is it cost effective?
-
Are the benefits worth it?
-
What are the unintended consequences?
Why bother?
-
Supports continuous program improvement
-
Increases understanding of the program – how are activities and strategies linked to results?
-
Leads to improved planning and management
-
Provides shared understanding of program
Guidelines for Conducting Successful Evaluations
-
Invest heavily in planning early on
-
Use knowledgeable, experienced evaluators
-
Integrate evaluation into ongoing activities of the program
-
Select key questions
P5
Typical Evaluations
Needs Assessment
-
-
What is nature & extent of the issues program should address?
-
Planning phase
-
Process Evaluation
-
-
Is program being conducted & producing output as planned?
-
How can process can be improved?
-
Outcome Evaluation
-
-
Extent to which a program’s goals have been met?
-
Sample Study Questions
Sample Study Questions
Why should you care?
If you don’t know where you’re going, any road will take you there.
- Lewis Carroll, Alice in Wonderland
-Illustration by Sir John Tenniel, eBooks@ Adelaide, 2004
Evaluation Standards
Utility
Evaluations should serve the practical information needs of a given audience
Feasibility
Evaluations take place in the field and should be realistic, prudent, diplomatic and frugal
Propriety
The rights of individuals affected by evaluations should be protected
Accuracy
Evaluations should produce and convey accurate information about a program’s merit and/or worth
For more information on standards, see Joint Cmte on Standards on Educational Evaluation at http://www.wmich.edu/evalctr/jc/
Key Steps in Evaluation
-
Engage stakeholders
-
Describe the program
-
Focus the evaluation design
-
Gather credible evidence
-
Justify conclusions (present data, analysis used, and findings)
-
Ensure use and share lessons
1. Engage Stakeholders
Who are the stakeholders?
Those involved in program operations, those affected by the program operations, and primary users of evaluation results
“I think you should be more explicit here in Step Two.”
By Sidney Harris, Copyright 2007, The New Yorker
2. Describe the program
What is the goal and purpose of the program?
What problem or need is it designed to address?
What are the objectives? What are the strategies to achieve the objectives?
What are the expected effects?
What are the resources and activities?
How is the program supposed to work?
Activity ≠ Program
You cannot evaluate a program by assessing only an activity
Remember:
Model of a Training Program
Resources Activities Impact
Short term
Knowledge
Skills
Attitudes
Intermediate
Behaviors
Practices
Long term
Enter PhD Program
Faculty & Staff
Money
Equipment & Technology
What is invested?
(Inputs) (Outputs) (Outcomes)
What is invested?
What is done?
What are the changes or benefits?
3. Focus the evaluation design
-
What do you want to know? (key questions)
-
Who will be involved in or affected by the evaluation or use the findings? (stakeholders)
-
To focus an evaluation, consider its purpose, uses, questions, methods, roles, budgets, deliverables etc.
An evaluation cannot answer all questions
for all stakeholders.
4. Gather credible evidence
Methodology & Sample
5. “Justify” conclusions
Consider data:
-
Analysis and synthesis
- determine findings
-
Interpretation
- what do findings mean?
-
Judgments
- what is the value of findings based on accepted standards?
-
Recommendations –
- what claims can be made?
- what are the limitations of your design?
An evaluation plan should include:
-
Program Description (with goals)
-
Purpose of and rational for evaluation
-
Evaluation Design (study questions, pop’n, key variables, conceptual framework)
-
Data Collection & Analyses (include ethical issues and limitations of data)
-
Products of evaluation and their use
-
Project Management (implementation, advisory committees, timeline)
-
Budget estimate
If you remember nothing else…
Evaluation is a tool to help you make decisions about program management