Program Evaluation 101

Juliana M. Blome, Ph.D. , MPH

Chief, Office of Program Analysis and Evaluation

National Institutes of Health, USDHHS

Bethesda, MD

April 18, 2008

Bridges to the Baccalaureate

Technical Assistance Workshop

National Institute of General Medical Sciences

“Program Evaluation…!?”

Evaluation: What is it?

Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working.

- #GAO/GGD-00-204 Program Evaluation

Program Evaluation Answers Questions Like….

Does it work?
How well does it work?
Does it do what we want it to?
Does it work for the reasons we think it does?
Is it cost effective?
Are the benefits worth it?
What are the unintended consequences?

Why bother?

Supports continuous program improvement
Increases understanding of the program – how are activities and strategies linked to results?
Leads to improved planning and management
Provides shared understanding of program

Guidelines for Conducting Successful Evaluations

Invest heavily in planning early on
Use knowledgeable, experienced evaluators
Integrate evaluation into ongoing activities of the program
Select key questions

Typical Evaluations

Needs Assessment

- What is nature & extent of the issues program should address?
- Planning phase

Process Evaluation

- Is program being conducted & producing output as planned?
- How can process can be improved?

Outcome Evaluation

- Extent to which a program’s goals have been met?

Sample Study Questions

Why should you care?

If you don’t know where you’re going, any road will take you there.

- Lewis Carroll, Alice in Wonderland

-Illustration by Sir John Tenniel, eBooks@ Adelaide, 2004

Evaluation Standards

Utility

Evaluations should serve the practical information needs of a given audience

Feasibility

Evaluations take place in the field and should be realistic, prudent, diplomatic and frugal

Propriety

The rights of individuals affected by evaluations should be protected

Accuracy

Evaluations should produce and convey accurate information about a program’s merit and/or worth

For more information on standards, see Joint Cmte on Standards on Educational Evaluation at http://www.wmich.edu/evalctr/jc/

Key Steps in Evaluation

Engage stakeholders
Describe the program
Focus the evaluation design
Gather credible evidence
Justify conclusions (present data, analysis used, and findings)
Ensure use and share lessons

1. Engage Stakeholders

Who are the stakeholders?

Those involved in program operations, those affected by the program operations, and primary users of evaluation results

“I think you should be more explicit here in Step Two.”

2. Describe the program

What is the goal and purpose of the program?

What problem or need is it designed to address?

What are the objectives? What are the strategies to achieve the objectives?

What are the expected effects?

What are the resources and activities?

How is the program supposed to work?

Activity ≠ Program

You cannot evaluate a program by assessing only an activity

Remember:

Model of a Training Program

Resources Activities Impact

Short term

Knowledge

Skills

Attitudes

Intermediate

Behaviors

Practices

Long term

Enter PhD Program

Faculty & Staff

Money

Equipment & Technology

What is invested?

(Inputs) (Outputs) (Outcomes)

What is invested?

What is done?

What are the changes or benefits?

3. Focus the evaluation design

What do you want to know? (key questions)
Who will be involved in or affected by the evaluation or use the findings? (stakeholders)
To focus an evaluation, consider its purpose, uses, questions, methods, roles, budgets, deliverables etc.

An evaluation cannot answer all questions

for all stakeholders.

4. Gather credible evidence

Methodology & Sample

5. “Justify” conclusions

Consider data:

Analysis and synthesis

- determine findings

Interpretation

- what do findings mean?

Judgments

- what is the value of findings based on accepted standards?

Recommendations –

- what claims can be made?

- what are the limitations of your design?

An evaluation plan should include:

Program Description (with goals)
Purpose of and rational for evaluation
Evaluation Design (study questions, pop’n, key variables, conceptual framework)
Data Collection & Analyses (include ethical issues and limitations of data)
Products of evaluation and their use
Project Management (implementation, advisory committees, timeline)
Budget estimate

If you remember nothing else…

Evaluation is a tool to help you make decisions about program management