|
Office of the Assistant Secretary for Planning and
Evaluation |
|
Policy Information Center |
Appendix
C
Evaluation Review
Panel's
Criteria for
Assessing Program Evaluations
OVERALL
SIGNIFICANCE
- The study addresses a
significant issue of policy relevance.
- Evaluation findings are
likely to be useful.
CONCEPTUAL CRITERIA
Conceptual Foundations
- A literature review is
included.
- The project is shown to
be logically based on previous findings; the report uses either theory, or
models, or both.
- The program assumptions
are stated.
- The evaluation draws
from any previous evaluation.
- The report is linked
with a program and describes the program.
- The report presents
multiple perspectives.
- Multiple relevant
stakeholders are consulted and involved.
- The timing is
appropriate because the program is ready for evaluation.
Questions for
Evaluation
- The aims of the
evaluation are clear, well-specified, and testable.
- The questions are
feasible, significant, linked to the program, appropriate for the resources and
audience, and derive logically from the conceptual
foundations.
- The questions show
ingenuity and creativity.
Findings and
Interpretation
- The conclusions are
justified by the analyses.
- The summary does not go
beyond what the data will support.
- The appropriate
qualifiers are stated.
- The conclusions fit the
entire analysis.
- Equivocal findings are
handled appropriately.
- The initial questions
are answered.
- The interpretation ties
in with the conceptual foundation.
- The report notes that
the findings are either consistent with or deviate from the relevant
literature.
- The presentation is
understandable.
- The results have
practical significance.
- The extent of program
implication is assessed.
Recommendations
- The recommendations
follow from findings, are worth carrying out, and are affordable, timely,
feasible, useful, and appropriate.
- The recommendations are
shown to be relevant to the questions asked.
- The breadth of
specificity of the recommendations is addressed.
- Any recommendations for
either future evaluations, or improvements, or both are clearly
presented.
METHODS
Evaluation Design
- Design considerations
include overall appropriateness, soundness, feasibility, funding and time
constraints, generalizability, applicability for cultural diversity, assessment
of the extent of program delivery, validity, feasibility for data collection,
reliability of selected measurements, use of multiple measures of key concepts,
and appropriateness of the sample.
- Variables are clearly
specified and fit with the questions and concepts.
- The design permits
measurement of the extent of program implementation and answering of the
evaluation questions.
Data Collection
- Data are collected
using appropriate units of measurement for analysis, controls for participant
selection and assignment bias, and proper handling of missing data and
attrition.
- Data collection is
characterized by use of an appropriate comparison group of control; adequate
sample size, response rate, and information about the sample; a data collection
plan; data collection that is faithful to the plan; attention to and
cooperation with the relevant community; project confidentiality; and
consistency.
- The quality of the data
(including the quality of any extant data sets used in the study) and the
efficiency of sampling are addressed.
- The data collection is
appropriate to evaluation questions.
Data Analysis
- The data analysis
addresses the handling of attrition, the matching of the analysis to the
design, the use of appropriate statistical controls, the use of methodology and
levels of measurement appropriate to the type of data, and estimation of effect
size.
- The analysis shows
sensitivity to cultural categories.
- The analysis makes
appropriate generalizability of inferences.
- The chosen analysis
type is simple and efficient.
CROSS-CUTTING FACTORS
The following are
cross-cutting factors that are likely to be important at all stages of a
report: clarity, presentation, operation at a state-of-the-art level,
appropriateness, understandability, innovation, generalizability, efficiency of
approach, logical relationships, and discussion of the report's limitations.
The report should also address ethical issues, possible perceptual bias,
cultural diversity, and any gaps in study execution.
Table of Contents