|
Standards for
Effective Program Evaluation |
"A standard is a principle mutually agreed to by people engaged
in a professional practice, that, if met, will enhance the quality and fairness of that
professional practice, for example, evaluation."
---
Joint Committee on Educational Evaluation
The second element of the framework is a set of 30
standards for assessing the quality of evaluation activities; these standards are
organized into the following four groups:
These standards, adopted from the Joint
Committee on Standards for Educational Evaluation, answer the question,
"Will this evaluation be effective?" and are recommended as criteria for judging
the quality of program evaluation efforts in public health. They are an approved
standard by the American National Standards Institute (ANSI) and have been endorsed by the
American Evaluation Association and 14 other professional organizations
Public health professionals will recognize that the basic steps of the framework for
program evaluation are part of their routine work. In day-to-day public health practice,
stakeholders are consulted; program goals are defined; guiding questions are stated; data
are collected, analyzed, and interpreted; judgments are formed; and lessons are shared.
Although informal evaluation occurs through routine practice, having standards help to
assess whether a set of evaluative activities are well-designed and working to their
potential.
The standards also make conducting sound and fair evaluations practical. They are
well-supported principles to follow when faced with having to compromise regarding
evaluation options. The standards help avoid creating an imbalanced evaluation (e.g., one
that is accurate and feasible but not useful, or one that would be useful and accurate but
is infeasible).
Furthermore, the standards can be applied while planning an evaluation and throughout
its implementation. The Joint Committee is unequivocal in that, "the standards are
guiding principles, not mechanical rules. . . . In the end, whether a given standard has
been addressed adequately in a particular situation is a matter of judgment."
To facilitate use of the standards, however, the Joint
Committee's report discusses each with an associated list of guidelines and
common errors, as well as applied case examples.
The specific standards are as follows:
Utility
The utility standards are intended to ensure that an
evaluation will serve the information needs of intended users. These standards are
as follows.
- Stakeholder Identification: Persons involved
in or affected by the evaluation should be identified, so that their needs can be
addressed.
- Evaluator Credibility: The persons conducting the
evaluation should be both trustworthy and competent to perform the evaluation, so that the
evaluation findings achieve maximum credibility and acceptance.
- Information Scope and Selection: Information
collected should be broadly selected to address pertinent questions about the program and
be responsive to the needs and interests of clients and other specified stakeholders.
- Values Identification: The perspectives,
procedures, and rationale used to interpret the findings should be carefully described, so
that the bases for value judgments are clear.
- Report Clarity: Evaluation reports should
clearly describe the program being evaluated, including its context, and the purposes,
procedures, and findings of the evaluation, so that essential information is provided and
easily understood.
- Report Timeliness and Dissemination:
Significant interim findings and evaluation reports should be disseminated to intended
users, so that they can be used in a timely fashion.
- Evaluation Impact: Evaluations should be planned,
conducted, and reported in ways that encourage follow-through by stakeholders, so that the
likelihood that the evaluation will be used is increased.
Back to top
Feasibility
The feasibility standards are intended to ensure that an
evaluation will be realistic, prudent, diplomatic, and frugal. The standards are as
follows:
- Practical Procedures: The evaluation
procedures should be practical, to keep disruption to a minimum while needed information
is obtained.
- Political Viability: The evaluation should be
planned and conducted with anticipation of the different positions of various interest
groups, so that their cooperation may be obtained, and so that possible attempts by and of
these groups to curtail evaluation operations or to bias or misapply the results can be
averted or counteracted.
- Cost Effectiveness: The evaluation should be
efficient and produce information of sufficient value, so that the resources expended can
be justified.
Back to top
Propriety
The propriety standards are intended to ensure that an
evaluation will be conducted legally, ethically, and with due regard for the welfare of
those involved in the evaluation, as well as those affected by its results. These
standards are as follows:
- Service Orientation: Evaluation should be
designed to assist organizations to address and effectively serve the needs of the full
range of targeted participants.
- Formal Agreements: Obligations of the formal parties
to an evaluation (what is to be done, how, by whom, when) should be agreed to in writing,
so that these parties are obligated to adhere to all conditions of the agreement or
formally to renegotiate it.
- Rights of Human Subjects: Evaluation should be
designed and conducted to respect and protect the rights and welfare of human subjects.
- Human Interactions: Evaluators should respect
human dignity and worth in their interactions with other persons associated with an
evaluation, so that participants are not threatened or harmed.
- Complete and Fair Assessment: The evaluation
should be complete and fair in its examination and recording of strengths and weaknesses
of the program being evaluated, so that strengths can be built upon and problem areas
addressed.
- Disclosure of Findings: The formal parties to
an evaluation should ensure that the full set of evaluation findings along with pertinent
limitations are made accessible to the persons affected by the evaluation, and any others
with expressed legal rights to receive the results.
- Conflict of Interest: Conflict of interest should be
dealt with openly and honestly, so that it does not compromise the evaluation processes
and results.
- Fiscal Responsibility: The evaluator's allocation
and expenditure of resources should reflect sound accountability procedures and otherwise
be prudent and ethically responsible, so that expenditures are accounted for and
appropriate.
Back to top
Accuracy
The accuracy standards are intended to ensure that an
evaluation will reveal and convey technically adequate information about the features that
determine worth or merit of the program being evaluated. The standards are as
follows:
- Program Documentation: The program being
evaluated should be described and documented clearly and accurately, so that the program
is clearly identified.
- Context Analysis: The context in which the program
exists should be examined in enough detail, so that its likely influences on the program
can be identified.
- Described Purposes and Procedures: The purposes and
procedures of the evaluation should be monitored and described in enough detail, so that
they can be identified and assessed.
- Defensible Information Sources: The sources of
information used in a program evaluation should be described in enough detail, so that the
adequacy of the information can be assessed.
- Valid Information: The information gathering
procedures should be chosen or developed and then implemented so that they will assure
that the interpretation arrived at is valid for the intended use.
- Reliable Information: The information
gathering procedures should be chosen or developed and then implemented so that they will
assure that the information obtained is sufficiently reliable for the intended use.
- Systematic Information: The information collected,
processed, and reported in an evaluation should be systematically reviewed and any errors
found should be corrected.
- Analysis of Quantitative Information: Quantitative
information in an evaluation should be appropriately and systematically analyzed so that
evaluation questions are effectively answered.
- Analysis of Qualitative Information: Qualitative
information in an evaluation should be appropriately and systematically analyzed so that
evaluation questions are effectively answered.
- Justified Conclusions: The conclusions reached in an
evaluation should be explicitly justified, so that stakeholders can assess them.
- Impartial Reporting: Reporting procedures should
guard against the distortion caused by personal feelings and biases of any party to the
evaluation, so that evaluation reports fairly reflect the evaluation findings.
- Metaevaluation: The evaluation itself should be
formatively and summatively evaluated against these and other pertinent standards, so that
its conduct is appropriately guided and, on completion, stakeholders can closely examine
its strengths and weaknesses.
Back to top
Citation: Joint Committee on Educational Evaluation, James R. Sanders
(chair). The program evaluation standards: how to assess evaluations of
educational programs. 2nd edition. Sage Publications, Thousand Oaks, CA.
1994.
Back to top
Purchase Copies From:
|