Stage 4: Assessing Effectiveness and Making Refinements
Questions to Ask and Answer Why Outcome Evaluation Is Important Revising the Outcome Evaluation Plan Conducting Outcome Evaluation Refining Your Health Communication Program Common Myths and Misconceptions About Evaluation Selected Readings
-
How can we use outcome evaluation to assess the effectiveness of our program?
-
How do we decide what outcome evaluation methods to use?
-
How should we use our evaluation results?
-
How can we determine to what degree we have achieved our communication objectives?
-
How can we make our communication program more effective?
In Stage 3, you decided how to use process evaluation to monitor and adjust your
communication activities to meet objectives. In Stage 4, you will use the
outcome evaluation plan developed in Stage 1 to identify what changes (e.g., in
knowledge, attitudes, or behavior) did or did not occur as a result of the
program. Together, the progress and outcome evaluations will tell you how the
program is functioning and why. (If you combine information from the two types
of evaluation, be sure that you focus on the same aspects of the program, even
though you look at them from different perspectives.) This section will help
you revise your plans and conduct outcome evaluation.You should begin planning
assessment activities either before or soon after you launch the program.
Outcome evaluation is important because it shows how well the program has met
its communication objectives and what you might change or improve to make it
more effective. Learning how well the program has met its communication
objectives is vital for:
-
Justifying the program to management
-
Providing evidence of success or the need for additional resources
-
Increasing organizational understanding of and support for health communication
-
Encouraging ongoing cooperative ventures with other organizations
During Stage 1, you identified evaluation methods and drafted an outcome
evaluation plan. At that time, you should have collected any necessary baseline
data. The first step in Stage 4 is to review that plan to ensure it still fits
your program. A number of factors will influence how your communication
program’s outcomes should be evaluated, including the type of
communication program, the communication objectives, budget, and timing. The
outcome evaluation needs to capture intermediate outcomes and to measure the
outcomes specified in the communication objectives. Doing so can allow you to
show progress toward the objectives even if the objectives are not met.
Examples of Effectiveness Measures for Health Communication Programs
|
Knowledge
A public survey conducted before and after NCI’s 5 A Day campaign found
that knowledge of the message (a person should eat 5 or more servings of fruits
and vegetables each day for good health) increased by 27 percentage points.
Attitude
In 1988, the U.S. Surgeon General sent a pamphlet designed to influence
attitudes on AIDS to every U.S. household. An evaluation conducted in
Connecticut showed no change in attitude between residents who read the
pamphlet and those who did not.
Behavior
The Pawtucket Heart Health Program evaluated a weight-loss awareness program
conducted at worksites. More than 600 people enrolled, and they lost an average
of 3.5 pounds each compared with their preprogram weight.
|
Consider the following questions to assess the Stage 1 outcome evaluation plan
and to be sure the evaluation will give you the information you need:
-
What are the communication objectives?
What should the members of the intended audience think, feel, or do as a result of the health communication plan in contrast to what they thought, felt, or did before? How can these changes be measured?
-
How do you expect change to occur?
Will it be slow or rapid? What measurable intermediate outcomes (steps toward the desired behavior) are likely to take place before the behavior change can occur? The behavior change map you created in Stage 1 should provide the answers to these questions.
-
How long will the program last?
What kinds of changes can we expect in that time period (e.g., attitudinal, awareness, behavior, policy changes)? Sometimes, programs will not be in place long enough for objectives to be met when outcomes are measured (e.g., outcomes measured yearly over a 5-year program). To help ensure that you identify important indicators of change, decide which changes could reasonably occur from year to year.
-
Which outcome evaluation methods can capture the scope of the
change that is likely to occur?
Many outcome evaluation measures are relatively crude, which means that a large percentage of the intended audience (sometimes an unrealistically large percentage) must make a change before it can be measured. If this is the case, the evaluation is said to "lack statistical power." For example, a public survey of 1,000 people has a margin of error of about 3 percent. In other words, if 50 percent of the survey respondents said they engage in a particular behavior, in all likelihood somewhere between 47 percent and 53 percent of the population represented by the respondents actually engages in the behavior. Therefore, you can conclude that a statistically significant change has occurred only if there is a change of 5 or more percentage points. It may be unreasonable to expect such a large change, and budgetary constraints may force you to measure outcomes by surveying the general population when your intended audience is only a small proportion of the population.
-
Which aspects of the outcome evaluation plan best fit with your
organization’s priorities?
Only rarely does a communication program have adequate resources to evaluate all activities. You may have to illustrate your program’s contribution to organizational priorities to ensure continued funding. If this is the case, it may be wise to evaluate those aspects most likely to contribute to the organization’s mission (assuming that those are also the ones most likely to result in measurable changes).
Quantitative Versus Qualitative
Evaluation |
Quantitative research is used to gather objective
information by asking a large number of people a set of identical questions.
Results are expressed in numerical terms (e.g., 35 percent are aware of X and
65 percent are not). If the respondents are a representative random sample,
quantitative data can be used to draw conclusions about an intended audience as
a whole. Quantitative research is useful for measuring the extent to which a
knowledge set, attitude, or behavior is prevalent in an intended audience.
Qualitative research is used to gather reactions and
impressions from small numbers of intended audience members, usually by
engaging them in discussion. Results are subjective and are not described
numerically or used to make generalizations about the intended audience.
Qualitative research is useful for understanding why people react the way they
do and for understanding additional ideas, issues, and concerns.
Quantitative research methods are usually used for outcome evaluation because
they provide the numerical data necessary to assess progress toward objectives.
When evaluating outcomes, qualitative research methods are used to help
interpret quantitative data and shed light on why particular outcomes were (or
were not) achieved. See the Communication Research Methods section for detailed
explanations of quantitative and qualitative research methods and the
circumstances under which you should use each.
|
Conduct outcome evaluation by following these steps:
-
Determine what information the evaluation must provide.
-
Define the data to collect.
-
Decide on data collection methods.
-
Develop and pretest data collection instruments.
-
Collect data.
-
Process data.
-
Analyze data to answer the evaluation questions.
-
Write an evaluation report.
-
Disseminate the evaluation report.
Evaluation Constraints
|
Every program planner faces limitations when conducting an outcome evaluation.
You may need to adjust your evaluation to accommodate constraints such as the
following:
-
Limited funds
-
Limited staff time or expertise
-
Length of time allotted to the program and its evaluation
-
Organizational restrictions on hiring consultants or contractors
-
Policies that limit your ability to collect information from the public
-
Difficulty in defining the program’s objectives or in establishing consensus on them
-
Difficulty in isolating program effects from other influences on the intended audience in "real world" situations
-
Management perceptions of the evaluation’s value
These constraints may make the ideal evaluation impossible. If you must
compromise your evaluation’s design, data collection, or analysis to fit
limitations, decide whether the compromises will make the evaluation results
invalid. If your program faces severe constraints, do a small-scale evaluation
well rather than a large-scale evaluation poorly. Realize that it is not
sensible to conduct an evaluation if it is not powerful enough to detect a
statistically significant change.
|
See a description of each step below.
1. Determine What Information the Evaluation Must Provide
An easy way to do this is to think about the decisions you will make based on
the evaluation report. What questions do you need to answer to make those
decisions?
2. Define the Data You Need to Collect
Determine what you can and should measure to assess progress on meeting
objectives. Use the following questions as a guide:
-
Did knowledge of the issue increase among the intended audience (e.g., understanding how to choose foods low in fat or high in fiber, knowing reasons not to smoke)?
-
Did behavioral intentions of the intended audience change (e.g., intending to use a peer pressure resistance skill, intending to buy more vegetables)?
-
Did intended audience members take steps leading to the behavior change (e.g., purchasing a sunscreen, calling for health information, signing up for an exercise class)?
-
Did awareness of the campaign message, name, or logo increase among intended audience members?
-
Were policies initiated or other institutional actions taken (e.g., putting healthy snacks in vending machines, improving school nutrition curricula)?
3. Decide on Data Collection Methods
The sidebar Outcome Evaluation Designs describes some common outcome evaluation
designs, the situations in which they are appropriate, and their major
limitations. (See the Communication Research Methods
section for more information.) Complex, multifaceted programs often employ a
range of methods so that each activity is evaluated appropriately. For example,
a program that includes a mass media component to reach parents and a
school-based component to reach students might use independent cross-sectional
studies to evaluate the mass media component and a randomized or
quasi-experimental design to evaluate the school-based component.
The following limitations can make evaluation of your communication program
difficult:
-
Lack of measurement precision (e.g., available data collection mechanisms cannot adequately capture
change or cannot capture small changes). Population surveys may not be able to identify the small number of people making a change. Self-reported measures of behavior change may not be accurate.
-
Inability to conclusively establish that the communication
activity caused the observed effect.
Experimental designs, in which people are randomly assigned to either receive an intervention or not, allow you to assume that your program causes the only differences observed between the group exposed to the program and the control group. Outcome evaluations with experimental designs that run more than a few weeks, however, often wind up with contaminated control groups, either because people in the group receiving the intervention move to the control group, or because people in the control group receive messages from another source that are the same as or similar to those from your program.
The more complex your evaluation design is, the more you will need expert
assistance to conduct your evaluation and interpret your results. The expert
can also help you write questions that produce objective results. (It’s
easy to develop questions that inadvertently produce overly positive results.)
If you do not have an evaluator on staff, seek help to decide what type of
evaluation will best serve your program. Sources include university faculty and
graduate students (for data collection and analysis), local businesses (for
staff and computer time), state and local health agencies, and consultants and
organizations with evaluation expertise.
Outcome Evaluation Designs Appropriate for Specific Communication Programs
|
Programs Not Delivered to the Entire Population of the Intended Audience
|
Evaluation Design
|
Major Limitations
|
Randomized experiment. Members of the intended audience are randomly
assigned to either be exposed to the program (intervention group) or not
(control group). Usually, the same series of questions is asked pre- and
postintervention (a pretest and posttest); posttest differences between the two
groups show change the program has caused.
|
-
Not appropriate for programs that will evolve during the study period.
-
Not likely to be generalizable or have external validity because of tight controls on program delivery and participant selection. Delivery during the evaluation may differ significantly from delivery when the program is widely implemented (e.g., more technical assistance and training may be available to ensure implementation is proceeding as planned).
-
For programs delivered over time, it is difficult to maintain integrity of intervention and control groups; group members may leave the groups at different rates of attrition.
-
Often costly and time-consuming.
-
May deprive the control group of positive benefits of the program.
|
Quasi-experiment. Members of the intended audience are split into
control and intervention groups based simply upon who is exposed to the program
and who is not.
|
-
Same as randomized experiments.
-
Difficult to conclude that the program caused the observed effects because other differences between the two groups may exist.
|
Before-and-after studies. Information is collected before and after
intervention from the same members of the intended audience to identify change
from one time to another.
|
-
Difficult to say with certainty that the program (rather than some unmeasured variable) caused the observed change.
|
Independent cross-sectional studies. Information is collected before
and after intervention, but it is collected from different intended audience
members each time.
|
-
Cannot say with certainty that the program caused any observed change.
|
Panel studies. Information is collected at multiple times from the
same members of the intended audience. When intended audience members are
differentially exposed to the program, this design helps evaluators sort out
the effects of different aspects of the program or different levels of
exposure.
|
-
Generalizability may be compromised over time. As participants age, leave, or respond to repeated questions on the same subject, they may no longer closely represent the intended audience.
-
Can be difficult to say with certainty that the program caused the observed change.
|
Time series analysis. Pre- and postintervention measures are collected
multiple times from members of the intended audience. Evaluators use the
preintervention data points to project what would have happened without the
intervention and then compare the projection to what did happen using the
postintervention data points.
|
-
Large number of pre- and postintervention data points are needed to model pre- and postintervention trends.
-
Normally restricted to situations in which governmental or other groups routinely collect and publish statistics that can be used as the pre- and postintervention observations.
|
Examples of Outcome Evaluation for Communication Programs
|
NCI’s Cancer Information Service
Customer satisfaction surveys are one means of gathering data about a
program’s effects. Surveys of telephone callers to NCI’s Cancer
Information Service (CIS) have shown that:
-
Eight out of 10 callers say they take positive steps to improve their health after talking with CIS staff.
-
Seventy percent of those who call about symptoms say the CIS information was helpful in their decision to see a doctor.
-
Fifty-five percent of those who call about treatment say they use CIS information to make a treatment decision.
-
Two-thirds of callers who are considering participation in a research study talk with a doctor after calling the CIS.
|
Note. From Morra, M. E. (Ed). (1998). The impact and
value of the Cancer Information Service: A model for health communication. In Journal
of Health Communication, Vol. 3, Number 3, Supplement 1, pp.
7–8. Copyright Taylor & Francis Co. Adapted with permission.
|
The Right Turns Only Program
Right Turns Only is a video-based drug education series produced by the Prince
George’s County, Maryland, school system. The effects of this series
(including collateral print material) on student knowledge, attitudes, and
behavioral intentions were tested among approximately 1,000 seventh grade
students.
Twelve schools were assigned to one of four groups: three intervention groups
and one control group. One intervention group received only the video-based
education, a second received both the video-based and a traditional drug
education curriculum, a third received only the traditional curriculum, and the
control group received no drug abuse prevention education. All interventions
were completed within a 3-week period.
The six outcomes measured included: 1) knowledge of substance abuse terminology,
2) ability to assess advertisements critically, 3) perception of family, 4)
conflict resolution, 5) self-efficacy in peer relationships, and 6) behavioral
intentions related to substance use/abuse prevention.
Changes were measured using data from questionnaires completed by students
before and after the interventions. The data were analyzed to identify
differences based on gender, race, grades (self-reported), and teacher. Groups
that received drug education scored higher than the control group on all
posttest measures except self-efficacy. On two of the six measures, the group
receiving the combination of the video series and traditional curriculum scored
significantly higher than other groups.
The evaluation demonstrated that instructional videos (particularly when used in
conjunction with print materials and teacher guidance) could be an effective
tool for delivering drug education in the classroom.
|
Note. Adapted from Evaluating the Results of
Communication Programs (Technical Assistance Bulletin), by Center for
Substance Abuse Prevention, August 1998, Washington, DC: U.S. Government
Printing Office. In the public domain.
|
NIDDK’s "Feet Can Last a Lifetime" Program
In 1995 the National Institute of Diabetes and Digestive and Kidney Diseases
(NIDDK) developed a feedback mechanism for the promotion of its kit, "Feet
Can Last a Lifetime," that was designed to reduce the number of lower
extremity amputations in people with diabetes. The first printing of the kit
included a feedback form for health care providers to comment on the materials.
Based on the feedback, NIDDK revised the kit in 1997. The new kit’s
contents were then pretested extensively with practitioners for technical
accuracy, usefulness, and clarity. The original kit was developed primarily for
providers; based upon evaluation results, the revised kit also includes
materials for patients. These include an easy-to-read brochure; a fact sheet
with "foot care tips" and a "to do" list that contains steps
for patients to follow to take care of their feet; and camera-ready, laminated
tip sheets for providers to reproduce and give to patients.
|
4.
Develop and Pretest Data Collection Instruments
Most outcome evaluation methods involve collecting data about participants
through observation, a questionnaire, or another method. Instruments may
include tally sheets for counting public inquiries, survey questionnaires,
interview guides. Select a method that allows you to best answer your
evaluation questions based upon your access to your intended audience and your
resources. To develop your data collection instruments—or to select and
adapt existing ones—ask yourself the following questions:
Which Data?
The data you collect should be directly related to your evaluation questions.
Although this seems obvious, it is important to check your data collection
instruments against the questions your evaluation must answer. These checks
will keep you focused on the information you need to know and ensure that you
include the right measures. For example, if members of your intended audience
must know more about a topic before behavior change can take place, make sure
you ask knowledge-related questions in your evaluation.
From Whom?
You will need to decide how many members of each group you need data from in
order to have a sufficiently powerful evaluation to assess change. Make sure
you have adequate resources to collect information from that many people.
Realize that you may also need a variety of data collection instruments and
methods for the different groups from whom you need information.
How?
Before you decide how to collect your data, you must assess your resources. Do
you have access to, or can you train, skilled interviewers? Must you rely on
self-reports from participants?
Also consider how comfortable the participants will be with the methods you
choose to collect data. Will they be willing and able to fill out forms? Will
they be willing to provide personal information to interviewers? Will the
interviews and responses need to be translated?
Conducting Culturally Competent Evaluation
|
When you evaluate communication programs, you form a set of assumptions about
what should happen, to whom, and with what results. Recognize that these
assumptions and expectations may vary, depending on the cultural norms and
values of your intended audiences.
You may need to vary your methods of gathering information and interpreting
results. Depending on the culture from which you are gathering information,
people may react in different ways:
-
They may think it is inappropriate to speak out in a group, such as a focus group, or to provide negative answers. (This does not mean that you should not use focus groups within these cultures; observance of nonverbal cues may be more revealing than oral communication.)
-
They may be reluctant to provide information to a person from a different culture or over the telephone.
-
They may lack familiarity with printed questionnaires or have a limited ability to read English.
Remember that the culture of the evaluator your program uses can inadvertently
affect the objectivity of your evaluation. When possible, try to use culturally
competent evaluators when you examine program activities. If your program cuts
across cultures and you adapt your evaluation methods to fit different groups,
you may find it difficult to compare results across groups. This type of
evaluation is more complicated, and if you plan to conduct one, enroll the help
of an expert evaluator.
|
5. Collect Data
Collect postprogram data. You should have collected baseline data during
planning in Stage 1, before your program began, to use for comparison with
postprogram data.
6. Process Data
Put the data into usable form for analysis. This may mean organizing the data to
give to professional evaluators or entering the data into an evaluation
software package.
7. Analyze the Data to Answer the Evaluation Questions
Use statistical techniques as appropriate to discover significant
relationships.Your program might consider involving university–based
evaluators, providing them with an opportunity for publication and your program
with expertise.
8. Write an Evaluation Report
A report outlining what you did and why you did it, as well as what worked and
what should be altered in the future, provides a solid base from which to plan
future evaluations.Your program evaluation report explains how your program was
effective in achieving its communication objectives and serves as a record of
what you learned from both your program’s achievements and shortcomings.
Be sure to include any questionnaires or other instruments in the report so
that you can find them later.
See Appendix A for a sample evaluation report. As
you prepare your report, you will need someone with appropriate statistical
expertise to analyze the outcome evaluation data. Also be sure to work closely
with your evaluators to interpret the data and develop recommendations based on
them.
Why?
Writing an evaluation report will bring your organization the following
additional benefits:
-
You will be able to apply what you've learned to future projects. Frequently, other programs are getting under way when evaluation of an earlier effort concludes, and program planners don’t have time to digest what has been learned and incorporate it into future projects. A program evaluation report helps to ensure that what has been learned will get careful consideration.
-
You will show your accountability to employers, partners, and
funding agencies. Your program’s evaluation report showcases the program’s accomplishments. Even if some aspects of the program need to be modified based on evaluation results, identifying problems and addressing them shows partners and funding agencies that you are focused on results and intend to get the most benefit from their time and money.
-
You will be able to give evidence of your program and
methods’ effectiveness. If you want other organizations to use your materials or program, you need to demonstrate their value. An evaluation report offers proof that the materials and your program were carefully developed and tested. This evidence will help you explain why your materials or program may be better than others, or what benefits an organization could gain from using its time and resources to implement your program.
-
You will provide a formal record that will help others. A comprehensive evaluation report captures the institutional memory of what was tried in the past and why, which partners had strong skills or experience in specific areas, and what problems were encountered. Everything you learned when evaluating your program will be helpful to you or others planning programs in the future.
Evaluation Report Helps CIS Promote Program Areas, Strengths
|
NCI’s CIS used an evaluation report, "Making a Difference," to
show its partners, the research community, NCI/CIS leadership, and the media
that its programs are effective. The document both quantified CIS results
(e.g., making 100,000 referrals a year to research studies, providing
information on breast cancer to 76,000 callers in 1996, providing information
that increased fruit and vegetable consumption among callers) and put a human
face on the calling public. Quotations from callers and leaders in the cancer
community illustrated the personal impact of the service on people’s
lives and health.
The report was written in lay language and used pullouts and simple charts to
explain statistics. Ideas for using the report with regional partners, the
media, and community leaders were included with the copies sent to each CIS
office. To maximize opportunities for using the report, CIS has also made it
available on computer disk and as a PowerPoint® slide presentation.
|
How?
Consider the Users
Before you write your evaluation, consider who will read or use it. Write your
report for that audience. As you did when planning your program components in
Stage 1, analyze your audiences for your report before you begin to compose. To
analyze your audience, ask yourself the following questions:
-
Who are the audiences for this evaluation report?
-
Public health program administrators
-
Evaluators, epidemiologists, researchers
-
Funding agencies
-
Policymakers
-
Partner organizations
-
Project staff
-
The public
-
The media
-
How much information will your audience want?
-
The complete report
-
An executive summary
-
Selected sections of the report
-
How will your audience use the information in your report?
-
To refine a program or policy
-
To evaluate your program’s performance
-
To inform others
-
To support advocacy efforts
-
To plan future programs
Consider the Format
Decide the most appropriate way to present information in the report to your
audience. Consider the following formats:
-
Concise, including hard-hitting findings and recommendations
-
General, including an overview written for the public at the ninth-grade level
-
Scientific, including a methodology section, detailed discussion, and references
-
Visual, including more charts and graphics than words
-
Case studies, including other storytelling methods
Selected Elements to Include
Depending on your chosen audience and format, include the following sections:
-
Program results/findings
-
Evaluation methods
-
Program chronology/history
-
Theoretical basis for program
-
Implications
-
Recommendations
-
Barriers, reasons for unmet objectives
9. Disseminate the Evaluation Report
Ask selected stakeholders and key individuals to review the evaluation report
before it is released so that they can identify concerns that might compromise
its impact. When the report is ready for release, consider developing a
dissemination strategy for the report, just as you did for your program
products, so the intended audiences you’ve chosen will read it.
Don’t go to the hard work of writing the report only to file it away.
Letting others know about the program results and continuing needs may prompt
them to share similar experiences, lessons, new ideas, or potential resources
that you could use to refine the program. In fact, feedback from those who have
read the evaluation report or learned about your findings through conference
presentations or journal coverage can be valuable for refining the program and
developing new programs.You may want to develop a formal mechanism for
obtaining feedback from peer or partner audiences. If you use
university–based evaluators, the mechanism may be their publication of
findings.
If appropriate, use the evaluation report to get recognition of the
program’s accomplishments. Health communication programs can enhance
their credibility with employers, funding agencies, partners, and the community
by receiving awards from groups that recognize health programs, such as the
American Medical Writers Association, the Society for Technical Communication,
the American Public Health Association, and the National Association of
Government Communicators. A variety of other opportunities exist, such as
topic–specific awards (e.g., awards for consumer information on
medications from the U.S. Food and Drug Administration) and awards for specific
types of products (e.g., the International Communication Association’s
awards for the top three papers of the year). Another way to get recognition is
to publish articles about the program in professional journals or give a
presentation or workshop at an organization meeting or conference.
The health communication planning process is circular. The end of Stage 4 is not
the end of the process but the step that takes you back to Stage 1. Review the
evaluation report and consider the following to help you identify areas of the
program that should be changed, deleted, or augmented:
-
Goals and objectives:
-
Have your goals and objectives shifted as you’ve conducted the program? If so, revise the original goals and objectives to meet the new situation.
-
Are there objectives the program is not meeting? Why? What are the barriers you’re encountering?
-
Has the program met all of your objectives, or does it seem not to be working at all? Consider ending the program.
-
Where additional effort may be needed:
-
Is there new health information that should be incorporated into the program’s messages or design?
-
Are there strategies or activities that did not succeed? Review why they didn't work and determine what can be done to correct any problems.
-
Implications of success:
-
Which objectives have been met, and by what successful activities?
-
Should successful communication activities be continued and strengthened because they appear to work well or should they be considered successful and completed?
-
Can successful communication activities be expanded to apply to other audiences or situations?
-
Costs and results of different activities:
-
What were the costs (including staff time) and results of different aspects of the program?
-
Do some activities appear to work as well as, but cost less than, others?
-
Accountability:
-
Is there evidence of program effectiveness and of a continued need to persuade your organization to continue the program?
-
Have you shared the results of your activities with the leadership of your organization?
-
Have you shared results with partners?
-
Do the assessment results show a need for new activities that would require partnerships with additional organizations?
Once you have answered the questions above and decided what needs to be done to
improve the program, use the planning guidelines in Stage 1 to help determine
new strategies, define expanded or different intended audiences, and
rewrite/revise your communication program plan to accommodate new approaches,
new tasks, and new timelines. Review information from the other stages as you
plan the next phase of program activities.
Myth: We can’t afford an evaluation.
Fact: Rarely does anyone have access to adequate
resources for an ideal health communication program, much less an ideal
evaluation. Nevertheless, including evaluation as a part of your work yields
the practical benefit of telling you how well your program is working and what
needs to be changed. With a little creative thinking, some form of useful
evaluation can be included in almost any budget.
Myth: Evaluation is too complicated. No one here knows how to do it.
Fact: Many sources of help are available for designing
an evaluation. Several pertinent texts are included in the selected readings at
the end of this section. If your organization does not have employees with the
necessary skills, find help at a nearby university or from someone related to
your program (e.g., a board member, a volunteer, or someone from a partner
organization). Also, contact an appropriate clearinghouse or Federal agency and
ask for evaluation reports on similar programs to use as models. If the program
has enough resources, hire a consultant with experience in health communication
evaluation. Contact other communication program managers for recommendations.
Myth: Evaluation takes too long.
Fact: Although large, complicated outcome evaluation
studies take time to design and analyze (and may require a sufficient time
lapse for changes in attitudes or behavior to become clear), other types of
evaluation can be conducted in a few weeks or months, or even as little as a
day. A well-planned evaluation can proceed in tandem with program development
and implementation activities. Often, evaluation seems excessively
time-consuming only because it is left until the end of the program.
Myth: Program evaluation is too risky. What if it shows our funding source
(or boss) that we haven’t succeeded?
Fact: A greater problem is having no results at all. A
well-designed evaluation will help you measure and understand the results
(e.g., if an attitude or a perception did not change, why not?). This
information can direct future initiatives and help the public health community
learn more about how to communicate effectively. The report should focus on
what you have learned from completing the program evaluation.
Myth: We affected only 30 percent of our intended audience. Our program is a
failure.
Fact: Affecting 30 percent of the intended audience is
a major accomplishment; it looks like a failure only if your program’s
objectives were set unrealistically high. Remember to report your results in
the context of what health communication programs can be expected to
accomplish. If you think the program has affected a smaller proportion of the
intended audience than you wanted, consult with experts (program planning,
communication, or behavioral) before setting objectives for future programs.
Myth: If our program is working, we should see results very soon.
Fact: Results will vary depending on the program, the
issue, and the intended audience. Don’t expect instant results; creating
and sustaining change in attitudes and particularly in behavior or behavioral
intentions often takes time and commitment. Your program may show shorter term,
activity-related results when you conduct your process evaluation; these
changes in knowledge, information seeking, and skills may occur sooner than
more complex behavioral changes.
Academy for Educational Development. (1995). A tool box for building health
communication capacity.
Washington, DC.
Agency for Toxic Substances and Disease Registry. (1994). Guidelines for
planning and evaluating environmental health education programs. Atlanta.
Center for Substance Abuse Prevention. (1998). Evaluating the results of
communication programs [Technical Assistance Bulletin].Washington, DC:
U.S. Government Printing Office.
Flay, B. R., & Cook, T. D. (1989). Three models for evaluating prevention
campaigns with a mass media component. In R. E. Rice & C. K. Atkin (Eds.), Public
communication campaigns (2nd ed.). Thousand Oaks, CA: Sage.
Flay, B. R., Kessler, R. C., & Utts, J. M. (1991). Evaluating media campaigns.
In S. L. Coyle, R. F. Boruch, & C. F. Turner (Eds.), Evaluating AIDS prevention
programs. Washington, DC: National Academy Press.
Morra, M. E. (Ed.). (1998). The impact and value of the Cancer Information
Service: A model for health communication. Journal of Health Communication,
3(3) Suppl.
Muraskin, L. D. (1993). Understanding evaluation: The way to better prevention
programs. Washington, DC: U.S. Department of Education.
Rice, R. E., & Atkin, C. K. (2000). Public communication campaigns (3rd
ed.). Thousand Oaks, CA: Sage.
Rossi, P. H., Freeman, H. E., & Lipsey, M.W. (1998). Evaluation: A systematic
approach (6th ed.). Thousand Oaks, CA: Sage.
Siegel, M., & Doner, L. (1998). Marketing public health: Strategies to promote
social change. Gaithersburg, MD: Aspen.
Windsor, R.W., Baranowski, T. B., Clark, N. C., & Cutter, G. C. (1994). Evaluation
of health promotion, health education and disease prevention programs (2nd
ed.). Mountain View, CA: Mayfield.
Back to Top
< Previous Section | Next Section > |