Communication Research Methods
Types of Communication Research Differences Between Qualitative and Quantitative Research Methods Qualitative Research Methods Quasi-Quantitative Research Methods: Pretesting Messages and Materials Quantitative Research Methods Additional Research Methods
Research into intended audiences’ culture, lifestyle, behaviors and
motivations, interests, and needs is a key component to a health communication
program’s success. This section describes communication research methods
commonly used throughout program planning. See the chart Types of Research and
Evaluation for more detail about research conducted in each of the stages of
health communication program planning.
Most programs use more than one research method. For example, conducting
exploratory focus groups with an intended audience at the start of program
planning can orient you to the types of approaches, messages, and channels that
are most likely to be successful with a particular group. In some cases, focus
groups might be augmented with in-depth interviews to learn more about intended
audience members’ motivations. Later, messages and materials might be
pretested approximate how an individual would encounter them in "real
life." Theater-style testing also approximates reality, using a simulated
television-viewing environment. Clearly, some methods are better suited to
specific purposes than others. Using multiple methods can help ensure that you
get an accurate picture of your intended audience members and their likely
responses to your program.
There are two basic types of research you might conduct with intended audiences:
qualitative and quantitative. You will use methods from one of these two types
depending upon what you want to learn. See the sidebar below, Qualitative
Versus Quantitative Methods, for common distinctions between qualitative and
quantitative research.
Qualitative Versus
Quantitative Methods |
Qualitative |
Quantitative |
Provides depth of understanding |
Measures level of occurrence |
Asks "Why?" |
Asks "How many?" and "How
often?" |
Studies motivations |
Studies actions |
Is subjective; probes individual
reactions to discover underlying motivations |
Is objective; asks questions without
revealing a point of view |
Enables discovery |
Provides proof |
Is exploratory |
Is definitive |
Allows insights into behavior and
trends |
Measures levels of actions and trends |
Interprets |
Describes |
Note.
From Methodological Review: A Handbook for Excellence in Focus Group
Research by M. Debus. Copyright 1988 by The Academy for
Educational Development, Washington, DC. Reprinted with permission. |
In this section, you will learn when to use each type of research, how to
conduct research with members of your intended audience, and how you can use
the data you collect to inform your project. Qualitative, quasi-quantitative,
and quantitative research methods are discussed separately.
Qualitative Research
Use qualitative research methods when:
-
You are planning a communication program and developing materials for it
-
When the goal of your research is to explore a topic or idea
-
When the goal of your research is to gain insights into an intended audience’s lifestyle, culture, motivations, behaviors, and preferences
Conduct qualitative research by:
-
Selecting a small group of people chosen for particular characteristics
-
Convening a discussion (i.e., a focus group or in-depth interview) or observing individuals’ behaviors in schools, malls, supermarkets, etc.
-
Keeping the discussion fairly unstructured, so that participants are free to make any response and are not required to choose from a list of possible responses
-
Choosing which question to ask next based on your participants’ previous responses
Qualitative research results cannot be:
-
Quantified or subjected to statistical analysis
-
Projected to the population from which the respondents were drawn because participants are not selected randomly (to be representative of the population as a whole) and because not all participants are asked precisely the same questions
Quantitative Research
Use quantitative research methods when:
-
You are planning a communication program (e.g., to measure the prevalence of a particular behavior) or assessing a program already in place
-
The goal of your research is measurement of particular variables Conduct quantitative research by:
-
Selecting a large group or groups of people
-
Using a structured questionnaire containing predominantly forced-choice or closed-ended questions
Quantitative research results can be:
-
Analyzed using statistical techniques
-
Considered representative of the population from which the respondents were drawn if each person in the population had an equal chance of being included
Use qualitative research methods during the following parts of your program:
-
Stage 1—to find out more about your intended audiences and to learn what the priorities and approach should be for trying to influence their awareness, knowledge, attitudes, intentions, or behaviors
-
Stage 2—to determine whether your materials communicate the intended messages effectively and persuasively
-
Stage 3—to understand why the program is or isn’t working as expected
-
Stage 4—to learn more about what worked and what didn’t, and why certain outcomes occurred
Types of Research and
Evaluation |
Stage |
Type of Research/Evaluation |
Benefits |
Stage 1 |
Consumer Research, Market Research |
|
Planning and Strategy Development |
Provides information on the problem,
intended audiences, and barriers to and opportunities for change |
Answers questions such as:
What dimensions of the health problem do we need to address?
How should the population be segmented and which intended audience should be
targeted?
What are the best ways of reaching the intended audience?
What benefits would be credible and appealing to the intended audience?
What images should be conveyed?
What barriers need to be overcome?
What actions can the intended audience take? |
Stage 2 |
Pretesting |
|
Developing and Pretesting Concepts,
Messages, and Materials |
Assesses reactions to proposed
messages or materials |
Tests and refines messages and
materials prior to production |
|
Pilot or Field Testing |
|
|
Assesses program activities in limited
areas and/or time periods |
Tests and refines activities prior to
full-scale implementation |
Stage 3 |
Process Evaluation |
|
Implementing the Program |
Documents and assesses implementation;
quantifies what was done; when, where, and how it was done; and who was reached |
Identifies areas for improvement as implementation proceeds
Documents progress of implementation |
Stage 4 |
Outcome Evaluation |
|
Assessing Effectiveness and Making
Refinements |
Measures whether, and to what extent,
a program or activity had the planned effects |
Documents the extent of the
campaign’s success or failure
Documents success to support replication
Determines any need to improve the existing program or future efforts |
|
Impact Evaluation |
|
|
Measures whether, and to what extent,
a program contributed to long-term goals |
Is not often used for health
communication activities (improving health status usually requires multifaceted
approaches—e.g., communication plus changes in health care service
delivery and relevant policies—and it is generally not possible to
isolate a particular communication program’s contribution to achieving
longer-term goals) |
Focus groups and in-depth interviews are the most common methods used in
qualitative communication research. However, there are many innovative methods,
some described here, that can help you learn about an audience. Because the
methodologies for each are very similar, they are discussed together here,
using instructions for focus groups as a guide.
About Focus Groups
Working from a discussion guide, a skilled moderator facilitates a 1- to 2-hour
discussion among 6 to 10 participants, which can be conducted either in person
or by telephone (ideally in person). The moderator keeps the session on track
while participants talk freely and spontaneously. As new topics related to the
material emerge, the moderator asks additional questions to learn more.
Common Uses
-
Developing a communication strategy:
-
Learning about feelings, motivators, and past experiences related to a health topic
-
Exploring the feasibility of various potential actions (from the intended audience’s viewpoint)
-
Identifying barriers to those actions
-
Exploring what benefits the intended-audience members find compelling and what results they expect from taking a particular action
-
Learning about the intended-audience’s use of settings, channels, and activities
-
Capturing the language used by the intended audience to discuss issues
-
Exploring reactions to message concepts (concept testing):
-
Identifying concepts that do or do not resonate and understanding why
-
Triggering the creative thinking of communication professionals
-
Illustrating to others how the intended audience thinks and talks about a health issue
-
Developing hypotheses (or broad issues) for quantitative studies and identifying the range of responses that should be included in closed-ended questionnaires
-
Providing insights into the results of quantitative studies by obtaining in-depth information from individuals typical of the intended audience to help understand why individuals responded in certain ways
Pros
-
Group interaction can help elicit in-depth thought and discussion.
-
Group interaction can help with brainstorming because respondents can build off one another's ideas.
-
Moderators have considerable opportunity to probe responses.
-
Focus groups yield richer data than surveys about the complexities of an intended audience’s thinking and behavior.
Cons
-
Findings are not generalizable to the population.
-
Focus groups can be labor intensive and expensive, especially if sessions are conducted in multiple locations.
-
Group responses do not necessarily reflect individuals’9 opinions because some individuals in the group may dominate the discussion or may influence others’ opinions.
-
Each person is limited to about 10 minutes of talking.
Working With Market Research
Professionals |
You may need to hire or contract with
two kinds of market research professionals as you design, conduct, and analyze
your concept and materials testing:
1. Someone to design the research and data instruments (e.g., questionnaires,
discussion guides, screeners), to analyze the results, and to prepare a report
2. A vendor to handle the fieldwork (i.e., recruiting and hosting focus groups;
administering telephone, mail, or in-person surveys)
|
Ideally, these professionals will have
a background in health communication or, if not, a background in marketing or
advertising research. You can get the best service from these professionals by:
-
Providing clear research objectives and appropriate background information, including
the creative brief.
-
Learning enough about common communication research methods to understand their strengths and limitations, so that you don’t ask for more than a given method can deliver (e.g., asking, "What percentage of the American public does that represent?" when a focus group study was conducted).
-
Letting market researchers’ expertise guide your selection of methods. Rather than saying, "We want to focus test this," explain your research objectives, timing, budgetary constraints, and any additional factors (such as the need for a publication to be tested with people from a wide range of cultures). Then let the experts propose methods to you and explain their rationale.
-
Being realistic about timelines, quantity of information, materials to be tested at one time, and the level of "proof" you need. Pretesting is diagnostic; it can provide guidance on what needs to be improved, but it can’t tell you how successful something will be. Other factors, such as the final production of your message, the number of people who see it, the frequency with which it is seen, and the presence of competing messages will all influence your message’s success.
-
Recognizing that there are inherent differences between testing advertising and other commercial communication materials versus testing health communication materials, even if the channel and activity (e.g., a television spot) are the same. Individuals trained in commercial concept development and copy testing will be able to draw on their commercial experience for selecting the appropriate methodology. However, they often have little experience assessing reactions to complex health messages; they are more familiar with assessing efforts to direct an existing behavior toward use of a particular product brand than with assessing efforts to completely change a behavior.
|
Sometimes, one individual or
organization can play both roles; at other times, you may have internal staff,
a consultant, or staff at a health communication firm to handle the first role
but contract externally for the second. The American Marketing
Association’s Green Book lists suppliers and services
geographically throughout the United States. Other sources include the
Marketing Research Association, the Association of Public Opinion Researchers,
the Qualitative Research Consultants Association, and faculty at university
departments of marketing, communication, health education, psychology, and
sociology. |
About
In-Depth Interviews
The process, benefits, and drawbacks of in-depth interviews are similar to those
of focus groups, except that the interviewer speaks with one person at a time.
In-depth interviews can take place at a central facility or at the
participant’s home or place of business. As with focus groups, when
individual interviews cannot be conducted in person, they can be conducted by
phone or over the Internet. Although these interviews are more time intensive,
one of their key benefits is that each respondent is isolated from other
respondents and therefore not influenced by what others say.
How to Design and Conduct a Focus Group or In-Depth Interview Study
To design and conduct a qualitative research study, complete the following
steps.
Plan the Study
Determine the following:
-
What you want to learn. Determine the objectives of your study at the outset, and then check to make sure that the moderator’s/interviewer’s guide includes lines of questioning that will provide the answers.You may also use the objectives to help analyze the results of the discussions and to organize the focus group or in-depth interview report.
-
When you need to have that information.
-
How you will apply what you learn. It is important to decide how you will use your focus group or in-depth interview results before you conduct your study.
-
Your budget.
-
Your criteria for who should participate.
Select people who are:
-
Typical of your intended audience (the same behavioral, demographic, and psychographic characteristics).You may want to conduct separate groups with "doers" (those who already engage in the desired behavior) and "nondoers" (those who do not) to help identify what actions the doers take, and why, so that those approaches can be explored with the nondoers.
-
Not experts. Exclude market researchers and advertising professionals (because of their familiarity with the methodology) and those who have, or might be perceived by other group members as having, expertise in the subject matter (e.g., exclude health professionals from focus groups when the topic is a health issue).
-
Relative newcomers to focus groups or interviews, so that their reactions will be spontaneous. This will help you avoid "professional" respondents (i.e., those who have participated in many previous focus groups or individual interviews before) who may lead or monopolize the discussion.
-
The number of groups you will convene.
-
Divide participants into different focus groups based on their gender, race, age, level of formal education, or any other variable likely to hinder their freedom of expression (e.g., teenage girls will be more comfortable discussing sexual activity if teenage boys or college-age women are not in the group).
-
Conduct a minimum of two focus groups with each intended audience segment (e.g., if you are conducting separate groups with men and women, you will need at least four groups–two with men, two with women). If intended audience perceptions vary or the audience feedback is unclear, you may want to conduct additional groups with each segment, especially if you revise the discussion guide to more fully explore unresolved issues.
-
If you are using in-depth interviews, conduct approximately 10 interviews per intended audience segment. If common themes do not emerge or the interviews, especially if you revise the interview guide in between interviews.
Choose the Location
You can convene focus group discussions or in-depth interviews in a variety of
ways:
-
Commercial focus group facilities can recruit participants for you (for both focus groups and interviews) and offer audio and video recording equipment as well as observation rooms with one-way mirrors for viewing. However, these facilities are usually available only in larger metropolitan areas.
-
Teleconference services can set up telephone focus groups for you. Most allow observers to listen without being heard, and some provide remote viewing programs to allow the moderator to see a list of participant names (with a symbol next to the one currently speaking) or notes sent in by a technician from observers listening to the call. Some teleconference services can recruit participants; with others, you will have to recruit participants or contract with a recruiter separately.
-
You can also conduct focus groups or in-depth interviews in meeting rooms at churches, office buildings, or other locations. If an observation room with a one-way mirror is not available, staff may still listen in through speakers hooked up in a nearby room or by audiotaping or videotaping the session. In some cases, one or two quiet observers may be allowed in the room to take notes.
See the sidebar Pros and Cons of Different Formats below for the advantages and
disadvantages of different formats for focus group and in-depth interview
research.
Pros and Cons of Different
Formats |
Format |
Pros |
Cons |
Face-to-Face |
|
|
Moderator/interviewer and participants
are in one room, usually around a table; observers (members of the research
team) are behind a one-way mirror |
Can assess body language If
videotaped, can share with others who couldn’t attend
Have participants’ undivided attention |
Responders lose some anonymity
Higher travel expenses due to multiple locales
Usually excludes people in rural areas or small towns |
Telephone |
|
|
Moderator/interviewer and participants
are on a conference call; observers listen in |
More convenient for participants and
observers
Can easily include people in rural areas, in small towns, and who are homebound
For professional groups, may be easier to gain participation because it is less
likely participants will know each other
Relative anonymity may result in more frank discussion of sensitive issues |
Can’t assess nonverbal reactions
More difficult to get reactions to visuals (although they can be sent ahead of
time)
Participants can be distracted by their surroundings |
Internet Chat Sessions |
|
|
Moderator and participants
"chat"; observers watch |
Complete record of session instantly
available
Relative anonymity may result in more frank discussion of sensitive issues |
Only useful for participants
comfortable with this mode of communication
Relatively slow pace limits topics that can be covered
No way to assess if participants meet recruitment criteria
Can’t assess body language or tone of voice
More difficult to get reaction to visuals |
Draft a Recruitment Screener
A recruitment screener is a short questionnaire that is administered to
potential participants, typically by telephone, to ensure that they meet the
criteria you developed during step 1. Ideally, the screener should help you
exclude participants who are already familiar with the specific subject of the
sessions, or who know each other. If participants know the subject in advance,
they may formulate ideas or may study to become more knowledgeable about the
subject than the typical intended audience member. If participants know each
other, they may speak less freely. See Appendix A
for a sample screener.
Recruit Participants
Recruit participants by telephone one to three weeks in advance of the sessions.
You can identify potential participants in different ways depending upon the
type of people you are seeking and the resources you have available. Identify
members of the public through focus group facility databases, by running an ad
in a local publication, by working with community organizations, or by using
the phone book (although the latter will be extremely time consuming if you
have stringent recruitment criteria). Identify professionals through a relevant
association or mailing list service or through a focus group facility’s
recruiting databases. Depending on your budget and internal resources, you may
choose to recruit in one of the following ways:
-
Use your own organization’s staff to recruit participants.
-
Hire a focus group facility or independent recruiter (to identify such facilities, consult a directory such as the American Marketing Association’s
Green Book or the Marketing Research Association’s Blue Book). If you have many facilities and recruiters to choose from, consider getting recommendations from local corporations or organizations that conduct qualitative research.
-
Get help from a university marketing research or advertising class.
-
Work through gatekeepers such as teachers (for students), health care providers (for patients, physicians, or nurses), religious institutions or community organizations (a small donation may encourage them to participate), and instructors of English as a second language.
Regardless of how the recruiting is done, ensure that the screener is followed
carefully so that only individuals who qualify for participation will be
included.
Contracting With Commercial Facilities |
Before you contract with a commercial
facility to conduct in-depth interviews or focus groups, prepare a
specification sheet detailing all of the services you need and, if you will be
asking the facility to recruit, a profile of your intended audience. Vendors
will use this information to estimate the cost of the project and to develop
bids.
Decide whom to approach by using the following checklist to determine which
vendors will fit your needs. Each vendor should provide the following
information:
-
Descriptions of past projects
-
Descriptions of, or a list of, clients
-
Location of facility (Is the facility conveniently located? Is parking available? Is the facility accessible by public transportation? If not, does vendor provide transportation assistance, such as taxi money or van service? What does this add to cost?)
-
Diagram of table/seating arrangement (What size and shape are tables? Rooms?)
-
Description of the size and features of observation rooms
-
Details about audio and video recording arrangements and costs
-
Details about food arrangements for participants and clients
-
Description of moderator services
-
Description of recruitment methods and geographic coverage
-
Recommendations for participant incentives
-
Reasonable rates for the services they will provide (ask for nonprofit rates, if appropriate)
|
Getting People to Show Up
To ensure that enough people show up, offer an incentive (usually money) and
recruit two or three more people than you actually need. If all show up, select
those who best fit the screening criteria, thank the extra participants, give
them the agreed-upon incentive, and dismiss them. Other ways to increase
participation include:
-
Scheduling sessions at times that are convenient for your potential participants (e.g., at lunch or after work)
-
Choosing a safe and convenient site with easy parking
-
Providing transportation (or reimbursement for agreed-upon transportation costs)
-
Arranging for childcare, if necessary
-
Letting them know you will be serving refreshments
Recruiting Patients and Their Families
Recruiting patients and their families requires special consideration. Contact
clinics, hospitals, or local HMOs for help and make adequate plans to ensure
that the participants and their family members are not inconvenienced or upset.
Some organizations may require institutional review board (IRB) approval of
your research. Gaining IRB approval is often a long process, so be sure you
check with the organization early in the planning stage of your study to find
out whether you will need IRB approval.
Recruiting for Telephone Interviews
If you are recruiting for in-depth interviews to be conducted on the telephone,
create a spreadsheet that includes spaces for the following information about
each potential participant:
-
Time zone in which the person is located
-
Date, time, and telephone number at which he or she should be called for the interview
-
Disposition of each call (e.g., scheduled an interview, no answer, busy, refused)
Develop a Moderator’s Guide
The moderator’s guide tells the moderator what information you want from
the groups and helps him or her keep the discussion on track and on time. It is
only a guide, however. During the focus groups, experienced moderators flow
with the conversation, asking questions in the prescribed language and sequence
when possible but sometimes deviating from the guide to avoid awkward
transitions or unnecessary back-and-forth between topics.
Before you draft the moderator's guide, answer the following questions:
-
What do we want to learn from the focus group?
-
How will we apply what we learn?
-
What tools (e.g., descriptive information, message concepts, or other draft creative work) will we need to provide?
Then, write questions for the guide that relate to the purposes you have
identified. Make most questions open-ended so that participants can provide
more in-depth responses than just "yes" or "no," but make sure
the questions are not leading. This will help you get answers that reflect
participants’ true feelings and not what they think you would like to
hear. The amount of time and depth of questions devoted to each issue should
reflect the value of the issue to the research. See Appendix
A for an example of a moderator’s guide.
Do not include questions for group discussion when you need individual
responses. However, you can have the moderator give self-administered
questionnaires to each participant to be completed prior to conducting a focus
group, or participants can be asked to individually rank items on
paper—such as potential actions, benefits, or message
concepts—during a group to obtain both individual and group reactions.
Working With Community Organizations to
Conduct Focus Groups |
Once you have identified potentially cooperative community groups (see the
following sidebar for a list of groups you might approach), contact an official
within each group (e.g., the president or program director) to request
cooperation. You may make these initial contacts by telephone and follow up
with a formal written request that includes the following:
-
Description of your agency or organization
-
Description of the material/topic to be discussed and its purpose
-
Details regarding the participants to be recruited and how you will protect their confidentiality
-
Outline of the activities involved
-
Incentives you are offering the organization and the participants
-
Detailed explanation of why the organization official should not reveal details about the nature of the discussion to participants in advance, unless the organization is to recruit participants
-
If and how you will share the information learned
Once you have an agreement with a community organization, decide how you will
recruit participants. One possibility is to conduct your research as part of
the group's regularly scheduled meeting.
The advantages of this approach are:
-
Little extra effort is required to recruit participants.
-
You may need to provide only minimal or no incentives.
-
The group’s regular and familiar meeting place can be used.
The disadvantages of this approach are:
-
You have little control over the number of people who will come or the composition of the group.
-
It is difficult to place a 1- to 2-hour focus group on the agenda of a regularly scheduled meeting.
-
Many organizations set their calendars months ahead of time (it may be difficult to schedule the focus group within a reasonable time frame).
An alternative is to recruit the group’s members to a special meeting.
Schedule this meeting immediately before or after the group’s regular
meeting to make it most convenient for the participants. If you use this
alternative, contact members in advance on behalf of the group and ask them to
participate. A person from the community group can also ask others to
participate. To ensure that participants attend and stay through the whole
meeting, let them know in advance that you will be providing refreshments
(assuming that you are doing so).
The advantages of recruiting participants to a special meeting are:
-
It provides an opportunity to screen participants on relevant characteristics and to eliminate market researchers or other experts who should not participate.
-
It helps reduce participant fatigue because the entire meeting will be devoted to your research.
If you recruit participants yourself, you will have more control over what the
participants are told about the focus group, and you will be able to screen
potential participants to make certain that they fit special intended audience
characteristics. However, recruiting participants takes a significant amount of
time, and it is possible that people would be more likely to participate if
asked by someone whom they know.
If the member organization recruits participants, it is essential that you
provide the recruiter with detailed instructions for carrying out the task.
These instructions must include a written description of the focus group topic,
which should be read to potential participants verbatim, and a questionnaire to
screen participants on relevant criteria.
|
Conduct the Focus Groups
Focus groups typically begin with the moderator welcoming participants and
briefing them on the process (e.g., all opinions welcome—there are no
right or wrong answers; the presence of audio- and videotaping and observers;
the importance of speaking one at a time; confidentiality). Participants
introduce themselves to the group by first name, usually including some
information relevant to the topic of discussion (e.g., number of years with
glaucoma, amount/type of insulin used each day). Next, the moderator asks a few
simple "ice-breaker" questions to help participants get used to the
group process and to reduce participant anxiety. This also helps the moderator
develop rapport with the participants.
Examples of Community Organizations to
Contact for Help Recruiting Participants |
American Legion
B’nai B’rith Women
Business groups
Jaycees
Junior League
Knights of Columbus
League of Women Voters
Lions Club
NAACP
National Council of Negro Women
National Urban League
Parent Teacher Associations
Religious organizations
Rotary Club
Schools
Senior citizen centers
Unions
Veterans of Foreign Wars
YWCA/YMCA |
Continuing to follow the moderator’s guide, the moderator manages the
group and ensures that all topics are covered without overtly directing the
discussion. Participants are encouraged to express their views and even
disagree with each other about the discussion topics. The moderator does not
simply accept what participants say but probes to learn more about
participants’ underlying thinking and attitudes. The moderator also seeks
out opinions from all participants so that all are heard and a few do not
dominate the discussion.
Near the end of the discussion, the moderator will often give participants an
activity or simply excuse him- or herself from the room for a moment to check
with the observers and obtain any additional questions. Alternatively or
additionally, notes can be sent in to the moderator while the group is in
process if the observers would like different questions asked or other changes
made to the group.
One advantage of focus group methodology is that the moderator’s guide,
and any materials presented, can be revised between groups if necessary.
Analyze Results
The easiest and most thorough way to analyze focus groups is by reviewing
transcripts, although groups can also be analyzed (albeit less thoroughly) by
reviewing notes taken during the discussion. In many analyses, the goal is to
look for general trends and agreement on issues. At the same time, it is
important to note divergent opinions. Don’t ignore individual comments
that raise interesting ideas or concerns such as lack of cultural sensitivity
or difficulty in comprehension. In some instances, the goal is to capture the
range of opinions about an issue, rather than to look for evidence of agreement
or consensus.
Avoid counting or quantifying types of responses (e.g., "75 percent of
participants preferred concept A"). Attempting to quantify the
results—or suggesting in other ways that they represent the opinions of
the intended audience as a whole—is inappropriate for qualitative
research.
Some commonly used communication research methods, such as central-location
intercept interviews and theater tests, are best termed quasi-quantitative.
While these methods are used in situations in which the goal is measurement and
typically involve a questionnaire with mostly forced-choice questions, the
results cannot be projected to the population as a whole (as with true
quantitative surveys) because of the way in which participants are selected.
For centrallocation intercept interviews, the only people who have a chance to
participate are those who go to the location where the interviews are being
held and who go there during the times they are conducted; this is not a truly
representative sample of the intended audience. For theater tests, the only
people who have a chance to participate are those who are recruited for the
test, and recruitment does not follow a truly representative sampling design.
Quasi-quantitative methods are most often used during Stage 2 to pretest
messages and materials. If your intended audience is geographically dispersed
or it is difficult for them to get to a central facility, you can use telephone
interviews and send participants any materials in advance. This type of pretest
typically resembles an in-depth interviewing project in price and number of
interviews, although there may be more closed-ended questions and the question
sequence may be adhered to more closely.
The Moderator's Role |
The moderator does not need to be an expert in the subject of your research but
must have experience facilitating group discussions. A good moderator builds
rapport and trust and probes, without reacting to or influencing, participants'
opinions. The moderator must be able to lead the discussion and not be led by
the group. He or she must emphasize that there are no right or wrong answers to
the questions that are posed. A good moderator understands the process of
eliciting comments, keeps the discussion on track, and figures out other ways
of approaching a topic if the first way is unproductive. Good moderators
understand what you are looking for and what you need to do with the
information, and they are able to probe and guide the discussion accordingly.
Go over the guide with the moderator to point out any topics or concerns you
want emphasized or discussed in depth. By the end of the focus group or
interview, the moderator should ensure that all agreed-upon topics are covered
sufficiently.
If your organization plans to conduct focus groups regularly, consider hiring a
skilled, experienced moderator to train your internal staff to moderate focus
groups. Use local advertising agencies, the American Marketing Association's
Focus Group Directory, or the Qualitative Research Consultants Association to
identify a good moderator.
|
Using Software to Analyze Qualitative
Data |
If you have conducted a large number of focus groups or interviews with many
intended audience subsets-and are interested in analyzing results by different
cultural groups, age groups, or economic groups within the overall respondent
population-you may want to use computer software to do a comparative analysis
of your results. If your results are from only a few groups, however, computer
analysis will be too time consuming to benefit your program. Be sure to
supplement computer analysis with "human" analysis, since the strength of
qualitative research is that it can uncover unexpected human reactions that
software cannot properly capture or weigh.
Before you decide to use computer software to analyze your qualitative data,
assess the following advantages and drawbacks of this type of analysis.
Advantages
-
The ability to highlight sections of the transcript that are important to the project and to eliminate "noise" or sections of the transcript that are not important to answering your research questions (of course, a wordprocessor's cut-and-paste functions can also accomplish this).
-
The ability to quickly access and compare information on one topic or questions across several transcripts.
Disadvantages
-
You must tape all interviews
-
It costs time and money to transcribe focus group sessions and in-depth interviews.
-
Coding the transcripts is both time intensive and expensive. If more than one person will be coding, you will need to train the coders, periodically assess intercoder reliability, and retrain as necessary.
-
While all comments on a particular topic can be gathered, they are taken out of context in the process. Sarcasm and other tonal characterizations may be lost.
-
Analysis by software will help you organize information, but will also produce overwhelming amounts of paper.
Steps
If you decide to use software to analyze your qualitative data, follow these
steps:
-
Transcribe the focus group discussions or interviews into an electronic format that can be read by the analysis software. If you decide to use qualitative data analysis software, check the requirements of the package you choose.
-
Develop and apply the codes you will use to organize the information in the transcripts. A code is a word or number that represents a research objective, research question, theory, or idea you are testing. The codes you develop will be unique to your research. For example, if your first research question is to find out how many vaccine shots a parent is willing to give his or her infant, you might code all lines of your transcript that include comments on that question as "A,"' signifying that they correspond to your first research question. Applying the code requires that you review the transcripts and use a mouse to indicate on the computer screen all the lines of the transcript that pertain to that code word.
-
Generate reports using the codes you developed. To do this, you instruct the program to search for a particular code or a combination of codes. The program then presents a single report showing all the lines of the transcript bearing the requested codes.
For a complete review of qualitative data analysis software, refer to Computer
Programs for Qualitative Data Analysis: A Software Sourcebook, by Eben
A. Weitzman and Matthew B. Miles, 1995, Thousand Oaks, CA: Sage. A new edition
of this book is due out in 2002.
|
Central-Location Intercept Interviews
Central-location intercept interviews consist of stationing interviewers at a
point frequented by individuals from your intended audience and asking the
individuals to participate in a study. If they agree, they are asked specific
screening questions to see whether they fit the study criteria. If so, the
interviewer takes them to the interviewing station (a quiet spot at a shopping
mall or other site), shows the pretest materials, and then administers the
pretest questionnaire.
For intercept interviews to be effective, you must obtain results from a minimum
of 60 to 100 respondents from each intended audience segment you want to test.
Pros
-
You can connect with harder-to-reach respondents in locations convenient and comfortable for them.
-
The interviews can be conducted quickly. (The interview should be no longer than 15 to 20 minutes.)
-
The interviews are a cost-effective means of gathering data in a relatively short time.
-
If you choose an appropriate location, you will increase your chance to interview respondents who are among your intended audience.
Cons
-
You must train interviewers.
-
Your results are not representative or generalizable.
-
Intercept interviews are not appropriate for sensitive issues or potentially threatening questions.
-
Intercept interviews do not allow you to probe easily for additional information.
Central-location intercept interviews should not be used if respondents must be
interviewed in depth or on emotional or sensitive subjects. The intercept
approach also may not be suitable if respondents are likely to be resistant to
being interviewed on the spot. In cases in which central-location intercepts
will not work well, schedule interviews with respondents instead.
Estimated Costs of Focus Groups and In-Depth Interviews, 2002
|
These estimated costs are included to suggest how you should budget for focus
groups and in-depth interviews if you are using commercial research firms. Your
actual costs will vary depending upon your geographic location, the intended
audience to be recruited, and the amount of time donated by staff, companies,
and participants. Be sure you do not jeopardize the quality of your results
with a budget that is too small.
The focus group cost estimate in the table assumes that you conduct two groups,
each composed of 10 members of the general public. A group size of 6 to 8 is
sometimes preferred, because it is easier to engage all participants in the
conversation. This estimate is also based on the assumption that each group
session is 2 hours long, is conducted in English, and includes audiotapes.
Staff travel, food for participants, and videotaping are not included.
The in-depth interview estimate assumes a total of ten 30-minute interviews
conducted in English and audiotaped.
|
|
2 Focus Groups
|
10 In-Depth Interviews
|
Develop screener |
$ 920 - 1,380 |
$ 920 - 1,380 |
Develop discussion guide |
$ 920 - 1,840 |
$ 920 - 1,840 |
Recruit |
$1,725 - 2,875 |
$ 860 - 1,725 |
Rent facility |
$ 705 - 1,380 |
$ 0 - 1,150 |
Provide respondent incentives |
$ 690 - 1,380 |
$ 0 - 575 |
Compensate moderator or interviewer to
conduct |
$1,000 - 2,200 |
$ 500 - 1,000 |
Analyze and report results |
$1,840 - 2,760 |
$1,840 - 2,760 |
Total |
$7,800 - 13,815* |
$5,040 - 10,430** |
*Add an additional $400-600 for transcribing focus group audiotapes (optional).
**Add an additional $300-400 for transcribing in-depth interview audiotapes
(optional).
|
Estimated Costs of Central-Location Intercept Interviews, 2002
|
The central-location intercept interview cost estimate below assumes that you
question 100 respondents from the general population for 15 to 20 minutes each.
|
Develop questionnaire |
$ 460 - 4,025 |
Produce questionnaire, schedule
facility, phones, and mail |
$ 400 - 600 |
Screen/conduct interviews |
$1,955 - 4,025 |
Provide respondent incentives |
$ 345 - 575 |
Code/enter data/tabulate |
$ 805 - 1,495 |
Analyze and report results |
$1,495 - 4,025 |
Total |
$5,460 - 14,745 |
Questionnaire
Contents
Unlike focus groups or in-depth interviews, the questionnaire used in
central-location intercept pretesting is highly structured and contains
primarily multiple choice or closedended questions to permit quick response.
Open–ended questions, which allow free–flowing answers, should be
kept to a minimum because they take too much time for the respondent to answer
and for the interviewer to record. Questions that assess the intended
audience's comprehension and perceptions of the pretest materials form the core
of the questionnaire. A few additional questions, tailored to the specific item
or items being tested ("Do you prefer this picture—or this
one?"), may also be included to meet your program planners' particular
needs. The questionnaire should be pretested before it is used in the field.
See Appendix A
for a sample questionnaire.
Interview Setup
A number of market research companies throughout the country conduct
central–location intercept interviews in shopping malls.You can also
conduct these interviews in clinic waiting rooms, religious institutions,
Social Security offices, schools, work sites, train stations, and other
locations frequented by members of your intended audience. Be sure to obtain
permission well in advance of the time you want to set up interviewing stations
in these locations.
If you are using a market research company to conduct the interviews, you will
need to provide screening criteria, test materials, and the questionnaire. In
some cases, market research companies have offices in shopping malls, and you
can watch the testing through a one-way mirror.
Participant Recruitment
If you or someone in your organization is recruiting the participants, you will
need to develop a script and provide training in approaching members of the
intended audience. For example, if you are recruiting participants in a clinic
waiting room, the interviewer should be familiar with the screening criteria
(e.g., women under 60 years of age) and approach only those people who appear
to fit the criteria. When, after screening, individuals do not qualify to
participate, the interviewer should thank them for their time and indicate that
this study is not the right fit for them but that their willingness to
participate is appreciated. If they do qualify, the interviewer can bring them
to a designated location (e.g., another room or corner of the waiting room) and
proceed with the study.
University and college departments of marketing, communication, or health
education may be able to provide interviewer training or trained student
interviewers. Pretesting is an excellent real–world project for a faculty
member to adopt as a class project or for a master's student to adopt as a
thesis project. However, this approach may mean that it takes longer to
accomplish the research, and you could compromise the quality of the results if
the individuals are not experienced in this type of research.
Theater-Style Tests
Theater testing is often used in the commercial arena to test advertisements for
products and services. Theater testing can also be used to test the
effectiveness of PSAs. In this methodology, participants are invited to a
central location to respond to a pilot for a new television show; in the midst
of viewing the TV pilot, they are shown your PSA or advertisement along with
other ads. Participants complete a questionnaire following the presentation,
first answering questions about the show and then answering questions about how
effectively your message was communicated to them and what their overall
reactions were.
Common Uses
Theater–style tests are most commonly used to test TV advertisements and
PSAs. For theater–style tests to be effective, you must obtain results
from 50 to 100 respondents from each segment you want to test.
Pros
-
You can obtain responses from a large number of respondents at the same time.
-
Theater–style tests more closely replicate what goes on in someone's home when they are watching TV, so you can accurately judge people's reactions to your message.
-
Theater–style tests can be cost–effective if you use donated facilities and equipment.
Cons
-
It is expensive to rent a facility and equipment (if necessary).
-
Your results are not representative of the general population.
General Format
Individuals typical of your intended audience are invited to a conveniently
located meeting room. The room should be set up for screening a television
program. Participants should not be told the real purpose of the session, only
that their reactions to a television program are being sought.
At the session, participants watch a television program. The program can be any
entertaining, nonhealth video approximately 15 to 30 minutes in length. The
videotape is interrupted about halfway through by a sequence of four
commercials. Your message should be inserted between the second and third
commercials. See Appendix A for a description of
how to create a roughcut video for theater-testing your message.
At the end of the program, participants receive a questionnaire and answer
questions designed to gauge their reactions, first to the program and then to
the advertisements. Finally, your ad is played again and participants complete
several questions about your ad. The majority of these questions should be
closed-ended to enable an easy and accurate summary of participant responses.
In more sophisticated theater testing, participants use automated intended
audience response systems to answer questions. Participants are provided with a
small device that has response keys. Once a question is asked, they push a key
to respond and the data are automatically tabulated.You have instant access to
the numbers using this system. In addition, questions can be instantly added or
deleted from the questionnaire based on the previous responses. Using an
automated system is much more costly than using a standard paper-and-pencil
questionnaire.
Other Media You Can Test
This methodology can also be used to test videos by asking participants to view
a series of videos in which yours has been included. Examples of videos that
might be tested include a 15- to 30-minute breast cancer awareness video that
will be played in a clinic or a "how-to" video on administering epinephrine.
These testing sessions will, of course, last longer than those testing ads.
Participants evaluate the videos as described above.
Print advertisements can also be tested using a variation of this methodology.
Several ads, including yours, are inserted into a magazine. Participants are
given an adequate amount of time to read through the article, which includes
your ad and others. After reading the article, participants receive a
questionnaire and answer questions designed to gauge their reactions, first to
the article and then to the advertisements. Finally, your ad is displayed alone
and participants complete several additional questions.
Designing and Conducting a Theater-Style Pretest
The process for conducting a theater–style test includes the following
steps:
-
Planning the pretest
-
Developing the questionnaire
-
Recruiting respondents
-
Preparing for the pretest
-
Conducting the pretest
-
Analyzing the pretest
You may find step 2 also useful for central–location intercept interviews.
Plan the Pretest
Determine:
-
The purpose of the study (e.g., what do you want to learn?)
-
When you need to produce results
-
What your budget is
-
The type and number of people who should participate in the pretest
-
The locations where the pretest will be conducted
To conduct theater testing, you must have a large enough space to accommodate
all of your participants at the same time. You must also ensure that you have
several video monitors so that all participants can adequately view the
program. Space constraints may be overcome by seeking out low-cost facilities
such as a school auditorium or church hall.You may be able to borrow the
audiovisual equipment from these facilities as well. You can also rent space,
such as a hotel ballroom, if you want to test a large number of people. Hotels
often rent audiovisual equipment as well. Reserve facilities and equipment well
in advance of your pretest.
Develop the Questionnaire
To gather useful information from the pretest, you must carefully construct the
questionnaire. See the sidebar Components Used in Most Questionnaires on the
next page for general guidelines. Once you have written your questionnaire, be
sure to test and revise it before you use it with a large number of
respondents.
Recruit Respondents
Participants may be recruited through a market research facility or through
local community organizations. In either case, you will need to provide an
incentive for participants. If using a market research facility, you will also
incur recruiting expenses. If you are working with a community organization,
you may choose to make a donation.
ESTIMATED COSTS OF THEATER TESTING, 2002*
|
Develop questionnaire |
$ 460 – 2,760 |
Produce questionnaire |
$ 400 – 600 |
Recruit |
$ 0 – 5,750 |
Rent audiovisual equipment |
$ 0 – 2,300 |
Conduct theater test |
$ 0 – 920 |
Provide respondent incentives |
$ 285 – 2,875 |
Code/enter data/tabulate |
$ 920 – 3,680 |
Analyze and report results |
$1,840 – 3,680 |
Total |
$3,905 – $22,565+ |
* Estimates assume 50
participants. They exclude facility rental costs. The costs of large
facilities (e.g., hotel ballrooms) vary widely by geographic location. Check
with local facilities for approximate costs. |
Components Used in Most Questionnaires
|
Although the following components should be used in most questionnaires, these
descriptions are specific to a theater-style pretest.
Program Questions
Program questions elicit general intended audience reactions to the program
viewed. Use the questions in Part I of the sample Theater–Style Pretest
Questionnaire in Appendix A as the first page of
your questionnaire.
Recall, Main Idea, and General Reaction Questions
These three standard questions that assess your message’s ability to
attract attention, convey its main point, and create a positive response should
appear on the second page of your questionnaire. Use the questions in Part II
of the sample questionnaire in Appendix A.
These standard questions should be incorporated into the pretest for several
reasons. The questions address the most important indicators of a
message’s potential effectiveness: 1) whether it attracts intended
audience attention (recall), 2) whether it communicates your main point (main
idea), and 3) what respondents thought and felt when they viewed the ad
(reaction).
If you test many ads and always use the same questions, you can develop a
database of results to allow you to assess the relative strength of various
ads.
Specific intended audience Reaction Questions
These provide answers to specific questions you have about your message.
Develop questions that address specific concerns you have about your message.
For example, suppose your message asks viewers to call a toll-free number for
more information. You may want to include a question that asks, "What
action, if any, does the message ask you to take?" A related question may
be, "Did the telephone number appear on the screen long enough for you to
write it down (or remember it)?"
It is best to develop one or more questions addressing each characteristic of
your message. The list below includes various characteristics commonly found in
messages. Note the characteristics that apply to your test message and then
develop questions that focus on those characteristics.
-
Use of music (with or without lyrics)
-
Use of famous spokesperson
-
Use of telephone number/Web site address
-
Request for a particular action
-
Instructions for adopting a specific health behavior
-
Presentation of technical or medical information
-
Presentation of new information
-
Promotion of a sponsoring organization or event
-
People intended to be typical of the intended audience
-
Use of a voice-over announcer
-
Presentation of controversial or unpleasant information
See Part III of the sample in Appendix A
for examples of questions you can use for each message characteristic listed
above. These questions are just examples and should be adapted to your needs.
Remember that the objective of pretesting is to uncover any problems with your
ad before final production.
Demographic Questions
These questions help to record the characteristics of the participants (e.g.,
their sex, age, level of education, health status).
Once you have written your questionnaire, be sure to test it before you use it
with a large number of respondents. After any necessary revisions, you are
ready to make copies for the pretest participants. You will need a cover page
that instructs participants not to open their questionnaires until they are
asked to do so by the meeting host. Place a cover sheet between each part of
the questionnaire and instruct participants not to continue until they are
asked to do so by the meeting host.
|
Prepare for the Pretest
Rehearse the testing session at your own office to anticipate and avoid any
problems before actual pretesting. Review the following questions to be sure
that your session will go as smoothly as possible:
-
Is the pretesting videotape ready for use? Are the video and audio portions clear?
-
Is the videotape equipment—recorder and television monitors—functioning properly?
-
Is the facility set up? Is the room reserved? Are there enough chairs? Are extra chairs available in case more people show up than you expect? Do you need another monitor so that everyone will be able to see the program? Is the heat or air conditioning working properly? Do you know where the light switches are? If a microphone is needed, is it set up and functioning properly?
-
Have you made all the necessary staffing arrangements? Are your assistants coming to the session? Do they have transportation and directions for getting there?
-
Have you made enough copies of your pretest questionnaire (including some extras)? Is each questionnaire complete (with no pages missing)? Do you have pencils for participants? Will they need clipboards or pads?
-
Has participant recruitment taken place as scheduled? Did you call and remind participants to attend? Do they have transportation and directions?
-
Has the moderator rehearsed?
Conduct the Pretest
The procedures to follow during the pretest session are relatively simple. The
keys to a successful testing session are to:
-
Be friendly and courteous to participants from the moment they arrive until they leave (remember to say "thank you").
-
Keep calm and cool-headed throughout the session.
-
Anticipate problems in advance (conducting a rehearsal to make sure that both equipment and timing work is a good idea).
The test session should take no more than 1 hour and 15 minutes if you are
organized and well prepared. Follow the steps below to conduct your test:
-
Encourage participants to take a seat as they arrive. Close the doors no later than 10 minutes after the
scheduled starting time.
-
When everyone is seated, introduce yourself by your name only (assuming you are the host). Do not
tell participants the name of your organization during the session because it might bias their responses to your test ad.
-
Thank participants for coming and assure them that the evening should be enjoyable and that they will have
a chance to give their views to the producers of "new" television program material. Discourage participants from talking to one another during the session. Tell them you are interested in their own individual views and that there are no right or wrong answers. Also, encourage them to write their answers clearly in the space provided on the questionnaire.
-
After your introductory remarks, have your assistants hand out the questionnaires (see
Appendix A for a sample), pencils, and clipboards (if needed). Instruct the participants not to open the questionnaire until you ask them to do so. Turn on the video recorder and monitor to begin the test session.
-
Be attentive and watch for any problems with the sound or picture on the monitor. Be sure that the equipment is functioning properly throughout the program.
-
Be prepared to stop the recorder when the television program has ended. Introduce the questions, and thank the participants for their help so far. Ask them to open their questionnaires and complete the questions on the first page.
-
When the participants have finished Part I of the questionnaire, tell them that you would like to gather their reactions to the messages/PSAs that were shown during the program. Have them turn to Part II and instruct them to fill out the questions about the messages.When they have completed these questions, tell them that you want to obtain their reactions to one particular message in the series of messages they viewed.
-
Start the video. (Note: To avoid an awkward pause in the session’s pace, be sure there is not too much lead tape before the message starts.) After your PSA/ad has been replayed, ask participants to turn to the next page of the questionnaire and complete the remaining questions. Encourage them to answer every question and to avoid giving more than one answer, except when this option is indicated on the questionnaire.
-
Circulate through the room to monitor progress and to be sure participants are not discussing their responses. Collect the questionnaires as participants finish.
-
Thank participants for their cooperation. If you have incentives or token gifts, distribute them to participants as they leave. If you have provided a donation to a group in lieu of payment to participants, mention that you hope the group will find the donation helpful.
Analyze the Pretest
Analyze the questionnaires in two steps. First, tabulate or count how many
participants gave each possible response to each question. Next, look for
patterns in the responses to both closed– and open–ended questions.
The patterns will help you draw conclusions about the effectiveness of your
message. See Appendix A for detailed instructions
on tabulating closed- and open-ended questions and for a table of average
ratings to help interpret standard question responses.
At this point, look at the overall results:
-
What did you learn from the pretest?
-
Did your message receive a favorable and appropriate intended audience reaction?
-
Did your message fulfill its communication objectives?
-
What are your message's strengths? Weaknesses?
-
Did answers to any particular question stand out?
Use your answers to these questions to decide whether your message is both
effective and appropriate and whether you need to revise your message prior to
program implementation.
Diaries and Activity Logs
Other tools you can use to evaluate your program are diaries and activity logs.
If you plan to use these tools to gauge the quality of program planning or
execution, be sure to start keeping the diaries and activity logs as soon as
you begin program planning. For each activity, request information in a
specific format from program managers or participants. This information may
cover issues such as the quality of program components or track how your
intended audience uses the components.
Common Uses
-
Track program implementation
-
Assess effectiveness of program implementation
-
Monitor whether planned activities are being conducted on schedule and within budget
-
Learn what questions program participants had
-
Learn what technical assistance was needed by program staff
-
Track intended audience exposure to program components
Pros
-
Allow respondents flexibility in their responses
-
Enable researchers to observe behavior over time, rather than only once
Cons
-
Require considerable effort on respondents' parts (for this reason, consider offering incentives for completion of the diaries/logs)
-
Require staff able to code voluminous and challenging incoming data
-
Are not appropriate for respondents who have low literacy skills or who have poor writing skills or penmanship
Instituting Diary/Activity Log Use
Steps in instituting the keeping of diaries and activity logs are:
-
Planning the study
-
Identifying who will participate
-
Developing and pretesting the form you will use
-
Collecting data
-
Analyzing results
Follow the steps below to institute the keeping of diaries and activity logs.
Plan the Study
Determine the following:
-
What you want to learn
-
How much information you need to collect
-
How you will apply what you learn
-
When you need the information
-
What your budget is
-
Your criteria for who should participate
Identify Who Will Participate
The sample you select depends on the goals of your study. If you are focusing
on program implementation, you will want the diaries/logs to be completed by
program staff (e.g., nurses in a clinic). In this case, you may have some
control over the quality of responses you receive.
When planning the study, you must obtain permission from a manager or supervisor
on site for staff to complete the diaries/logs during the study.You should
provide an estimate of the amount of time and effort participation will entail
(e.g., 15 minutes per day, 1 hour per day). Share drafts of the diaries/logs
and get input from the supervisor prior to the study. This will help to ensure
cooperation during the study.
Before the study begins, you should train staff to complete the diaries/logs.
Even if it seems obvious to you, it is essential that you explain exactly what
you want recorded in the diary/log. (See the sample log in
Appendix A.) In addition, you should provide detailed, written
instructions for future reference. These instructions can be used in lieu of
training if you cannot physically get to the study site.
If you are focusing on participant experience with a program, you will want the
diaries/logs to be completed by people who were exposed to program components.
In this case, you will have much less control over the quality and quantity of
responses.
Obtaining cooperation from participants may also be more difficult in this
situation. For example, people attending an educational program on nutrition
might be recruited to complete a diary of what they eat for a week and send it
back to the researchers.You will likely need to provide an incentive (e.g., a
gift certificate upon receipt of the completed diary), and you may also need to
remind participants to send back the diaries at the end of the study period.
Develop and Pretest the Form You Will Use
Once you have identified what you want to learn and who will complete the
diaries/logs, you must create a user-friendly document to collect the necessary
information.
Create questions. Write questions that are specific to your study
objectives.
Examples of the types of information you might collect include:
-
For a toll-free hotline activity log:
-
Date?
-
What is the gender of the caller?
-
Where did the caller get the number?
-
Did the caller request any educational materials?
-
What questions did the caller ask?
-
For a health education program diary:
-
Date?
-
Which module was used?
-
Which of the suggested activities were completed?
-
How long did it take to complete the suggested activities?
-
Would you use these activities again?
-
Why or why not?
-
For a clinic observation activity log:
-
Date?
-
How many people passed by the display containing the new breast cancer information?
-
How many stopped to look at the information?
-
How many took a brochure?
Pretest the diary/log. Once you have created the draft diary/log, you
must pretest it with individuals who represent your intended audience. Describe
the scenario for them before the pretest. For example, in the case of a
hotline, you might say, "You are an operator on a hotline. People will be
calling in, and you will need to fill out this activity log as you complete
each call." Sit together with them and ask them to read each question aloud
and tell you what they think they are supposed to do. Do not correct them if
they do not say what you intended. This probably means that your diary/log is
unclear. Continue through the entire diary/log and then ask them if there was
anything that they found confusing or unclear. Pretest the diary/log with
everyone as planned before you make any changes.
Revise the diary/log. Revise questions that people found confusing
during the pretest. If a question was confusing only to one person, use your
judgment to decide whether to change the question. Ask yourself whether there
is something you can easily fix that would have helped that one person
understand the question (e.g., providing an example). If so, you may be able to
make a simple change or addition to clarify the question. Also consider whether
this respondent found many of the questions confusing while other respondents
had no problem with them. If this is the case, you may not want to make
changes.You will have to decide on a case-by-case basis. If you make
substantial changes to the diary/log, you should conduct another pretest before
finalizing the form.
Collect Data
Produce diaries/logs in sufficient quantities so that respondents have several
extra forms in case they make errors or need more space. Deliver the
diaries/logs to respondents, along with detailed written instructions, prior to
training (if applicable) or at least 1 week before the study begins. If you are
asking program participants rather than program staff to complete diaries/logs
for you, you will have to distribute the materials on site. Give respondents a
fixed time frame to complete the diaries/logs and provide them with a means
(envelope/postage) to return the data to you. If your study is longer than a
week or two, you may want to ask respondents to ship the first week of data to
you so that you can review the logs for accuracy and completeness and even
begin to tally some of the information.
Analyze Results
In the planning phase, you determined what you wanted to learn from the study.
Now you will have the chance to look through the diaries/logs to answer these
questions. Diaries generally contain qualitative information (e.g., how food
choices were made that day, evaluation of programs completed). Activity logs
may contain several types of information—quantitative information you can
tabulate easily (e.g., how many people called a hotline each day, whether
people picked up a brochure) as well as qualitative information (e.g., reasons
that students liked or participated in an activity).
Analyzing qualitative responses. The best way to analyze qualitative
information is to read through the information, searching for similarities and
differences between diaries. You will need to consider all of the questions
that you determined were important in the planning phase. Once you have
reviewed several diaries, you should be able to pull out general themes or
patterns from the information. The best way to analyze these themes is to
develop categories for the responses. For example, if you want to know why
teachers thought their students liked or disliked a certain educational module
in your program, you might group responses into categories such as
"challenging," "fun," "too much work,"
"boring." Continue reading through the remaining diaries and see how
many responses fall into these categories. As you go along, you may come up
with additional categories or decide to collapse several categories
together.You can certainly make inferences (e.g., "Teachers liked the
module becau...") about diary information, but resist the temptation to
quantify this information.
Analyzing quantitative responses. The easiest way to analyze these
types of responses is to create a coding sheet for each quantitative question.
Use a separate sheet for each question, writing the question at the top and
creating columns for each possible response. For example, for a question about
how many people picked up particular brochures, you could create columns for
the following categories: 0, 1–5, 6–10, 11–15, 16–20,
>20.
Use the following procedure to record the responses:
-
Take the first activity log and record the response by making a check mark in the appropriate column.
-
Repeat this procedure for every questionnaire.
-
Tally the total number of check marks in each column and then calculate the percentage of participants who gave each type of response.
Use quantitative research methods during the following parts of your program:
-
Stage 1—to obtain information on prevalence of relevant knowledge, attitudes, behaviors, and behavioral intentions
-
Stage 3—to monitor usage of materials and intended audience awareness of the communication program and its various tactics
-
Stage 4—to measure progress toward objectives
Two different quantitative research methods, surveying and readability testing,
can be used.
Surveys
Surveys are characterized by large numbers of respondents (100 or more) and
questionnaires that contain predominantly forced-choice (closed-ended)
questions.
Common Uses
Used in planning and assessment to obtain baseline and tracking information on
knowledge, attitudes, behaviors, and behavioral intentions
Pros
-
Provides generalizable results (to generalize to a broader population, you must have a statistically valid random sample)
-
Can be anonymous (useful for sensitive topics)
-
Can incorporate visual material (e.g., can pretest prototype materials)
Cons
-
Limited ability to probe answers
-
Potential bias from possible respondent self-selection
Most surveys are custom studies that are designed to answer a specific set of
research questions. Some surveys, however, are omnibus studies, in which you
add questions about your topic to an already existing survey. A number of
national and local public opinion polls offer this option.
Pros and Cons of Various Survey Formats
|
Survey Formats
|
Pros
|
Cons
|
Mail
|
Can be used to cost–effectively access difficult–to–reach
populations (e.g., the homebound, rural populations)
Respondents can answer questions when most convenient for them
|
Not appropriate for respondents with limited literacy skills
Low response rate diminishes value of results
May require extensive/ expensive followup by mail or telephone to increase
response rate
Respondents may return incomplete questionnaires
Can be difficult to read responses
May take long time to receive sufficient numbers of responses
Postage may be expensive if sample is large, questionnaire is long, or multiple
reminder cards are needed
|
Telephone
With interviewer using paper-and-pencil questionnaires
|
Appropriate for those with limited literacy skills
Results in more complete responses because interviewer fills out questionnaires
Can control question sequence
|
Potential respondents without phones cannot participate
Respondents may hang up if they believe the survey is part of a solicitation
call or they don’t want to take time to participate
|
Using computer-assisted telephone interviewing (CATI)
|
Useful for complex questionnaires because "skip patterns" can be
programmed in
Data entry is eliminated
|
Requires CATI software and computers
Requires extensive interviewer training
Requires time to program questionnaire into CATI
|
In Person
Interviewer-administered
|
Face-to-face persuasion tactics can be used to increase response rates
Can be used with those with limited literacy skills
Useful with difficult-to-reach populations (e.g., homeless, rural) or when
intended audience cannot be sampled using other data collection methods
Interviewer can clarify questions for respondents
Results in more complete responses because interviewer fills out questionnaires
|
More expensive than self–administered or telephone data collection
Not appropriate for sensitive, threatening, or controversial questions
(respondents may not answer as truthfully in person)
|
Written, self-administered
Respondents are asked to complete survey at a location frequented by the target
population (e.g., during a conference, in a classroom, after viewing an exhibit
at a health fair)
|
Can connect with harder-to-reach respondents in locations convenient and
comfortable for them
Can be conducted quickly
Cost-effective means of gathering data in relatively short time
Can result in increased number of respondents within intended audience if
appropriate location chosen
|
Must be able to reach respondents in person at a central location or a
gathering
Respondents must have adequate literacy skills and be self-motivated to respond |
Computerized, self–administered
Questionnaire is displayed on computer screen and respondents key in their
answers
|
Useful for complex questionnaires because computerized "skip patterns"
can be used
Can control question sequence
Eliminates data entry and provides quick summary or analysis of results
|
Not appropriate for intended audiences with limited literacy skills or those
unfamiliar or uncomfortable with computers
Requires expensive technical equipment that may not be readily available or may
be cumbersome in many settings
|
Internet
Computerized, self–administered
Questionnaire is displayed on respondent’s computer screen via a Web site
|
Useful for complex questionnaires because computerized "skip patterns"
can be used
Can control question sequence
Eliminates data entry and provides a quick summary or analysis of results
|
Not appropriate for audiences with limited literacy skills or those unfamiliar
with or uncomfortable with computers
Respondents must have access to the Internet and be somewhat familiar with
using computers
No way to confirm the validity of information provided by respondents
|
Follow these steps to conduct a survey:
-
Plan the study
-
Determine how the sample will be obtained and contacted
-
Develop and pretest the questionnaire
-
Collect the data
-
Analyze results
Sampling size and composition, questionnaire design, and analysis of
quantitative data are complex topics beyond the scope of this book. If you are
planning a quantitative study, see the reference list at the end of this book
for additional information.
Gatekeeper Reviews
Public and patient education materials are often routed to their intended
audiences through health professionals or other individuals or organizations
that can communicate with these audiences for you. These intermediaries act as
gatekeepers, controlling the distribution channels that reach your intended
audiences. Their approval or disapproval of materials can be a critical factor
in your program’s success. If they do not like a poster or a booklet or
do not believe it to be credible or scientifically accurate, it may never reach
your intended audience.
Common Uses
Gatekeeper review of rough materials is important and should be considered part
of the pretesting process, although it is not a substitute for pretesting
materials with intended audience members. Neither is it a substitute for
obtaining clearances or expert review for technical accuracy; these should be
completed before pretesting is undertaken. Sometimes, telling gatekeepers that
technical experts have reviewed the material for accuracy will reassure them
and may speed their approval of your material.
Methodology
The methodology you should use for gatekeeper review depends upon your available
resources, time, and budget. Common methods include:
-
Self-administered questionnaires—
Participants are sent the materials and the questionnaire at the same time.
-
Interviewer-administered questionnaires—
Typically, an appointment for the interview is scheduled with the gatekeeper, and the materials are sent for review prior to the interview.
Develop questionnaires that ask about overall reactions to the materials and for
assessment of the information’s appropriateness and usefulness.
In some cases, you might not use a formal questionnaire (especially if you
don’t think the reviewer will take the time to fill it out) but will
instead schedule a telephone conversation or a meeting about the materials. If
you are not using a questionnaire, consider in advance what kind of questions
you want to ask in the meeting or interview and determine whether you need
formal approval of the materials. A discussion with gatekeepers (e.g., a
television public service director, the executive director of a medical
society) at this point can also be used to solicit their involvement in a
variety of ways that extend beyond materials development.
Readability Testing*
Readability formulas often are used to assess the reading level of materials.
Fry, Flesch, FOG, and SMOG are among the most commonly used. Applying these
formulas is a simple process that can be done manually or by using a computer
software program. Each method takes only a few minutes.
Typically, readability formulas measure the difficulty of the vocabulary used
and the average sentence length. In addition, computer software programs
analyze a document’s grammar, style, word usage, and punctuation, and
assign a reading level. These formulas, however, do not measure the
reader’s level of comprehension.
Readability software programs are available at computer stores. Some software
programs, such as Microsoft Word, include a readability-testing function.
(Note: Mention of software products does not constitute an endorsement by the
National Cancer Institute.)
Researchers James Pichert and Peggy Elam suggest three principles for using
readability formulas effectively:
-
Use readability formulas only in concert with other means of assessing the effectiveness of the material.
-
Use a formula only when the text’s intended readers are similar to those on whom the formula was validated.
-
Do not write a text with readability formulas in mind.
Before you choose a readability testing method, decide on an appropriate reading
level for the materials you’ve written. Then use readability testing to
determine whether your text corresponds to the reading level you want.
The term reading level refers to the number of years of education
required for a reader to understand a written passage. Some experts suggest
aiming for a level that is two to five grades lower than the highest average
grade level of your intended audience to account for a probable decline in
reading skills over time. Others note that a third- to fifth-grade level is
frequently appropriate for low-literacy readers. Keep publications as simple as
possible to increase reader comprehension of the material.
Readability Testing Methods
You can test readability easily using a formula such as Fry, Flesch, FOG, or
SMOG. These tests can be done quickly to indicate any problems with the drafted
text. They do not involve the intended audience.
SMOG
To calculate the SMOG reading grade level of a written sample, begin with the
entire written work that is being assessed, and follow these four steps:
-
Count off 10 consecutive sentences each near the beginning, in the middle, and near the end of the text.
-
From this sample of 30 sentences, circle all of the words containing three or more syllables (polysyllabic), including repetitions of the same word, and total the number of words circled.
-
Estimate the square root of the total number of polysyllabic words counted. Do this by finding the nearest perfect square and taking its square root.
-
Finally, add a constant of three to the square root. The resulting number is the SMOG grade or the reading grade level that a person must have reached to fully understand the text being assessed.
A few additional guidelines will help to clarify these instructions:
-
A sentence is defined as a string of words punctuated with a period, an exclamation point, or a question mark.
-
Hyphenated words are considered one word.
-
Numbers that are written out should also be considered, and if in numeric form in the text, they should be pronounced to determine whether they are polysyllabic.
-
Proper nouns, if polysyllabic, should be counted, too.
-
Abbreviations should be read as unabbreviated to determine whether they are polysyllabic.
Not all pamphlets, fact sheets, or other printed materials contain 30 sentences.
To test a text that has fewer than 30 sentences:
-
Count all of the polysyllabic words in the text.
-
Count the number of sentences.
-
Find the average number of polysyllabic words per sentence as follows:
Average = Total # of polysyllabic words
Total # of sentences
-
Subtract the total number of sentences from 30.
-
Multiply that number by the average.
-
Add that figure to the total number of polysyllabic words.
-
Find the square root and add a constant of 3.
Perhaps the quickest way to administer the SMOG test is by using the SMOG
conversion table. Simply count the number of polysyllabic words in 30 sentences
(chains of 10 each from the beginning, middle, and end of the text) and look up
the approximate grade level on the chart. See the sidebar Example Using the
SMOG Readability Formula on the next page for an example of how to use the SMOG
Readability Formula and the SMOG conversion table. In the sidebar, each of the
3 sets of 10 sentences is marked with brackets.
Readability Testing With the Intended Audience*
Other methods of evaluating reading levels and comprehension include having your
intended audience pretest your materials. The WRAT or the Cloze technique can
be used to do this. These types of testing are useful when you suspect that the
intended audience may encounter difficulties with the material. Including
pretest participants who have the same characteristics as the lowliteracy
intended audience you are trying to reach is critical to the validity of your
pretest results. Recruiting participants through groups or settings that
include people with limited literacy skills is a logical starting point. But
the only way to be sure your pretest volunteers read at the same level as your
intended audience is to test their reading skills.
The Wide Range Achievement Test (WRAT) is used to measure reading levels, and
the Cloze technique is used to measure comprehension. To avoid offending or
causing discomfort to those whose reading ability you are testing, you can
integrate a WRAT or a Cloze test into the pretest interview. For example, in a
recent pretest conducted by the National Cancer Institute, the interviewers
introduced the WRAT test as the last part of the pretest. They stated,
"Thank you for helping with the questions on the chemotherapy booklet.We
need your help with one last part—a word list. This will take only a few
minutes. The word list will help us understand how difficult the words are in
the chemotherapy booklet." This integrated approach spared participants the
pressure or potential embarrassment of "failing a reading test."
The WRAT is based on word recognition and does not measure comprehension or
vocabulary. The WRAT is an efficient way to determine reading levels and takes
only a short time to administer.
The WRAT involves listening to the participant read from a prepared list of
words, arranged in increasing order of difficulty. Pronouncing the word
correctly shows that the reader recognizes the word. The WRAT focuses on
recognition because, at the most basic level, if a person does not recognize a
word, comprehension is impossible.
The test is over after the reader mispronounces 10 words. The test administrator
notes the level at which the last mispronunciation occurred. The "stop"
level equates to a grade level of reading skills.You can compare this level
with the reading level of your intended audience to see if your pretest readers
are a representative match for that audience.
The Cloze technique measures the reader’s ability to comprehend a written
passage. Because it requires readers to process information, it may take up to
30 minutes to administer.
In a Cloze test, text appears with every fifth word omitted. The reader tries to
fill in the blanks. This task demonstrates how well he or she understands the
text. The reader’s ability to supply the correct word also reflects his
or her familiarity with sentence structure.
While packaged Cloze tests are available, Leonard and Cecilia Doak’s Teaching
Patients with Low Literacy Skills explains how to make up and score a
Cloze test yourself, based on the materials you are pretesting. The book also
discusses use of the WRAT to assess reading levels.
Example Using the SMOG Readability
Formula |
Benign Prostate Hyperplasia (BPH)
[Benign prostatic hyperplasia (BPH) is an enlarged prostate. Benign means
noncancerous and hyperplasia means excessive growth of tissue. BPH is the
result of small, noncancerous growths inside the prostate. It is not known what
causes these growths, but they may be related to hormone changes that occur
with aging. By age 60, more than half of all American men have microscopic
signs of BPH, and by age 70, more than 40 percent will have enlargement that
can be felt on physical examination.
The prostate normally starts out about the size of a walnut. By the time a man
is age 40, the prostate may already have grown to the size of an apricot; by
the age of 60, it may be as big as a lemon.
BPH, which usually does not affect sexual function, is a troublemaker because
the prostate, as it enlarges, presses against the bladder and the urethra,
blocking the flow of urine.
A man with BPH may find it difficult to initiate a urine stream or to maintain
more than a dribble. He also may need to urinate frequently, or he may have a
sudden, powerful urge to urinate.] Many men are forced to get up several times
a night; others have an annoying feeling that the bladder is never completely
empty.
Straining to empty the bladder can make matters worse; the bladder stretches,
the bladder wall thickens and loses its elasticity, and the bladder muscles
become less efficient. The pool of urine that collects in the bladder can
foster urinary tract infections, and trying to force a urine stream can produce
back pressure that eventually damages the kidneys. The kidneys are where urine
is formed as waste products are filtered from the blood.
BPH sometimes leads to problems. [For instance, a completely blocked urethra is
a medical emergency requiring immediate catheterization, a procedure in which a
tube called a catheter is inserted through the penis into the bladder to allow
urine to escape. Other serious potential complications of BPH include bladder
stones and bleeding.
Diagnosing BPH
A detailed medical history focusing on the urinary tract—kidneys, ureters
(the pair of tubes that carry urine from the kidneys to the bladder), the
bladder, and the urethra— allows the doctor to identify symptoms and to
evaluate the possibility of infection or other urinary problems.
The initial medical workup typically includes a physical exam called a digital
rectal examination (DRE), a urinalysis to check for infection or bleeding, and
a blood test to measure kidney function. Some physicians may also check the
amount of prostate-specific antigen (PSA) using a PSA test to help rule out the
likelihood of cancer. PSA is a protein that is produced by the cells of the
prostate gland.
In addition, other tests may help a urologist—a doctor who specializes in
disorders of the urinary tract and the male reproductive tract—to
determine if BPH has affected the bladder or kidneys. These include tests that
measure the speed of urine flow, pressure in the bladder during urination, and
how much urine is left in the bladder after urinating.
Some other tests that are widely used, according to an expert panel sponsored by
the United States Public Health Service (USPHS) practice guidelines, are
expensive, sometimes risky, and, for most men, unnecessary. These include
cystoscopy, in which the doctor inserts a viewing tube up the urethra to get a
direct look at the bladder; an x-ray called a urogram, in which urine is made
visible on an x-ray after dye is injected into a vein; and ultrasound, which
obtains images of the kidneys and bladder after a probe is placed on the
abdomen.]
Treating BPH
About half of men with BPH develop symptoms serious enough to warrant
treatment. BPH cannot be cured, but its symptoms can be relieved by surgery or
by drugs in many cases.
BPH does not necessarily grow worse. According to one review, mild to moderate
symptoms worsened in only about 20 percent of the cases. They improved (without
any specific treatment) in another 20 percent and stayed about the same in the
rest.
[Men whose symptoms are mild enough often opt for an approach called watchful
waiting. This means that they report for regular checkups and have further
treatment only if and when their symptoms become too bothersome.
The USPHS Clinical Practice Guidelines call watchful waiting "an appropriate
treatment strategy for the majority of patients." Men who choose watchful
waiting should have regular, perhaps annual, checkups, including DREs and
laboratory tests.
For those who choose watchful waiting, a number of simple steps may help to
reduce bothersome symptoms. These include limiting fluid intake in the evening,
especially beverages containing alcohol or caffeine, which can trigger the urge
to urinate and can interfere with sleep; taking time to empty the bladder
completely; and not allowing long intervals to pass without urinating.
Men monitoring prostate conditions should also be aware that certain medications
they are taking for other ailments may make their symptoms worse. These include
some over-the-counter cough and cold remedies, prescribed tranquilizers,
antidepressants, and drugs to control high blood pressure. Switching to a
different prescription may help.
Watchful waiting, of course, is not always enough for BPH, and surgery or drug
therapy may be required.]
|
Readability Test Calculations
|
Total Number of Polysyllabic Words |
= 104 |
Nearest Perfect Square |
= 100 |
Square Root |
= 10 |
Constant |
= 3 |
SMOG Reading Grade Level |
= 13 |
We have calculated the reading grade level for this example. Compare your
results to ours, and then check both with the SMOG conversion table:
|
SMOG Conversion Table*
|
Total Polysyllabic Word Counts
|
Approximate Grade Level
(± 1.5 Grades)
|
0 — 2 |
4 |
3 — 6 |
5 |
7 — 12 |
6 |
13 — 20 |
7 |
21 — 30 |
8 |
31 — 42 |
9 |
43 — 56 |
10 |
57 — 72 |
11 |
73 — 90 |
12 |
91 — 110 |
13 |
111 — 132 |
14 |
133 — 156 |
15 |
157 — 182 |
16 |
183 — 210 |
17 |
211 — 240 |
18 |
* Developed by Harold C. McGraw, Office of Educational Research, Baltimore
County Schools, Towson, Maryland.
|
* Adapted from Clear and Simple: Developing Effective Print Materials for
Low-Literate Readers (NIH Publication No. 95-3594), by the National
Cancer Institute, 1994. Bethesda, MD. In the public domain.
Back to Top
< Previous Section | Next Section > |