Steps for Effective Prevention Planning and Evaluation

Unfortunately, because most administrators associate evaluation with the measurement of final results, prevention planners often fail to think about evaluation until after their programs and policies are up and running. Instead, the evaluation should be planned as the prevention program is being developed.

Thinking about evaluation as part of the planning process will sharpen everyone's thinking about the program: its mission, its goals, its objectives, and the activities designed to meet those objectives. Used in this way, evaluation planning can be a valuable management tool. Many prevention planners are finding it useful to view program and policy development and evaluation as an iterative process, with evaluation findings informing later alterations.

The process for developing and evaluating prevention programs and policies can be divided into five basic steps:

1. Identifying specific goals and objectives
2. Reviewing the evaluation research
3. Outlining how the intervention will work
4. Creating and executing a data collection plan
5. Providing feedback to the intervention program.

Basic considerations for each step are described below. We realize that program planning does not always proceed with these steps in sequence. Indeed, it is typical for earlier steps to be revisited as planners refine their thinking.

1. Identify Specific Goals and Objectives

The problem—student drinking—is obvious, but exactly which goals and objectives should be specified to guide campus prevention efforts is not. Is the goal to eliminate college student drinking? Limit excessive consumption of alcohol? Eliminate alcohol-related behavior problems? Protect student drinkers from harm? Should the prevention effort focus on student drinking on campus, or should it also cover off-campus behavior?

How college officials answer these questions will depend on several factors: the philosophy and academic mission of the institution, the nature of the student alcohol problem, the level of prevention resources available, the views and opinions of key constituencies, the characteristics of the surrounding community, and the cultural and political context in which the school operates.

This is the time at which one would conduct a so-called “needs assessment.” While this assessment may be conducted with varying degrees of formality, and comprise focus groups, large-scale surveys, or open-ended interviews with members of the (extended) campus community, the purpose is to better understand not only the nature and scope of the problem for any specific campus, but ideally, to also understand the broader social, economic, and physical context that might shape both the problem and the range of appropriate strategies that might be adopted to reduce it. Conducting a needs assessment and communicating its results may also be required in order to achieve agreement among key campus stakeholders, including the students.

Unfortunately, it is common for college officials to jump into prevention work without taking time to explore what their goals and objectives are. This often happens when program planners choose to replicate programs and policies from other campuses without thinking through exactly what they are intended to accomplish.

For example, consider social norms marketing campaigns. Using local campus media, these campaigns are designed to reduce heavy alcohol consumption by communicating accurate information about student drinking levels, thereby correcting the common misperception that “everyone” drinks. Such a program is less likely to be successful on a campus with older commuter students who do not identify with the institution or their student peers. In addition, such campaigns are also more likely to foster moderation rather than total abstinence, an outcome that may be philosophically unacceptable at some institutions. Even so, many prevention specialists have been swept along by the current enthusiasm for social norms marketing without considering whether such a program is a good fit for their campus.

There is the risk, then, that program planners may end up pursuing a set of disconnected programs and policies in service of multiple goals and objectives, some of which may even contradict one another. Instead, having an evaluator be part of the planning process from the beginning will help college officials consider a full range of options for what they want to accomplish, and have articulated specific goals and objectives. In turn, these goals and objectives can be translated into specific outcomes to be assessed through the evaluation process. This is a good example of how planning an intervention can also help shape the evaluation.

Importantly, having an evaluator be part of the planning process will help a college’s officials reach consensus on their specific goals and objectives. Absent this discussion, it is common for program planners to develop a broadly stated goal that can mean different things to different people. For example, a typical goal might be “to reduce student alcohol problems.” Some officials will interpret that to mean abstinence is the goal, whereas other officials will think it means the development of programs and policies to protect students from the consequences of their heavy drinking. When evaluators are brought in to assess an established program, it is common for them to discover that different officials have widely varying opinions about what they are trying to achieve, a fact that had previously gone unrecognized.

In listing goals and objectives, specificity is key. Evaluators will push program planners to develop precise and measurable objectives, meaning that the achievement of those objectives can be measured and readily observed. For example, stating that “alcohol consumption” will be reduced is too imprecise. Instead, a specific goal might be to increase the percentage of underage students who abstain from alcohol, or to decrease the number of separate occasions on which students consume alcohol per month, or to decrease the percentage of students who report having three or more drinks the last time they consumed alcohol. In some cases, it will be important to specify the time and place where the changes will be observed. For example, it might make sense to concentrate on reducing alcohol problems that occur during specific campus events (e.g., homecoming) or at off-campus taverns and bars.

2. Review the Research on College Drinking Interventions

The next step is to review program and policy options that might be applied to achieve the outlined goals and objectives. Many types of prevention programs and policies are being implemented in the name of reducing alcohol-related problems on campus. A typology of programs and policies developed by the U.S. Department of Education’s Higher Education Center for Alcohol and Other Drug Prevention provides a useful way to categorize existing efforts, identify missing program elements, and guide new strategic planning.

A social ecological framework used in public health work defines one dimension of the typology, with programs and policies classified into one of the following levels: 1) individual, 2) group, 3) institution, 4) community, and 5) State and Federal public policy.

The typology’s second dimension concerns the key areas of strategic intervention, each of which is linked to a particular definition of the college alcohol problem. There are four alternative areas of strategic intervention to be considered:

1. Changing people’s knowledge, attitudes, and behavioral intentions regarding alcohol consumption
2. Eliminating or modifying environmental factors that contribute to the problem
3. Protecting students from the short-term consequences of alcohol consumption (“health protection” or “harm reduction” strategies)
4. Intervening with and treating students who are addicted to alcohol or otherwise show evidence of problem drinking.

This typology is consistent with the “3-in-1 Framework” to comprehensive and integrated prevention programs espoused by the NIAAA Task Force on College Drinking in its report, A Call to Action: Changing the Culture of Drinking at U.S. Colleges. The Task Force divided programs and policies according to three broad levels: 1) the student population as a whole; 2) the broader college and community environment; and 3) individual students. The value of both the “3-in-1” and ecological frameworks is that they can be a useful introduction to encourage presidents, administrators, college prevention specialists, and community members to think in a broad and comprehensive fashion about college drinking.

Many areas of strategic intervention can be pursued at one or several levels in the social ecological framework. For example, consider intervention activities focused on the objective of increased observance and enforcement of the minimum drinking age law (also known as the age-21 law):

• At the State or community level, the alcohol control commission could increase the number of decoy (or “sting”) operations at local bars and restaurants.
• At the community level, local police could implement a protocol for notifying college officials of all alcohol-related violations involving students.
• At the college itself, the campus pub could require that all alcohol servers complete a training course in responsible beverage service.
• At the group level, the college might require that residential groups and special event planners provide adequate controls to prevent alcohol service to underage students.
• At the individual level, a media campaign could publicize these new policies, the stepped-up enforcement efforts, and the consequences of violating the law.

Implementing multiple strategies at these various levels would greatly increase the likelihood of the objective being achieved.

Major findings from the review of the literature on college-focused preventions are presented below, organized according to the social ecological framework.

Individual-Level Interventions. One set of programs is designed to increase student awareness of alcohol-related problems, change attitudes and beliefs, and foster each student’s determination to avoid high-risk drinking. Typical among these efforts are freshman orientation, alcohol awareness weeks and other special events, and curriculum infusion, where faculty introduce alcohol-related facts and issues into their regular academic courses. The assumption behind these approaches is that, once students are presented with the facts about alcohol’s dangers, they will make better informed and therefore healthier decisions about drinking. Rigorous evaluations of these educational programs are rare, but work in elementary and secondary school-based settings suggests that, while these types of awareness programs are necessary, information alone is usually insufficient to produce behavior change.

A second set of programs is designed to intervene with students whose pattern of alcohol use puts them at risk for serious negative consequences. There is little evidence that standard awareness and values clarification programs alone can reduce alcohol consumption by college students. There are new approaches being studied that do hold promise, however, including:

1. Expectancy-challenge procedures. In this approach, didactic or experiential manipulations are used to show students that many of the effects they anticipate from drinking, such as sociability and sexual attractiveness, are due to their expectations, not to the alcohol per se. Students may be given a placebo drink but led to think that it contains alcohol, or they may observe others who have consumed alcohol or placebo drinks in a social setting.

2. Cognitive-behavioral skills training. In these programs, students are taught several ways to reduce their risk of heavy drinking, including managing stress, documenting daily alcohol consumption and planning ahead of time how much to drink before attending social events. Such programming works best when coupled with a motivational enhancement intervention.

3. Brief motivational enhancement intervention. Delivered in a student health center, hospital emergency room, or other setting, this personalized intervention involves giving individual students feedback about their drinking behavior in comparison with others, information on the true drinking norms on campus, and a review of the negative consequences they are likely to suffer if they continue to drink at current levels. Research shows that receiving this feedback via computer rather than in-person can also work.

As identified in the Task Force’s report, these strategies have been shown to be effective, and will become more refined with further study to determine the most effective combination of program components. The ultimate challenge, however, may be in figuring out how to scale up these programs to impact the behavior of large numbers of students, not just a small number of research participants.

Group-Level Interventions. Programs in this category have focused primarily on fostering peer-to-peer communication to change student social norms about alcohol use. Peer education programs, for example, train student leaders to implement a variety of awareness and educational programs and to serve as role models for other students. Well-structured evaluations of peer education are rare, so such programs remain an unproven strategy for reducing student alcohol consumption.

Social norms campaigns are another prevention strategy in this category. This approach is grounded in the well-established observation that college students greatly overestimate the percentage of their peers who drink heavily. Because this misperception drives normative expectations about alcohol use, which in turn influence actual use, a viable prevention strategy is to correct the misperception. A social norms campaign attempts to do this by using campus-based mass media to provide more accurate information about actual levels of alcohol use on campus. Preliminary studies at several institutions suggest that this approach to changing the social environment may be promising, but more definitive research is still needed to gauge its impact.

Most recently, there are a number of programs aimed at groups of students who share an affiliation (e.g., members of an athletic team, or fraternity members). Underlying such programs is the idea that prevention might take advantage of the social bonds formed by these groups to create peer sentiment for safer drinking practices. This is another area in which programs have outpaced evaluation, so little is known about how well these programs work.

Institutional-Level Interventions. On campus, a task force should conduct a broad-based examination of the college environment, looking not only at alcohol-related policies and programs, but also at the academic program, the academic calendar, and the entire college infrastructure. The objective is to identify ways in which the environment can be changed to clarify the college’s expectations for its students, better integrate students into the intellectual life of the college, change student norms away from alcohol and other drug misuse, or make it easier to identify students in trouble with substance use.

There are five strategic objectives that can be pursued at the institutional level:

1. Offer and promote social, recreational, extracurricular, and public service options that do not include alcohol and other drugs
2. Create a social, academic, and residential environment that supports health-promoting norms
3. Limit alcohol availability
4. Restrict marketing and promotion of alcoholic beverages
5. Develop and enforce campus policies.

Each of these objectives can be met through a variety of programs and policies. Consider limiting alcohol availability. Potential strategies include, among many others, banning or restricting the use of alcohol, banning delivery or use of kegs or other common containers, requiring use of registered and trained alcohol servers, and instituting responsible server training programs.

Community-Level Interventions. Student alcohol problems are not a problem of the campus alone, but also of the surrounding community. Work at the community level can be accomplished through a campus and community coalition. Community mobilization, involving a coalition of civic, religious, and government officials, is widely recognized as key to the successful prevention of alcohol problems. Higher education officials, especially college and university presidents, can take the lead in forming these coalitions and moving them toward an environmental approach to prevention.

A chief focus of a campus-community coalition should be to curtail youth access to alcohol and to eliminate irresponsible alcohol sales and marketing practices by local bars, restaurants, and liquor outlets. Potential strategies include limiting the number and concentration of alcohol outlets near campus, limiting the days or hours of alcohol sales, and instituting responsible server training programs.

State and Federal Public Policy. College officials should also work for policy change at both the State and Federal level. There are several potentially helpful laws and regulations that can be considered, including:

1. Distinctive and tamper-proof licenses for drivers under age 21
2. Increased penalties for illegal service to minors
3. Prohibition of “happy hours” and other reduced-price alcohol promotions
4. Restricted hours of sales
5. Reduced density of retail outlets
6. Increased excise tax rates on alcohol.

Some communities have the ability to pursue these strategies locally through either local licensing laws, business permits, or through voluntary initiatives such as a code of responsible business practices that local owners and managers would agree to sign.

3. Outline How the Intervention Will Work

A review of available research, plus consultations with other college and university prevention specialists, will suggest a set of program and policy options that can be adopted. The next planning step is to outline the chain of events that will lead from implementation of each component program or policy to its specific (and measurable) objective. Describing this chain of events is often called building the “logic model” for the intervention. Very often drawing a flow chart is the clearest and most economical way of presenting this information (see Figure 1 for an example).

There are several reasons why this step is important:

First, a logic model will pinpoint areas of uncertainty, confusion, or disagreement among members of the planning team. It is common for different people to have varying theories about why a particular type of intervention should reduce student alcohol problems. In some cases, members of the planning team may even have a theory about why an intervention will have unintended negative effects. These competing expectations and theories need to be discussed and sorted out.

Second, a logic model can expose any false assumptions that need to be addressed. For example, orientation programs for first-year students often assume that these students are unaware of alcohol’s dangers. Given that today’s students have been given this basic information since they were in junior high school, that is very unlikely to be the case. However, there may be other legitimate reasons for providing this information. For example, students may not be aware of how heavy alcohol consumption can interfere with learning.

Third, a logic model will help guarantee that all program activities and policies can be logically linked to the achievement of specific objectives. Items that cannot be so linked will be discarded from the plan. Work at this step may lead to a reconsideration of the objectives. For example, it may be that only a small number of intervention components can be organized to achieve a certain objective, due to considerations of available resources, political obstacles, or other barriers. In such cases, it might make sense to abandon that objective altogether and concentrate instead on other objectives that can be more easily achieved.

Fourth, a logic model can later serve as an educational and communications tool when a new program or policy is being implemented. The logic model not only specifies what is being done, but also why.

Fifth, a logic model can be a tool for tracking changes in the intervention or its implementation. As a concrete manifestation of institutional memory, a sequence of modified logic models will inform all parties to the intervention (including those new to the program) about how it was shaped along the way. Sometimes, a prevention team or administrator may need to be reminded of how the program or intervention was modified and for what reasons.

A final reason for developing a logic model is that it helps inform the evaluation. In essence, the logic model makes clear the intervening steps that are hypothesized to lead from specific activities to specific outcomes. Data can be collected to document progress at each step. With this information in hand, evaluators can diagnosis what went wrong if a program or policy fails to meet its ultimate objective.

Consider again the example of a social norms marketing campaign. The underlying premise of this campaign is that accurate information about drinking norms, conveyed through credible sources, will help students realize that the majority of their peers drink far less than they once thought was the case, which in turn will reduce perceived pressure to drink heavily and drive down actual consumption. Figure 1 shows a simple logic model for this program.

Figure 1: Simple Logic Model for Normative Education Intervention
Starting Point Students Overestimate Prevalence of Binge Drinking Among Peers		Intervention Normative Education Campaign Newspaper Editorials Posters Contests Group Trainings		Proximal Outcomes Students Aware of Message Students Accept Message Students More Accurate in Estimating Peer Drinking		Long-term Outcomes Lower Prevalence of Binge Drinking Lower Prevalence of Negative Consequences More Support for Other Prevention Interventions

Starting from the left, the diagram shows a starting point at which students overestimate the prevalence of heavy drinking among their peers, a very common finding of student surveys. The educational campaign would be designed to change those beliefs, perhaps with several coordinated components—for example, newspaper advertisements, editorials, and letters to the editor; posters; electronic mail messages from student health services; contests and other promotional events; and group meetings and presentations. Note that a more fully developed logic model might also specify the frequency and duration of these and other related activities.

The campaign, if successful, should result in several immediate (or “proximal”) outcomes. Specifically, students will be able to identify the campaign’s main message and will report that the message is credible. In addition, students will report more accurate estimates of peer drinking and will cite less social pressure to drink heavily. Long-term (or “distal”) outcomes will include both a lower prevalence of heavy drinking and fewer negative consequences of alcohol consumption (e.g., academic failure or unintentional injury).

An additional outcome might be higher levels of support for other prevention initiatives, such as parental notification or tougher policies to reduce alcohol availability. Here the hypothesis would be that such initiatives will gather support as students better understand that only a minority of students will be impacted rather than a majority.

A good evaluation will assess whether each of the intermediate effects was achieved. In this example, a student survey could determine whether, prior to the campaign, students actually misperceived drinking norms on their campus. Absent that initial misperception, a social norms campaign should have little effect. Subsequent surveys could assess whether students remember the campaign message and find it to be credible. If not, then we would not expect the campaign to lead to a change in student beliefs about peer drinking norms. Survey questions would also be included about the other intermediate steps and the immediate as well as long-term outcomes.

It is important to see that the usefulness of the evaluation is in large part dependent on its following the logic model. If the evaluation were to only measure the final outcome, and the intervention fell short of its aims, the evaluation would be unable to answer the fundamental question of whether the program effects were smaller than hoped because the fundamental concept behind the intervention was wrong, the implementation was flawed, or one piece of the intervention sequence fell apart. From a program manager's viewpoint, these are crucial distinctions, as the answers will suggest different directions to take in the future to improve the intervention.

4. Create and Execute a Data Collection Plan

Self-report surveys are a primary data source for program and policy evaluations, especially if the goal is to reduce consumption or alcohol-related problem behaviors. That said, too many program planners assume that evaluation data and student survey data are one and the same. As a result, they may lose interest in evaluation when there is insufficient funding to mount a survey. Even worse, they may conduct a poorly administered survey in the hope that it will still provide useful data, rather than concentrate their resources on developing other potential data sources.

A student drinking survey must meet several requirements to be considered scientifically valid. The questions themselves must be both valid and reliable, meaning that they must truly measure what is intended (validity), and that, with a repeated administration, students will provide the same answer to the same question (reliability). Fortunately, there are several alternative survey instruments available that can be used as sources of questions.

In addition, the survey should be administered at a time that reflects typical drinking patterns. Surveys conducted shortly after the start of the school year, after traditional social events (e.g., homecoming), or close to mid-terms or final exams will not provide representative data. Hence, most national student drinking surveys are conducted in the early part of the spring semester before spring break.

Most important, the sample of students asked to participate in the survey must be randomly selected. It may be tempting to administer the survey in classrooms, but this will not result in a sample that is truly representative of all students. There must also be a set of procedures in place to boost the response rate. Achieving a response rate of 70 percent or more for student surveys is extremely difficult. More typical are rates between 50 and 60 percent.

It should be remembered, however, that a student survey is not the only source of useful data, and in some cases may not even be the best source. A key limitation is that many of the serious negative consequences of drinking are not frequent enough to be well estimated by a typical self-report survey involving a sample of only 200 to 1,000 students. Even so, a sizable university will experience these adverse events with some regularity.

Ideally, colleges and universities will institute a system for recording a wide range of alcohol-related incidents involving students. For example, it would be of great value to have a record of each instance in which a student is brought in for urgent or emergency care, including an indication of whether alcohol (or other drugs) were involved. How this might be done best will vary from school to school, depending on the type of student health center that is available, the number of nearby hospitals, or the manner in which insurance claims are processed.

Similarly, incident reporting forms used by the campus police should require officers to indicate whether a student being investigated, cited, or detained has been using alcohol. A direct reading of blood alcohol content (BAC) using a “passive” breathalyzer, which analyzes exhaled air in front of the mouth, would be the best means of assessment. Absent that, the officers can be asked to make a judgment about alcohol involvement. Such judgments can be difficult to make, yet despite their fallibility, having such data available over a long period of time will still reveal relative changes in alcohol involvement that might be attributable to new programs and policies.

Additional examples of potentially useful records are listed below. Which types of records are monitored will depend on the specific goals and objectives being pursued.

• Student health services: Costs associated with provided medical care when alcohol is involved.
  
• Student counseling services: Patient history of alcohol use.
  
• Residence facilities: Alcohol involvement in noise/nuisance complaints, property damage, calls for police or emergency services.
  
• Athletic department: Alcohol involvement in spectator injuries, nuisance complaints, or disciplinary actions.
  
• Greek student office: Records of alcohol involvement in student injuries, noise complaints, contacts with police or fire departments, and property damage.
  
• University discipline: Alcohol involvement in charges heard in disciplinary proceedings.

On many campuses, the problem is that data are recorded but are not easily accessed. In some cases, program planners may not even be aware of useful records being kept by several sub-units of the college or university. In other cases, there may only be hard-copy records made, which makes compiling the data for an evaluation too time-consuming to be practicable. On many campuses this situation is improving as offices move toward using computerized databases and automated data entry. As these systems are put in place, administrators should ensure that records of campus problems make note of alcohol involvement.

5. Communicating Evaluation Results: Feedback

Our emphasis here has been on the value of evaluation for program planning and management. Thus, we again stress that the full value of any evaluation is not likely to be realized if the information it comprises is not used to inform the campus community of what is happening with the intervention. In some cases, evidence of the program’s impact may be mandatory for it to continue. There is also the possibility that initial program impact may fade with time or that program fidelity will not be maintained.

All communities are deeply interested in whether the intervention is effective, but often there are more specific questions, too, regarding the effectiveness of each component, who is being affected and in what way, and how much effort is expended to achieve the impact. Support for the intervention often hinges on gathering and reporting this information. More broadly, feedback tends to engender support even when results are mixed or disappointing at first, because providing the information builds confidence that the people running the program or policy are interested in following through to either improve the intervention, or transfer resources to an alternative strategy for good reasons.

Previous | Back to Table of Contents | Next

Historical document
Last reviewed: 9/23/2005