Evaluate the Program

In the 18 months after Zippy’s put in place a new drug-free workplace policy that required unannounced urine tests, swing shift manager Thomas Kehoe reported that, despite some staff shortages, morale and productivity had improved. “Even though we’re sometimes short, it’s worth it. You don’t have nearly as many problems.” Those who apply for jobs “are a much better bunch of people. They’re very reliable and they’re more punctual. They help out a lot more.”[1]

Zippy’s is not unusual. Many businesses and organizations get positive results after they begin drug-free workplace policies and programs. But how do they know they are getting positive results? How can they be sure? Just as a policy and program don’t have to be complex and costly to be effective, evaluation doesn’t have to be as complicated as you might think. This section will discuss some simple steps you can take to make sure your efforts are worthwhile, whatever those efforts may be—whether they are encouraging changes in the workplace in informal ways, or training, testing, and assisting employees as part of an officially established drug-free workplace program.

Evaluation is important

We all know the importance of evaluation in our own lives. Many of us have yearly physicals for ourselves and regular checkups for our cars. And most organizations have short-term and long-term business plans that they periodically review, evaluate, and update. Evaluation can help you detect and remove barriers to high-level performance.[2] Evaluation can tell you if what you are doing is working and whether, depending on the cost, it is worthwhile. Evaluation is often critical to success. For your drug-free workplace initiative, evaluation is essential.

It doesn’t have to be costly

For a large organization with a sizable budget, an evaluation might be elaborate and expensive. “However, the expense of an evaluation is relative,” the Centers for Disease Control point out in a framework for evaluating public health programs. “The cost depends on the questions being asked and the level of certainty desired for the answers. A simple, low-cost evaluation can deliver valuable results.”[3] For a small business, the evaluation phase can be simple, cost effective, and informative. As mentioned in the “Assess Your Workplace” section, you can get the information, insight, and understanding that you need using relatively simple and inexpensive methods. The key is to be systematic, deliberate, and informed.

Top of page

Steps to effective evaluation

Ideally, an evaluation protocol should be developed before the program is designed and implemented. This way you can know where you are, where you are going, and how you expect to get there. On the road to success, evaluation can help you see if you have achieved the milestones along the way, and can help you identify opportunities for improvement. Evaluation should occur at every major step. This systematic approach to evaluation enables you to make informed improvements to the program.[4]

Some employers hire consultants to conduct evaluations.[5] One of the most compelling reasons for hiring outside consultants is that your employees may feel more comfortable disclosing sensitive information to an outsider. However, this may not be an option for organizations with limited resources.

This section of the Kit gives organizations with limited resources some tips on how to conduct a basic evaluation without bringing in outside assistance. “Lay the Groundwork” describes what to do before rolling out the policy and starting the program. “Plant the Seeds” describes the types of evaluations that should be conducted during the implementation phase. “Reap the Rewards” explains how to evaluate the entire process. “The Seasons of Evaluation” describes the ongoing cycle of evaluation.

1. LAY THE GROUNDWORK

Articulate the Goals

Deciding what it is you are trying to accomplish is key when designing an evaluation.[6] You cannot evaluate whether your policy and program have succeeded unless you have a clear idea of what you are trying to achieve with them.

Link Each Goal to Something Measurable

If one goal is to “have employees who are drug-free,” you should determine what this means and how you intend to measure it. Such broad goals are often measured by defining and measuring more concrete objectives that support the goal.

Get a Baseline

You should try to document what your workplace looked like before you began activities related to creating a drug-free workplace policy and program. This “pretest” information gives you something to compare with after you have developed your policy and program and set them in motion. (For more information, see “Assess the Workplace.”)

• Review the qualitative data. What were your supervisory and nonsupervisory employees saying about the safety, health, and productivity of your workplace before the program? What was the morale of the organization before your drug-free workplace policy and program were implemented?
  
  
• Analyze the quantitative data. What were the numbers on turnover, absenteeism, tardiness, use of health care benefits, and workers’ compensation claims? Has there been a change over time? What was the percentage of loss—including theft, accidents, and poor-quality goods?

Top of page

Estimate the costs of these factors in dollars, if possible. Try to get the data for past years. If this is not possible, start tracking trends now.

Describe the Process

Perhaps you followed the suggested steps in this Kit. For instance, perhaps you developed consensus by building a team. You may have revised your policy several times to incorporate feedback from your drug-free workplace team. Perhaps you addressed a variety of legal requirements, which will help protect you against lawsuits. If you took the time to assess your employees’ strengths and their areas in need of improvement, you should note that too.

Document the steps taken before, and while, developing and implementing the policy and program. For example, have a team member take minutes at each meeting. Recording this information will help you remember what actions were taken and will help you determine how the process can be improved.

Find a Way to Compare

You can simply compare what your workplace looked like before and after you implemented your policy and program. Another powerful way to show change is by comparing your efforts against the efforts of another workplace. Match your organization to a comparison organization that is not making the same changes. Or compare one of your job sites that received an extensive program with another of your job sites that received only basic training.

2. PLANT THE SEEDS: EVALUATE WHILE IMPLEMENTING

At the beginning, you may not know what is going to contribute to positive change. For example, when a large automotive manufacturing company implemented a weekly lunch talk on substance abuse treatment and prevention, it was not at all clear to management why no one showed up. But when the weekly talk was renamed and reworked to speak more broadly about health-related issues, employees were lining up at the doors.

Some industrious researchers do a process check after every module or session by asking questions such as the following:

• How many attended the session?
  
• Did you like this session?
  
• Did you find it useful?
  
• Do you have ideas for improving it?

This process check gives immediate feedback and helps pinpoint what is working and what is not.

Top of page

3. REAP THE REWARDS: EVALUATE THE PROGRAM

After implementing the policy and program, it is time to review. There are four levels of evaluation: 1) reaction, 2) learning, 3) behavior, and 4) results.[7] More and more, employers are under pressure to move away from simply measuring reactions.[8, 9] It is not enough to say that employees “liked the training” or “learned a lot.” Evaluation at different levels is important. Also important is that your employees’ responses be kept confidential. If you respect their right to confidentiality, you will be more likely to get better information and a more complete picture.

Reactions

This level of evaluation is the easiest, and it consists of simply surveying employee feelings about important elements of the program. You can ask employees to complete a short evaluation form at the end of the program.

Evaluation questions usually concentrate on what was most helpful, what was least helpful, how competent the trainer was, and how well the sessions were paced. The survey can ask basic questions, such as: Was the program useful? Did the program meet your expectations? What could have been done better?

Learning

This level of evaluation assesses whether participants have learned the ideas, facts, or processes that were presented. In a basic drug-free workplace training program, you may want to see if participants learned the details of the policy and the consequences of not adhering to the policy. In a drug-free workplace program embedded in a health initiative, you will most likely have different learning goals. For example, you could measure the extent to which your employees have learned productive ways of dealing with job stress.

Behavior

This level of evaluation concentrates on whether employees have used what they have learned to change various behaviors. In the Workplace Managed Care (WMC) project, researchers used the Health Behavior Questionnaire (HBQ) to assess changes in supervisors and employees behaviors. The HBQ emphasizes substance use, health, and related factors such as stress-relief strategies, alcohol consumption, and use of employee assistance program (EAP) services. This type of measurement is called “self-report,” as it asks the respondent about his or her behaviors. Another measure of behavior is to observe changes. One example is to enumerate supervisor referrals to EAP services before and after the program. But changes in behavior are, in general, more difficult to achieve and measure than changes in knowledge.

Results

This level of evaluation measures various outcomes such as decreased absenteeism, decreased loss of inventory, or increased morale. In the WMC project mentioned above, results were measured by looking at changes in substance abuse–related medical claims. This helped in assessing whether the worksite drug-prevention/health-promotion programs had any effect in encouraging employees to seek and get help in the short term and in decreasing substance abuse–related medical claims in the long term. Results evaluations are the most effective for determining the actual benefits of a program.

Top of page

After you have completed the evaluation, it is time to compare the results with your baseline and the goals of the program. If you have a comparison site that did not receive the program, you also can compare the evaluation results with data from the comparison site.

Cost savings may take some time to accumulate, but employee morale may be evident right away. A drug-free workplace policy and program can serve multiple functions. The main function, which is most obvious, is to reduce or eliminate workplace drug use and its effects. Another function may be to increase the safety in the workplace and the public it serves. Yet other functions of a drug-free workplace policy and program may be less obvious. Promoting healthy behaviors at work, encouraging workers to develop a team spirit, and creating a supportive work environment may all be functions or byproducts of the drug-free workplace policy and program, whether these were your intentions or not. Make sure to measure all of these when evaluating the program’s success.

Another thing to keep in mind is the adage “formal evaluation can kill useful programs.” This is not to imply that formal evaluations are not important, but just that they tend to concentrate on the intended functions and that they may not pick up on and give credit to functions and achievements that were not considered important when the program was first designed and implemented. So, if the evaluation turns out negative, the drug-free workplace team may conclude that the program did not work, when, in practice, the program produced some significant favorable results. To avoid this trap, evaluations should concentrate on multiple processes and multiple outcomes instead of a single criterion.

4. THE SEASONS OF EVALUATION: AN ONGOING EVALUATION CYCLE

Evaluation should be ongoing, much like assessment. Do not be discouraged if you do not see results right away. The cycle of evaluation encourages ongoing reassessment of the goals, objectives, means for achieving those objectives, and ways of measuring processes and outcomes. An initiative can fail at any point in the cycle, so it is important to continue the evaluation process. The evaluation results will help you and your drug-free workplace team make informed decisions about whether, and in what ways, your policy and program should be changed.

Top of page

Stages in the Cycle

1. Articulate the goals of the drug-free workplace policy and program and develop a set of objectives that should lead to the accomplishment of those goals. The goals may be as simple as responding to the requirements of legislation.
   
   
2. Design strategies that will accomplish the objectives with the resources available. (See “Plan and Implement a Program.”)
   
   
3. Determine the key processes and outcomes (that is, the major kinds of activities and accomplishments for which you will be evaluating change). Try to select assessment methods that can measure your organization’s progress toward achieving your objectives and goals. And assess not only the outcomes but also the processes that contribute to those outcomes.
   
   
4. Gather, analyze, and interpret assessment data.
   
   
5. Continually improve processes and outcomes, using the results of the assessment. Often overlooked, this is an important step in the cycle.

Evaluation Checklist: Questions to Ask Your Team

——— Was there an initial assessment? Was it completed? Does it seem to have been done well?

——— Did the assessment identify workplace needs? Do these seem to have been correctly identified?

——— Does the program have objectives that address the identified workplace needs?

——— Are the right employees getting the program? For example, if your program is about safe driving behaviors, perhaps the drivers need to attend but the employees in the IT department do not.

——— Does the program seem to be working as it was intended? That is, is the content being delivered?

——— Are the employees learning what you intended them to learn?

——— Are the employees actually applying the knowledge learned?

——— Is the workplace achieving the established goals?

——— What can you do better next time?

Top of page

Conclusion

Evaluation does not need to be costly or elaborate. However, for a drug-free workplace initiative to fully succeed, some kind of evaluation is essential. Without any evaluation, you can easily spend time and money on programs that fail. By contrast, with a systematic approach that includes evaluation, you can more clearly see where you are, where you are going, how you can get there, and, in the end, whether you got there. In short, an evaluation can help you improve your efforts to create a drug-free workplace that is safe, healthy, and productive.

For more information about how to evaluate the costs and benefits of having (or not having) a Drug-Free Workplace policy and program, see http://www.dwp.samhsa.gov/YIW/Pages/Cost%20Analysis.aspx/.

References

1. Dan Nakaso. 2005. “Zippy’s New Drug Policy Identifies ‘Problem’ Workers.” The Honolulu Advertiser, June 26.
   
2. Paul Connolly and Kathleen Groll Connolly. 2006. Employee Surveys: Practical and Proven Methods, Samples, Examples (Second Edition). Old Saybrook, Connecticut: Performance Programs, Inc.
   
3. Centers for Disease Control and Prevention. 1999. Framework for Program Evaluation in Public Health. MMWR 48 (No. RR–11). Available at http://www.cdc.gov/eval/framework.htm/.
   
4. William K. Kurtines, Laura Ferrer–Wreder, Stephen L. Berman, Carolyn Cass Lorente, Ervin Briones, Marilyn J. Montgomery, Janene Bussell, Ondina Arrufat, and Sara K. Swenson. 2007 (in press). “Promoting Positive Youth Development: The Impact of Outreach Research on New Directions in Developmental Theory—The Miami Youth Development Project (YDP).” Journal of Adolescent Research.
   
5. Paul Connolly and Kathleen Groll Connolly. 2006. Employee Surveys: Practical and Proven Methods, Samples, Examples (Second Edition). Old Saybrook, Connecticut: Performance Programs, Inc.
   
6. Dana Gaines Robinson and James C. Robinson. 1989. Training for Impact: How to Link Training to Business Needs and Measure the Results. San Francisco, California: Jossey Bass Publishers.
   
7. Donald L. Kirkpatrick.1994. Evaluating Training Programs: The Four Levels (Second Edition). American Society for Training and Development.
   
8. Jeffrey S. Hornsby and Donald F. Kuratko. 2005. Frontline HR: A Handbook for the Emerging Manager. Crawfordsville, Indiana: R.R. Donnelley.
   
9. Laurie J. Bassi and Mark E. Van Buren. 1999. “The 1999 ASTD State of the Industry Report.” Training and Development Journal (Supplement):1–26.
   

Top of page