Print-friendly version with easy to fill worksheets (PDF 131 KB)

Evaluation Manual: Step 1 - Engage Stakeholders

Introduction to Program Evaluation for Public Health Programs
Evaluating Appropriate Antibiotic Use Programs
Released April 2006

The first step in the CDC Framework approach to program evaluation is to engage the stakeholders. Stakeholders are people or organizations that are invested in the program, are interested in the results of the evaluation, and/or have a stake in what will be done with the results of the evaluation. Representing their needs and interests throughout the process is fundamental to good program evaluation.

Key stakeholders for evaluations of public health programs fall into three major groups:

• Those involved in program operations: Management, program staff, partners, funding agencies, and coalition members.
• Those served or affected by the program: Patients or clients, advocacy groups,
  community members, and elected officials.
• Those who are intended users of the evaluation findings: Persons in a position to make decisions about the program, such as partners, funding agencies, coalition members, and the general public or taxpayers.

Clearly, these categories are not mutually exclusive; in particular, the primary users of
evaluation findings are often members of the other two groups, i.e., the program management or an advocacy organization or coalition. While you may think you know your stakeholders well, these categories help you to think broadly and inclusively in identifying stakeholders.

Back to top

Stakeholders can help (or hinder) an evaluation before it is conducted, while it is being
conducted, and after the results are collected and ready for use. Because so many public health efforts are complex and because public health agencies may be several layers removed from frontline implementation, stakeholders take on particular importance in ensuring that the right evaluation questions are identified and that evaluation results will be used to make a difference. Stakeholders are much more likely to support the evaluation and act on the results and recommendations if they are involved in the evaluation process. Conversely, without stakeholder support, your evaluation may be ignored, criticized, resisted, or even sabotaged.

In reviewing the long list of stakeholders that might be generated in the three generic categories, use of some or all of the evaluation standards will help identify those who matter most.

Use of results will be enhanced if you give priority to those stakeholders who

• Can increase the credibility of your efforts or your evaluation
• Are responsible for day-to-day implementation of the activities that are part of the
  program
• Will advocate for or authorize changes to the program that the evaluation may
  recommend
• Will fund or authorize the continuation or expansion of the program.

In addition, to be proper/ethical and accurate, you need to include those who participate in the program and are affected by the program or its evaluation.

The worksheets at the end of this chapter are intended to help you identify key stakeholders. For example, in using the worksheets with the Childhood Lead Poisoning Prevention (CLPP) program, we identified the stakeholders in the sample worksheet 1A (see Table 1.1). Note that some stakeholders appear in more than one column; these are not exclusive classes of stakeholders so much as four ways of thinking about stakeholders to ensure we were thinking as broadly as possible. Second, note that not all categories have the same number of stakeholders. Indeed, for a simple project, there may be very few stakeholders and some categories may have none at all. The sample worksheet 1B (see Table 1.2) helped us identify the perspectives and needs of these key stakeholders and the implications for designing and implementing our evaluation. Note in the CLPP example that while all stakeholders may applaud our efforts to reduce EBLL in children, several stakeholders put priority on outcomes that might or might not agree with our priorities. For example, private physicians are most interested in “yield” of their screening efforts, while Congress cares about cost-effectiveness. Note that advocacy groups, in addition to specific outcomes that may be priorities for them, also have some preferences related to data collection—expressing a preference for methods other than surveys. All of these insights are helpful at the start of an evaluation to ensure that the evaluation goes smoothly and the results are used.

Table 1.1 - CLPP Example: Identifying Stakeholders

Who are the key stakeholders we need to...

• Increase credibility of our efforts:
  • Physician associations
• Community associations
• Implement the interventions that are central to this effort:
  • State and local health departments
  • Housing authorities
• Advocate for changes to institutionalize this effort:
  • Advocacy groups
  • Maternal and child health groups
  • Physician associations
  • Community associations
• Fund/authorize continuation or expansion of this effort:
  • Legislators and policymakers at federal and state levels
  • CDC
  • Private industry
  • Court system

Table 1.2 - CLPP Example: What Matters to Stakeholders

Six stakeholders and what component of intervention/outcome matters most to them?

1. Physician associations
   • Sufficient “yield” of EBLL children to make their screening
     efforts “worth their time.”
   • Clear referral mechanisms that are easy and work.
2. Community associations
   • Cleaning up housing in their neighborhood.
   • Support for families with EBLL children.
3. Housing authorities
   • No additional monetary and time burden for toxic clean-ups.
4. State and local health departments
   • Efforts lead to improved health outcome for EBLL children.
5. Advocacy groups
   • EBLL is seen as a housing problem and not a “failure” or example of bad child-rearing by poor families.
   • No survey data collection with families.
6. Congress and policymakers
   • Efforts lead to improved health outcomes.
   • “Cost-effectiveness” of the effort.

Back to top

Throughout the evaluation planning process, you will be asking some or all stakeholders the following questions:

• Who do you represent and why are you interested in this program?
• What is important about this program to you?
• What would you like this program to accomplish?
• How much progress would you expect this program to have made at this time?
• What do you see as the critical evaluation questions at this time?
• How will you use the results of this evaluation?
• What resources (i.e., time, funds, evaluation expertise, access to respondents, and access to policymakers) might you contribute to this evaluation effort?

Back to top

Stakeholder perspectives may influence every step of the CDC Framework. Obviously, stakeholder input in “describing the program” ensures a clear and consensual understanding of the program’s activities and outcomes. This is an important backdrop for even more valuable stakeholder input in “focusing the evaluation design” to ensure that the key questions of most importance will be included. Stakeholders may also have insights or preferences on the most effective and appropriate ways to collect data from target respondents. In “justifying conclusions,” the perspectives and values that stakeholders bring to the project are explicitly acknowledged and honored in making judgments about evidence gathered. Finally, the considerable time and effort spent in engaging and building consensus among stakeholders pays off in the last step, “ensuring use,” because stakeholder engagement has created a market for the evaluation results. Stakeholders can be involved in the evaluation at various levels. For example, you may want to include coalition members on an evaluation team and engage them in developing questions, data collection, and analysis. Or consider ways to assess your partners’ needs and interests in the evaluation, and develop means of keeping them informed of its progress and integrating their ideas into evaluation activities. Again, stakeholders are more likely to support the evaluation and act on results and recommendations if they are involved in the evaluation process.

In addition, it can be beneficial to engage your program’s critics in the evaluation. In some cases, these critics can help identify issues around your program strategies and evaluation information that could be attacked or discredited, thus helping you strengthen the evaluation process. This information might also help you and others understand the opposition’s rationale and could help you engage potential agents of change within the opposition. However, use caution: It is important to understand the motives of the opposition before engaging them in any meaningful way.

This emphasis on engaging stakeholders mirrors the increasing prominence in the research community of participatory models or “action” research. A participatory approach combines systematic inquiry with the collaboration of diverse stakeholders to meet specific needs and to contend with broad issues of equity and justice. As noted earlier, The Study of Participatory Research in Health Promotion, commissioned by the Royal Society of Canada, has published a set of guidelines for use by evaluators and funding agencies in assessing projects that aspire to be participatory. (Green LW, George MA, Daniel M, Frankish CJ, Herbert CP, Bowie WR, et al. op cit.) The guidelines emphasize that traditional ways of conducting health research in populations must adapt to meet the educational, capacity-building, and policy expectations of more participatory approaches if the results of the research are to make a difference.

Standards for Step 1: Engage Stakeholders

• Utility - Who will use these results?
• Feasibility - How much time and effort can be devoted to stakeholder engagement?
• Propriety - Which stakeholders need to be consulted to conduct an ethical evaluation, for example, to ensure we will identify negative as well as positive aspects of the program?
• Accuracy - How broadly do we need to engage stakeholders to paint an accurate picture of this program?

Back to top

__ Identify stakeholders, using the three broad categories discussed: those affected, those involved in operations, and those who will use the evaluation results.

__ Review the initial list of stakeholders to identify key stakeholders needed to improve credibility, implementation, advocacy, or funding/authorization decisions.

__ Engage individual stakeholders and/or representatives of stakeholder organizations.

__ Create a plan for stakeholder involvement and identify areas for stakeholder input.

__ Target selected stakeholders for regular participation in key steps, including writing the program description, suggesting evaluation questions, choosing evaluation questions, and disseminating evaluation results.

Worksheet 1A - Identifying Key Stakeholders

1. Category: Who is affected by the program?
   Stakeholder: _________________

2. Category: Who is involved in program operations?
   Stakeholder: _________________

3. Category: Who will use evaluation results?
   Stakeholder: _________________

Which of these are key stakeholders we need to engage to:

• Increase credibility of our evaluation
• Implement the interventions that are central to this evaluation
• Advocate for changes to institutionalize the evaluation findings
• Fund/authorize the continuation or expansion of the program

Worksheet 1B - What Matters to Stakeholders

Fill in the blanks.

Back to top

Step 1: Engage Stakeholders

Stakeholders for appropriate antibiotic use programs may include:

Those involved in program operations:

• Program managers and staff
• Local, state, and regional coalitions interested in reducing inappropriate antibiotic use
• State and local health departments
• Funding agencies, such as national and state governments

Those served or affected by the program:

• Physicians, nurse practitioners, pharmacists, and other healthcare providers
• Healthcare systems and the medical community
• Managed care organizations and healthcare delivery organizations
• Healthcare insurers and insurer organizations
• Schools and educational groups
• Universities and educational institutions
• Parent Teacher Associations (PTAs)
• Childcare providers and organizations of childcare providers
• Community organizations
• Consumer advocacy groups
• Patients and the general public

The intended users of the evaluation results will vary with each specific evaluation, and often the users comprise a subset of the individuals and groups listed in the prior two categories. Hence, the many potential users of a specific evaluation’s results might include:

• Program managers and staff
• Local, state, and regional coalitions interested in reducing inappropriate antibiotic use
• State and local health departments
• Funding agencies, such as national and state governments

Why Stakeholders Matter

Evaluations of appropriate antibiotic use efforts, like evaluations of other public health efforts, will benefit greatly from the involvement of diverse groups of partners and stakeholders. When appropriate antibiotic use programs are planned and implemented by coalitions, coalition members should also be engaged in the planning and implementation of the program’s evaluation. Target populations such as patients and providers should be involved in program evaluation to ensure that the evaluation focus meets their needs and that the evaluation is ethical. Engaging target populations in planning evaluation activities will also help ensure that the evaluation is feasible and accurate. For example, patient questionnaires will yield much better information on patients’ knowledge and attitudes if groups of patients have reviewed the questions to make sure they are clear and understandable. Similarly, providers will be far more likely to complete a questionnaire if they have helped design a plan for implementation that does not disrupt clinic flow.

Often, groups of stakeholders will define program success differently; therefore, it is important to understand stakeholders’ different interests and expectations from the start. Worksheet 1B can help you determine which components of the program and which outcomes matter most to various stakeholders. Epidemiologists and other health department staff may assume that a “successful” program would result in reductions in antibiotic resistance rates or slower increases in these rates as compared to a control group. Managed care organizations and other health delivery organizations may look to reduced costs as a measure of success (e.g., decreased prescriptions for antibiotics or decreased number of office visits) in addition to improvements in quality of care. Health educators often look at changes in knowledge, attitudes, and behaviors as indicators of success, especially when these intermediate outcomes are quicker to change or easier to measure than more long-term outcomes.

The Role of Stakeholders in Program Evaluation

As discussed earlier, stakeholders can be involved at various levels of program evaluation. Stakeholders can contribute to the program description, suggest or choose evaluation questions, and disseminate evaluation results. Including stakeholders can inspire a change in focus during program planning or program evaluation. For example, a group of healthcare providers may cite high patient demand for antibiotics when describing the problem of antibiotic resistance and inappropriate antibiotic use. Their definition of the problem might lead to a program based on educating consumers about the risks of overuse of antibiotics. If consumers were involved in efforts to define the problem, the resulting program could look quite different. Consumers may say that their medical providers don’t listen to their complaints or explain their diagnosis and treatment and that they feel rushed by short office visits. While consumer education would still be an important component, consumer input to the definition of the problem illustrates the need to examine the structure of office visits as well as provider skills in communicating with patients.

Stakeholders can also play an important role in crafting evaluation tools. Healthcare providers can provide useful insight when drafting and selecting evaluation questions for participants in provider education components of appropriate antibiotic use programs. Finally, stakeholders can play key roles in disseminating evaluation results. For example, professional medical societies and managed care organizations can distribute evaluation findings to providers through newsletters, mailings, and other contact with members.

Back to top