Department of Health and Human Services
Centers for Disease Control and Prevention
 |
|||||||||||||
|
|
|
|
|
||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
State Program Evaluation Guides:
|
|||||||||||||||||||||||||||
 |
|
|
|
|
| Guides: | |
 |
Writing SMART Objectives |
|
|
Developing and Using a Logic Model |
|
|
Developing an Evaluation Plan |
|
|
Fundamentals of Evaluating Partnerships |
|
|
|
 |
|
The evaluation guide Logic Models offers a general overview of the development and use of logic models as planning and evaluation tools. A feedback page is provided at the end of this guide. We will appreciate your comments.
Logic models are tools for planning, describing, managing, communicating, and evaluating a program or intervention. They graphically represent the relationships between a program’s activities and its intended effects, state the assumptions that underlie expectations that a program will work, and frame the context in which the program operates. Logic models are not static documents. In fact they should be revised periodically to reflect new evidence, lessons learned, and changes in context, resources, activities, or expectations.
Logic models increase the likelihood that program efforts will be successful because they
State programs should develop logic models to describe
Electronic logic model templates can be created fairly easily in either a Microsoft Word table or a Microsoft Excel work sheet. A sample template is provided as an appendix.
As with many aspects of evaluation, people use a variety of terms to describe logic models and their components. A logic model can also be visually represented in a variety of ways, including as a flow chart, a map, or a table. The only “rule” for a logic model is that it be presented on one page. The basic components of a good logic model are:
A basic logic model (Figure 1) typically has two “sides”—process and outcome. The process section describes the program’s inputs (resources), activities, and outputs (direct products). The outcome section describes the intended effects of the program, which can be short term, intermediate, and/or long term. Assumptions under which the program or intervention operates, and the contextual factors can also be included in a logic model. They are often noted in a box below or on the left side of the logic model diagram. Figure 1, below, illustrates the components of a logic model.
Figure 1. Layout of a General Logic Model
Inputs are the resources that go into a program or intervention—what we invest. They include financial, personnel, and in-kind resources from any source. For example, inputs could include
Various funding sources for your program.
Your partners.
Staff time and technical assistance.
Activities are events undertaken by the program or partners to produce desired outcomes—what we do. You could include a clear identification of “early” activities and “later” activities. Examples of activities include
Create a state-level partnership.
Train health care partners and staff in clinical guidelines.
Develop a community health communication campaign on signs and symptoms of stroke, and to call 9–1–1.
Outputs are the direct, tangible results of activities—what we get. These early work products often serve as documentation of progress. Examples include
State-level partnerships created.
Health care professionals trained in clinical guidelines.
Community health communication campaigns developed.
Outcomes are the desired results of the program—what we achieve.
Describing outcomes as short, intermediate, or long term depends on the objective, the length of the program, and expectations of the program or intervention. What is identified as a long-term outcome for one program could be an intermediate outcome for another.
Short-term outcomes are the immediate effects of the program or intervention activities. They often focus on the knowledge and attitudes of the intended audience. Examples include
Increase partner knowledge of HDSP priorities and strategies.
Increase physician knowledge of clinical guidelines.
Increase knowledge of signs and symptoms of stroke and of the need to call 9–1–1.
Intermediate outcomes are behavior, normative, and policy changes. Examples include
HDSP State Plan has been developed and published with partner involvement.
Health systems implement clinical guidelines.
Decrease transport time to treatment for stroke victims.
Long-term outcomes refer to the desired results of the program and can take years to accomplish. Long-term outcomes include
Increase in statewide policy and environmental strategies for HDSP.
Increase in blood pressure control in a health center population.
Increase in early treatment for stroke.
Impacts refer to the ultimate impacts of the program. They could be achieved in a year or take 10 or more years to achieve. These may or may not be reflected in the logic model, depending on the purpose and audience of the logic model. A logic model that portrays an HDSP intervention may show expected long-term outcomes, such as a state-level system change, and impact, such as a population-wide reduction in death rate. Examples of impacts include
Decrease in the rate of death due to heart disease.
Eliminate disparities in treatment for stroke between general and priority populations.
Assumptions are the beliefs we have about the program or intervention and the resources involved. Assumptions include the way we think the program will work—the "theory" we have used to develop the program or intervention. (See the subsequent section on Theories of Change.) Assumptions are based on research, best practices, past experience and common sense. The decisions we make about implementing a program or intervention are often based on our assumptions. Examples of assumptions we sometimes make include
Funding will be secure throughout the course of the project.
Because we teach information, it will be adopted and used in the way we intended.
Professionals will be motivated to attend learning sessions.
External funds and well-placed change agents can facilitate institutional change.
Staff with the necessary skills and abilities can be recruited and hired.
Partnerships or coalitions can effectively address problems or reach into areas we cannot.
Policy adoption leads to individual behavior change.
In developing your logic model, you should explore and discuss the assumptions you are making. Often, an in-depth discussion is included as a narrative that accompanies your logic model. Inaccurate or overlooked assumptions could be a reason that your program or intervention did not achieve the expected level of success.
Contextual factors describe the environment in which the program exists and external factors that interact with and influence the program or intervention. These factors may influence implementation, participation, and the achievement of outcomes. Contextual factors are the conditions over which we have little or no control that affect success.
Examples include
Competing or supporting initiatives sponsored by other agencies.
Socioeconomic factors of the target audience.
The motivations and behavior of the target population.
Social norms and conditions that either support or hinder your outcomes in reaching disparate populations, such as the background and personal experiences of participants.
Politics that support or hinder your activities.
Potential barriers or supports that could affect the success of your project.
In program or intervention planning and development, we should consider contextual factors that are likely to affect our activities and either address them or collect data on them as part of the process evaluation.
Determine the purpose of the logic model, who will use it
and for what? Is your purpose to develop a work plan, to talk with
stakeholders about the program or intervention, or to develop an evaluation
plan?
Convene stakeholders. Who should participate? Program planners
and managers, epidemiologists, and groups with a stake in program outcomes.
Determine a focus for the logic model. Will the logic model
depict a single intervention, a multiyear intervention, or a comprehensive
picture of your HDSP program? Determine what level of detail is needed to make
this a useful tool.
Understand the situation. Use the program objective or goal as
your anchor. Set priorities and clarify expectations.
Explore the research, knowledge base, and what others have
done/are doing. Compile research findings and lessons learned, applicable
program theory, and resources. Identify and discuss assumptions you are making
and contextual factors.
Construct a series of linked activities and outcomes or statements using a “left-to-right” or “right-to-left” approach. Then connect the activities with arrows to show linkages.
One way to proceed is using a “left-to-right” process by connecting a series of “If, then” statements that help you identify and connect activities and anticipated outcomes.
Ask yourself how you can complete the following to describe your program:
If we have ________ and, we can (do) ______ and ______, which will result in ________ and _______.
The first two blanks list the resources available to conduct your program, the third and fourth blanks describe the activities to be conducted, and the final two blanks list the expected outputs of those activities.
Example: “If we have program funding and participating clinics, we can inform our clinic partners of the need to implement clinical practice guidelines and sponsor training for clinic teams on the chronic care model, which will then increase the number of clinic teams who are aware of clinical practice guidelines and who implement the chronic care model.
By asking other similar questions, you can determine your short-, intermediate-, and long-term outcomes.
If we educate clinic teams and train them in the chronic care model in clinics, then we will see _________ and __________ occur in the short-term.
Example: “If we educate clinic teams and train them in the chronic care model in clinics, we will see implementation of the chronic care model resulting in appropriate treatment for patients with high blood pressure.”
Continuing with the flow of the logic model, you should next complete:
If clinics implement the chronic care model and have an increase in appropriate treatment for high blood pressure (short-term outcomes), then we will see ____________ occur (intermediate outcomes).
Example: “If clinics use the chronic care model and increase appropriate treatment for patients with high blood pressure, then we will see an increase in the number of patients with high blood pressure under control.”
Next, consider what the accomplishment of intermediate outcomes will lead to:
If there is an increase in the number of current clinic patients whose high blood pressure is under control, then we expect that to lead to ____________ (long-term outcomes).
Example: “If there is an increase in the number of current clinic patients whose high blood pressure is under control, then we will see a reduction in heart disease and stroke among these patients.”
Finally, identify contextual factors and assumptions that should be considered and stated when developing the logic model and interventions. In the example above, although we expect that controlling high blood pressure in an individual will reduce their risk for heart disease and stroke, when we apply this theory to a population, there are a number of confounding factors
Risk factors for high blood pressure such as obesity and
diabetes are increasing in prevalence. This is likely to cause an increase
in the prevalence of high blood pressure and the number of heart disease or
stroke patients.
We assume in this model that once control of high blood pressure
has been achieved, it will be maintained. This might not be the case.
We assume that once the chronic care model is implemented and clinic-based changes occur, the changes are maintained.
If we put this all together in a logic model, it would look like this
As you develop your logic model, remember the amount and types of resources, activities, and outcomes depicted can vary and are particular to each program. Some programs will have an abundance of resources that allow a variety of activities and other programs may choose to conduct fewer activities. The activities and expected outcomes are based on the type of program or intervention you are implementing, the resources you have available and their distribution, the needs and desires of your program or department, and your partners.
In a logic model, arrows are drawn to indicate the links between resources, activities, and outcomes. A theory of change is used to provide a rationale for the expected links between program resources, activities, and outcomes. It explains how and why activities are expected to lead to outcomes in the particular order depicted.
Health promotion and prevention activities are based on numerous theories of change—a reasonable explanation of why and how a certain set of activities leads to certain outcomes. These theories are based on our beliefs, expectations, experience, and conventional wisdom. They describe the set of assumptions that explain both the steps that lead to long-term objectives and the connections between program activities and outcomes that occur at each step of the way.
Several common theories of change are used in health programming. To learn more about theories of change, the following Web sites will be useful:
http://www.csupomona.edu/~jvgrizzell/best_practices/bctheory.html*
http://www.cancer.gov/theory/pdf [PDF–3.4M]
Theories of change allow us to hypothesize that a program’s intermediate and long-term outcomes are a result of short-term outcomes, which are a result of the activities implemented. The logic model for the State Heart Disease and Stroke Prevention Program is based on a socioecological model that links environmental and policy or systems changes with individual-level behavioral changes. The “systems” interventions of HDSP result in policy or environmental change that can lead to changes in knowledge and attitudes that reinforce behavior change among individuals and gatekeepers. For example, implementing the Chronic Care Model in a health care system would include use of electronic medical records that remind physicians of services needed to increase the number of patients who have their high blood pressure under control. This, in turn, leads to changes in patient behavior that result in better management of their high blood pressure.
As a planning tool, a logic model clarifies the sequence of outcomes and the relationship between activities and specific outcomes. It helps you
Examine/refine the program mission and vision, goals and
objectives, preferably with stakeholders.
Identify the most important desired outcomes.
Identify the “critical path.” If efforts must be reduced, which
paths are most effective, are likely to get you there quickest, and/or are most
cost-effective?
Identify existing and needed, or weak and strong, components of the program and ways to enhance performance.
Much of the benefit of constructing program logic models comes from the process of discussing, analyzing, and justifying the expected relationships and linkages between activities and expected outcomes with staff and partners.
A logic model is often used to guide evaluation planning. It can help you
Determine what to evaluate.
Identify appropriate evaluation questions based on the program.
Know what information to collect to answer these questions—the indicators.
Determine when to collect data.
Determine data collection sources, methods, and instrumentation.
Using a logic model we can identify four areas, or domains, on which we can focus evaluation activities. The four evaluation domains embedded within the logic model shown in Figure 2 are
Implementation (Process): Is the program or
intervention implemented as planned? Were all of the activities carried out
as expected?
Effectiveness (Outcome): Is the intervention achieving
its intended short-, intermediate-, and/or long-term effects/outcomes?
Efficiency: How much “product” is produced for a given
level of inputs/resources?
Causal Attribution: Is progress on outcomes due to your program or intervention? In public health practice, causal attribution is often difficult to ascertain, especially for your more distant outcomes. However, determining causality between your activities/outputs and your short-term outcomes can often be accomplished without too much effort. Usually, surveys and interviews, or analysis of records can establish causality at that level. And the brief time duration for short-term outcomes usually insures that causal results can be determined in a relatively small amount of time. By using theories of change to develop your logic model you can assume, with more confidence, that intermediate and long-term outcomes are a result of your short-term outcomes. Therefore, it is important to establish causality between at least the activities (and resulting outputs) you carry out and the short-term outcomes.
Figure 2: Evaluation Domains
The boxes and arrows in Figure 2 indicate evaluation points or places where it is logical to ask evaluation questions. As the program or intervention progresses through the logic model—as the intervention matures—new series of evaluation questions can be identified. Outcome evaluation looks back over the entire model. If based on a good process evaluation, the logic model can help identify reasons for less than successful interventions by asking “where did the model break down?”
Using this thinking, the logic model can facilitate mapping evaluation questions and indicators as shown in Figure 3.
Figure 3: Mapping Evaluation Questions and Indicators to the Logic Model
The Healthy People 2010 objectives for heart disease and stroke are national goals to unify and focus work done by states, federal agencies, and nonprofit agencies. State HDSP programs are not directly responsible for these long-term, high-level outcomes; however, state interventions and accomplishments contribute to achieving them. Typically, surveillance data are used to track progress on such long-term outcomes.
The CDC HDSP program logic model is provided in Appendix 1. The logic model was developed to describe the processes and events that are expected from combined state and federal resources and activities to prevent heart disease and stroke. CDC and State activities are outlined in terms of capacity building, surveillance, and interventions. These activities and outcomes result in changes in policy and environmental supports (intermediate outcomes), which in turn influence system or population changes and improve health status (long-term outcomes). A population decrease in premature death and disability (impact) is the ultimate result of program activities. As programs focus efforts on disparate populations, these activities are also expected to eliminate disparities between general and priority populations.
To learn more about logic models, the following sources are helpful:
Taylor–Powell E, Jones L, Henert E. Enhancing Program
Performance with Logic Models; 2002. Retrieved November 2005 from the
University of Wisconsin–Extension Web site:
http://www1.uwex.edu/ces/lmcourse/*
Kellogg Foundation Logic Model Development Guide. Retrieved from
W.K. Kellogg Foundation Evaluation Toolkit: Retrieved October 2005 from
http://www.wkkf.org/default.aspx?tabid=101&CID=281&CatID=281&ItemID=2813669&NID=20&LanguageID=0*
US Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to Program Evaluation for Public Health Programs: A Self Study Guide. Atlanta, GA: Centers for Disease Control and Prevention; 2005.
Appendix 1: CDC HDSP Program Logic Model
Click HERE to view a text version
of this graphic.
Appendix 2: Logic Model Template
This guide was developed for the Division for Heart Disease and Stroke Prevention under the leadership of Susan Ladd and Jan Jernigan in collaboration with Nancy Watkins, Rosanne Farris, Belinda Minta, and Sherene Brown.
State Heart Disease and Stroke Prevention programs were invaluable in the development and fine-tuning of this guidance document. Their review contributed significantly to the clarity and utility of this guide. Special thanks are extended to:
Susan Mormann, North Dakota Department of Health,
Ghazala Perveen, Kansas Department of Health and Environment,
Ahba Varma, North Carolina Department of Health and Human Services, and
Namvar Zohoori, Arkansas Department of Health and Human Services.
We encourage readers to adapt and share the tools and resources in the document to meet program evaluation needs. For further information, contact the Division for Heart Disease and Stroke Prevention, Applied Research and Evaluation Branch at cdcinfo@cdc.gov or (990) 488–2424.
American Heart Association. Heart Disease and Stroke Statistics–2006 Update. Dallas, Tex: American Heart Association; 2006.
Centers for Disease Control and Prevention. Prevention Works: CDC Strategies for a Heart-Healthy and Stroke-Free America. Atlanta, GA: U.S. Department of Health and Human Services; 2003. Available at http://www.cdc.gov/DHDSP/library/prevention_works/index.htm.
*Links to non–Federal organizations are provided solely as a service to our users. Links do not constitute an endorsement of any organization by CDC or the Federal Government, and none should be inferred. The CDC is not responsible for the content of the individual organization Web pages found at this link.
Page last reviewed: October 15, 2008
Page last modified: October 15, 2008
Content source: Division for Heart Disease and Stroke
Prevention,
National Center for Chronic Disease Prevention and
Health Promotion
|
|
|
||||||||||||
|
||||||||||||
|
||||||||||||
|
||||||||||||
|
||||||||||||
|
||||||||||||
|
This document is also available in
