|
|
|
State Program Evaluation Guides:
|
Guides: | |
Writing SMART Objectives | |
Developing and Using a Logic Model | |
Developing an Evaluation Plan | |
Fundamentals of Evaluating Partnerships | |
The evaluation guide Writing SMART Objectives is aimed at helping states develop realistic and measurable objectives. This guide describes the components of a SMART objective and provides examples. An exercise at the end offers an opportunity to work through the development of SMART objectives. We will appreciate your comments on the utility and applicability of this guide.
As part of the program development process, states propose objectives and develop a work plan and an evaluation plan. Both these plans are based on proposed objectives so it is important that they are developed using the SMART approach. Such objectives offer specific, relevant, and measurable benchmarks to measure achievement of your state’s HDSP program goals and will serve as the foundation for your program activities.
In program planning, we often think of what we want to accomplish in terms of goals and objectives so that we can easily explain our expectations to others.
A goal is a statement that explains what the program wishes to accomplish. It sets the fundamental, long-range direction. Typically, goals are broad general statements.
Example: Improve control of high blood pressure in (state).
Objectives break the goal down into smaller parts that provide specific, measurable actions by which the goal can be accomplished. Objectives define for our stakeholders and partners the results we expect to achieve in our program or intervention. For our program expectations to be clear, we must write clear, concise objectives.
The two general types of objectives are process and outcome. Process objectives focus on the activities to be completed in a specific time period. They enable accountability by setting specific activities to be completed by specific dates. Process objectives explain what you are doing and when you will do it. They describe participants, interactions, and activities.
Example: By June 30, 2006, provide training for 20 community health center administrators in the use of electronic medical records.
Outcome objectives express the intended results or accomplishments of program or intervention activities. They most often focus on changes in policy, a system, the environment, knowledge, attitudes, or behavior.
Objectives can also be thought of as short-term, intermediate or long-term.
Short-term objectives are generally expected immediately and can occur soon after the program or intervention is implemented, very often within a year.
Intermediate objectives result from and follow short-term outcomes.
Long-term objectives state the ultimate expected impact of the program or intervention.
Example: Let’s put all these concepts together to form a series of related objectives.
Goal | Improve control of high blood pressure in the state. |
Long-term Objective | By December 30, 2010, increase from baseline to 65% the percentage of hypertension patients at community health centers whose blood pressure is under control. |
Intermediate Objective | By June 28, 2008, increase from 2 to 10 the number of community health centers that have implemented use of electronic medical records with provider reminders of high blood pressure treatment guidelines. (There are currently 15 community health centers in the state.) |
Short-term Objective | By June 28, 2006, increase from 2 to 5 the number of community health centers that have provided staff training on the JNC7 guidelines. |
For the short-term objective listed above, activities or process objectives could include the following:
Objectives are meant to be realistic targets for the program or project. They are written in the active voice and use action verbs such as plan, write, conduct, and produce (rather than more vague terms like learn, understand, feel). Well-written objectives will always answer the following question:
WHO is going to do WHAT, WHEN, and TO WHAT EXTENT?
States develop an HDSP work plan using short-term, intermediate, and long-term objectives for capacity building and interventions. Capacity building objectives will likely focus on the achievement of products or processes, such as developing a state plan, a partnership, or a burden document that will help the state reduce heart disease and stroke. Intervention objectives should focus on processes such as training, or on outcomes such as systems change or health status change. The state work plan will also include activities or processes needed to achieve proposed objectives.
One way to develop well-written objectives is to use the SMART approach. Developing specific, measurable objectives requires time, orderly thinking, and a clear picture of the results expected from program activities. The more specific your objectives are, the easier it will be to demonstrate success.
SMART stands for
Specific
Measurable
Attainable/Achievable
Relevant
Time bound
Specific—What exactly are we going to do for whom?
The “specific” part of an objective tells us what will change for whom in concrete terms. It identifies the population or setting, and specific actions that will result. In some cases it is appropriate to indicate how the change will be implemented (e.g., through training, or through implementation of the Chronic Care Model). Coordinate, partner, support, facilitate, and enhance are not good verbs to use in objectives because they are vague and difficult to measure. On the other hand, verbs such as provide, train, publish, increase, decrease, schedule, or purchase indicate clearly what will be done.
Measurable—Is it quantifiable and can WE measure it?
Measurable implies the ability to count or otherwise quantify an activity or its results. It also means that the source of and mechanism for collecting measurement data are identified, and that collection of these data is feasible for your program or partners.
A baseline measurement is required to document change (e.g., to measure percentage increase or decrease). If the baseline is unknown or will be measured as a first activity step, that should be indicated in the objective as “baseline to be determined using HRSA database, 2005.” The data source you are using and the year the baseline was obtained should always be specified in or adjacent to your objective statement. If a specific measurement instrument is used, you might want to incorporate its use into the objective. For example, “By June 2007, increase the proportion of physicians at the Green Clinic who are 100% compliant with the JNC7 Guidelines from 70% to 80% as measured by the Physician Guideline Self-Assessment Tool.” specifies not only the performance measure, but the data source as well.
Another important consideration is whether change can be measured in a meaningful and interpretable way given the accuracy of the measurement tool and method. For example, to estimate population awareness of the signs and symptoms of heart attack, we estimate awareness using a sample of the state population. Since this is an estimate, there is a chance of error associated with it—usually expressed by a confidence interval (the point estimate, plus or minus an estimate of variability). Projecting a very small change in population awareness, although measurable, might not be meaningful because the change projected falls within expected variability or within the bounds of the confidence interval for population awareness.
Attainable/Achievable—Can we get it done in the proposed time frame with the resources and support we have available?
The objective must be feasible with the available resources, appropriately limited in scope, and within the program’s control and influence.
Sometimes, specifying an expected level of change can be tricky. To help identify a target, talk with an epidemiologist, look at historical trends, read reports or articles published in the scientific or other literature, look at national expectations for change, and look at programs with similar objectives. Consult with partners or stakeholders about their experiences. Often, talking to colleagues in other states who have implemented similar programs or interventions can provide you with information about expected change.
In some situations, it is more important to consider the percentage of change as a number of people when discussing impact. Will the effort required to create the amount of change be a good use of your limited resources?
For example, our intervention might be intended to increase awareness of the symptoms of stroke and the need to call 9–1–1 among patients in a statewide health clinic system. If as a result of our intervention we measure a 5% increase in awareness among all clinic patients, but 5% of our population is a very small number, we might want to consider the cost of the intervention relative to the number of people affected. We could choose to enhance the intervention for a greater impact or not implement that intervention again.
Relevant—Will this objective have an effect on the desired goal or strategy?
Relevant relates to the relationship between the objective and the overall goals of the program or purpose of the intervention. Evidence of relevancy can come from a literature review, best practices, or your theory of change. For state HDSP programs, the objective should accomplish one of the following:
For example, although it may be important to public health, assessing the height and weight of high school students does not directly lead to change in an HDSP priority area for an at-risk population. Also, the intervention focuses at an individual level rather than on a system change.
Time bound—When will this objective be accomplished?
A specified and reasonable time frame should be incorporated into the objective statement. This should take into consideration the environment in which the change must be achieved, the scope of the change expected, and how it fits into the overall work plan. It could be indicated as “By December 2010, the HDSP program will” or “Within 6 months of receiving the grant,...”
Writing SMART objectives also helps you to think about and identify elements of the evaluation plan and measurement, namely indicators and performance measures.
An indicator is what you will measure to obtain observable evidence of accomplishments, changes made, or progress achieved. Indicators describe the type of data you will need to answer your evaluation questions. A SMART objective often tells you what you will measure.
Consider the example “By February 15, 2006, increase by four the number of community health centers in [State] that have incorporated into the clinic system electronic medical records with reminders of treatment protocols.” The indicator is
the number of community health centers in [State] that have incorporated electronic medical records with reminders of treatment protocols into the clinic system.
A performance measure is the amount of change or progress achieved toward a specific goal or objective. SMART objectives can serve as your performance measures because they provide the specific information needed to identify expected results.
Consider the example, “By February 15, 2006, increase by four the number of community health centers in [State] that have incorporated into the clinic system electronic medical records with reminders of treatment protocols.” The performance measure is
increase by four the number of community health centers in [State] that have incorporated electronic medical records with reminders of treatment protocols into the clinic system.
To develop SMART objectives, use the template below and fill in the blanks:
By_____/_____/_____,
___________________________________________________
[WHEN—Time bound] [WHO/WHAT—Specific]
from _____________________ to
__________________________________________
[MEASURE (number, rate, percentage of change and baseline)—Measurable]
Next, review your objective to assure that it is achievable and relevant to your overall program goals.
Here are some examples of SMART objectives:
Each of these objectives is either a capacity building activity or directly relates to one of the HDSP program areas and will be a policy and systems-level change.
Take the following objectives and “make them SMART.”
1. Increase the number of HDSP partners.
______________________________________________________________
______________________________________________________________
2. Train physicians on clinical practice guidelines.
______________________________________________________________
______________________________________________________________
3. Enhance EMS policy to decrease stroke deaths in 2002.
______________________________________________________________
______________________________________________________________
4. By June 2007, increase by 10% the percentage of state residents that know
the signs and symptoms of stroke and heart attack.
______________________________________________________________
______________________________________________________________
5. Increase the number of work sites that adopt heart-healthy insurance
options.
______________________________________________________________
______________________________________________________________
There are many ways that these objectives could be made SMARTer. For each example, several things to think about as you review your answer are provided below.
This guide was developed for the Division for Heart Disease and Stroke Prevention under the leadership of Susan Ladd and Jan Jernigan in collaboration with Nancy Watkins, Rosanne Farris, Belinda Minta, and Sherene Brown.
State Heart Disease and Stroke Prevention programs were invaluable in the development and fine-tuning of this guidance document. Their review contributed significantly to the clarity and utility of this guide. Special thanks are extended to the following:
Susan Mormann, North Dakota Department of Health,
Ghazala Perveen, Kansas Department of Health and Environment,
Ahba Varma, North Carolina Department of Health and Human Services, and
Namvar Zohoori, Arkansas Department of Health and Human Services.
We encourage readers to adapt and share the tools and resources in the document to meet program evaluation needs. For further information, contact the Division for Heart Disease and Stroke Prevention, Applied Research and Evaluation Branch at cdcinfo@cdc.gov or (990) 488–2424.
American Heart Association. Heart Disease and Stroke Statistics–2006 Update. Dallas, Tex: American Heart Association; 2006.
Centers for Disease Control and Prevention. Prevention Works: CDC
Strategies for a Heart-Healthy and Stroke-Free America. Atlanta, GA:
U.S. Department of Health and Human Services; 2003. Available at
http://www.cdc.gov/DHDSP/library/prevention_works/index.htm.
Page last reviewed: October 15, 2008
Page last modified: October 15, 2008
Content source: Division for Heart Disease and Stroke
Prevention,
National Center for Chronic Disease Prevention and
Health Promotion
|
||||||||||||
|