Chapter 4. Evaluation Design

The evaluation design of this initiative was to answer the following questions:

1. What were the results at the clinic level in increasing goal-setting performance?
   1. Summary of Action Plan data.
   2. Assessment scores of clinic progress during the initiative.
2. What were lessons learned at the clinic level in implementing this patient self-management intervention for each component of the Chronic Care Model? For the Hispanic population?
3. What is the potential for spread within clinics at the end of this initiative? Likelihood of changes being institutionalized; further data collection being conducted to demonstrate the impact of goal setting on the control of the disease; and continuation and spread within the clinic?
4. What are the plans and likelihood of regional/national spread of what has been learned?
5. What were the key lessons learned relative to conducting this learning network initiative for community health centers?

Return to Contents

1. Clinic-Level Results for Intervention

1A. Increasing Goal-Setting Rates

Self-management of a chronic condition relies primarily upon the ability of patients to change those behaviors that can improve the control of the condition. Self-management is more than patient education and includes communication between the patient and provider that promotes interactive goal setting and problem solving. Through the goal-setting process the patient can often develop the self-confidence needed to achieve the selected goal.

Improving the goal-setting rate in patients with diabetes was the key measure of this intervention. Table 1 summarizes characteristics of the six clinics, differences in their implementation strategies, and their progress with the intervention.

Table 1. Clinic characteristics, goal-setting rates, and HbA1c levels

Clinic	FQHC	Chronic Disease Registry	Intervention Target Population	Team Members	Primary Clinician	Goal-Setting Rate	HbA1c Level	Additional Information
District of Columbia Metropolitan Area
Clinic A	No	Yes	All patients in clinic with diabetes	Medical director, nurse practitioner (NP), and diabetes educator (DEd)	NP, who performed interactive goal setting, but often referred to DEd 2 days each week for additional counseling and goal setting.	Post-intervention goal-setting rate was 39%.	Clinics A and B had combined baseline HbA1c level of 9% Post-intervention HbA1c level for Clinic A was 8.6%.	DEd, who was a physician in her Central American country, conducted classes at Clinic A for persons with diabetes.
Clinic B	No	Yes	All patients in clinic with diabetes	Same three individuals as for Clinic A	Medical director Most interactive goal setting performed by DEd who spent 3 days in Clinic B and held classes at night or on weekend for persons with diabetes.	Increased from 9% to 41%.	Combined baseline HbA1c level for Clinics A and B was 9%. Post-intervention HbA1c level for Clinic B was 8%.	Clinic B completed most Action Plans, with almost all involving DEd.
Clinic C	No	Yes	All patients in diabetes school during intervention period	Clinic director/administrator and DEd (clinic physician present for first Learning Session only)	No clinician was involved in goal setting and no one-on-one interactive goal setting occurred.	Not available (Clinic has since implemented electronic medical record [EMR]; plans to evaluate impact of goal setting in diabetes school.)	Not available	Clinic used Action Plan for goal setting only in diabetes school. Trainer was not able to provide one-on-one interactive goal setting. Instead, patients used form to select goals while in class. Copies of Action Plans were not placed in medical records to allow followup by clinicians.
Colorado Region
Clinic D	Yes	Yes	Diabetic patients of one physician assistant (PA)	PA and quality coordinator	PA, whose performance on many indicators exceeded that of other clinicians in clinic. PA achieved patient continuity and goal-setting rate of 76%. PA worked closely with medical assistant for intervention.	No significant change in rate (75%)	No change in 8.1% level	Other clinicians participated in on-site patient self-management training to facilitate spread. Willing to try interactive goal setting after training because they learned that process did not have to be time consuming.
Clinic E	Yes	Yes	All patients in clinic with diabetes	Quality improvement coordinator and medical director	Mid-level practitioners and physicians	Increased from 25.9% to 49%.	Decreased from 8.2% to 8.0%.	Did not implement Action Plan because similar tool was unsuccessful. Instead, redesigned processes to facilitate interactive goal setting by clinic team with clinicians providing one-on-one support. Also, on-site training was provided to clinicians on patient self-management and goal setting. Clinic began implementation of EMR during latter part of intervention.
Clinic F	No	No	All patients in clinic with diabetes	Clinic director only participant	Volunteer physicians (clinic heavily dependent on volunteers)	Not available	Not available	Unable to implement Action Plan or any goal-setting strategy during intervention period.

Reviewing the data for Clinics D and E from July 2005 through January 2007 provided additional information. As shown for Clinic E in Table 2, the number of patients with diabetes in the registry has remained constant, but the goal-setting rate was higher a year before the intervention started, and after the intervention it is still almost 10 percentage points lower than in July 2005. However, the HbA1c has steadily declined from 8.3 to 8.0, raising concerns that the goal-setting rate had little to do with the outcome results or control of diabetes for this clinic. This same finding has been noted for other clinics with a registry, and one potential explanation offered by clinics not participating in this intervention is that more goal setting occurs than is documented in the registry. They doubt that the goal-setting rate truly falls from 60 percent or higher to 25 percent or lower in a year, but instead believe that clinics document goal setting better when they are participating in an intervention focusing on patient self-management.

Clinic D presents a different picture. For the same time period, 72 patients with diabetes have been added to the registry for a 52 percent increase. The percentage of these patients with a goal has gradually declined about 9 percentage points, which is understandable given the increase in patients. However, the HbA1c remained the same over the 19 months.

Table 2. Additional HbA1c and goal-setting data for Clinics D and E

Date	DM Patients in Registry	HbA1c	No. with Goals	% With Goals
Clinic D
July 2005	136	8.1	88	64.7
July 2006	193	8.1	118	61.0
January 2007	208	8.1	116	55.8
Clinic E
July 2005	356	8.3	209	58.7
July 2006	351	8.2	91	25.9
January 2007	359	8.0	176	49.0

Overall, clinics with strong team participation in the learning network were more successful. Having a strong clinic champion improved the likelihood for success. Furthermore, having a medical director as the clinic champion may be the best scenario for intervention support. Following are additional findings from the intervention:

• Clinics with a diabetes educator (even part time) completed the greatest number of Action Plans for patients with diabetes during the intervention.
• Clinics with volunteer clinicians perceive greater difficulty implementing any quality improvement strategies with the volunteer staff.
• Clinics without registries are at a disadvantage in collecting data needed to assess progress with quality improvement interventions.
• Clinics may use the same chronic disease registry but vary in the amount of data they enter into the system and use routinely for performance improvement.
• A clinic can redesign its clinic processes and increase the goal-setting rate similar to a clinic using a tool such as the Action Plan. Clinic team acceptance of a tool is important for success. Implementing a paper tool in any intervention may meet more resistance as clinics move to electronic medical records (EMRs).
• A high-performing clinician with a goal-setting rate already in the highest quartile may not benefit significantly from a tool such as the Action Plan.
• An improvement in the process and outcome measures of an intervention may not provide evidence that the intervention was effective in achieving the outcome. Ongoing monitoring and analysis of clinic data in the context of the environment are necessary to know when the intervention is the primary reason for the outcome results. Patient goal setting is a common measure for community clinics, but few clinics have designed a methodology to determine the effectiveness of their goal setting in changing patient behaviors.

1B. Analysis of Action Plan and Activation Tool Data

Action Plan. During the intervention, 211 Action Plans were completed, with more than 96 percent by Hispanic patients. The three most frequently chosen goals are shown in Table 3.

Table 3. Most frequently identified Action Plan goals

Type of Goal	Number of Action Plans With Goal Identified	% of Action Plans
Diet	89	42.2
Exercise	74	35.1
Medication	24	11.3
Total	187	88.6

Differences in patient goals by clinic. The diabetes educator in Clinic B with 91 Action Plans had 51 patients or 56 percent choosing diet as their goal. Other clinics varied from 22 percent to 42 percent for diet. Additional findings included:

• Clinic D with physician assistant performing interactive goal setting had the highest percentage (27 percent) choosing reduction of stress as a goal.
• Clinic C did not provide one-on-one interactive goal setting and had patients in diabetes school complete the Action Plan without assistance. Table 4 highlights the difference in rates for goals chosen in Clinic C compared with those for all clinics.

Table 4. Rates for Action Plan goals chosen by all clinics versus Clinic C

Type of Goal	% of Action Plans
	All Clinics	Clinic C
Diet	42.2	28.0
Exercise	35.1	39.0
Medication	11.3	30.0

Clinicians in Clinic C referred newly diagnosed patients with diabetes and those having the most difficulty with control to the diabetes school classes. Without linking the Action Plans with the medical records, it was not possible to determine if this group was significantly different from the patients targeted in the other clinics. The higher rate for medication goal setting for this group raises the concern that patients may be more likely to acknowledge lack of medication adherence when completing the Action Plan alone than when completing with a clinic team member.

Comfort levels chosen by patients indicated confidence in achieving their selected goal. If a form had a goal specified, but no comfort rating was provided, this form was dropped from calculations on comfort level. Every clinic had a few Action Plans without a comfort level noted.

However, an outlier was one clinic with 40 percent of Action Plans having no comfort-level rating.

• For the four clinics implementing the Action Plan, the percentage of forms with a "10" rating ranged from 44 percent to 70 percent with the latter in Clinic C without a one-on-one goal-setting process. The lowest was in the clinic where comfort level was not rated in 40 percent of the Action Plans.
• No Action Plan had a rating less than 5; the second most frequent rating was 5, with the third and fourth being ratings of 7 or 8.

Activation Assessment Tool. All clinics agreed from the beginning of the intervention that although this tool had been validated in Spanish, it was above the literacy level of their clinic population. However, a greater concern was that the concepts contained within the survey were not familiar to their Hispanic population. One clinician explained that Hispanics tend to be emotional and rate their feelings and well-being at the extreme poles of any rating scale and rarely select the middle ratings.

The two Spanish Catholic clinics continued to use the Activation Tool for 3 months to give the instrument a trial.

At Clinic A, 26 Activation Assessment Tools were completed as follows: 8 by the diabetes educator, 13 by the nurse practitioner, and 5 by the medical director. Table 5 provides the percentage of total responses (4 x 26 = 104) for each rating on the 10-point scale under the four areas of patient confidence.

Table 5. Clinic A: Percentage of total responses for each scale rating

1-2	3-7	8	9	10
0	7.6%	10.5%	14.4%	67.3%

At Clinic B, 74 forms were completed primarily with the assistance of the diabetes educator. Table 6 provides the percentage of total responses (4 x 74 = 296) for each rating on the 10-point scale under the four areas of patient confidence.

Table 6. Clinic B: Percentage of total responses for each scale rating

1-4	5	6	7	8	9	10
4%	12.8%	1%	8.1%	6.4%	6.7%	60%

In addition, of the total responses for Clinic B:

• 51.4 percent answered 10 under all four areas of patient confidence.
• 64.8 percent rated all four areas of patient confidence with the same number.
• 81.2 percent rated all four areas of patient confidence between 7 and 10.

For Clinics A and B, Latinos were more likely to choose ratings demonstrating a high confidence level in managing their own care and understanding their treatment.

One drawback in having six clinics with such varying characteristics and intervention designs is that the findings may be interesting for consideration and further study, but may not be applicable to other clinics. For example:

• The patients in a diabetes school that completed the Action Plan themselves without one-on-one interactive goal setting were more likely to choose medication for their goal than patients setting a goal with a team member. They were also more likely to rate their comfort with reaching their goal at a "10" (70 percent).
• Having a diabetes educator in the clinic for clinicians to refer patients for goal setting might result in more patients having a goal.
• The discipline of the team member performing interactive goal setting might influence the type of goal selected by the patient.
• Hispanic patients' responses for the Action Plan on a scale from 0 to 10 relating their comfort or the likelihood of reaching their goal were predominately from 7 to 10. A concern might be that the patients were expressing confidence to please the clinic team.

Return to Contents

2. Lessons Learned in All Components of Chronic Care Model

The Chronic Care Model is a framework for improving the health care of individuals with chronic conditions and includes six components:

• Delivery System Design.
• Decision Support.
• Clinical Information System.
• Patient Self-Management.
• Health System Organization.
• Community Resources and Policies.

Though this intervention focused on self-management, Tables 7 through 12 summarize lessons earned in each of the components of the Chronic Care Model.

Table 7. Lessons learned in patient self-management

Patient Self-Management

Tools/Steps	Resources	Tips/Lessons Learned
Select tools proven successful in implementing intervention to improve goal-setting rates.
1. Use Action Plan to facilitate patient self-management and goal setting.	Tool is in public domain. Delmarva translated into Spanish, produced as triplicate forms, and delivered to clinics.	Most clinics felt Action Plan facilitated change process and decreased paperwork (copy for patient was important for motivation; copy to record for followup; copy to Delmarva for data collection). Clinics would like Action Plan modified for use with all chronic diseases and obesity. Some felt Action Plan would be easier to use with section to note progress on followup visits. Others preferred completing new Action Plan for each visit and sending copy home with patient each time. Starting with small steps in goal setting avoids setting patient up for failure.
2. Use Activation Assessment Tool to assess patient's comfort level in making behavioral changes. Knowing this level helps provider understand where to begin in goal-setting process and in education.	Tool validated in Spanish by Stanford University and available to use without charge. Delmarva made copies and provided them to clinics.	All clinics agreed tool was above reading level of their population. Low literacy level was considered for any materials produced. Simple drawings are effective. It was also felt that concepts embedded in tool were not understood by their Hispanic populations. Suggestion made that AHRQ promote research to determine if using graduated tool to assess comfort of Latinos produces same distribution of responses as with other cultures. Additional suggestion made that before assessing self-activation level for goal, provider should assess importance of goal to patient during goal-setting process.
3. Make clinic team aware of potential benefit of intervention and assure they have skills for patient self-management.	On-site training offered to clinics on patient self-management. Texas Association of Community Health Centers video used during training.	Clinics found training helpful in convincing clinicians that goal setting did not have to be time consuming. Video felt to be good example of provider/patient interactive goal setting. Medical assistants may not feel empowered to help with goal setting. May require training and regular meetings for discussions and support.
4. Identify clinic team members involved in intervention and define their roles and responsibilities to maximize resources and patient outcomes.	In one clinic medical assistant who routinely worked with one provider was involved in helping with goal setting. In two clinics patients referred to DEd that divided her time between clinics for help with education and goal setting. This team member also held diabetes classes on Saturday. Another clinic had diabetes school for educating their patients and used these classes for goal-setting process due to time constraints on clinicians.	Regardless of type of team approach used, goal-setting process should be documented in patient record and involve clinician in followup and support of patients' goal-setting efforts. Every encounter with patients should be utilized to discuss challenges and progress with their goals. If separate classes for diabetes education are used for goal setting, following must be considered: One-on-one time needed for interactive goal setting. Documentation of goal chosen must be incorporated into patient's record for clinician followup at subsequent visits. Culture affects diet and lifestyle. Diet is very important to Latinos; helping them prepare their favorite food in healthier way is important. Many Latinos have shown interest in how other cultures prepare food and have been open to learning these methods. Walking is easily accessible exercise activity, but Latinos do not routinely swim, play tennis, golf, or go to gyms for exercise. Dancing is also well received for exercise.
5. Followup for progress on goals should not wait until next scheduled visit but occur within 2 weeks.	Usually medical assistant, health educator, or promotora makes telephone contacts.	Contacting patient by telephone may be difficult during day. After work hours is better. Patient cell phone has proven to be better means of making telephone contact.
6. Invite family to participate in patient care.		Family is important to Hispanics. Invite family members to classes, specifically one that lives with patient. If patient is male and female in home is responsible for food preparation, it is important for her to attend class and clinic to learn about diet and how to prepare healthier foods. Active recruitment of family members when care is not going well has been successful in two clinics.
7. Create comfortable, caring atmosphere for patients in clinic and classes.		Creating trust and respect in clinic population is critical and can often trump other efforts to help patients. If clinic is not orderly and high tech in appearance, it is probably more comfortable. If clinic feels like part of Latino community, patients are more relaxed.
8. Patient and all members of provider team must accept responsibility for care and outcomes.		Lesson taught by DEd is that several unsuccessful attempts to make telephone contact do not mean fulfillment of responsibility to patient. Try again and/or use other methods. DEd tries again at different hour and after work or on Saturday, calls cell phone, and calls emergency number given. She has a high success rate for telephone contacts because she feels responsible for making followup contacts.
9. Debunk myth in Latino population that insulin can worsen symptoms and complications of diabetes.		Provide good explanation of how lack of control of disease produces complications that many relate to use of insulin. Explain that delaying insulin when diet, exercise, and medication do not succeed in controlling glucose can allow complications. Be careful not to imply that if patients adhere to care plan they will never have to use insulin. If they have the disease long enough, need for insulin is a real possibility.

Table 8. Lessons learned in decision support

Decision Support

Tools/Steps	Resources	Tips/Lessons Learned
Determine any changes that could support better decision making in clinic.
1. Develop standing orders for medical assistants to use in ordering blood tests, procedures, and referrals for patients with diabetes.	Clinics developed standing orders from clinical guidelines to remind medical assistants when HbA1c, LDL, foot exams, eye exam, etc. should be done. This allowed medical assistants to have appropriate blood tests drawn and make referrals (or begin paperwork).	Standing orders facilitated clinical guidelines being met because medical assistants and clinicians performed better than clinician alone.
2. Develop sheet for front of patient record with diabetes clinical guidelines stating what tests or procedures should be performed at every visit and periodicity of others.	Developed and copied in clinic.	Reminder helped all clinic team members to adhere to clinical guidelines.
3. Use screening tools for behavioral health problems, especially depression, to facilitate detection in all patients with chronic diseases. Screening for substance abuse has been added in one clinic.	Several tools are available: PHQ-9, PHQ-2, and CAGE.	If clinics translate tools, they should remember to avoid words like "depression" in Latino population to get more accurate responses. Finding mental health resources may be quite difficult. One clinic formed partnership with mental health agency and has bilingual Licensed Clinical Social Workers coming to provide services. This decreases incomplete referral rate.

Return to Contents
Proceed to Next Section