Chapter 3. Findings (continued)

3.3.4. Level 4: Impact on Patient Outcomes

This Tunis and Stryer impact level demands an "actual impact on health outcomes (clinical, economic, quality of life, satisfaction)." There are sublevels for impacts on outcomes in a limited study population (4a) and impacts identified outside a formal research context (4b). The extent to which CERTs were able to impact patient outcomes is difficult to attribute (as for research in general and pharmacoepidemiology in particular). However, in a few intervention studies, the Portfolio investigators were able to demonstrate some impact. Limited CERT research has been conducted at the clinical level and even fewer studies have focused on impacting patient outcomes (although there has been much research on understanding patient outcomes and outcomes research). The Portfolio research that has had an impact on patient outcomes has been at Level 4a — impact on outcomes in a limited study population. The examples of Portfolio work that directly impact patient outcomes were intervention studies, including:

Reducing antimicrobial resistance:

A 3-year intervention for physicians, parents, childcare providers, and anyone we could engage to reduce unnecessary antibiotic prescribing as a community. One of the things that was fairly unusual is that we were doing a rigorous evaluation of the kind of coalition building that's often done in public health projects. The coalition was the: the state Department of Public Health and 4 of the major health plans, including the Medicaid program, and the physicians in those communities. Those kinds of coalitions to tackle public health programs do happen but they're rarely evaluated in this kind of randomized control trial... Could we affect antibiotic prescribing rates for children? To test that we measured very precisely with health plan data the antibiotics received by children 0-6 year old who were insured by the four health plans 16 communities. We found a dramatic trend in all the communities toward lower prescribing rates; both in the indirect and control communities there was a substantial drop, at least in some age groups, in antibiotic prescribing.

Improving antibiotic use:

The primary objectives were to examine two different types of community based educational strategies to improve antibiotic use.... The first strategy was office and household based education, and the second strategy was a mass media campaign.³⁶ We identified approximately 12 practices located in a specific suburb area of the Denver metro area. We developed household mailings that included a refrigerator magnet, brochures, and other material about the topic of appropriate antibiotic use for respiratory infections. In addition, we developed similar materials that were branded in the same way and which were placed in the physicians' offices, so that patients would be getting the message at home and in the physician's office. The physician would also be seeing the message on exam room wall posters and displays in their waiting rooms. Our evaluation showed that there was an absolute decrease in antibiotic prescribing for adults with acute bronchitis of approximately 10% – a relative decrease of approximately 20%. However, in children with pharyngitis, which was our other target population, we didn't see any change in prescribing, but part of that was due to the fact that prescribing rates had already been down to close to optimal level probably due to a number of factors. Our conclusion was that patient and household education (small-scale education) was effective for adults with bronchitis and but not for children with pharyngitis.

The second major component of this project was to evaluate a mass media campaign, because that would be the most easily generalizable way to improve antibiotic use, although it would also be the most expensive. It was important to have a good cost-effectiveness component, and which we incorporated into our design. We designed ... the mass media campaign to change physician prescribing behavior. While we didn't observe a change in physician prescribing behavior, we observed a fairly large decrease in antibiotic utilization in our mass media area, which can only be attributable to reduced office visits for respiratory infections. We did see reduced visits for respiratory infections primarily among children and parents of young children. The campaign cost $180,000, and the amount of antibiotic savings for the total Denver metro was in the millions of dollars. If you look just at the two health plans that provided their data and hypothesize that only those two plans paid for the entire Denver metro area, it was still cost saving. We estimate that those two health plans saved about $400,000 in antibiotic costs.

Intervening in emergency departments:

We used a computer kiosk to deliver culturally and language concordant pre-visit information to patients and ... we showed that we did reduce antibiotic use for adults with acute bronchitis ... the main outcome there was reducing antibiotic use for adults with URIs and acute bronchitis—Round 1. Round 2 was going to evaluate a diagnostic test intervention in the ER setting (e.g. the C Reactive Protein test). Many clinicians are asking for is empirical evidence that the patient doesn't need antibiotics. We observed a huge drop in antibiotic prescribing for bronchitis regardless of the test...we believe that is a very strong Hawthorne effect – by randomizing at the patient level, one sees providers change their behavior. On one hand it says that ED physicians really can reduce their prescribing for bronchitis to very low levels...but the problem is that once you've reduced antibiotic use in the comparison group, the CRP test doesn't have any marginal benefit. It's only helpful if people are over prescribing... this is the first series of studies to improve antibiotic use in the ER setting, so we've definitely increased attention in the emergency medicine literature.

3.3.5. Impact Case Studies

Four CERTs research projects were selected and highlighted here as "impact case studies" to illustrate how the individual projects and the CERTs program in general have had an impact on policy and ultimately on improving the use of medications. The cases also provide insight into mechanisms that may have resulted in an impact. Detailed descriptions of each of the four case studies are provided below including descriptions of the research, timeline, dissemination, and levels and mechanisms of impact. For the dissemination descriptions, the impact factor (IF)³⁷ of each journal from the ISI Citation Report 2005 is included. The case study descriptions are derived from initial and follow-up interviews with the investigators, partner or outside expert/policymaker interviews, the publications and other dissemination materials, progress reports, and other CERT program documents.

To further delineate the impact of the case studies, the Research Impact Framework and its key descriptive categories were used to categorize the areas of impact (e.g. research, policy) and the relevant descriptive categories within each of those.³⁸ Although the case studies were selected based on the specific project that fell within the evaluation timeframe (2002-2005); other projects may have been critical precursors of a selected project and are therefore included in the case study descriptions.

The 4 studies were:

Study Title	CERT	PI
FDA Drug Prescribing Warnings: Is the Black Box Half Empty or Half Full? (Case 1)	HMO	A. Wagner
Evaluation of physicians' understanding of the QT interval and medications that may alter it (Case 2)	Duke	A. LaPointe; S. Al-Khatib
Effect of AAP Guidelines on Vitamin D Supplementation recommendations in Practice (Case 3)	UNC	M. Davenport and A. Calikoglu
Tensions between patient and public health values in generalists use of antibiotics (Case 4)	Penn	J. Metlay

Impact Case Study I: Black Box Warnings

The Black box warning project has been eye opening. (CERT PI)

Project Title: FDA Black Box Warnings³⁹
PI: Anita Wagner
CERT: HMO Research Network
Project Description: Assess the overall frequency of prescribing at variance with FDA 'black box' warnings or commonly accepted clinical guidelines.

Formation of the Research Question and Study Methods

Dr. Wagner described the background of the black box warning study⁴⁰:

We are interested in improving medication use by patients and prescribers. Appropriate medication use requires that information about risks associated with drugs is effectively communicated to prescribers and patients. Risk communication through drug labeling is one of the FDA's primary approaches to risk management. A black box warning is the strongest labeling requirement the FDA has, intended to alert prescribers to the high risks associated with certain drugs.

Dr. Wagner's project was one of the HMO Research Network grants entitled "Prescribing Safety Program." Dr. Wagner emphasized the importance of studying black box warning by pointing out that that there is much research on adverse drug events in the hospital setting but less in ambulatory settings. High-risk drugs are used in ambulatory populations and black box warning drugs are presumably the highest risk drugs.

Black box warnings (BBWs) are the Food and Drug Administration's (FDA) strongest labeling requirements for high-risk medicines. It is unknown how frequently physicians prescribe BBW drugs and whether they do so in compliance with the warnings. The purpose of the present study was to assess the frequency of use of BBW medications in ambulatory care and prescribing compliance with BBW recommendations.⁴¹

The study was retrospective and used automated claims data of 929,958 enrollees in 10 geographically diverse health plans in the United States. Frequency of use in ambulatory care of 216 BBW drugs/drug groups between 1/1/99 and 31/6/01 was estimated and dispensing compliance with the BBW requirements for selected drugs was assessed and reported.³⁴

Research Findings

The 30-month study findings included:

• Most non-compliance was associated baseline laboratory monitoring recommendations: 49.6% of all therapy initiations that should have been accompanied by baseline laboratory monitoring were not.³⁴
• Greater than 40% of enrollees received at least one medication with a BBW. There were few instances of prescribing during pregnancy of BBW drugs absolutely contra-indicated in pregnancy.
• There was almost no co-prescribing of contraindicated drugs with the two QT-interval-prolonging BBW drugs evaluated.

Dissemination

Publication:
Wagner AK, Chan KA, Dashevsky I, Raebel MA, Andrade SE, Lafata JE, Davis RL, Gurwitz JH, Soumerai SB, Platt R. FDA Drug Prescribing Warnings: Is the Black Box Half Empty or Half Full? Pharmacoepidemiol Drug Saf. 2006 Jun;15(6):369-86.

Times Cited: Lexis Nexis results returned 13 documents.⁴²
Journal IF⁴³: 0.750
Associated Commentary: "Thinking Outside the (Black) Box: A New Research Agenda" Paul J. Seligman, Director, Office of Pharmacoepidemiology and Statistical Science, Center for Drug Evaluation and Research, FDA. The journal asked Paul Seligman of the FDA to write a commentary with the publication of the article.
Key Citation: Article cited in IOM Report Preventing Medication Errors: Quality Chasm Series (2007), Chapter 2.
AHRQ: Patient Safety E-Newsletter, December 2, 2005, Issue No. 15 "1. New AHRQ study finds mixed compliance with medication warning labels"
CERTs: Announcement 11/18/05 "warning labels on high-risk drugs not always heeded by doctors.

Media:

11/18/05
Wall Street Journal Online "Drug-safety labels often go unheeded"
HealthDay Reporter "Doctors not heeding 'black-box' warnings on Rx drugs", posted on MedicineNet.com, MedicineOnline.com and KESQ.com (news channel Web site)
SeniorJournal.com "Warning labels on high-risk drugs inconsistently heeded by doctors"
Medscape Medical News "Inconsistent adherence to black box warnings: Newsmaker interview with Anita Wagner, PharmD, DPH"
U.S. News and World Report "Doctors often ignore 'black box' warnings on prescription drugs"
Drug Industry News at adrugrecall.com "doctors not heeding 'black box' warnings"
Drugwonks.com "the art of the black box"
National Center for Policy Analysis at ncpa.org "black box warnings often go unheeded"
11/22/05
The Harvard Crimson "Study: Rx Warnings Ignored"

Other publications:

Interviewed for article "Relevance of Black Box Warnings" in the APSF Also had interviews" Newsletter of Anesthesia and Patient Safety" (APSF) 2006 vol 21 #1 pg 16 & 18
Harvard Medical School Office of Public Affairs News Release 11/18/05 "Warning labels on high-risk drugs inconsistently heeded by doctors: Better means of communicating risks needed"

Dr. Wagner also mentioned that she spoke on a radio show. Dr. Wagner also reported many contacts upon release of the electronic version, with additional interest generated by the release of the print version. Dr. Seligman was contacted by the journal Pharmacoepidemiology and Drug Safety to write a commentary in the issue in which the BBW warning appeared.

Impact of the Findings

The potential implications of the findings were described in the article's conclusions:

Many individuals receive drugs considered to carry the potential for serious risk. For some of these drugs, use is largely consistent with their BBW, while for others it is not. Since it will not be possible to avoid certain drug-associated risks, it will be important to develop effective methods to use BBWs and other methods to minimize risks.

Level 1 Impact: Research
This case study had a level 1 impact; it contributes to future research in several ways. First, this study involved compiling a list of drugs with black box warnings, because no such list existed. Another perceived potential import of the research may be indicated by the fact that the editor asked Dr. Seligman of the FDA to write a commentary in that issue. Additionally, the findings of the extent of noncompliance with BBW and particularly regarding recommendations for baseline laboratory monitoring are important results.

Level 2 Impact: Policy
The impact of the Black Box warning study on policy or FDA regulation is difficult to assess, in part because the study results were only released in November 2005 and time can be an important factor. However, it has contributed to literature that advances the understanding of the use of black box warnings as a risk management tool and of potential ineffectiveness due to non-adherence to the warnings. An AHRQ representative characterized the to-date limited impact to the findings being outcomes that were already known, rather the more important question was what should and can be done about the limited attention to the black-box warnings.

Also, Dr. Seligman in his commentary stated, "What we really need to know is why and under what circumstances co-prescribing occurs, what patients are told, how they actually use the medicines, and whether such use is appropriate." An FDA representative who was a study respondent commented on the black box:

It's unique only in the sense that this is the kind of work that I really do believe should be done in medical practice everywhere ...in an organization monitoring the quality of care being provided.... this is really an issue about how well people are paying attention to issues of concern in drug therapy. I liked the study, more studies of this type should be done...but the real issues are—are people using the results of these studies to improve the way that care is delivered?

An FDA representative thought it was too early to tell if the black box study had an impact, as the article had just been published in June 2006. We cannot assess whether this study has yet affected clinical practice or outcomes. However, given that it has informed a national level risk management program used by the FDA, it has the potential to have an impact. When Dr. Wagner was asked why she thought her research had the large amount of media attention it did, she identified the following as potential factors: the aging population with many medical conditions that are amenable to effective treatment with medications, but because patients are sicker and treatment is more complex, there is an even greater need to be clear about risks and risk communication, especially in ambulatory care where most prescribing occurs. In addition, there may be a heightened awareness in the population of the risks of medicines.

Mechanisms of Impact

While the black box warning study is recent, it has received much media attention. Other indications of the importance of the study are its selection by the journal editor for an FDA commentary and that the study was cited recently in the Institute of Medicine Medication Errors report. The study had not been released for as long as some of the other case studies and did not have the time often needed for impact such as changing or informing policies. The media attention may be an indicator potential impact, and drug safety is an important topic for the public and the research community.

Impact Case Study II: QT Prolongation

Project Title: Evaluation of physicians' understanding of the QT interval and medications that may alter it
PIs: Nancy Allen LaPointe, Sana Al-Khatib
CERT: Duke
Brief Project Description: Evaluate physicians' knowledge of the QT interval and medications, diseases, and drug combinations that may alter the QT interval.

Formation of the Research Question

This project stemmed from previous studies at the Duke CERTs and the questions those studies raised. Some of the first studies the Duke CERTs conducted were looking at the risk management program the FDA instituted for the anti-arrhythmic drug dofetilide, a drug known to prolong the QT interval and cause torsades de pointes. As a result of these studies on dofetilide (seems like you should reference these here), the Duke CERTs investigators raised the questions of whether practitioners understand the meaning of QT interval prolongation, which drugs prolong the QT interval and how to measure the QT interval? Around the same time, studies were published about practitioners' continued use of another QT prolonging drug, cisapride, with contraindicated medications that increase the risk of QT prolongation despite warnings in the product labeling (insert reference). Therefore, the Duke CERTs investigators began to assess health care practitioners' knowledge of the QT interval and medications that may prolong it along with other studies to assess real world co-prescription of more than one QT prolonging drug (insert references—Curtis et. al and Allen LaPointe et. al).

One of the first studies to assess practitioner knowledge was a pilot survey conducted with a group consisting largely of cardiologists attending a cardiology symposium. The survey included a reproduction of a ECG complex and asked respondents to measure the QT interval. The survey also included questions about which drugs or drug combinations could prolong the QT interval and clinical factors associated with increased risk of QT prolongation.

Research Findings

From a total of 334 survey respondents, 157 (47%) were physicians; 271 (81%) stated that cardiology was their area of specialization. Most of the respondents (86%) said that they would check an ECG before and after starting QT-prolonging medications, but less than half (42%) of all respondents and only 60% of physician respondents were able to accurately measure a sample QT interval on the survey. Less than two-thirds (63%) of respondents were able to accurately identify possible QT-prolonging medications, while only about half (51%) could accurately identify medication combinations that might prolong the QT interval.⁴⁴

Given the findings, the investigators went on to conduct a more rigorous study to determine the generalizability of the results. They conducted a survey to assess health care practitioners' ability to correctly measure the QT interval, and to identify factors and medications that may increase the risk of QT-interval prolongation and torsade de pointes. Participants included practitioners attending internal medicine and psychiatry Grand Rounds at six academic institutions and practitioners at six community hospitals in the same geographical areas as the academic institutions.⁴⁵ The results were:

Of approximately 826 attendees, 517 (63%) completed the survey. Of about 608 attendees of internal medicine conferences, 371 (61%) responded, and of about 208 attendees of psychiatry conferences, 146 (67%) responded. Of a total number of 20 questions, the median number of correct answers for the whole group was 10 (interquartile range 7-13). The median number of correct answers for internists was 12 (interquartile range 9-13), for psychiatrists 10 (interquartile range 7-13), and for other specialists 10 (interquartile range 5-13). Respondents who graduated between 1990 and 1999 and academicians performed significantly better overall than other respondents. Of the 517 respondents, 224 (43%) measured the QT interval correctly. Physicians in training and academicians were more likely to measure the QT interval correctly.⁴

Given the finding of limited knowledge of the QT interval and QT prolonging medications, the Duke CERTs investigators developed a QT educational module to address the knowledge deficit (insert reference for Web site). The module was designed to provide information to healthcare practitioners on how to measure the QT interval, what the QT interval means, and what drugs and other clinical factors could change the QT interval. The module was designed to be not only a case-based learning tool, but to serve as a resource practitioners could return to when questions arose in their daily practice (e.g. does this drug prolong the QT interval?). Prior to development of the module, the investigators solicited the advice and opinions of experts in cardiac repolarization to refine the module contents. Results of this expert survey and collective knowledge regarding the QT interval was submitted as a manuscript and published in 2003 in JAMA. Prior to the official public launch of the module, numerous practitioners including some of the experts in cardiac repolarization were asked to review the module and provide feedback. The module was revised and underwent pilot testing to determine if knowledge of the QT interval improved among a group of medical residents who completed the module.

The QT educational module was launched (with an announcement of its release) on the Duke CERTs' Web site initially. To reach a larger audience, the Duke CERTs investigators leveraged their relationship with the American Heart Association (AHA) to post the module to the AHA Web site.

Dissemination

Publication I: Allen LaPointe NM, Al-Khatib SM, Kramer JM, Califf RM. Knowledge deficits related to the QT interval could affect patient safety. Annals of Noninvasive Electrocardiology. 2003 Apr;8(2):157-60.
Journal IF: 0.790

Publication II: Al-Khatib SM, Allen LaPointe NM, Hammill BG, Chen AY, Kramer JM, Califf RM. A survey of health care practitioners' knowledge of the QT interval. Journal of General Internal Medicine 2005;20:392-396
Journal IF: 3.013

Publication III⁴⁶: Al-Khatib SM, Allen LaPointe NM, Kramer JM, Califf RM. What clinicians should know about the QT interval. JAMA. 2003. 289:2120-2127.
Journal IF: 23.332

Publication IV: Curtis LH, Ostbye T, Sendersky V, Hutchison S, Allen LaPointe NM, Al-Khatib SM, Yasuda SU, Dans P, Wright A, Califf RM, Woosley RL, Shulman KA. Prescription of QT-prolonging drugs in a cohort of about 5 million outpatients. American Journal of Medicine 2003;114:135-141.
Journal IF: 4.388

Publication V: Nancy M. Allen LaPointe. Frequency of High-Risk Use of QT-Prolonging Medications. Pharmacoepidemiology & Drug Safety 2006;15:361-368
Journal IF: 0.750

Media: U.S. News & World Report
Presentations: Presented at the American College of Cardiology (ACC) meeting in 2002 and 2004. Presentations also provided in medical centers titled, "what you should know about the QT interval." A presentation on the dofetilide risk management program was also given at the Drug Information Association meeting in 2002.

Impact of the Findings

Level 1 Impact: Research
This research was an important contribution to the understanding of medication risks and safety, particularly the role of the prescribers. From the first study the researchers identified knowledge deficits of various prescribers. The second published study suggested that many healthcare practitioners cannot correctly measure the QT interval and cannot correctly identify factors and medications that prolong the QT interval. The findings suggest that greater attention to the QT interval is warranted to ensure safer use of QT prolonging medications. The potential impacts of the findings on future research are numerous, including: the potential to examine the impact of the educational module on physician knowledge and prescribing behavior.

Level 2 Impact: Policy
Although this research did not have direct impact on policies, it has important regulatory implications. For example, drugs that prolong the QT interval with limited clinical efficacy beyond other treatment options have a poor risk-benefit profile, particularly if one takes into consideration the findings of these studies—that practitioners have a knowledge deficit of QT prolongation and QT prolonging medications.

Level 3 Impact: Clinical Practice
The QT educational module has the potential to impact health care practitioners prescribing, drug monitoring, and drug utilization review (DUR) behaviors if their knowledge gaps are addressed; however, these studies cannot be directly attributed with impacting clinical practice, either in terms of practitioners or patients.

Mechanisms of Impact

There were several mechanisms that supported the impact of this research:

• Connections of stakeholders to a specialty organization (i.e. AHA).
• Recognition of the gap and responsiveness of the researchers.
• Exploration of the problem via a pilot study.
• Conceptualization of the problem in the context of a larger population to understand its extent.
• Recognition of the need for education regarding the QT interval and medications that cause QT prolongation.
• Context, given the withdrawal of cisapride from the market largely due to its QT prolonging effects.

Return to Contents
Proceed to Next Section