Chapter 3. Findings (continued)

3.3. Outcomes and Impacts of Portfolio Research

Portfolio research and awards have contributed to the knowledgebase of diverse areas in therapeutics, clinical practice, and research methodologies. The research findings of the Portfolio funded grants and research funded through the CERTs program also have contributed to the field of therapeutics research. The research outcomes relate to the specific areas of CERTs specialization, including drugs, biologics, and devices. The contributions are aimed at various end users including: patients, consumers, health care providers, HMOs, PBMs, government agencies, professional organizations, and others. The research has focused on children, women, elderly, and racial and ethnic minorities and has included studies on various diseases and organ systems (e.g. cardiovascular, musculoskeletal systems).

In the following section, we use the "Levels of Impact Framework" developed by Tunis and Stryer to classify what levels of impact Portfolio research has had. This framework "outlines an idealized process by which basic findings in outcomes and effectiveness research are linked over time to increasingly concrete impacts on the health of patients."³¹ The levels of impact illustrate how research may ultimately contribute to a patients' health. Below we describe the levels, and we provide examples from the Portfolio.³² The tables summarize the studies (note that the source of these summaries includes documents (DOC) and interviews (INT), so the amount and type of information available for each study varies considerably) and include the study population or population impacted by the findings. These summaries are followed by several in-depth case studies.

3.3.1. Level 1: Research Findings

Level 1 impacts are "effects of research studies that do not represent a direct change in policy or practice" and include examples such as new tools and methods for research, instruments and technique to assist in clinical decision-making. Level 1 impacts also occur when studies produce results in conflict with "current clinical paradigms, and stimulate rethinking and questioning within a clinical specialty."³³ The CERTs and Portfolio projects have contributed to research findings in the following areas: data and methods; adherence; medication safety; medication errors; trends; cost and economics of therapeutics; QT prolonging medications; health information technology; and antibiotics and antimicrobials.

Data and Methods

The CERTs researchers have not only contributed to the knowledge base in their specific content areas, but some have also contributed to the advancement of research methods in therapeutics, epidemiology, and health services research. Besides investigators' use of data, measures, and methods the Portfolio has also produced material on the utility and limitations of certain data for therapeutics research, has compiled datasets, and has developed methods and algorithms. Understanding the utility and limitations of available data and databases is important for future research on therapeutics. Themes include:

The development of large datasets as a resource for the CERTs to shape research questions, and assessment of the limitations of electronic data for surveillance and research.

• Linkage of clinical and administrative data and the utilization of other types of data sources and algorithms to leverage the use of databases and registries.

Development of methods to enhance the understanding and practice of therapeutics including registries to conduct pharmacoepidemiologic studies (and which potentially can contribute to understanding the genetic/genomic basis of therapeutic response).

• Advancement of methods in therapeutics research and pharmacoepidemiology through educational efforts at the CERTS and through seminars, published books, discussions.

One study, for example, found that automated claims and pharmacy databases are not sufficient on their own for assessing appropriate renal dosing to determine prescribing errors of QT interval prolonging medications. Another project developed the Web-based drug-induced arrhythmias registry which has been attributed with providing the information that contributed to identifying the cardiotoxic effects of methadone. Table 1 provides additional examples.

Adherence

Non-adherence to medications is a pervasive problem. A number of Portfolio funded projects examined patient adherence in different populations and medical conditions, expanding the understanding of adherence for eventual use in practice. One study, for example, found that 10-20% patients discharged from hospitals with myocardial infarction or heart failure and prescribed follow-up medication do not fill the hospital prescription and therefore do not receive the benefits. Table 2 provides additional examples.

Medication Safety

The Portfolio has conducted much research on medication safety, including prescribing, therapeutic risks, and risk management and assessment. The CERTs jointly applied for an AHRQ Patient Safety grant under the auspices of the HMO CERT, whose PI described the prescribing safety program as having identified safety issues in the outpatient setting. The research generated data potentially useful for future projects. Themes of Portfolio research on medication safety include:

• Age-specific risk in pediatric and elderly populations.
• Medication use among pregnant women.
• Culture-specific research.
• Medications not previously of concern.

Table 3 provides additional examples.

Medication Errors

With the release of the Institute of Medicine (IOM) Report "Crossing the Quality Chasm"³³ the prevalence of medical errors was brought to the attention of practitioners, researchers, and policymakers. In late 2006 the IOM released the pre-publication findings of the Medication Errors³⁴ report. The CERTs and Portfolio grants have conducted research to further the understanding of medication errors. Themes included inappropriate prescribing practices leading to medical error, variation in error rates by mediation class, and patient-physician relationships. Examples of CERT research findings in this area included:

• Errors in which the medication reaches the patient but does not cause harm account for the majority of errors reported.
• Leading types of errors reported through one type of reporting system were omission errors, improper dose/quantity, and wrong time. Consistently, the leading causes of errors seen in pediatric medication errors are performance deficit, procedure/protocol not being followed, and communication.

Table 4 provides more examples.

Trends

The CERTs have contributed to the study of trends in the use of therapeutic agents. For example, one study found that the annual prevalence of use for life saving cardiovascular therapies increased between 1995 and 2002, and another found that use of psychotropics has increased by youths in IPA-model health plans, with consistency across health plans in different geographic regions. Go to Table 5 for additional examples.

Racial Disparities

The CERTs have conducted limited research explicitly on racial and ethnic disparities. Two studies identified racial disparities. An HMO CERT study of Caucasian and African-American women found a lack of physicians prescribing therapy for women with osteoporosis. The trend was more prevalent in black women than in white women. The Alabama CERT in a study of Caucasian and African-American women found that Caucasian women were more than twice as likely to receive osteoporosis treatment and bone density measurement than African-American women, even among those who had suffered hip, rib, or wrist fractures.

Cost and Economics of Therapeutics

The cost and economics of therapeutics has also been a part of the research Portfolio of the CERTs and grantees. For example, increasing the co-pay to more that $10 for a thirty-day supply of oral hypoglycemics was associated with significantly reduced use. Another study found that Glucocorticoid users face non-negligible incremental health care costs compared to non-users. For an estimated one million chronic glucocorticoid users, the costs to the health care system were estimated as over $1.2 billion annually in the U.S. alone for treatment of adverse effects. Additional examples of research findings the economics of therapeutics area are presented in Table 6.

QT Prolonging Medications

Four of the seven CERTs have contributed to knowledge and understanding of QT prolongation and therapeutics. The CERTs have contributed extensively to the knowledge base on medications that affect the QT interval, including research on concomitant use of medications that affect the QT interval, creating education modules, and maintaining a registry to further understand the extent of the QT prolongation affect. For example, one CERT developed and maintains an international registry of drug-induced arrhythmias that led to the detection of unexpected drug toxicity (methadone) and a new risk of toxicity from a group of intravenously administered drugs. Contrary to recent reports in the literature, one CERT concluded from its analysis that prolonged QT and TdP (Torsade de Pointes) can occur over a wide range of methadone dosages including those recommended for addiction treatment. Additional examples are provided in Table 7.

Health Information Technology (HIT)

The use and advancement of health information technology is a growing area and is represented in the CERTs program. As stated in an annual report, "As we move into our fifth year, more and more of the CERTs projects focus on technology" (CERTs AR Y4). The CERTs have focused on the use of information technology to improve the safety and effectiveness of therapeutics in practice. This has included research on Computerized Physician Order Entry (CPOE) and other decision support technologies for practice. Findings from some of these studies are provided below:

• In a study of cardiac patients, the Vanderbilt CERT showed that a computerized physician order entry system (CPOE) at the hospital has improved the care of patients with myocardial infarction and heart failure, suggesting that the CPOE system significantly improved discharge planning.
• An HMO CERT study exploring the laboratory monitoring alerts within a CPOE in an HMO found that passive alerts for appropriate laboratory monitoring for specific medications ordered in a CPOES do not improve adherence to monitoring recommendations.
• Another HMO CERT Study examining patient-specific order entry intervention found that clinicians prefer decision support alerts that are concise, clear, easy to navigate, and that provide minimal information in the alert text.

Antibiotics and Antimicrobials

With the ever-increasing use of antibiotics and identification of drug resistant organisms, and an entire CERT committed to antimicrobials, there has been much pertinent research in therapeutics in this area within the Portfolio. A grantee praised the Portfolio's involvement in trying to "promote judicious antibiotic use in which, to their credit, AHRQ is involved." An AHRQ representative stated that this area is "a very high priority area for the government and there are multiple agencies interested in this including the CDC and FDA because it is a really important public health problem." Also, this is a PART goal focus area. The Portfolio research has included an emphasis on anti-infectives including not only appropriate and judicious use but also further understanding of resistance and how to address it. The research on antibiotic and anti-infective therapeutics has included examining prescribing patterns.

In one example, a CERT found that the proportion of the primary bloodstream infections accounted for by gram-negative pathogens has increased significantly over the past five years. These changes have great implications for empiric antimicrobial therapy for suspected bloodstream infections, and for fostering the development of new agents with expanded gram-negative activity. Another CERT found that patients with extended-spectrumß-lactamase producing Escherichia coli and Klebsiella (ESBL-EK) infection were 5/8 times more likely to have had exposure to an extended spectrum cephalosporin within the thirty days prior to infection and also were more likely to be female, had infection caused by a Klebsiella species, and received steroids in the thirty days prior to infection. These findings have the potential to contribute to limiting the emergence of ESBL-EK infections in children. Additional examples are presented in Table 8.

Devices

The CERTs program has also demonstrated a commitment to research on devices through its support of a new CERT focused on devices and through the Coordinating Center's sponsorship of a meeting on devices. A few studies have examined implantable cardiovascular defibrillators (Duke) and blood glucose monitors (UNC), and the UNC CERT is studying a device in the pediatric population because "most devices are inappropriately just scaled down to children as if they were miniature adults" a UNC investigator said. The Coordinating Center in 2003 sponsored a think tank workshop to develop a research agenda for the evaluation of the health impact of diagnostic and therapeutic devices.

3.3.2. Level 2: Impact on Policies and Change Agents

Tunis and Stryer state that Level 2 research impact requires "a policy or program that is created as a direct result of the research, including use of information by health plan, professional organizations, legislative bodies, regulators, accrediting organizations... etc." The level 2 impacts of the Portfolio research are described in the following areas, including: standardized quality of care performance measures, professional guidelines, drug labeling, and drug withdrawals from the market. A Steering Committee member describes:

The concept of policy is complicated because you have to define "policy' Is it the inclusion of a drug in a formulary, is it the inclusion of a warning in a label, is it translation of a warning in a label into an action program to make a difference?

An Alabama researcher stated, "there have been some terrific policy pieces that have come out of some of the risk workshops that have been initiated by the CERTs." A UNC investigator described:

We've decided that we should work with existing national organizations (AAP, American Board of Pediatric Medicine) and help them do better getting educational material into practice... Our goal with the Board is to try to get our findings inserted into guidelines and the practitioner re-certification process.

HEDIS and Quality Indicators/Measures OR Changing Measures

HEDIS™³⁵ measures illustrate an important area for potential policy impact because of the wide use of these measures by managed care plans and their potential impact on plan members. The Duke CERTs collaborated with the Council for Affordable Quality Healthcare (CAQH) in developing a national initiative to evaluate long-term use of beta-blockers in patients with previous myocardial infarction. A Duke investigator described "working with the project manager from CAQH who began talking to NCQA (National Committee for Quality Assurance) about adding long term adherence measures to HEDIS."

One of the grantees that held two Portfolio grants on antibiotic use and resistance attributes his AHRQ-funded research with affecting his subsequent work on developing a HEDIS measure. He indicated that "it was work from this project that led to my participation as their expert consultant to develop an adult measure for appropriate antibiotic use, and that just finished the testing phase last year and is being released as a new measure." (Grantee). The grantee continues:

My work has been focused on adults with acute bronchitis, because that's where the biggest quality gap is. This is a very important mechanism for disseminating the importance of appropriate antibiotic use, specifically for reducing overuse of antibiotics for adults with acute bronchitis where there's strong evidence to show that antibiotics make no difference but prescription rates remain 50-70%. Many major commercial health plans, most Medicaid managed care plans, and many Medicare plans seek NCQA accreditation and report on the HEDIS measure, so this will be a very powerful tool to get the message to physicians that this is something that's important.

The UAB CERT has contributed to the development of quality indicators for osteoporosis and arthritis. A CERT Annual Report stated that the UAB CERT "worked with RAND and the National Committee for Quality Assurance to develop quality-of-care indicators for the management of patients with osteoporosis." (UAB ARY3) Additionally, the UAB CERT developed quality-of-care indicators for gout management and updated guidelines for the Arthritis Foundation Quality Indicator Project. (UAB ARY5)

Drug regulation

Some CERTs maintain data on drugs post market and offer examples of their work having at least informed if not directly impacted the withdrawal of medications from the market. The CERTs also have conducted research on the understanding of prescription labels, including black-box warnings and assessment of the risk management programs of specific medications. These contributions appear to have had an indirect impact on policy; as an AHRQ representative clarified, "I wouldn't say that's policy per se, it's regulation, which is probably better than policy." The drug withdrawals in which CERTs investigators claim a role for their research are described below.

The Vanderbilt CERT described in a progress report, "In 1998 a regulatory action was taken by FDA to limit prescribing of cisapride and drugs that affect cisapride metabolism. CERT study compared prevalence of contraindicated use of cisapride before and after 1998. Little difference was found. This study, with others, led to the voluntary withdrawal of cisapride in July 2000." (VAN PR 01-02). A Vanderbilt CERT investigator said:

We demonstrated that a regulatory intervention designed to make use of Cisapride safer actually failed and we think that led to the drugs being withdrawn from the market... so that was a big change. The FDA put in a black box warning and publicized it highly. The FDA thought that solved the problem, but we did a study with the HMO research network site, and we found that the black box warning did nothing and as much as 20% of the use was in people for whom the drug was very dangerous. Since it was a drug for treating heartburn (and not the best drug for treating heartburn), this called into question its continued presence in the market. Shortly after the preliminary findings of our study were released to the FDA, the manufacturer withdrew Cisapride from the market.

In September 2004, the COX-2 inhibitor Vioxx (rofecoxib) was withdrawn from the market. Some of the CERTs research centers identified their research as having contributed to that outcome. The Vanderbilt CERT described a study on rofecoxib in the progress report for 2001-2002, stating, "We noted that 17% of rofecoxib users were using this higher dose and that this proportion did not vary among those with cardiovascular disease, who would be particularly susceptible to dose-related adverse effects" (VAN PR 01-02). An investigator from Vanderbilt further stated:

One of the early studies to look at some of the Cox 2 effects came out of this CERT. That really began to open up the possibility of looking further into this so further studies were done. There was then regulatory action at the FDA on the Cox 2s, and that had a direct effect.

Another Vanderbilt investigator said, "We were among the first to identify the risk associated with Vioxx." (VAN) A CERT Steering Committee member said that the CERTs, "were able to contribute to the debate relating to Vioxx; there was major policy impact... we were able to add science to political rhetoric." (SC) However, in another interview one of the CERT investigators said that it was a challenge to the CERTs diversity to take a stance on a particular, critical topic and he identified Vioxx as one such topic.

The Arizona CERT in a progress report indicated that it has "... pioneered the concept of using an Internet based registry for the study of rare adverse drug reactions. The first project... has resulted in the removal from the market of five major drugs in the past four years" (AZ PR01-02). As a result of the Web-based International Registry for Drug-Induced Arrhythmia, the Arizona CERT discovered methadone to be a cause of lethal arrhythmia (AZ ARY3). Additionally, the Arizona CERT PI stated that their work was also instrumental in getting Orlam (levo-methadyl acetate HCl) off the market, because it appeared to cause arrhythmia.

Besides the work on the medications that were withdrawn, the CERTs contributed to the FDA and drug regulation in other capacities typically with regard to the change in drug labeling. There are a few examples that were identified in the CERTs progress reports and stated by the investigators:

Retrospective study of antipsychotics and sudden cardiac death. "Among cohort members with severe cardiovascular disease, current moderate-dose users had a 3.3 fold increased rate relative to comparable nonusers, resulting in 367 additional deaths per 10,000 person-years of follow-up. These data were reviewed by the FDA prior to the thioridazine label change" (VAN PR 01-02)

ACE inhibitor medications early in pregnancy. One investigator had just published a paper in the New England Journal looking at the effects of ACE inhibitor medications early in pregnancy. The FDA held a press conference about a public warning or advisory before the paper was released. This led to a follow-up AHRQ grant to the HMO Network to look at prescribing of ACE inhibitors and ARBs during pregnancy. (VAN)

Selective serotonin reuptake inhibitors (SSRI) and the serotonin-norepinephrine reuptake inhibitors. Another CERT reported in their progress report that they were contacted by scientists in the Division of Neuropharmacologic Drug Products of the FDA and asked if they had any "comparative information on the risk of TdP (torsades de pointes) with the newer selective serotonin reuptake inhibitors (SSRI) and the serotonin-norepinephrine reuptake inhibitors" (AZ PR04-05)

Additionally, as described earlier in this report, the CERTs program held five think tanks on risk, as described by a CERTs steering committee member:

The FDA recognized that it had mandated under the Prescription Drug User Fee Act a program to govern a new and evolving concept called therapeutic risk management. There were no rules of engagement, there was no experience, and it wasn't clear what FDA needed to do. The FDA came to the CERTs and asked us to convene a national think tank to help them deliberate about appropriate implementation... Not only did we do it, but after we convened the first one, it was clear that we needed to convene many, and we ended up convening five in that first wave of think tanks, which created the context for guidance development by the FDA leadership.. the National Guidance on Risk Management was configured in the CERTs think tanks.

An AHRQ representative indicated that the "CERTs have generated regulatory activity on the part of the FDA in terms of labeling changes." In addition to the aforementioned studies identifying the deleterious effects of medications and their impending withdrawal, the CERTs also evaluated a risk management program for dofetilide. Additionally, the CERTs have spoken to the FDA on at least a few occasions; for example, the CERT Coordinating Center PI spoke to the FDA about "Risk management of prescription drugs." Also, CERTs educational modules are accessible through the FDA's CDER Web site.

Guidelines

Clinical practice guidelines can guide practitioners' delivery of care and are indicative of an impact on policy if research effects a change in the guidelines. Many CERTs have contributed to the development, improvement, and evaluation of clinical practice guidelines as part of their research. The CERTs have been involved with assessing the extent to which practitioners adhere to clinical guidelines. For example, the Alabama CERT examined adherence to Carpal Tunnel Syndrome guidelines. CERTs have also assessed the appropriateness of guideline components. For example, the Penn CERT examined outcomes of alternative strategies for diagnosing and treating pediatric acute otitis media. The Duke CERT conducted an evaluation of American College of Cardiologists (ACC) guidelines and indicated that this was "the first study to document that practitioners had concerns about conflict of interest,"(Duke KW) which resulted in the ACC changing this. Also, UNC has been "working with the ACC on more rapidly updating guidelines" (Duke NAL). The Penn CERT investigators suggested that their research has changed antibiotic guidelines. One example is the finding regarding changing endocarditis guidelines for dental prophylaxis for cardiac abnormalities. As a result of this research the CDC and American Thoracic Society changed their guidelines for antibiotic prophylaxis. Penn also published in the Archives of Dermatology widely disseminated guidelines on the use of antibiotics for acne.

Medication Errors

In addition to their contribution to the knowledge base for understanding medication errors, the CERTs have also impacted policies relating to medication errors. The UNC CERT investigators worked with the United States Pharmacopeia (USP) to use the MEDMARX system where medication errors are reported. They conducted an analysis of errors in pediatrics, which led to the drafting of an addition to the USP recommendations, Error Avoidance Recommendations for Pediatric and Neonatal Medicine Use (UNC ARY3). A year later in their Annual Report, the UNC CERT stated that using the MEDMARX Web-based reporting system (error reports), it made five sets of recommendations that the U.S. Pharmacopeia placed on its Web site in April 2003 (UNC ARY4).

3.3.3. Level 3: Impact on Clinical Practice

Level 3 impact level, which is difficult for researchers in general to achieve, involves "a change in what clinicians or patients do, or changes in a pattern of care" (Tunis & Stryer). Tunis and Stryer also included a Level 3a and Level 3b, to indicate when impacts in clinical practice are "demonstrated in a limited study population as a result of a specific intervention" and when then "impacts are trends identified outside a formal research context," respectively. Little Portfolio research has had an impact on clinical practice. The nature of the study may be a predictor of the potential level of impact a study can have on clinical practice. For example, the studies that had a level 3 impact were intervention studies and controlled trials. The following CERTs intervention studies and their effect on behavior are briefly described below:

The UAB CERT in their intervention study of steroid-associated bone disease reported a change in prescribing behavior by physicians associated with a national health plan.

• Several interventions have been designed to reduce errors at various points in the ambulatory care setting: Patient Specific order entry intervention; Drug Specific Order Entry intervention; Communicating with Patients about Error; Quality Improvement intervention. (HMO PR 02-03).
• Additionally, the HMO CERT found in a study of medication alerts that "after the alerts were implemented, a steady decline in the number of apparent errors was noted .drug-specific alerts resulted in a 22% reduction in the rate of use of the target medications." The study observed a 14.9% reduction in warfarin medication interactions, due to alerts. (HMO Final Report).

Academic detailing and printed educational materials achieved their intended purpose of reducing the odds of patient requested antibiotics for upper respiratory infection (Penn PR 02-03).

The UAB CERT reported in a study with a large national managed care organization that physicians who participate in Web-based continuing medical education coupled with wider feedback of their performance may see slight improvement in their quality of care (despite barriers to changing provider behavior).

The Penn CERT found: that while physician-based interventions reduce inappropriate prescribing of antibiotics, there is substantial opportunity for improvement. That CERT designed interventions to identify patients who gratuitously seek antibiotics and designed a brochure about why not to seek antibiotics for a cold.

Additionally, two of the Portfolio grant recipients conducted research that changed medical practices. One stated that he believes even beyond the impact of his study intervention that "others have picked up on the materials after the intervention period ended." Another grantee's work on antibiotic use in Colorado involved a coalition:

It's really wonderful to see... that the Health Department took over sponsorship of this program so that it's still the "get Smart" campaign. Now there is an individual in the Colorado State Health Department who is the coordinator for these activities. She keeps the plans involved, and they've been doing a lot more outreach... they're doing a lot more local activities around health fairs. In fact they are planning a new mass media Spanish language campaign.

What I feel most excited about is that a critical mass that came together after the grant ended to keep things moving, because they saw value from the public health perspective as well as from what's important from the business perspective and the community perspective. It has found a way to become self-sustaining.

The impact on clinical practices is difficult to identify and measure; however there is an implicit assumption that some of the level 2 impacts, such as a drug being withdrawn from the market would ultimately impact prescribing and clinical practice by changing available therapeutic options. Additionally, time for diffusion of findings is needed to recognize an impact on clinical practice and significant resources would be necessary to measure and identify impact at this level.

Return to Contents
Proceed to Next Section