Pharmaceutical Research

Journal supplement explores alternative research approaches to test drug safety and effectiveness

Though the prescription drug benefit (Part D) received a large amount of media attention when Congress passed the Medicare Modernization Act of 2003, Section 1013 of the act has important health implications as well. Because the Federal government became a major payer for prescription drugs, that section called upon the skill sets of the Agency for Healthcare Research and Quality (AHRQ). As the lead Federal agency for improving the quality, safety, and effectiveness of health care, the Agency was asked to conduct research to increase knowledge about the risks and benefits of drug therapies to improve health care decisions.

To implement Section 1013, AHRQ sponsored a 2-day symposium in June 2006 for researchers to present approaches to designing studies, identifying data sources, and developing methods for studying outcomes, safety, and effectiveness. In October 2007, the journal Medical Care 45(10) published the symposium proceedings as a supplement. The supplement is a resource for scientists studying the safety and effectiveness of treatments. It brings together national experts in the field to spark a national discussion on how science can provide information on improving health outcomes and public health. The papers written by AHRQ and AHRQ-supported researchers are briefly summarized here.

Smith, S., "Preface," pp. S1-S2.

AHRQ created the Effective Health Care program to meet the requirements of Section 1013. The author of the supplement's preface, who works for AHRQ's Center for Outcomes and Evidence, explains how the program's three components work together to compile, evaluate, develop, and disseminate scientific evidence on the effectiveness of treatments. Evidence-Based Practice Centers review scientific evidence about the effectiveness of treatments and pinpoint gaps in the evidence. Filling those gaps with new research and new methods falls to the DEcIDE (Developing Evidence to Inform Decisions about Effectiveness) research network and the Centers for Education and Research on Therapeutics. Finally, the John M. Eisenberg Clinical Decisions and Communications Science Center takes scientific findings and puts them into lay language to share with diverse audiences.

Lorh, K.N., "Emerging methods in comparative effectiveness and safety: Symposium overview and summary," (AHRQ contract 290-05-0036-1), pp. S5-S8.

In this symposium summary, the author explains that current methods for collecting safety and effectiveness data are lacking, because they either limit generalizability or are based on subjective observations. Cluster randomized trials, which can be employed in different regions or health care settings and systems, are valuable for evaluating outcomes from actual use. Databases can also be mined to look at the pros and cons of medications when they are prescribed, and if they are taken. With more drugs being pulled from formularies because of their negative health effects, surveillance systems for monitoring adverse events can serve as sentinels for safety problems. Finally, using research approaches that reduce bias (error caused by encouraging one outcome over another) and confounding (interference by a variable that affects the study's conclusions) can allow researchers to make recommendations on the safest, most effective therapies.

Strom, B.L., "Methodologic challenges to studying patient safety and comparative effectiveness," (AHRQ contract 290-05-0036-1), pp. S13-S15.

Study methodologies that address patient safety and drug effectiveness can encounter many difficulties. The author explains the issues of selection bias, misclassification of exposure or outcomes, confounding, and logistical challenges. Data access, in particular, continues to be a logistical problem. For example, researchers like the idea of having access to data collected for Medicare Part D (the prescription drug benefit) and Medicare claims data, because they contain numerous variables and cover a large, stable population. However, no access has been granted to the Federal data, nor is there any indication that it will be. Other logistical challenges include obtaining Institutional Review Board approval and convincing people that studies on patient safety and comparative effectiveness need to be conducted.

Atkins, D., "Creating and synthesizing evidence with decision makers in mind: Integrating evidence from clinical trials and other study designs," pp. S16-S22.

Just as bacteria cultures have been the gold standard for laboratories, randomized controlled trials (RCTs) remain the gold standard for testing the efficacy of new drugs and medical procedures. However, the RCT format cannot provide physicians complete information on risks and benefits for patients or help assess the repercussions of adopting policies on certain medical therapies. The author, who works in AHRQ's Center for Outcomes and Evidence, demonstrates the limitations of RCTs. The researcher sifted through the results of multiple RCTs to determine what would seem to be simple answers. He posed the question of whether low-dose daily aspirin would help a 60-year-old woman with mildly high blood pressure. (Answer: It wouldn't.) He also looked at RCT results to determine if the risks of performing carotid artery surgery outweighed the benefit of preventing a stroke if the patient had no symptoms. (Answer: It depends on the patient.) He suggests that improvements in care would occur if cohort and case-control designs, disease and intervention registries, outcomes studies using administrative databases, and quality improvement methods were used.

Reprints (AHRQ Publication No. 08-R012) are available from the AHRQ Publications Clearinghouse.

Sedrakyan, A. and Shih, C., "Improving depiction of benefits and harms: Analyses of studies of well-known therapeutics and review of high-impact medical journals," pp S23-S28.

Researchers presenting their findings on therapeutics in medical journals would better serve their readers by consistently presenting clear proof of the benefits and harms the therapy offers. During a literature review, the authors from AHRQ's Center for Outcomes and Evidence found that one in three studies that reports this information does not use the same measurement tools. This hampers clinicians' ability to accurately communicate with patients. The authors look at well-known therapeutics and side effects, such as hormone replacement therapy and stroke, to show that relative risk estimates (high risk and low risk) are not helpful in communicating to patients the chances of having an adverse event.

Reprints (AHRQ Publication No. 08-R013) are available from the AHRQ Publications Clearinghouse.

Mazor, K.M., Sabin, J.E., Boudreau, D., and others, "Cluster randomized trials: Opportunities and barriers identified by leaders of eight health plans," (AHRQ grant HS10391), pp. S29-S37.

When researchers test therapeutics for safety and efficacy, they commonly conduct randomized controlled clinical trials and use observational methodologies. Cluster randomized trials (CRTs) have been offered as an alternative to these traditional methods. This is because they evaluate outcomes by studying actual use in clusters of study subjects who can be dispersed, for example, throughout a region or among different care settings. The authors interviewed 34 health plan leaders from 8 health plans and learned that these individuals agree on the need to study the effectiveness of therapeutics in real-world situations. However, they also zeroed in on barriers to conducting these studies, including costs, perceptions of these trials, physician prescribing habits, and formulary changes.

Maclure, M., Carleton, B., and Schneeweiss, S., "Designed delays versus rigorous pragmatic trials: Lower carat gold standards can produce relevant drug evaluations," (AHRQ contract 290-05-0016), S44-S49.

The authors discuss the merits of designed delayed pragmatic randomized trials (PRTs). These trials allow policymakers to have a large say in how trials are conducted. For example, the trials are conducted with patients in real-world settings and let factors, such as the stop and start dates, fluctuate after negotiation with policymakers. After conducting these types of trials, the researchers found that policymakers are receptive to design delay trials because they are simple and don't cost as much as traditional trials. Issues surrounding this kind of trial include ethics concerns, timing, and the actual policies being tested in the trial. Further, research results on drug safety and effectiveness may be less generalizable because they are not conducted with selected patients in purer settings.

Horn, S.D. and Gassaway, J., "Practice-based evidence study design for comparative effectiveness research," (AHRQ grant HS15350), pp. S50-S57.

In keeping with this issue's theme of offering alternatives to traditional clinical studies, the author describes a study method called practice-based evidence for clinical practice improvement (PBE-CPI). This approach captures real-world variation in clinical practice using data documented by medical providers. Data are able to show the effectiveness of combinations of interventions in various patients, something traditional trials do not test. The low cost of using existing data and point-of-care documentation makes PBE-CPI less expensive than traditional studies. For example, PBE-CPI studies to date have had sample sizes ranging from 1,000 to 2,500 patients and have cost between $1 and $5 million. Traditional randomized controlled trials with similar sample sizes can cost more than 20 times as much and answer fewer questions.

Crystal, S., Akincigil, A., Bilder, S., and Walkup, J., "Studying prescription drug use and outcomes with Medicaid claims data: Strengths, limitations, and strategies," (AHRQ grants HS16097 and HS11825), pp. S58-S65.

Data housed in Medicaid databases offer abundant information on prescription drug use among low-income, minority, elderly, and disabled populations that often is not captured by randomized clinical trials. After reviewing studies that relied on Medicaid data, the researchers developed a list of "best practices" for using the data and checking its validity, for choosing statistical models, and for improving Medicaid datasets. The researchers propose a program that links multiple databases that house claims data, disease registries, surveys, electronic medical record data, and birth and death records to provide researchers a national research database on prescription drug use and safety.

Grijalva, C.G., Chung, C.P., Arbogast, P.G., and others, "Assessment of adherence to and persistence on disease-modifying antirheumatic drugs (DMARDS) in patients with rheumatoid arthritis," (AHRQ contract 290-05-0042-1), pp. S66-S76.

Disease-modifying antirheumatic drugs (DMARDs) are expensive alternatives that help patients whose rheumatoid arthritis does not respond to common drugs. Though randomized clinical trials have shown these drugs to be effective, no studies have examined their effectiveness in a clinical setting. These researchers, providing an example of how researchers can exploit the information stored in Medicaid databases, located 6,018 patients who were prescribed DMARDS for their rheumatoid arthritis. They studied the information in a Tennessee Medicaid database to determine if the drugs were used consistently once they were prescribed. Patients seemed to consistently use the DMARD methotrexate; however, fewer patients consistently used the DMARD sulfasalazine.

Nebeker, J.R., Yarnold, P.R., Soltysik, R.C., and others, "Developing indicators of inpatient adverse drug events through nonlinear analysis using administrative data," (AHRQ grant HS11885), pp. S81-S88.

Adverse drug events are among the most costly of patient safety problems. The researchers used a method called hierarchically optimal classification tree analysis (HOCTA), which has demonstrated an ability to create predictive models that are more accurate than traditional linear models. The researchers used HOCTA to develop an accurate model to predict the rate of adverse drug events involving bleeding and clotting problems caused by medication.

Lieu, T.A., Kulldorff, M., Davis, R.L., and others, "Real-time vaccine safety surveillance for the early detection of adverse events," (AHRQ contract 290-05-0036), pp. S89-S95.

Keeping in the theme of adverse event reporting, the authors of this article set out to create and test a surveillance system for early detection of adverse events once a new vaccine enters the marketplace. The Vaccine Safety Datalink Project, sponsored by the Centers for Disease Control and Prevention, examined data files from eight health plans every week to detect adverse events involving a new meningococcal vaccine for adolescents. Resources used to track adverse events included dynamic data files, aggregation of data, and sequential analysis methods. The researchers determined that this method of surveillance offers a useful, adaptable approach to detecting adverse events early.

Curtis, L.H., Hammill, B.G., Eisenstein, E.L., and others, "Using inverse probability-weighted estimators in comparative effectiveness analyses with observational databases," (AHRQ grant HS10548), pp. S103-S107.

When researchers conduct drug or treatment trials, they want to gather as much information as they possibly can. Often, they seem to reach for the impossible: to know what would happen if the same person had been exposed to both treatments being tested. The researchers show how the tool of inverse probability-weighted estimation can complement observational data to provide answers on the effectiveness of two or more treatments. A major drawback to using the method is the lack of statistical software. However, because the method yields useful information and is flexible, the authors expect to see more inverse probability-weighted estimators in medical and epidemiology literature.

Samore, M.H., Shen, S., Greene, T., and others, "A simulation-based evaluation of methods to estimate the impact of an adverse event on hospital length of stay," (AHRQ contract 290-05-0036-1), pp. S108-S115.

The authors of this paper compared conventional analytic methods with inverse probability weighting (IPW) to study the problem of time-varying confounding. This problem occurs when an outcome and exposure are both influenced by a third variable that changes over time, for example, when disease severity influences the decision to start drug therapy. The researchers compared conventional analytic methods with IPW in a simulated group of hospitalized patients to determine the effect an adverse event has on the length of stay in a hospital. They found that, unlike conventional regression methods, IPW had less bias.

Schneeweiss, S., Patrick, A.R., Sturmer, T., and others, "Increasing levels of restriction in pharmacoepidemiologic database studies of elderly and comparison with randomized trial results," (AHRQ grant HS10881), pp. S131-S142.

To ensure they compare apples to apples and avoid bias, researchers restrict who can participate in a study to create a similar study group with similar characteristics. The authors of this paper undertook a large database study of 122,406 patients who use statins to control blood cholesterol levels. They winnowed out participants by establishing five restrictions. In the end, they came up with results that closely reflected those reported in randomized clinical trials. The authors suggest that putting restrictions in place does not significantly diminish the generalizability of research findings.

Segal, J.B., Griswold, M., Achy-Brou, A., and others, "Using propensity scores subclassification to estimate effects of longitudinal treatments: An example using a new diabetes medication," (AHRQ contract 290-05-0034), pp. S149-S157.

Researchers comparing medication effectiveness using observational data must consider patient characteristics that cause treatments to change over time. Armed with claims data for 131,714 patients with diabetes, the authors developed a method that generates potential outcomes using propensity scores at multiple time points. They used their method to estimate the effects of treatments for diabetes at different time points to compare outcomes between a new drug (exenatide) and traditional drugs (insulin and oral medications). They found there were no differences in the new drug's outcomes compared with existing therapies. The authors suggest that their method will make large observational databases more useful for comparing the effectiveness of new drugs and treatments.

Return to Contents
Proceed to Next Article