Navigation

About this book Advances in Patient Safety: From Research to Implementation Volume 3. Implementation Issues Sections and articles in Advances in Patient Safety volume 3: Where the Rubber Meets the Road Using Specialized Information Technology to Reduce Errors in Emergency Cardiac Care Improving Patient Safety with the Military Electronic Health Record Decision Support System Design and Implementation for Outpatient Prescribing: The Safety in Prescribing Study Looking for Trouble in All the Right Places: The Legal Implications Associated with "Electronic Signatures" and High-risk Clinical Situations Clinical Informatics and Its Usefulness for Assessing Risk and Preventing Falls and Pressure Ulcers in Nursing Home Environments Barriers Associated with Medication Information Handoffs Using Reported Primary Care Errors to Develop and Implement Patient Safety Interventions: A Report from the ASIPS Collaborative The AAFP Patient Safety Reporting System: Development and Legal Issues Pertinent to Medical Error Tracking and Analysis Lessons Learned from the Evolution of Mandatory Adverse Event Reporting Systems Shared Learning and the Drive to Improve Patient Safety: Lessons Learned from the Pittsburgh Regional Healthcare Initiative Identifying Barriers to the Success of a Reporting System "Near-miss" Reporting System Development and Implications for Human Subjects Protection The Impact of a Web-based Reporting System on the Collection of Medication Error Occurrence Data Ten Considerations for Easing the Transition to a Web-based Patient Safety Reporting System Identifying, Understanding, and Communicating Medical Device Use Errors: Observations from an FDA Pilot Program Implementation of a Data-based Medical Event Reporting System in the U.S. Department of Defense The Impact of Personal Digital Assistant Devices on Medication Safety in Primary Care Implementation of an Evidence-based Protocol for Surgical Infection Prophylaxis Reducing the Use of Short-acting Nifedipine by Hypertensives Using a Pharmaceutical Database Can an Academic Health Care System Overcome Barriers to Clinical Guideline Implementation? Implementing a Systems Engineering Intervention for Improving Safety in Outpatient Surgeries Improving the Safety of Heparin Administration by Implementing a Human Factors Process Analysis Establishing a Culture of Patient Safety Through a Low-tech Approach to Reducing Medication Errors Cost Effectiveness of a Multifaceted Program for Safe Patient Handling Patient Safety Data Sharing and Protection from Legal Discovery Does Medical Error Disclosure Violate the Medical Malpractice Insurance Cooperation Clause? Beyond the Dusty Shelf: Shifting Paradigms and Effecting Change From Insight to Implementation: Lessons from a Multi-site Trial of a PDA-based Warfarin Dose Calculator Lessons in Safety Climate and Safety Practices from a California Hospital Consortium Systemwide Deployment of Medical Team Training: Lessons Learned in the Department of Defense Barcode Medication Administration: Lessons Learned from an Intensive Care Unit Implementation Institutional Review Board Approval of Practice-based Research Network Patient Safety Studies Behind the Scenes: Patient Safety in the Operating Room and Central Materiel Service During Deployments Surgical Safety: Addressing the JCAHO Goals for Reducing Wrong-site, Wrong-patient, Wrong-procedure Events Voluntary Hospital Coalitions to Promote Patient Safety Suicide Risk Response: Enhancing Patient Safety Through Development of Effective Institutional Policies A New Model of Tracheostomy Care: Closing the ResearchPractice Gap Peer ReviewersVolume 3

Advances in Patient Safety: From Research to Implementation Advances in Patient Safety: From Research to Implementation Volume 3. Implementation Issues

From Insight to Implementation: Lessons from a Multi-site Trial of a PDA-based Warfarin Dose Calculator

Richard L. Kravitz, Jonathan D. Neufeld, Michael A. Hogarth, Debora A. Paterniti, William Dager, Richard H. White

Abstract

Clinical decision support (CDS) systems show promise for enhancing patient safety, but they require rigorous evaluation before they can be implemented widely. We developed a software application for use with personal digital assistants (PDAs) that models patient-specific dose responses to help physicians predict steady-state warfarin dosing requirements and steer patients to a therapeutic level of anticoagulation as quickly and safely as possible. We also designed a randomized, controlled multi-site trial to evaluate the effectiveness of the Warfarin Dosing and Communication System (WARFDOCS) in reducing warfarin-related errors. Numerous obstacles delayed implementation of the CDS system and completion of the trial. To better understand the causes that led to the delay, we interviewed key informants at participating hospitals; reviewed study protocols, administrative records, and meeting minutes; and held discussions to review the data and their interpretation. Salient themes were identified by consensus of the research team and these were corroborated by key informants. Four major themes emerged. First, agreement to participate in the trial reflected very different levels of commitment. Sites participating in CDS system evaluations must be managed actively. Second, the enthusiasm of end-users for a CDS system was derived from a complex calculus of perceived benefits and burdens. Unfortunately, the most relevant appeal (that such a system would markedly improve patient safety) could not be made in advance of the trial. Third, research changes everything. Valid research procedures (e.g., informed consent, randomization, and intrusive data collection) may be necessary, but can themselves affect a key outcome of most CDS system evaluations: user uptake. Fourth, strong "center effects" (i.e., the CDS system proved effective at some sites, but not at others) should be expected. If "all politics is local," then much of patient safety research is localized as well.

Introduction

Clinical decision support (CDS) systems are computer-based tools designed to assist clinicians in making higher quality and more cost-effective medical decisions. They have been defined operationally as "active knowledge systems which use two or more items of patient data to generate case-specific advice." ¹ CDS systems are proliferating rapidly and exhibit considerable potential for improving the realm of patient safety. ² However, despite the consensus opinion that rigorous assessment of these systems is imperative, the pace of evaluation has lagged far behind implementation. One Internet-based product catalog alone lists 73 medical expert and knowledge-based systems, ³ yet published articles on clinical computing remain "a descriptive feast but an evaluative famine." ⁴ This is unfortunate, given the high capital costs of CDS systems development and the real possibility that such systems could produce harm as well as benefit. ⁵

Evaluators of CDS systems face numerous scientific and technical challenges, including the application of suitable controlled designs in the face of rapidly advancing technologies, difficulties in selecting the unit of allocation and analysis (i.e., patient vs. group or cluster), meaningful outcomes measurement, and the interpretation of results. ⁴ However, researchers face equally daunting, albeit less heralded, pragmatic challenges in their efforts to implement and evaluate CDS systems. These challenges arise from the organizational context and the human beings working in it. ⁶ As Lorenzi described it, "People who have low psychological ownership in a system and who vigorously resist its implementation can bring a 'technically best' system to its knees." ⁷ Understanding the roots of resistance is critical to managing both uptake (by users) and assessment (by evaluators).

Warfarin is being prescribed to more than two million Americans and is among the most hazardous of drugs commonly prescribed today. ^{8 13} Dosing errors that result in excessively high or low anticoagulant effects are common, particularly in the first month of treatment. Sources of correctable dosing errors have been identified and include poor dosing judgment, failure to recognize well-described drug interactions, and failure to communicate important dosing information to the patient during the transition between hospital and outpatient followup. ^{8, 11, 13, 14}

Establishing the maintenance (i.e., steady-state) dose of warfarin is a major clinical problem. The between-patient variation in daily warfarin dose requirements is large; patients can require anywhere from about 0.5 mg per day to more than 15 mg per day. Given today's frequently short hospital stays, pharmacologically sensitive patients may be discharged from the hospital with their international normalized ratio (INR) still rising. By the time they are seen in the outpatient setting, they may already be bleeding.

Clinical researchers have developed several approaches to enhancing the safety of warfarin therapy. In the ambulatory setting, specialized anticoagulation clinics lead to more consistent anti-coagulation and fewer adverse outcomes; similar benefits have been achieved through an Internet-based "virtual clinic." ¹⁵ In the hospital, proven strategies for improving warfarin dosing include the use of dosing nomograms, ^{16, 17} daily consultations with a pharmacist, ¹⁸ and various desktop computer-based decision support tools. ¹⁹

While effective, the existing approaches to anti-coagulation control are cumbersome and have proved difficult to implement in practice. In an effort to address these drawbacks, members of our research group created the Warfarin Dosing and Communication System (WARFDOCS). At the core of WARFDOCS is a computerized decision-support aide for the initiation of warfarin therapy ²⁰ that can be loaded on a Palm OS-based personal digital assistant (PDA). The PDA platform was chosen to allow users greater mobility and complete independence from existing hospital information technology infrastructures. The WARFDOCS system requires only the PDA running the WARFDOCS application and an infrared-enabled printer to provide its entire range of benefits. No other connections (e.g., to a desktop computer or network) are necessary.

We developed a framework for evaluating the WARFDOCS program's effects on patient safety, but numerous obstacles slowed and sometimes threatened to halt our progress. To better understand the causes behind the delay, we interviewed key informants, examined study records, and held team meetings to assess the data and various interpretations of it. Although we began with an open-minded curiosity with regard to "what went wrong," our inquiry eventually focused on four narrower questions:

In this report, we will describe the difficulties encountered in the launch of a multi-center controlled trial of WARFDOCS, emphasizing the perceptions of CDS system end-users as they pertain to the organizational contexts affecting the implementation. Our focus on evaluation is motivated by the recognition that rigorous assessment of health information technologies is essential for sifting the worthwhile from the useless, and ultimately for effectively disseminating high-value interventions. Our findings have implications for evaluation planning, interpretation, and generalizability.

Background and methods

The WARFDOCS clinical decision support system

The WARFDOCS program is a PDA-based system that provides a daily dose-response estimation using the Bayesian pharmacokinetic-pharmacodynamic model previously implemented in a desktop system named DrugCalc. ²² The system models a patient's response to warfarin and allows clinicians to "test out" various dosing plans that include changes in the current day's dose and the subsequent day's dose. Pharmacists can use the system to recommend a course of anticoagulation care that minimizes the time required to achieve the target INR and the likelihood that the INR will rise to a value above 3.5 (at which point the risk of bleeding increases sharply). A Drug Interaction Knowledge Base on the PDA provides information on clinically meaningful drug interactions, but these interactions are not incorporated into the actual computer model.

Our randomized controlled trial evaluating the effectiveness of WARFDOCS relied heavily upon inpatient clinical pharmacists. The protocol calls for pharmacists to: 1) identify potential study patients using the pharmacy's computer system (or a paper system, in the absence of a computer); 2) review the patient's medical record and then complete an eligibility form stored on the PDA; 3) enroll consenting patients after first discussing the study with them and randomizing the patients to either the WARFDOCS group or the standard care group; 4) make daily rounds on enrolled patients, entering data on daily warfarin doses, INR values, and concurrent medications; 5) use the PDA to make a daily dosing recommendation and communicate this information to the ordering clinician; and 6) print out an Anticoagulation Discharge Summary Report and deliver it to the physician, clinic, or hospital assuming responsibility for subsequent outpatient anticoagulation care, on the day of the patient's discharge. This process was designed to enhance the safety of warfarin use by ensuring that physicians maintained dosing control, that INRs were monitored daily, and that all critical information was transferred to the responsible outpatient provider.

Evaluation design

In the ongoing WARFDOCS trial, patients at each participating hospital are assigned randomly to either the WARFDOCS system or to usual care. The hospitals were chosen for convenience and were not matched, so randomization by site was not possible. The principal outcome of interest is the proportion of patients who experience bleeding (caused by over-anticoagulation) or thrombosis (caused by under-anticoagulation) during the 4-week period following their enrollment in the study. Baseline clinical data are obtained by chart review. Warfarin doses and INRs are tracked daily throughout the hospital stay for intervention and control patients; however, INR recommendations are generated for intervention patients only. Outcomes data are collected via chart review and follow-up telephone interviews with patients are conducted 3060 days after their discharge from the hospital.

Settings

Nine hospitals initially agreed to participate in the WARFDOCS project. The essential characteristics of the hospitals are summarized in Table 1. Each facility had a unique history and culture with regard to warfarin care. Pharmacists at Hospital A have been involved in warfarin management for more than a decade. Anti-coagulation pharmacists routinely review the hospital charts of the vast majority of inpatients started on warfarin, and are available for additional consultation as needed. Inpatient pharmacists at Hospitals B and C consult regularly with physicians on clinical issues; however, no formal warfarin service exists at either facility.

Table 1. Characteristics of hospitals originally committed to participate

Hospital	Type	Location	Number of beds	Final status
A	University	Urban mid-sized city	540	Currently enrolling patients
B	Group model health maintenance organization	Suburban	350	Currently enrolling patients
C	Non-profit community hospital	Suburban	254	Currently enrolling patients
D	Nonprofit community hospital	Semi-rural	89	Currently enrolling patients
E	Non-profit community hospital	Urban core of a rural region	267	Currently enrolling patients
F	Non-profit community hospital	Rural	102	Withdrew before enrolling patients
G	Non-profit community hospital	Urban core of a rural region	195	Withdrew after change of management
H	Non-profit community hospital	Urban mid-sized city	208	Withdrew after reviewing detailed scope of work
I	Non-profit community hospital	Urban mid-sized city	304	Withdrew after reviewing detailed scope of work

There is little history of pharmacist-physician interaction at Hospitals D and E, with regard to clinical issues. A policy at Hospital F permitting warfarin orders "per pharmacy" (i.e., a pharmacist calculates and dispenses the appropriate dose) once existed, but was abandoned soon after its implementation; community physicians are currently responsible for warfarin dosing and all other aspects of inpatient prescribing. Hospitals G, H, and I dropped out of the study before any assessment of the prevailing work relations could be performed.

Data sources and analytic approach

This paper presents the results of a qualitative exploration of barriers encountered during the implementation phase of the WARFDOCS project. Most of the data for this analysis were collected through informal discussions with administrators, pharmacists, and physicians at the participating hospitals between the spring of 2002 and the fall of 2003. The study staff maintained detailed logs and field notes, which were reviewed during weekly project meetings. In addition, one investigator (RLK) conducted telephone interviews in March of 2004 with eight pharmacists from five of the participating hospitals. The interviewer used a set of guiding questions and probes to ensure consistency across the interviews. The guiding questions were open-ended and designed to elicit the pharmacists' experiences with and assessments of the new technology and its impact on their work as well as the effects of the research project (and its related tasks) on practice. The interviews averaged 30 minutes (range: 2040 minutes) in length and were audiotape recorded and transcribed for analysis.

Themes were identified through an iterative process of transcript review. A subset of team members each reviewed the interview transcripts and noted the predominant and recurrent themes in the pharmacists' definitions of and perspectives on the ways in which the new technology changed the nature of work and work relationships. This included the technology's effect on working relationships and the structure and flow of work. The project team members then held collective discussions on emergent themes and patterns and how they might be categorized. They resolved disagreements regarding categorization through discussion, and by noting differences in opinion among team members, and through reviews of positive and negative examples in the data. Members developed consensus opinions regarding salient categories for coding based on the data and its relevance to the pharmacists' use of PDAs for warfarin dosing.

Results and discussion

The nature of agreement (or discovering the meaning of "yes")

During the development of the grant application for the WARFDOCS trial, the principal investigator (RHW) received assistance from the California Institute for Health Systems Performance (CIHSP) (www.cihsp.org) in identifying potential evaluation sites across the Sacramento area. We then contacted hospital administrators, who in turn referred us to pharmacy managers. As noted, nine hospitals initially indicated their support for the study (Table 1).

Making the sale: a matter of timing

Nearly a year transpired between the submission of the WARFDOCS research application and the receipt of funding. Additional time elapsed as technical consultants labored to transfer the Bayesian predictor program from a PC to a PDA platform. Once the tool was ready, project staff contacted hospitals to organize meetings with top management and the chief hospital pharmacists. The hospital administrators gave these meetings a relatively low priority, so they took several months to schedule.

In the time frame between negotiation and implementation, some of the personnel who had committed to the study either left their respective hospitals or were replaced. A consulting firm was hired by one of the participating hospitals to create a more cost-effective pharmacy service. The resulting reorganization left WARFDOCS with little internal leadership support, and the hospital withdrew from the study. At other sites, participants had to be re-acquainted with the goals and methods of the project before the study could be initiated. Simultaneously, a regional shortage of pharmacists increased the work demands at nearly all the sites, making pharmacists less inclined to cooperate.

Biting-off versus chewing

Orientation meetings enabled the study staff to describe the aims of the study to the selected pharmacists and pharmacy managers, while introducing them to the PDA technology. While not everyone who might be affected by the project was part of these discussions, the initial participant "buy-in" was broad and seemingly enthusiastic. The focus of the early meetings was the best clinical use of the technology, and the pharmacists and administrators seemed to recognize its potential. But as the discussions shifted to the details of implementation and those aspects of the project necessary to properly evaluate the tool, the institutional commitments began to falter.

Hospitals H and I withdrew their support once the logistical demands of the project became evident. Perceived staffing problems at these hospitalsresulting from the regional and national pharmacist shortage ²³ were so severe that pharmacy managers decided their staffs would not have the time to participate, despite their expressed interest in warfarin dosing and patient safety. Offers of financial compensation (for lost time) and "in kind" support (to facilitate the project logistics) were to no avail.

Pharmacy managers at two other previously committed hospitals concluded that the WARFDOCS trial would require additional personnel. Project funds were made available, but union rules and other administrative barriers precluded the hiring of additional hospital staff. Hospital policies also prohibited study staff from having contact with patients in non-University of California-affiliated hospitals. The regional crisis involving the supply of pharmacists and pressures on administrators to implement other major programs substantially degraded the priority assigned to WARFDOCS. Every step forward thus depended upon research project staff. Pharmacy administrators, swamped by competing demands, were unavailable to help solve study-related problems from the inside. The project staff grew more and more creative in their plans for making the project more attractive to hospital sites.

Details of the protocol implementation and specific in-kind support for research tasks had to be renegotiated at several sites to re-secure cooperation agreements. It was not that the clinical sites reneged on their commitments. Rather, details presumed to be trivial or irrelevant were left unspecified by the parties to the original agreements, and administrators and staff members later charged with carrying out the protocols became reluctant or expressed an inability to provide the expected level of service.

Latent stakeholders

The WARFDOCS project garnered much support from the senior leadership of the hospitals we approached, but attempts to secure commitments from the more shadowy networks of "latent stakeholders" were less successful. This group included pharmacy managers and assistant managers faced with increasing time and staffing pressures; line pharmacists uncomfortable with the PDA technology, the computerized decision support system, or the need to interact more directly with physicians; and, in the case of one hospital, physicians with no historical precedent for collaboration with the pharmacy service.

Barriers to adoptability: pharmacists' perceptions

Although this paper focuses on factors affecting the implementation and generalizability of the WARFDOCS evaluation, it is impossible to ignore the reaction of the trial participants to the WARFDOCS CDS system itself. The ultimate safety benefit delivered by a CDS system depends upon not only its capacity to shape safer care, but also the extent to which the system is used. CDS systems disuse has been a longstanding concern. ^{24, 25} We found that users' perceptions of the WARFDOCS CDS system's value revolved around benefits, burdens, and time demands. Surprisingly, the changing role relationships of pharmacists and physicians were not a major issue for the users.

Benefits and burdens

Enthusiasm for the WARFDOCS CDS system itself (as opposed to enthusiasm for the project) appeared to reflect a careful calculus of perceived benefits and burdens, which varied by pharmacist experience and the prevailing social norms governing interactions between pharmacists and physicians. The potential benefits of a computerized warfarin dosing assistance program were most clear to clinical pharmacists who had relatively little experience with warfarin and who wanted to assist the physicians with warfarin dosing responsibilities. They regarded the program as an enhancement, enabling them to deliver better, safer care, and perhaps to interact more effectively with the physicians.

First, in general... I don't think we were doing it (warfarin) consistently or systematically... at all. And I think... that's probably what attracted me to this. I think that patients on warfarinwhether it was cardiac, ortho, or whateverI think we were just being taken care of by the physicians, and if the patient looked like they were getting in trouble with their INRs, then the pharmacists would intervene and make a call and say, you know, 'I am not sure what's going on here but how can I help.' pharmacist at Hospital C

More experienced pharmacists anticipated fewer gains, and a few expressed doubts with regard to the trustworthiness of the computer-driven recommendations:

I think the program is helpful when you have a... fair number of data points in it. But initially... I trust my own instincts because I know there is a lot of stuff that does not go into the PDA that is in the chart and is in my head. (The software) doesn't know about changes in antibiotics... advanced age, and albumin, and I know that. There are a few things that are in there, but there are a lot of other things that don't go into the program. pharmacist at Hospital A

The perceived burdens of using the program varied according to the pharmacist's prior experience with PDAs, the frequency of current interactions with physicians and their offices involving warfarin, and practical workplace details such as the physical size of the hospital, the location of patients taking warfarin, and the pharmacist staffing and schedules.

I have been involved with the study pretty much since day one, and it's getting easier to use it now... I'm a little more adept at using it... but it still isn't something I would normally pick up if I didn't have to. pharmacist at Hospital A

Because I'm a rotating pharmacist, I don't have a certain shift. (T)here are about five pharmacists who are doing this (protocol), and each of us will probably do it for three times a week or maybe none at all for one week. It has to do with our scheduling and staffing problems. pharmacist at Hospital B

Pressures of time: how fast is fast enough?

WARFDOCS is a tool designed to improve the quality of clinical decisions related to warfarin dosing. The time it took to use the tool was a source of occasional consternation. Commenting on the time required to input data and the delay that occurs when WARDOCS is generating predictions, one experienced anti-coagulation pharmacist complained early on:

Well, to be brutally honest... it's become murderously slow to do my job. Sometimes when I'd have four or five intervention patients to process through the Palm Pilot, it would take close to an hour just to put in their data points and come up with predictions... If there were an open window on the 6^th Floor, that Palm Pilot would be out on the street right now. pharmacist at Hospital A

Efforts to expedite the data processing (through the use of a more powerful PDA) mitigated the problem substantially, but did not abolish it. Using the WARFDOCS system simply takes longer than making an off-the-cuff clinical decision. Since the potential benefits of WARFDOCS are spread across the entire population of warfarin patients, it is not surprising that pharmacists had a tendency to accentuate the (known) burdens while discounting the (potential) benefits.

Interactions with physicians

Hospitals that participated in the study varied with respect to the norms and standards governing interactions between pharmacists and physicians involving warfarin. In general, physicians appeared to welcome increased pharmacy involvement.

I have to say that most of the time they (the physicians) are more than happy not to dose Coumadin... more than happy to have us do it rather than do it themselves. pharmacist at Hospital B

The impact of WARFDOCS on pharmacist-physician relations varied widely. For example, pharmacists at Hospital A have been involved in warfarin dosing since 1992. The addition of a handheld decision aide at this hospital had little impact on established patterns of interaction.

I don't think it changed interaction with physicians at all. You know, if we don't think the dose they are ordering is appropriate, we'll call them and change it, or we'll be even more proactive and recommend a dose... prior to them even writing it... We do that everyday anyway. pharmacist at Hospital A

At other hospitals, however, the expectation that pharmacists would communicate regularly with ordering physicians represented a pronounced departure from established patterns. Although challenging, such interactions could have an upside:

Before, we weren't even talking about dosing at all with the doctors... Before, all we would do is enter the warfarin order... I personally would rather recommend a dose than enter the order... It is a little bit more interesting. pharmacist at Hospital B

In this sense, WARFDOCS not only demanded that pharmacists learn new technical skills, it also expanded their professional role in new, and sometimes rewarding, ways.

The added burdens of research

The evaluation of clinical informatics tools is essential to the field's progress, yet evaluation tasks make demands on participants that are not integral to the CDS system itself. If and when WARFDOCS is integrated into clinical practice, future clinician-users will be expected to review patients' medical records to identify current medications, medical conditions, warfarin doses, and INR values. They also will be required to generate a dose prediction and make a subsequent decision or recommendation. Outside of the research setting, however, future users will not need to determine eligibility for the study, obtain consent from patients and/or families, or enter detailed information on demographics and medications. Furthermore, they will not interact regularly with the 50 percent of patients in the control arm of the study. In agreeing to participate in the study, the pharmacists could not have meaningfully anticipated the roughly 50 percent of the total workload that would be devoted to research overhead, rather than clinical care. Obtaining informed consent from patients was viewed as particularly taxing.

Well, the other main stumbling block was the process for consent. It was something that pharmacists hadn't done in the past... and having them take on extra things they're not used to doing, was kind of a hard sell. pharmacist at Hospital C

These observations form the outline of a paradox. CDS systems need to be evaluated. An important part of any meaningful evaluation is whether the system will be used. And yet the more carefully the system is evaluated, the greater the burden of initial adoption, and the less likely potential users are to embrace the technology itself. Moreover, attempts to solve the problem by embedding research personnel on site to handle logistical issues (as we did at select hospitals) can in itself be problematic, as the presence of outsiders may alter the way in which the system is used.

Limits to generalizability

At the time of this writing, we do not know if WARFDOCS is having a beneficial effect on warfarin-related care and patient outcomes at the participating hospitals. That conclusion awaits additional data collection and analysis. But let us assume that WARFDOCS is making care safer, on average, within the participating hospitals. In a moment of unrestrained optimism, let us even assume that WARFDOCS is associated with a decrease in the rate of major hemorrhage, and that the associated P-value is less than 0.05. Finally, let us assumebased on the randomized design, completeness of follow-up, and the blinded analysisthat the results can be trusted (i.e., that internal validity is assured). But what about external validity? To what extent can the results be generalized to other patient care settings?

We approached this question by reviewing how WARFDOCS was implemented at the five hospitals that actually enrolled patients (Table 2). We found notable variations between hospitals along three dimensions: 1) the pharmacists' prior involvement with warfarin care, 2) the degree of clinical interaction between pharmacists and physicians, and 3) workflow and logistical issues affecting implementation.

Table 2. Between-hospital variations affecting generalizability

Hospital	Prior inpatient pharmacist involvement with warfarin-related care	Prior pharmacist-physician clinical interactions	Workflow and logistical factors affecting implementation
A	Extensive	Well-established	Disruption minimized by assigning WARFDOCS duties to the "warfarin pharmacist" of the day
B	Minimal	Minimal in the inpatient setting (a pharmacy-run warfarin clinic handles about 50% of outpatients)	Pharmacists "assigned to station" and cannot leave without coverage
C	Moderate; pharmacists review INRs and potential drug interactions	Minimal	Pharmacists work 34 shifts per week, necessitating frequent handoffs of WARFDOCS responsibilities
D	Moderate; hospital computer system allows pharmacist to follow warfarin doses and subsequent INRs	Minimal	Substantial physical distance between pharmacy computer (source of dosing information) and nursing station (source of laboratory data)
E	Moderate; extensive in six months prior to launch of WARFDOCS	Moderate; warfarin declared a "quality improvement focus" by parent hospital chain in 2003, leading to "warfarin per pharmacy" in about half of patients	PDAs already in wide use (e.g., for assessing drug interactions), but this created own problems; pharmacists complained about multiple PDAs

Informatics tools are subject to substantial center or implementation effects, and these effects are not easy to explain or account for. They affect the quality of care by changing the processing and flow of informationin short, by changing the content and patterns of communication. Looked at in this way, the WARFDOCS intervention is not a single clinical trial but rather five related trialsone for each hospital. In hospitals like A and E, WARFDOCS is principally a computerized dosing guide that imposes additional rigor on an existing communication process between pharmacists and physicians. In the other hospitals, WARFDOCS provides new information and encourages a whole new mode of interaction between physicians and pharmacists. There is, therefore, no reason to expect that the effect of WARFDOCS will be homogeneous across all participating centers. Heterogeneity of effects is not unusual even for biomedical treatments. In one notable example, the Beta-Blocker Heart Attack Trial included 21 dominant centers (i.e., the effect of beta-blockers was the same as for the overall study) and 10 divergent centers (i.e., the effect of beta blockers was deleterious). ²⁶ Recognition of so-called center effects creates an imperative for reporting not only the estimated average effect of the treatment, but the estimated treatment effect dispersion as well. ²⁷ One approach is to fit hierarchical regression models that incorporate centers (sites) as random effects. Studies in which large "site effects" are anticipated should be appropriately powered to test for site-treatment interactions and for patient clustering within the sites. ²⁸

We suspect that WARFDOCS is not the sole clinical decision support system for which these observations are true. What may appear to be "random error" in the analysis of evaluation data may, in fact, be logically related to variations in the organizational context that are influenced or made manifest as a direct result of the CDS system implementation. The implementation of even relatively simple CDS systems, like WARFDOCS, requires the use of creative efforts to overcome barriers unique to each site. These malleable contextual dimensions, including clinical, social, and organizational factors, may be what makes or breaks a project at any given site. In light of this, it may be that the "quality of implementation" or even the "goodness of fit" between the site and the tool rise to the level of variables to be analyzed rather than nuisances to be randomized away in the search for an overall "generalizable" finding.

Conclusions

Our experience to date with the WARFDOCS project suggests four principal conclusions.

• Achieving participatory "buy-in" for a CDS system evaluation is an active process that requires investments of time, energy, and resources well beyond those typically budgeted for such purposes. Unless there is a miraculous alignment of organizational priorities and study objectives (such as that we were fortunate enough to experience with Hospital E), investigators and funders should be realistic about the need to actively manage organizational relationships and budget the work accordingly.
• From an end user's perspective, the acceptability of a CDS system reflects a calculus of benefits and burdens. Resistance to the system can be overcome by making the users' jobs more interesting or satisfying, minimizing disruptions in workflow, and perhaps most importantly, appealing to their professional commitment to patient safety. The most relevant appeal would involve a clear demonstration of the patient benefit, which cannot be accomplished until the end of the trial.
• A CDS system implementation may be greatly complicated by a formal system evaluation, especially if the evaluation is a randomized controlled trial. In the context of such a trial, there is no obvious way around the additional burdens imposed by the need to obtain consent, collect data, and involve control patients (who must complete all the study procedures, but will reap no benefit). These procedures are necessary but may themselves influence a key CDS system evaluation outcome: user uptake. There is no ready solution other than to describe the research proceduresand their perceived impact on end usersas completely as possible, so the results of the evaluation may be assessed in the proper context.
• In the evaluation of CDS systems, strong "center effects," manifested as qualitative interactions (i.e., positive effects at some sites and negative effects at others) should be expected. This is because informatics technologies always are embedded in a clinical and social context that can vary substantially from site to site. Aside from raising philosophical questions about the results of CDS systems trials and whether they are ever "generalizable," this observation underscores the importance of using a mixture of methods (i.e., quantitative and qualitative) to characterize potentially important contextual factors as thoroughly as possible.

In summary, when implementing tools to improve patient care, the experience of the journey may be as important as the ultimate destination.

Acknowledgments

Funding for the project was provided by the Agency for Healthcare Quality and Research (Grant No. R18HS11804).

Author affiliations

University of California, Davis, Center for Health Services Research in Primary Care, Sacramento, CA (RLK, JDN, DP). University of California, Davis, Department of Pathology and General Medicine, Sacramento, CA (MAH). University of California, Davis Health System, Pharmacy, Sacramento, CA (WED). University of California, Davis, Department of General Medicine, Sacramento, CA (RHW).

Address correspondence to: Richard L. Kravitz, MD, MSPH; University of California, Davis; Center for Health Services Research in Primary Care; 2103 Stockton Boulevard, Suite 2226; Sacramento, CA 95817. Phone: 916-734-1248; e-mail: rlkravitz@ucdavis.edu.

References

1. Wyatt J and Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. Proceedings of the 15th Symposium on Computer Applications in Medical Care; 1991; Washington, D.C. New York: McGraw Hill, Inc.; 1991: pp. 3 7.

2. Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003. 348(25):2526-34. (PubMed)

3. MedExpert www: a medical knowledge base server. Last updated October 28, 2004. Available at: http://medexpert.imc.akh-wien.ac.at/ . (Accessed Feb 9, 2005).

4. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97. BMJ 2001. 322(7281):279-82. (PubMed) (Full Text in PMC)

5. Morris A. Decision support and safety of clinical environments. Qual Saf Health Care 2002. 11:69-75. (PubMed) (Full Text in PMC)

6. Ash J. Organizational factors that influence information technology diffusion in academic health sciences centers. J Am Med Inform Assoc 1997. 4(2):102-11. (PubMed) (Full Text in PMC)

7. Lorenzi N, Riley R. Managing change: an overview. J Am Med Inform Assoc 2000. 7(2):116-24. (PubMed) (Full Text in PMC)

8. White R, Beyth R, Zhou H, et al. Major bleeding after hospitalization for deep-venous thrombosis. Am J Med 1999. 107(5):414-24. (PubMed)

9. Reardon M, Burns B, Brewer B, et al. Deaths associated with warfarin in elderly patients. Br J Clin Pract 1995. 49(6):322-3. (PubMed)

10. Meisel S, Sershon L, White D. Reducing adverse drug events and medication errors using rapid cycle improvement. Qual Manag Health Care 1998. 6(4):15-28. (PubMed)

11. McMahan D, Smith D, Carey M, et al. Risk of major hemorrhage for outpatients treated with warfarin. J Gen Intern Med 1998. 13(5):311-6. (PubMed) (Full Text in PMC)

12. Landefeld C, Beyth R. Anticoagulant-related bleeding: clinical epidemiology, prediction, and prevention. Am J Med 1993. 95(3):315-28. (PubMed)

13. Landefeld C, Anderson P. Guideline-based consultation to prevent anticoagulant-related bleeding. A randomized, controlled trial in a teaching hospital. Ann Intern Med 1992. 116(10):829-37. (PubMed)

14. Schuringa P, Yen D. Home prophylactic warfarin anticoagulation program after hip and knee arthroplasty. Can J Surg 1999. 42(5):360-2. (PubMed)

15. Kelly J, Sweigard K, Shields K, et al. Safety, effectiveness, and efficiency: a Web-based virtual anticoagulation clinic. Jt Comm J Qual Saf 2003. 29(12):646-51. (PubMed)

16. Anderson DR, Wilson SJ, Blundell J, et al. Comparison of a nomogram and physician-adjusted dosage of warfarin for prophylaxis against deep-vein thrombosis after arthroplasty. J Bone Joint Surg Am 2002. 84-A(11):1992-7. (PubMed)

17. White R, Dager W. Randomized trial of warfarin nomograms. Ann Intern Med 2004. 140:488-9.

18. Dager WE, Branch JM, King JH, et al. Optimization of inpatient warfarin therapy: impact of daily consultation by a pharmacist-managed anticoagulation service. Ann Pharmacother 2000. 34(5):567-72. (PubMed)

19. Chatellier G, Colombet I, Degoulet P. Computer-adjusted dosage of anticoagulant therapy improves the quality of anticoagulation. Medinfo 1998. 9(2):819-23. (PubMed)

20. Svec JM, Coleman RW, Mungall DR, et al. Bayesian pharmacokinetic/pharmacodynamic forecasting of prothrombin response to warfarin therapy: preliminary evaluation. Ther Drug Monit 1985. 7(2):174-80. (PubMed)

21. Fisher R and Ury W. Getting to yes: negotiating agreement without giving in. New York: Penguin Books; 1983.

22. Mungall DR, Oertel LB. Software applications in anticoagulation. In: Ansell J, Oertel LB, Wittkowsky A, editors. Managing oral anticoagulation. Gaithersburg, MD: Aspen Publishers; 1997.

23. Cooksey J, Knapp K, Walton S, et al. Challenges to the pharmacist profession from escalating pharmaceutical demand. Health Aff (Millwood) 2002. 21(5):182-8.

24. Kaplan B. Evaluating informatics applicationsclinical decision support systems literature review. Int J Med Inf 2001. 64(1):15-37.

25. Kaplan B. Evaluating informatics applicationssome alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf 2001. 64(1):39-56.

26. Horwitz RI, Singer BH, Makuch RW, et al. Can treatment that is helpful on average be harmful to some patients? A study of the conflicting information needs of clinical inquiry and drug regulation. J Clin Epidemiol 1996. 49(4):395-400. (PubMed)

27. Longford N. Selection bias and treatment heterogeneity in clinical trials. Stat Med 1999. 18(12):1467-74. (PubMed)

28. Brookes S, Whitely E, Egger M, et al. Subgroup analyses in randomized trials: risks of subgroup-specific analyses; power and sample size for the interaction test. J Clin Epidemiol 2004. 57(3):229-36. (PubMed)