Advances in Patient Safety: From Research to Implementation Volume 3. Implementation IssuesDecision Support System Design and Implementation for Outpatient Prescribing: The Safety in Prescribing Study
Adrianne C. Feldstein, David H. Smith, Nan R. Robertson, Christine A. Kovach, Stephen B. Soumerai, Steven R. Simon, Dean F. Sittig, Daniel S. Laferriere, Mara Kalter
AbstractBackground: Decision support (i.e., alerts and reminders) at the time of medication prescribing has been shown to be an effective method for reducing potential medication errors for inpatients, but much less is known about the effects in the outpatient setting. Using qualitative methods to inform our work, medication safety decision support and provider education interventions were designed to improve the use of medications in ambulatory care. Methods: This paper presents the study rationale, design, development, and implementation of the interventions. We include a summary of the qualitative findings, including usability testing of the decision support, and we describe how the qualitative findings enhanced the intervention design. We also describe our approach to clinician recruitment. We enumerate limitations of the existing electronic medical record to accommodating recommendations from the qualitative work and usability testing. Results: Our qualitative interviews suggest that clinicians prefer decision support alerts that are clear, concise, and easy to navigate, with minimal information in the alert text. In usability testing, we found that decision support alerts are followed less often when they appear at inappropriate times in workflow, are difficult to read, add to time pressure, and are canceled before being fully read. Our deliberate approach to clinician recruitment for the educational sessions achieved an impressive attendance rate of 85 percent. Conclusions: This study shows that careful consideration of alert design and provider education is critical. Future work will examine the effectiveness of the decision support and whether our educational intervention improves physician response.
The Institute of Medicine (IOM) report on medical errors identified computerization of medication prescribing as an important patient safety strategy.
1
Decision support through alerts and reminders at the time of prescribing has been shown to be an effective method for reducing potential medication errors for inpatients,
2
4
but much less is known about the effects in the outpatient setting. One preliminary study
5
examined the effect of basic computerized prescribing on medication errors in outpatients and concluded that more advanced decision support was necessary. We were unable to identify any studies about outpatient decision support of significant size and scope, or any assessing whether an education program enhances the effectiveness of computerized physician order entry (CPOE) systems intended to promote medication safety. Whether or not clinician education can improve the effectiveness of alerting systems is an important question, because educational outreach, often called "academic detailing," has been shown to be perhaps the single most consistently successful intervention for improving prescribing.
6
Using qualitative methods to inform our work, we designed an intervention study of computerized decision support at the time of medication order entry and supporting clinician education to improve medication safety in ambulatory care. This paper presents the study design and the intervention rationale, development, and implementation for the Safety in Prescribing (SIP) efficacy study and Phase I effectiveness trial.
7
The objective of the SIP study is to develop patient-specific computerized decision support for providers and measure its effectiveness to reduce prescribing errors. The SIP study, begun in 2004, is ongoing; this paper presents results of the study design process.
Setting and subjects
The study is being conducted in a large nonprofit group-model health maintenance organization (HMO) in the Pacific Northwest (northwest Oregon and southwest Washington) that cares for approximately 450,000 members. The group practice and decision support intervention includes 669 physicians practicing in all the major specialties of medicine and surgery; they supervise 417 allied health practitioners (nurse practitioners and physician assistants) at 20 clinical practice sites. Fifteen primary care practices in adult medicine, employing 281 family practice and internal medicine practitioners, agreed to participate in the SIP study randomized trial. The protocol for the study was approved by the HMO's institutional review board.
The HMO has used the EpicCare![[reg]](https://webarchive.library.unt.edu/web/20090830213910im_/http://www.ncbi.nlm.nih.gov/books/bookres.fcgi/system_default/reg.gif)
(Epic Systems, Madison, WI) electronic medical record (EMR), which includes computerized physician medication and other order entry, for all outpatient contacts since 1996. The EMR provides online access to medical problem lists, visit diagnoses, procedures, patient demographics, and visit progress notes. The EpicCare product provides two types of decision support relevant to this project: (1) drug-specific alerts that appear when the target drug is prescribed at the time of CPOE, and (2) patient-specific alerts created using the Best Practice Alerts tool. The latter were tailored with additional programming, as described below, to generate the alerts for this study. Prescribers had experience with both types of alerts at the time of our qualitative work with them.
The administrative and clinical electronic databases contain information on inpatient admissions, pharmacy dispenses, outpatient visits, laboratory tests, and outside claims and referrals. All of these databases are linked through a unique health record number that each member receives upon enrollment in the health plan. Because the study site is a closed, group-model HMO, the databases capture close to 100 percent of all medical care and pharmacy services members receive. Rates of dispensing errors in the clinical target areas are available through data from the outpatient pharmacy dispensing system, and supporting demographic, diagnostic, and procedure information is available through the HMO's other databases.
Decision support intervention
In a recent report, Bobb et al.
10
found that of 1,111 prescribing errors identified in an inpatient setting (62.4 errors per 1,000 medication orders), 64.4 percent were rated as likely to be prevented with CPOE (including 43 percent of the potentially harmful errors). Bobb et al. also found that 13.2 percent of the errors were unlikely to have been prevented with CPOE, and 22.4 percent could possibly have been prevented with CPOE. Findings were dependent on specific CPOE system characteristics. The authors concluded that incorporating advanced clinical decision support within the order entry routine is vital for achieving maximum medication safety. In a recent systematic review, Kaushal et al.
11
found five trials assessing CPOE with clinical decision support that met their criteria. Of these studies, two demonstrated a marked decrease in the serious medication error rate, one showed an improvement in corollary orders (e.g., requesting serum antibiotic levels when antibiotics are ordered), another had an improvement in five prescribing behaviors (i.e., selection of recommended drugs within a class, use of recommended doses and frequencies, reduction of excessive doses, use of corollary orders, and compliance with drug use guidelines),
12
and one had an improvement in nephrotoxic drug dose and frequency. As noted previously, there are no comparable data for the outpatient setting.
We developed and implemented an educational outreach intervention, popularly called academic detailing, to increase prescriber acceptance of evidence-based alerts and reminders in a computerized order entry system. Such programs draw on adult learning theory and target physicians with clear and professionally illustrated educational materials from a credible organization, combined with one-on-one, face-to-face visits.
13
Although most early studies of academic detailing employed individual meetings between detailers (either physicians or pharmacists) and prescribers, more recently studies have demonstrated the capacity of group detailing to improve practice behavior. By using small groups of four to eight physicians while retaining all other tenets of academic detailing, researchers have improved prescribing for various classes of medications.
14
17
In one comparative study, group detailing was as effective as individual academic detailing in reducing inappropriate prescribing.
17
Clinical target areas and guidelines
This project is part of a larger prescribing safety study being conducted by the HMO Research Network, Center for Education and Research on Therapeutics (CERT). 18 Therefore, our project investigators participated in and had access to clinical guidelines for prescribing safety that were generated by national project expert teams.
Locally, we sought to target potential prescribing errors that were of significant frequency and severity and that were preventable. Working with groups of expert pharmacists and physicians, we reviewed available data regarding the frequency and severity of potential prescribing errors to help us select clinical areas to target.
During our structured interviews with prescribers, we also elicited their preferences. A committee of investigators and expert pharmacists and physicians elected to focus on medications that are generally contraindicated in the elderly, require dose adjustment for renal insufficiency, and are commonly implicated in drug interactions. We assembled pharmacist and physician teams in the three target areas to choose specific drugs or drug pairs for intervention and to refine alert text and tools.
Before the project could be taken to the physicians for support, four functional groups within the HMO
research, applied medical informatics, pharmacy, and information technology (IT)had to agree upon design, accountabilities, resource allocation, training, implementation, and evaluation. Key committees and decisionmakers in quality management, patient safety, and IT were enlisted during the proposal and development phases. Senior management was enlisted to convene the groups and facilitate discourse to arrive at an integrated approach and resource reassignment as needed.
A coordination and implementation oversight committee advised on all aspects of the project and provided IT support. Alert programming and text went through two levels of review and approval. The first review was by a standing committee of HMO physicians and IT experts that reviews all alerts and reminders; final approval was through the Regional Formulary and Therapeutics Committee. Interestingly, several key managers were hesitant to withhold the educational intervention from some clinics as part of the study design. We assured them that all materials would be available for delivery to the control group after the end of the study, should the education be found effective.
Formative work appears to be an important predictor of intervention success with practicing clinicians, 8 and little literature exists in this area. The methods we used to design the decision support and educational interventions, based on our formative work, have been presented in detail elsewhere. 19 We recruited 20 primary care prescribers from family practice and internal medicine, using e-mail and phone followup. Interviews were conducted during paid work time. We developed a semistructured interview guide with closed and open-ended questions to elicit prescribers' prior experience with alerts and education activities related to decision support. We also elicited their opinions about proposed medication safety alerts and their preferences about how to learn about new decision support functions.
Interviewees provided informed consent. Interviews were transcribed for analysis. We developed a coding dictionary and used Atlas.ti 4.2 (Scientific Software Development, 1997), a qualitative research software package, for analysis.
Discount usability testing can be used to evaluate computer applications according to how easy they are to learn and remember, and their efficiency, error rate, and user satisfaction. This is a useful step before alert implementation because although real-time medication prescribing alerts can improve clinical performance and patient safety, other investigators have identified flaws in alert logic. For example, in one study, 37 of 43 (86 percent) sympathomimetic-tricyclic antidepressant interaction alerts were unjustified on the basis of scientific evidence. 20 In another study, the data triggering alerts were incomplete. For 29 percent (31 of 108) of the critical drug interaction alerts, one of the two interacting drugs contained "TOP" or "oint" or "shampoo" in the prescription, indicating that the drug was to be given topically rather than taken by mouth or injected. 21
There may also be flaws in the alert display. For example, in a study that did not demonstrate an effect of computerized reminders, the computer system notified clinicians only by means of a banner at the bottom of the screen stating that "there are suggested orders for this patient." In a subsequent study that demonstrated a significant effect, the computer immediately displayed the reminders to the physicians as full, prewritten orders and highlighted the suggested reminders with a distinctive color scheme, disabled the "escape" key, and set the default to "order," allowing the physician to accept the item simply by pressing the "enter" key. 22
For the SIP study, all 11 primary care clinicians from one clinic assigned to the intervention were asked to participate in a 30-minute usability session in their office. All usability sessions were conducted by trained usability evaluators and audiotaped for later review and qualitative analysis. Evaluators read a script that provided each clinician user with an introduction to the study, obtained their consent, and described seven hypothetical cases, complete with the patient's age, gender, and a statement of clinical condition. The clinician was instructed to perform a particular action using the electronic medical record system while "thinking aloud"verbalizing his/her reasoning process while performing the activity.
23
Evaluators recorded the start time, each click made, associated comments, and the end time for the activity.
In developing the educational intervention for this study, we adhered closely to the key principles of academic detailing. 13 The barriers to changing prescribing behavior identified in our interviews became the principal targets of the educational program, while the facilitating factors frequently served as useful counter-arguments to issues and concerns raised by participants. We established credibility by working through respected organizational sponsors and referencing authoritative and unbiased sources of information. We collected evidence from the peer-reviewed published literature as well as data from the HMO to demonstrate the clinical implications of prescribing alerts. The educational sessions were intentionally designed to be group discussions, rather than lectures or presentations.
Our qualitative results suggested that physicians preferred to receive information about prescribing safety from internal physician experts. In response, we recruited two physicians, both internists, who were well known and respected within the HMO for their clinical leadership skills and their prior experience as educators. They were provided with 2 hours of training. The educator training focused on techniques for eliciting barriers to complying with the alerts within the clinical decision support system. We provided the educators with a list of talking points that could be interspersed in the discussion, and we prepared them to "inoculate" the group by proactively addressing issues and concerns that we anticipated would arise. We equipped the educators with a list of arguments to use through the discussion. The arguments were derived chiefly from the interviews; counter-arguments were derived from our own research group's experience and the literature.
An education handout was created to accompany the academic detailing session. These materials emphasized the graphical and tabular presentation of information and incorporated color and easy-to-read typefaces. This material was also posted on a Web site, clearly identified on the organizational homepage, called "Safety in Prescribing." 1
The educators presented a practice run of the presentation to the co-investigator group, and piloted the educational session with the staff at a single clinic. A focus group followed. The education was generally well received during this practice run, and we refined several areas based on the feedback.
We recruited primary care clinicians to attend the educational sessions with an e-mail invitation from clinic physician leaders (primary care participants). A local clinician staffing clerk arranged time out of clinic for the meeting and offered alternate sessions for clinicians not able to attend the main scheduled session.
The handout is available from the corresponding author.
The primary outcome measure for SIP is the incidence rate of the targeted prescribing errors. For medications generally contraindicated in the elderly, the outcome measure is the number of prescriptions of the contraindicated medications, divided by the sum of the number of prescriptions of the contraindicated medication plus the number of prescriptions of the preferred medication alternatives (specific agents to vary for each medication targeted).
For dose adjustment in renal insufficiency, the outcome measure for patients with study-defined renal insufficiency is the number of prescriptions without dose adjustment divided by the total number of prescriptions of the targeted medications.
For drug interactions, the outcome measure is the number of the targeted contraindicated coprescribing events, divided by the sum of the number of the contraindicated coprescribing events plus the desired coprescribing alternatives.
Implementation and outcome measures
Fifteen primary care group practices are participating in the randomized portion of the trial. To ensure baseline comparability, we determined the 15 practices' baseline prescribing rate for several medications generally contraindicated in the elderly, including tertiary tricyclic antidepressants and long-acting benzodiazepines. The practice sites were then matched in pairs according to baseline prescribing rates, and each pair of clinics was randomized. One clinic of each pair was assigned to receive only the computerized decision support tool, and the other was assigned to receive the decision support tool plus clinician academic detailing.
For medications generally contraindicated in the elderly, we chose to target oral agents that had significant prescribing frequencies in our population between January 1, 2000, and June 30, 2001, and that were addressed through the Beers Criteria.
24
We selected long-acting benzodiazepines, tertiary tricyclic antidepressants, skeletal muscle relaxants, anti-inflammatory agents, and propoxyphene. For dose adjustment in renal insufficiency, patients with chronic kidney disease
25
were those with two measures of glomerular filtration rate (GFR) of 60 ml/min or less,
26
separated by at least 90 days. The team then selected drugs to target for intervention that were frequently not adjusted in this population and for which clear dosing guidelines were available. Allopurinol, ciprofloxacin, colchicine, trimethoprim/Sulfamethoxasole, gabapentin, metformin, nitrofurantoin, and probenecid were the final drugs selected for targeting. A similar approach was taken for drug interactions. We identified coprescribing that was thought to be clinically significant and of substantial frequency based upon 2001-2002 data. We targeted the following medication interactions: Warfarin and acetaminophen, or trimethoprim/sulfamethoxazole, fluconazole, metronidazole, nonsteroidal anti-inflammatory agents, Nitroglycerin and sildenafil, Statins and macrolide antibiotics and Theophylline and ciprofloxacin. Alternative medication recommendations and dosing guidelines were part of the educational handout.
These results have been described in detail elsewhere. 19 In brief, we interviewed primary care prescribers from 12 of the 15 primary care clinics included in the intervention. We found that patient safety-related alerts were seen as more helpful than more routine health maintenance alerts. Alerts that appeared in an inappropriate place in the workflow were subject to override, whereas alerts during medication prescribing were generally viewed as more helpful.
Prescribers particularly were enthusiastic about alerts related to drug interactions, appropriate medication dosing, and patient allergies. Prescribers we interviewed stated that it was important to keep alerts concise and clear, and they requested easy access to tools or information to allow them to resolve the clinical issue that led to the alert.
Prescribers preferred short, concise information in the alert text but also wanted a way to access more supporting evidence at a later time. They asked that safety alerts be clearly identified as such and not subject to clinician override.
Prescribers preferred small group educational sessions with supplementary written material and wanted education to be clear, succinct, and focused on clinical case studies. They preferred educators who were local physicians with expertise in the clinical issues and use of the electronic medical record.
During the discount usability testing, evaluators presented seven hypothetical cases. Five clinicians completed usability tests of all seven cases. All five clinicians took appropriate action when the alerts occurred immediately following the entry of an order for an alert-triggering medication. Four clinicians had trouble comprehending the warnings they received when they had not initiated an order for a medication (e.g., an alert appeared when a medical record was opened and targeted interacting medications were on the medication list).
Clinicians had difficulty reading the alert message text due to the length of text lines, use of upper and lower case letters, minimal contrast between the black text of letters and the gray background, and perceived time pressure during the simulated cases. They also tended to cancel the alert before reading it completely. The lack of consistency in the appearance of the messages limited the ability of users to transfer learning from one alert type to another.
We revised the alerts based on these findings, although we were constrained by limitations of the system. For example, the version of the software we used did not allow changes to color and limited when the alert could appear. To address clinician tendency to click though the alerts, we incorporated training and messages addressing this behavior into our academic detailing sessions.
A sample alert is presented in Figure 1. The first screen shot is the alert before usability testing, and the second is the alert after usability testing.
Several technological limitations, including alert formatting and workflow issues, prevented us from incorporating all the suggestions from our focus and usability testing. We were limited to the "built in" functionality provided by the EMR vendor.
Thus, we focused the intervention for reducing the use of inappropriate medications in the elderly on alerting for new prescribing of these agents. A program ran nightly to find plan members eligible for the alertthose over age 65 and without a previous prescription for the relevant medication. Their electronic medical record triggered an alert if the relevant drug was newly prescribed.
A similar approach was taken for renal dosing. Patients who had diminished GFR, meeting the study-defined criteria, received a flag on their record; this update occurred nightly. In the event that a medication requiring dose or drug modification was prescribed, an alert would present the best dosing recommendations for the given GFR range.
The drug interaction alerts did not depend upon prior identification of eligible patients. A real-time alert was triggered if a drug was prescribed for patients who had an interacting drug already on their medication list. These alerts were also triggered when a patient's record was opened, if two interacting drugs were on the patient's medication list.
Figure 2 presents a sample final alert in each of the clinical target areas. The alerts incorporate much of the feedback we heard from clinicians. All the safety alerts (1) are clearly identified as a safety alert, (2) provide a succinct description of the clinical issue or risk, (3) produce a clearly labeled recommended clinical action including a suggestion to check the medication list (for alerts that appear outside of medication order entry), and (4) offer a description of any tool available to assist in making a prescribing decision.
EpicCare EMR provides a selection list of alternatives during clinician order entry (Figure 3). For example, if the clinician types in "elderly," he or she is presented with a series of diagnostic options for specific treatment recommendations. In the case displayed inFigure 3, "elderly anxiety" is chosen, and the clinician is presented with a series of alternative medications with suggested dose and frequency of dose information.
The findings from our qualitative work, including piloting, led to a number of changes in the education intervention. In particular, we minimized the background scientific information supporting the evidence for the gap between medication safety and use of alerting systems to improve safety. Instead, the presentation focused on a walk-through with clinical case examples of the alerts and how to satisfy them.
Information was presented in a clinical "how-to" format using case examples and actual screen shots of alerts and tools to satisfy them. We trained the detailers to emphasize the key points of the intervention effort and to return frequently to these key points throughout the group discussion sessions. By identifying these key messages at the outset, returning to them during the session, and closing with them, detailers were able to highlight the specific goals of the intervention effort. Besides repeating important messages, detailers positively reinforced a small number of desired behaviors during the sessions. Session length was kept to a maximum of 40 minutes. The sessions were scheduled around the lunch break, lunch was provided, and the participants were given about 30 minutes of paid work time. At the detailing sessions, participants were given a handout describing the alerts and the EMR tools available, as well as a coffee mug and reminder note pads with a study logo.
The final outline of the educational session is available from the corresponding author. Clinician attendance by clinic was high and averaged 85 percent (range of 82100 percent). On a Likert rating scale ranging from 5 = excellent to 1 = poor, the average quality score for the program was 4.7.
 Brief introduction to medical errors and SIP project. |
|
Background and rationale for alerts at medication order entry. |
|
Educational objectives of program. |
|
Local data supporting improvement opportunity in SIP targeted prescribing areas. |
|
Case presentations for medications in the elderly, drug interactions, dose adjustment in renal insufficiency. |
|
Sample alert screen shots for each case and pathways to satisfy alert. |
|
Common features of all safety alerts. |
|
How to avoid false alerts. |
|
Alert limitations. |
|
Final message: "Take the time to make the safe choice." |
The decision support interventions have been well received. Concerns, expressed through the electronic feedback tool routinely used within the electronic medical record, have included questions regarding data accuracy, suggestions for improved alert design, requests to expand alternatives offered, and complaints that alerts fired for specialty physicians when the primary care doctor should address the warning. However, positive anecdotal feedback from physicians has stated that alerts seem relevant and helpful.
The maintenance of the alerts has been successfully transferred to the existing departments and committees that maintain the clinical content and monitor the alerts. A subset of the alerts has been adopted for use in the national electronic medical record application.
This paper describes our experience with developing and implementing a decision support intervention at the time of computerized medication order entry, along with an education program to support the implementation. It serves as an important case study for translating research into practice by describing the order in which the work and insights unfolded. We developed a successful process to come to clinical consensus regarding targeting clinical areas and clinical guidelines. Discount usability testing served as an effective means of evaluating our clinical alerts. We also identified several issues that might be important and common to other organizations.
The results of our qualitative work significantly changed our planned intervention. In particular, our findings led us to focus on programming and timing. We kept the alert text to the minimum and as clear as possible. We were limited in our ability to respond to all of the identified issues. Future work in this area would benefit from a method of prioritizing concerns in order to address the most important more quickly.
Our experience provides several important lessons for other organizations. Because of the high risk of alert overrides, qualitative work to understand the needs of users is essential. We found that small differences in alert text could significantly improve the understandability, and possibly acceptability, of alerts. We also found that multiple iterations were required to get to a final product, and technological limitations limited our ability to meet all user requests, so understanding user priorities is critical.
We experienced excellent participation in both the qualitative work and the education sessions. Our attendance rate of 85 percent is on the high end compared to other studies or physician academic detailing, where attendance has ranged from 60 to 92 percent. 27
We believe that several program design attributes contributed to this success. First, this program was part of the HMO's quality improvement program and was sponsored by the HMO's safety committees. Although attendance was not explicitly required, implicitly, attendance was expected. It was also helpful that the intervention was delivered during a period when patient safety work was prioritized and highly visible. During our study, for example, safety training, including root-cause analysis, was being provided to clinical teams, and the quality medical director routinely conducted safety walk-throughs in the inpatient setting.
Responding to clinician requests around scheduling also likely affected our success. Providing some paid work time for participation, providing lunch, and adding education to existing meetings at the local site were important program components.
Health care manager discomfort with random allocation of an intervention likely will be found at other health care institutions as well. This issue can be addressed through gentle and repetitive efforts to convey key messages regarding research design and research ethics. A randomized trial can be viewed and explained as staggered implementation of an intervention. We reinforced that, should the intervention (education) be found to be effective, it could be given to the other half of the clinical group. Often organizations have insufficient resources to provide programs to all staff immediately, and so this concept can be useful in meeting the needs of all stakeholders.
We successfully transferred the maintenance of the alerts to the local health plan during the project period. We believe that the key elements of this success were engaging key managers in the research, starting at the proposal phase and continuing their involvement through the design, implementation, and handoff phases.
The study has several limitations. We believe that the findings will be useful to other health care organizations that are planning decision support to improve medication safety. However, clinicians in different practice settings may have different needs. Specifics will vary depending upon the type of EMR utilized and the physician culture and organizational structure. However, we believe that many of our findings are generalizable.
We conclude that decision support for improved medication safety appears to be a priority for clinicians. However, careful consideration of alert design and supporting provider education is critical to participation and effectiveness.
This study was supported through grant #1 U18 HS11843 from the Agency for Healthcare Research and Quality. We would like to thank the many individuals who assisted with the design of the intervention and who were critical organizational advocates for the project: Dr. Michael Krall; Dr. Homer Chin; Dr. Ron Potts; Dr. Sean Jones; Dr. Maureen Wright; Nancy Louie-Lee, RPh; Lisa Shook; Norman Muilenburg, RPh; Diane Ditmer, RPh; Barbara Nye, RPh; and Emily Thomas, RPh. We would also like to thank Dr. Nancy Perrin for statistical support, Martha Swain for editorial assistance, and Debra Burch for administrative support.
Kaiser Permanente Center for Health Research (ACF, DHS, DFS, DSL, MK). Kaiser Permanente Northwest (NRR, CAK). Harvard Medical School and Harvard Pilgrim Health Care (SBS, SRS).
Address correspondence to: Adrianne C. Feldstein, M.D., M.S., Kaiser Permanente Center for Health Research, 3800 N Interstate Ave., Portland, OR 97227; phone: 503-335-6758; e-mail: Adrianne.C.Feldstein@kpchr.org.
1. Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000.
2. Bates DW, Kuperman G, Teich JM. Computerized physician order entry and quality of care. Qual Manag Health Care 1994. 2(4):18-27. (PubMed)
3. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998. 280(15):1311-6. (PubMed)
4. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999. 6(4):313-21. (PubMed) (Full Text in PMC)
5. Gandhi TK, Weingart SN, Seger AC, et al. Impact of basic computerized prescribing on outpatient medication errors and adverse drug events. J Am Med Inform Assoc 2002. 9(6 Suppl):S48-S49.
6. Thomson O'Brien MA, Oxman AD, Harvey EL. Outreach visits to improve health professional practice and health care outcomes. Cochran Library. Oxford: Update Software; 1998.
7. Garfield SA, Malozowski S, Chin MH, et al. Considerations for diabetes translational research in real-world settings. Diabetes Care 2003. 26(9):2670-4. (PubMed)
8. Davis DA, Thomson MA, Oxman AD, et al. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995. 274(9):700-5. (PubMed)
9. Gillings D, Makuc D, Siegel E. Analysis of interrupted time series mortality trends: an example to evaluate regionalized perinatal care. Am J Public Health 1981. 71(1):38-46. (PubMed) (Full Text in PMC)
10. Bobb A, Gleason K, Husch M, et al. The epidemiology of prescribing errors: the potential impact of computerized prescriber order entry. Arch Intern Med 2004. 164(7):785-92. (PubMed)
11. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003. 163(12):1409-16. (PubMed)
12. Teich JM, Merchia PR, Schmiz JL, et al. Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000. 160(18):2741-7. (PubMed)
13. Soumerai SB, Avorn J. Principles of educational outreach (academic detailing) to improve clinical decision making. JAMA 1990. 263(4):549-56. (PubMed)
14. Diwan VK, Wahlstrom R, Tomson G, et al. Effects of "group detailing" on the prescribing of lipid-lowering drugs: a randomized controlled trial in Swedish primary care. J Clin Epidemiol 1995. 48(5):705-11. (PubMed)
15. Lundborg CS, Wahlstrom R, Oke T, et al. Influencing prescribing for urinary tract infection and asthma in primary care in Sweden: a randomized controlled trial of an interactive educational intervention. J Clin Epidemiol 1999. 52(8):801-12. (PubMed)
16. Santoso B, Suryawati S, Prawaitasari JE. Small group intervention vs formal seminar for improving appropriate drug use. Soc Sci Med 1996. 42(8):1163-8. (PubMed)
17. Van Eijk ME, Avorn J, Porsius AJ, et al. Reducing prescribing of highly anticholinergic antidepressants for elderly people: randomised trial of group versus individual academic detailing. BMJ 2001. 322(7287):654-7. (PubMed) (Full Text in PMC)
18. Platt R, Davis R, Finkelstein J, et al. Multicenter epidemiologic and health services research on therapeutics in the HMO Research Network Center for Education and Research on Therapeutics. Pharmacoepidemiol Drug Saf 2001. 10(5):373-7. (PubMed)
19. Feldstein AC, Simon SR, Schneider J, et al. How to design computerized alerts to enhance safety of prescribing practices. Joint Commission Journal of Quality and Safety 2004. 30(11):602-613.
20. Weingart SN, Toth M, Sands DZ, et al. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003. 163(21):2625-31. (PubMed)
21. Payne TH, Nichol WP, Hoey P, et al. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002;602
6. 22. Dexter PR, Perkins S, Overhage JM, et al. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001. 345(13):965-70. (PubMed)
23. Davison GC, Vogel RS, Coffman SG. Think-aloud approaches to cognitive assessment and the articulated thoughts in simulated situations paradigm. J Consult Clin Psychol 1997. 65(6):950-8. (PubMed)
24. Beers MH. Explicit criteria for determining potentially inappropriate medication use by the elderly: an update. Arch Intern Med 1997. 157(14):1531-6. (PubMed)
25. Chronic Kidney Disease Workgroup. Kidney disease outcomes quality initiative clinical practice guidelines. Am J Kidney Dis 2002. 39(2):S17-S31.
26. Levey AS, Bosch JP, Lewis JB, et al. A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation. Modification of Diet in Renal Disease Study Group. Ann Intern Med 1999. 130(6):461-70. (PubMed)
27. Soumerai SB, Majumdar SR, Lipton HL. Evaluating and improving physician prescribing. In: Strom BL, editor. Pharmacoepidemiology. New York: John Wiley & Sons, Ltd.; 2000. pp. 483
503.