Clinical Decision Support Systems: State of the Art

Introduction

Clinical decision support (CDS) systems provide clinicians, staff, patients, and other individuals with knowledge and person-specific information, intelligently filtered and presented at appropriate times, to enhance health and health care.¹ The Institute of Medicine has long recognized problems with health care quality in the United States, and for more than a decade has advocated using health information technology (IT), including electronic CDS, to improve quality.^2-5 Since 2004, when the Federal Government promoted the importance of electronic medical records (EMRs),* there has been a slow but increasing adoption of health IT.^7-9 It must be remembered, though, that these health IT applications are a means to improve health care quality, not an end in themselves.¹⁰ Further, although EMRs with computerized provider order entry (CPOE) can improve accessibility and legibility of information, it is unlikely that there will be major improvements in the quality and cost of care from the use of health IT without proper implementation and use of CDS.^11-15

To illustrate this point, imagine the following scenario:

While his doctor is out-of-town, an elderly asthma patient who has developed severe knee pain sees another physician in his doctor’s office. An EMR provided documentation of the last visit, including recent laboratory results and a list of the patient’s medications. This information easily brought the doctor up to date on the patient’s condition. The doctor entered an order for medicine for the knee pain into the system, printed out a (legible) prescription for the patient, and sent him on his way. Unfortunately, within 2 months, the patient wound up in the emergency room with a bleeding ulcer caused by interaction of the pain medicine with the patient’s asthma medicine.

Problems of this kind occur frequently, as documented in reports from the Institute of Medicine.^4,5,16 Any of several types of CDS tools could have prevented this patient’s drug interaction. Examples include a pop-up alert to the potential drug interaction when the doctor prescribed the new medicine; clinical prediction rules to assess the risks of the pain medication for this patient; clinical guidelines for treatment of asthma; or reminders for timely followup. This scenario illustrates that EMRs are the foundation for patient safety and health care quality improvement, but CDS is an essential element in fully realizing these goals.

This review presents a summary of the state of the art of electronic CDS for clinicians. It includes background information on the types of CDS and focuses on the outcomes of deploying these CDS interventions. It also discusses the major issues and challenges of CDS implementation and evaluation. After reviewing what is known about implementing CDS, the impact from its use, and the knowledge gaps that remain, the review examines factors that can facilitate broader use of CDS, including the role of various stakeholders in influencing CDS adoption. This review uses both the peer-reviewed literature on implementation and outcomes of CDS and a variety of books, white papers, and recommendations put forth by national organizations in recent years.

Types of Clinical Decision Support

Early CDS systems were derived from expert systems research, with the developers striving to program the computer with rules that would allow it to “think” like an expert clinician when confronted with a patient.¹⁷ From this early research there was growing recognition that these systems might be useful beyond research, that they could be used to assist clinicians in decision making by taking over some routine tasks, warning clinicians of potential problems, or providing suggestions for clinician consideration.^5,18

This review focuses on CDS systems of a type known as knowledge-based CDS because they include compiled clinical knowledge. There have been several descriptions of types of CDS and their characteristics.^18-22 Osheroff and colleagues have provided a detailed taxonomy of CDS functions.¹⁸ Many of the early CDS systems provided expert consultation to the clinician for diagnosis and medication selection. CDS today also encompasses a range of options, from general references, through specific guidelines for a given condition, to suggestions that take into account a patient’s unique clinical data. CDS can include nationally recommended guidelines at one end of the continuum and customized order sets designed by an individual clinician at the other.

Technological Underpinnings

Common features of CDS systems that are designed to provide patient-specific guidance include the knowledge base (e.g., compiled clinical information on diagnoses, drug interactions, and guidelines), a program for combining that knowledge with patient-specific information, and a communication mechanism—in other words, a way of entering patient data (or importing it from the EMR) into the CDS application and providing relevant information (e.g., lists of possible diagnoses, drug interaction alerts, or preventive care reminders) back to the clinician. CDS can be implemented using a variety of platforms (e.g., Internet-based, local personal computer, networked EMR, or a handheld device). Also, a variety of computing approaches can be used. These approaches may depend on whether the CDS is built into the local EMR, whether the knowledge is available from a central repository (possibly outside the local site and accessed and incorporated locally when needed), or whether the entire system is housed outside the local site and is accessed, but not incorporated into the local EMR. In principle, any type of CDS could utilize any of these underlying computational architectures, methods of access, or devices. The choices among these elements might depend more on the type of clinical systems already in place, vendor offerings, workflow, security, and fiscal constraints than on the type or purpose of the CDS.

Target Area of Care

Many of the technology differences described in the previous section need not be apparent to the user. The following factors may be more relevant to the clinician user or those assisting with implementation: (1) the primary need or problem and the target area of care for which the CDS is being considered (e.g., improve overall efficiency, identify disease early, aid in accurate diagnosis or protocol-based treatment, or prevent dangerous adverse events affecting the patient); (2) to whom and how the information from the CDS will be delivered; and (3) how much control the user will have in accessing and responding to the information. A key decision is whether CDS can help solve the need or problem identified.

CDS can provide support to clinicians at various stages in the care process, from preventive care through diagnosis and treatment to monitoring and followup. CDS as implemented today can include, for example, order sets tailored for particular conditions or types of patients (ideally based on evidence-based guidelines and customized to reflect individual clinicians’ preferences), access to guidelines and other external databases that can provide information relevant to particular patients, reminders for preventive care, and alerts about potentially dangerous situations that need to be addressed.

The most common use of CDS is for addressing clinical needs, such as ensuring accurate diagnoses, screening in a timely manner for preventable diseases, or averting adverse drug events.²³ However, CDS can also potentially lower costs, improve efficiency, and reduce patient inconvenience. In fact, CDS can sometimes address all three of these areas simultaneously—for example, by alerting clinicians to potentially duplicative testing. For more complex cognitive tasks, such as diagnostic decisionmaking, the aim of CDS is to assist, rather than to replace, the clinician,^24,25 whereas for other tasks (such as presentation of a predefined order set) the CDS may relieve the clinician of the burden of reconstructing orders for each encounter.¹⁸ The CDS may offer suggestions, but the clinician must filter the information, review the suggestions, and decide whether to take action or what action to take. Table 1 below provides examples of CDS that address a range of target areas. For more examples of how various types of CDS can be applied to addressing specific improvement objectives, see the work of Osheroff and his colleagues.¹⁸

Table 1: Examples of CDS interventions by target area of care

Target Area of Care	Example
Preventive care	Immunization, screening, disease management guidelines for secondary prevention
Diagnosis	Suggestions for possible diagnoses that match a patient’s signs and symptoms
Planning or implementing treatment	Treatment guidelines for specific diagnoses, drug dosage recommendations, alerts for drug-drug interactions
Followup management	Corollary orders, reminders for drug adverse event monitoring
Hospital, provider efficiency	Care plans to minimize length of stay, order sets
Cost reductions and improved patient convenience	Duplicate testing alerts, drug formulary guidelines

Delivery of CDS Recommendations to Users

Key questions in designing or selecting CDS systems are whose decisions are being supported, what information is presented, when it is presented, and how it is presented to the user. Although it is usually assumed that the physician is the clinician whose decisions are being supported, in some cases it has been found that CDS is more effective if nurses and other clinicians receive the information.²⁶ As for timing, the most effective time to present some kinds of information may be immediately at the point of care—for example, delivering an alert about drug-drug interactions during the prescribing process. Other information, such as the names of patients being seen on a given day who need immunizations, may be less disruptive when delivered prior to the patient encounter. The information from the CDS can be presented automatically to the clinician or “on demand” (i.e., when the clinician chooses to access the information). Whatever the features of CDS delivery of information, the quality of the information and the evidence underlying it are the major determinants of the impact of CDS on patient safety and quality improvement. The description by Osheroff, et al. of what they call the “five rights” of CDS is a good summary of what is needed for effective delivery: CDS should be designed to provide the right information to the right person in the right format through the right channel at the right time (i.e., when the information is needed).¹⁸

User Control

CDS systems differ in how much control the user has over the decision to use CDS. These decisions involve not only whether the CDS is set up to be displayed on demand, so that users have full control over whether they choose to access it, but also the circumstances under which users can, after viewing the CDS information, choose whether to accept it. The two aspects of control are related and they connect with how closely the CDS advice matches a clinician’s intention. CDS may be designed to (1) remind clinicians of things they intend to do, but should not have to remember; (2) provide information when clinicians are unsure what to do; (3) correct errors clinicians have made; or (4) recommend that the clinicians change their plans. Conceived of in this way, it should be obvious that the users’ reactions to CDS may differ with these diverse intents.

An analogy can be seen in some of the functions of common desktop computer applications. When a user employs the calendar functions on the computer, the calendar alarm is an automatically presented reminder of something one intends to do. In this case the automatic notification is one of the most helpful features. The spell checker in a word processing application can both provide advice and correct errors, and can do so while one types (automatically) or after one is finished and the function is accessed to check the final document (on demand). Two other word processing features make suggestions to users about changing what they have done. The grammar checker, often accessed on demand, not only corrects obvious grammatical errors, but also makes suggestions for sentence revision, which may be ignored by writers who feel that they have expressed themselves exactly as they intended. Most users access the help function in their word processing program when they want advice on how to do something. However, as most people who use word processing programs attest, the automatic appearance of the help wizard (an example of automated decision support alerts) may lead the user to turn the automatic help function off immediately, if they have not already disabled it in advance. These reactions to nonclinical decision support have their parallels with CDS as well.

Achieving the five rights for CDS presents challenges, and the challenges differ depending on how closely the CDS is tied to what the clinician already intends to do. Clinicians may initially want certain reminders or, after performance assessments, agree that they need other reminders, but in either situation they are choosing to receive the reminders. The key issue in reminding the user about things they choose to be reminded about is the timing of the reminder. For instance, should reminders for preventive care be given to the physician in advance of the patient visit (e.g., the day before), or should the reminders appear during the patient’s visit?

Key issues for consultation that the user seeks out (on-demand CDS) are speed and ease of access. Users may recognize the need for information, but may be willing to access it only if they can do so efficiently. If access is too difficult or time-consuming, potential users may choose not to use the CDS.

The major issue involved in correcting errors or making suggestions that users change what they had planned is balancing clinicians’ desire for autonomy with other demands, from or on clinicians, such as improving patient safety or decreasing practice costs. Another question related to autonomy is how much control users have over how they respond to the CDS. This aspect of control relates to whether users are required to accept the CDS suggestion, whether they can easily ignore it, or whether it takes significant effort to override the advice. Table 2 below provides a summary of these points.

Table 2: CDS Intent and Key Issues

CDS Intent	Match to User’s Intention	Key Issues
Reminder of actions user intends to do, but should not have to remember (automatic)	High	Timing
Provide information when user is unsure what to do (on demand)	High	Speed and ease of access
Correct user’s errors and/or recommend user change plans (automatic or on demand)	Low	Automatic: timing, autonomy and user control over response On demand: speed, ease of access, autonomy and user control over response

While some of these issues have been addressed by research, there are no universally accepted guidelines regarding them, in part because clinicians often differ in their preferences. In addition, there are varying clinical approaches that are justified, which makes designing effective CDS a challenge. How these issues are addressed will influence the ultimate impact and effectiveness of CDS.

Impact and Effectiveness of CDS

This section focuses on evaluations of the impact of CDS on health care quality, using Donabedian’s classic definition of quality comprising structure, process, and outcomes of health care.²⁷ Donabedian advocated that organizational outcomes such as cost and efficiency, as well as individual patient health outcomes, be evaluated. Donabedian’s model is expanded by Carayon and her colleagues’ formulation of structure, which includes people, organization, technologies, tasks, and environment.²⁸ This expanded definition of structure is used here so that CDS impact on cost and efficiency are addressed and included as part of impact on structure. As outlined below, evaluation of impact includes care process and patient health outcomes. Structural outcomes are also addressed below.

Most published evaluations of the impact of CDS on health care quality have been conducted in inpatient rather than ambulatory settings, and most have been in large academic medical centers, often using “homegrown” systems, where there is a culture that is accustomed to their use and adequate resources (including expertise, time, infrastructure) to build and maintain them.^23,29-35 Although many commercial EMRs have CDS capabilities, there has been little systematic research on the outcomes or even on the implementation strategies of commercial CDS in community settings.^36,37 These omissions, and the narrow focus responsible for them, are particularly problematic since most hospitals will deploy commercial systems in the future, and their culture and resources are likely to differ from those of large academic medical centers. In addition, the impact of CDS in ambulatory settings needs more attention. Some of the projects within the AHRQ Ambulatory Safety and Quality Program are beginning to address this need.³⁸

The research on CDS has other noteworthy limitations. First, although a number of CDS studies have been published, comparatively few are randomized controlled trials (RCT). Second, most research has examined the effects of CDS on the process of care (rather than the outcomes or structure) and has focused primarily on clinician decisionmaking. Third, the diagnostic programs have had limited use in practice settings. Finally, the results of the research to date are mixed in terms of the effectiveness of CDS for particular conditions or particular types of CDS. These limitations point to gaps in the literature. Although RCTs are considered the gold standard for research studies, qualitative studies may be better able to determine why a CDS intervention succeeds or fails.³⁹

The following section reviews the results of RCT studies and other studies of CDS. Because most of the studies deal with process and patient health outcomes, these aspects are discussed first, followed by a discussion of structure.

Impact on Care Process and Patient Health Outcomes

In 2001, Trowbridge and Weingarten summarized the results of several systematic reviews or meta-analyses of CDS RCTs.^31,32,40,41 Since that paper, several new reviews and additional RCT studies have shown similar results.^{21,23,30,34,35,42,43} The meta-analyses of studies of alerts and reminders for decision support have been fairly consistent in showing that they can alter clinician decisionmaking and actions, reduce medication errors, and promote preventive screening and use of evidence-based recommendations for medication prescriptions. The data on how those decisions affect patient outcomes are more limited, although a number of studies have shown positive effects.^34,35,42 Overall, the results indicate the potential of CDS to improve the quality of care.

Although the studies showing the ability of CDS to prevent medication errors (incorrect decisions) have been consistently positive, the results of research studies on the ability of CDS to avert adverse drug events (harm to the patient) have tended to be mixed. Few of the studies examining the impact on health outcomes were RCTs, many studies were poorly designed, and not all studies showed statistically significant effects.^34,35,42 In terms of other outcomes, in one recent randomized controlled trial of the impact of CDS on use of deep vein thrombosis (DVT) prophylaxis, mortality was improved with CDS⁴⁴; however, well-designed studies of diabetes outcomes do not consistently show positive effects.^45,46 CDS studies that focus on providing diagnostic decision support have also shown mixed results, and fewer of these systems have been evaluated in practice settings.^17,23,47-50 However, studies comparing CDS diagnostic suggestions with expert clinicians’ analyses of challenging clinical cases have shown that the diagnostic CDS can remind even expert physicians of potentially important diagnoses they did not initially consider.^48,51,52

Some of the mixed results have resulted from methodological issues such as ceiling effects (performance was already very good prior to implementing CDS) or low statistical power to detect statistically significant effects for infrequently occurring events, such as adverse drug events.^53,54 In addition, there are often intervening factors between the clinician decision that is influenced by CDS and the outcome for the patient. For instance, physicians may prescribe a medication suggested by the CDS, but patients may fail to take it. But even when looking at physician actions alone, many studies have shown that even when CDS recommendations are accurate and delivered in a timely manner, physicians have frequently ignored or overridden them.

This issue of ignoring the advice of the CDS has been shown for a variety of types of CDS including those that provide diagnostic suggestions,^55,56 evidence-based treatment recommendations,^45,57,58 or alerts for potentially dangerous drug interactions.^37,54,59 The problem of overriding drug interaction alerts, in particular, has been shown in inpatient, long-term care, and outpatient settings.^{34,37,54,59-63} Until there is a better understanding of why clinicians either do not access, or choose to ignore, the CDS recommendations, assessing the effect of CDS on quality will be very difficult. Because clinician decisionmaking influences care processes, it is important to examine the literature on why clinicians fail to utilize CDS suggestions.

Match of CDS to user intentions. In discussing the types of CDS, a distinction was made between (1) systems that remind clinicians of things they intend to do, such as order sets that the physician has customized to his or her preferences, and (2) systems that provide suggestions to make clinicians reconsider what they intend to do. These latter suggestions may involve additional diagnoses to consider, a change in medications from what the physician initially prescribed, or reminders for tests that the physician did not initially intend to order. Most studies of CDS have focused on the types that suggest that clinicians change their actions (e.g., medication alerts), rather than the types that remind clinicians of their intentions (e.g., order sets). Studies of factors that make CDS effective have shown that it is more difficult to get users to change their plans than to remind them of what they already intend to do.^57,⁶⁰ On-demand CDS systems appear less likely to be overridden than automatic alerts, but are viewed less frequently than those that are automatically displayed.^56,60

The Institute of Medicine has emphasized that, to improve safety, health IT systems should be designed to make it “easy to do the right thing.”⁴ In a similar vein, Thaler and Sunstein in their book, Nudge, have focused on how “defaults” are set and advocated, making the default option (the option that does not require active choice on the part of the user) what is in the user’s best interest.⁶⁴ This is particularly challenging in terms of CDS design. Because alerts are often presented automatically during the ordering process and usually indicate problems of varying severity, attempts to improve attention to them have focused on a variety of ways to present such alerts. The options include allowing the user to choose to view the information (on demand) rather than presenting it automatically; presenting alerts so they are not interruptive; or turning off or not requiring a response for the less serious alerts. When users seek out CDS information they are less likely to override it than when it is automatically presented to them; however, they choose to access the information very infrequently,^59,65 reducing the overall impact.²³ Attempts to make the alerts less interruptive by displaying the information (rather than calling attention to it or requiring an action) have found that such passive display does not attract the attention of the clinician and, in general, does not change behavior.^62,66

User control, disruptiveness, and risk. Some have suggested turning off alerts that are frequently overridden, perhaps assuming that alerts that are ignored must be inaccurate or not needed. However, there is often lack of agreement about which alerts can be turned off without compromising safety.⁶⁷ One approach that has been demonstrated to improve positive responses to alerts is what has been termed “tiered alerts.”^18,68-70 In this approach the impacts of ignoring the alerts are rated for severity, with the display and users’ choices of action varying depending on the severity. For instance, alerts indicating a potentially life-threatening problem are presented automatically and may not allow overrides at all; those with less severe impact may be presented, but allow overrides with an explanation or rationale for the user’s decision; and those alerts with the least severe consequences if ignored may be presented passively.

Generally the alerts that are most frequently overridden—the majority of the alerts—are those that have a less severe impact when ignored. Most alerts fall into the less severe category because the current state of the art in CDS systems is such that the alerts are often very general, but in reality may be needed only by specific patient populations (e.g., elderly), by specific clinicians (e.g., less experienced), or in certain circumstances (e.g., first-time prescriptions). Another effective approach has been to design standing orders for the nurse as part of the discharge process for interventions that are not time-sensitive, rather than alerting the physician while he or she is focused on more immediate orders.⁷¹ These examples illustrate three of the five rights: recipient, timing, and format.¹⁸

Integration of CDS into work processes. Research has shown that CDS that fits into the workflow is more likely to be used. However, integrating CDS into the workflow often requires unique customization to local processes, and sometimes to changes in processes (when previous clinical processes were found to be inefficient or ineffective). CDS also needs to be minimally disruptive to the clinician’s “cognitive workflow” and this, too, can be a challenge. For instance, accessing the data needed for the CDS can be disruptive if the clinical systems are not well integrated or if the necessary data are not in a form that the CDS can use. If the lack of data leads to inappropriate alerts, these alerts may be overridden. In addition, to the extent that using CDS or following its advice is disruptive to the clinician’s work or thought processes, the CDS is likely to be ignored.

It is clearly a challenge to implement CDS effectively in a way that ensures that alerts are raised whenever needed but without inducing “alert fatigue.” A number of studies have identified the problem of overriding alerts and reminders, but further research is needed on methods to increase the specificity of the alerts and the effects of more specific alerts on physician overrides and patient outcomes. In addition, continuing research is needed on the design and impact of other types of CDS that may be less disruptive than alerts, such as order sets, other documentation tools, and infobuttons, which are CDS features that present context-sensitive information during the care process that the user can choose on demand.^66,72,73 These have been viewed positively by physicians and have shown promise in changing physician decisions.⁷³

Impact on Structure

The broader definition of structure used here includes people, organization, technologies, tasks, and environment.²⁸ Few studies have examined the structural impact of CDS. Berlin and Sim conducted a systematic review of CDS and found that when CDS is implemented, there is almost invariably an increase in the number of staff needed to deal with the CDS.²¹ Berlin and Sim pointed to the need for personnel to directly handle the implementation of CDS. In addition to the personnel needed for implementation, personnel are also needed to maintain the CDS knowledge base. Partners HealthCare, for example, has established an entire organizational unit devoted to updating of the knowledge needed for CDS.⁷⁴

The main focus of studies that looked at outcomes other than health care quality has been the effect of CDS on health care costs, with an emphasis on lowering costs by reducing adverse drug events (ADEs). Because ADEs have been shown to increase costs and because CDS can detect and potentially prevent ADEs it is assumed that CDS can reduce health care costs by helping to reduce ADEs.⁷⁵ There is some literature to show that CDS can reduce costs, although many of these analyses have used cost data related to known costs of ADEs (e.g., costs of increased length of stay, treatments, etc., that occur if a patient has an ADE), the costs of inappropriate prescriptions, or the costs of failing to prescribe antibiotics prior to surgery (e.g., costs similar to those used to determine ADE costs if a patient acquires an infection). These studies have then used their own data or the literature on the demonstrated effects of CDS to determine the extent of reduction of these adverse events (i.e., ADEs or infections). From these data they estimated the cost savings.^15,76-80 Most studies have either used modeling techniques based on the literature or have examined costs prior to and after implementation of CDS, rather than a direct assessment of actual cost savings that can clearly be attributed to the use of CDS.

Because many of these studies have been conducted at large academic medical centers where the CDS capabilities have been developed over many years and usually with grant support, it is difficult to determine the costs of developing the CDS., There have, however, been some attempts to estimate these figures.⁷⁶ A recent study by Field and colleagues estimated the development and initial implementation costs for a CDS targeted to medication prescribing for adults with renal insufficiency. They calculated the costs based on the time spent by different professionals involved in the development (including clinicians, programmers, informatics personnel, and project management personnel).⁸¹ They concluded that approximately half the cost of development was related to clinician review of the content. While some of the clinician time spent on development could be decreased by using existing databases for content, substantial time would still be needed for clinician review and adaptation to local conditions. Another study utilized an RCT to examine the effects of a diagnostic and therapeutic decision support system. The study found that the system increased costs, and that, although the physicians found it valuable for suggesting alternatives in diagnosis or treatment, they also found it time-consuming.⁵² For the most part, studies of the impact of implementing CDS on clinician time have been mixed, and the impact may depend on how well the system is designed, which clinician is expected to use the CDS (e.g., nurse or physician), and how well it is integrated into clinician workflow.^18,82

It is important to recognize that the development, implementation, and maintenance of CDS will have an impact on the structure or work system in which it will be used. The changes that the CDS will introduce need to be incorporated in the planning so that the impact on clinician time is not excessive. The research findings emphasize the diverse aspects that must be considered in CDS design, implementation, maintenance, and evaluation.

Design and Implementation of CDS

Planning for any new health IT system includes a number of key steps, such as identifying the needs and functional requirements (e.g., what the system is expected to do), deciding whether to purchase a commercial system or build the system, designing or configuring the system for use in the local environment, planning the implementation process, and determining how to evaluate how well the system has addressed the identified needs. In the case of CDS, the design and implementation issues are often interrelated.

There is a growing literature of best practices for CDS design and implementation.^{18,19,53,68,77,83} In addition to expert opinion, the literature also provides data on characteristics of successful CDS deployment. Kawamoto, et al. did a systematic review of the research literature and identified design characteristics that are associated with successful deployment of CDS.³⁰ Their review showed that:

1. Computer-based decision support is more effective than manual processes for decision support.

2. CDS interventions that are presented automatically and fit into the workflow of the clinicians are more likely to be used.

3. CDS that recommends actions for the user to take are more effective than CDS that simply provides assessments.

4. CDS interventions that provide information at the time and place of decisionmaing are more likely to have an impact.

A recent RCT explicitly used these guidelines to design a CDS for dyslipidemia. The authors compared automatic and on-demand CDS with a control group that did not have CDS. They found a significant effect compared to controls for both types of CDS, but automatic CDS was more effective than on demand for prompting both screening and appropriate treatment for dyslipidemia.⁶⁵ However, one caveat about these research-based design guidelines is that most of the literature on CDS implementation on which the guidelines are based comes from single-institution studies, prompting calls for more large-scale, multisite studies to validate these design and implementation recommendations.⁸⁴

As CDS and other health IT applications are used more frequently, reports have surfaced of their potential for harm.^85-87 Several authors have identified unexpected negative consequences related to the use of health IT, and Weiner, et al. coined the term “e-iatrogenesis” to describe unintended harm.^88-90 Although some intrinsic design flaws may lead to problems, most analyses of the studies of reported harm found that the problems were mainly the result of system implementation issues, rather than inaccurate recommendations or intrinsic system flaws.^91,92 In fact, The Joint Commission, in response to reports about harm to patients from information technology, issued a sentinel event alert that emphasizes proper implementation practices. These recommended practices include resolving workflow and process problems prior to implementation, involving users, training users well, monitoring the system to ensure that it is performing as expected, and addressing the errors that arise and correcting them if possible.⁹³

As described above, the research data demonstrate that CDS systems have great potential to improve the quality of care, but attention must be paid to implementation processes, not only for the quality improvement to be realized, but also to avoid negative effects of CDS.

Workflow Integration

The issue of workflow is one of the key issues both system designers and those implementing CDS must take into account. Workflow includes the structure or work system features and processes that support care.²⁸ While it may seem obvious that CDS that fits into the clinical workflow will be used more than CDS that does not, changes in the workflow may be needed to optimize care, either prior to the adoption of CDS or during the adaptation to CDS. Assessment of the workflow and how CDS will fit in should be done as one of the first steps in the development process, usually in the needs assessment phase where the CDS requirements are identified. If the needs assessment discovers processes that need redesign they should be fixed prior to implementing a CDS system, although in some cases the CDS may be part of the process redesign. In other words, congruence between clinicians’ workflow and CDS timing, structure, and design makes the CDS more likely to be accepted and effective, but to achieve that congruence, both the workflow and the CDS implementation may need to change.⁹⁴ That is why it is so important to engage the clinicians in the entire process of CDS design and implementation. As Osheroff said, “Do CDS with users not to them.”¹⁸ Guidelines for analysis and redesign of workflow are being addressed in another white paper in this series,⁹⁵ but we will briefly address some workflow issues here.

First of all, the workflow changes should be driven primarily by needs for process improvement, not solely by the specific CDS, since the CDS itself may not be optimally designed.¹³ Once the workflow is analyzed and a need for process improvement has been determined, it will be possible to make decisions on how to improve processes and how CDS can support improvement. Second, there may not be a single workflow pattern, as individual clinicians have often developed their own particular work styles. This is complicated by the fact that a clinician representative to the IT team may be speaking only for his/her own preferences, not representing a group of clinicians. As this suggests, in analyzing the workflow of a department or office, it is important to be aware of the variability among clinicians. Third, to the extent feasible, it is important to take the time to configure the system to meet users’ needs, as this will ultimately improve efficiency, ease of use, and usefulness.

CDS designers and implementers should also be aware of the issues they will face as they plan for implementation. In addition to the considerations discussed above there are other specific challenges that must be addressed.

Data Entry and Output

Most of the CDS systems related to drug interaction alerts and reminders are integrated into an EMR and draw their patient information from that record. They often provide the alert or reminder in the context of computerized provider order entry (CPOE) systems. However, there are also some CDS systems that are independent of the EMR, and it may be challenging to work with two different systems. This has been especially true of some diagnostic systems and other Web-based or handheld-computer-based CDS tools. In these cases, if the facility uses a paper chart (rather than an electronic record), or if the CDS cannot be integrated into the EMR, the user may have to enter patient information twice—once into the clinical record and again into the CDS. This is a workflow issue that can lead to failure to use the CDS routinely.

Another related issue is who enters the data and who receives the CDS advice. If the physician writes a paper prescription and a nonphysician enters it into to the “system,” how would a CDS recommendation to change medicines be handled? If a CDS notification (such as dropping hematocrit values over time) occurs when the clinician is no longer interacting with the electronic system, how will timely response be ensured? A variety of approaches to address feedback and use issues have been developed (e.g., email alerts to the physician’s pager), but, like user preferences for when to be alerted, these must be addressed as part of the needs assessment, design, and planning process and evaluated once the system has been implemented.

Standards and Transferability

In addition to the issue of technical integration with existing systems, users need to recognize that even EMRs with CDS capability may not be ready to use without additional work. For example, even if sets of reminders are built into the system, at the very least decisions need to be made as to which ones will be implemented. Miller and colleagues have emphasized that effective CDS implementation usually requires some degree of local customization, which may mean configuring a commercial CDS for local needs or, in some cases, paying for special features that are needed at the local site.⁹⁶ In addition, although efforts are ongoing to develop standards for information exchange, data quality, and desired functionalities of CDS,^97-100 because at present there are no national standards for the specific evidence-based guidelines or rules that should be built into CDS, users will have to select the rules and alerts that are most applicable to their site. Field, et al. estimated that approximately half the costs to develop the CDS involved clinician time in selection and design of content.⁸¹ Although some time can be saved if commercial knowledge bases or modules from other sites are used, there are often vocabulary differences among sites, as well as different standards for normal laboratory values, medication formularies, or norms for processes of care at different sites and within different CDS. As an example, in analyzing the underlying logic of four diagnostic decision support systems it was found that each system was based on a different norm for labeling a heart rate as “tachycardia” (abnormally rapid heart rate). In part this reflected the norms at the sites where the CDS systems were developed. One CDS considered a heart rate above 95 as abnormal, while another considered 120 as the cutoff of normal.⁵¹ What this means is that even when purchasing an EMR that has the capability for clinical decision support, sites must realize that they will still have to invest a considerable amount of time in understanding the logic of the CDS and, in some cases, will have to adapt the CDS to their unique needs.

Knowledge Maintenance

Two aspects of knowledge maintenance can be especially challenging. The first is maintaining the accuracy of the patient record. Studies that found high rates of alert overrides have shown that the medical records were often out-of-date (when for instance, a patient’s allergy information changed or medications were discontinued, but the changes were not updated in the medical record).^54,101 If the information the CDS is using to trigger the alert is inaccurate, the alerts will not be accurate and overriding them may make sense. Frequent inaccurate alerts can lead the clinicians to ignore all of the CDS advice. The problem of knowledge maintenance is important for all types of CDS, not just the alerts and reminders. For this reason, it is important to monitor the accuracy of the patient’s record and to address problems encountered.

Another issue in knowledge maintenance is related to the knowledge embedded within the CDS. Medical knowledge is expanding, new drugs and diagnoses are continually being discovered, and evidence-based guidelines change as new evidence is accumulated. One solution to this challenge is to utilize commercial knowledge bases that provide frequent updates. In deciding to purchase these commercial systems, users should investigate the source of the knowledge and the frequency of updates.^102,103 Another solution is to develop an in-house knowledge management process, as Partners HealthCare has done. Such a system may be needed even if commercial knowledge bases are purchased, but may require significant resources at the local site, which may be beyond the means of small physician practices.

AHRQ has funded two CDS projects that are approaching knowledge maintenance differently and illustrate some of the tradeoffs in the different approaches.¹⁰⁴ The GLIDES project has built CDS for management of pediatric asthma and obesity into two commercial EMRs and is implementing it in multiple sites across the country. Because the commercial systems are being customized in this manner, there is greater assurance that the CDS will be well integrated with each system. However, when the guidelines underlying the CDS change there is a need to rewrite the computer code and update multiple systems, an expensive and time consuming undertaking.

An alternative approach has been taken by the other project, the Clinical Decision Support Consortium (CDSC).¹⁰⁴ The CDSC is developing a Web-based repository of CDS knowledge in relation to hypertension and diabetes guidelines. The aim is to use what is known as service-oriented architecture (SOA) and to have a variety of types of materials, including guidelines that can be adapted to a local site, all the way to “plug and play” CDS, depending on the needs of a particular site.¹⁰⁵ What SOA allows is for the central site to maintain the knowledge but for local sites to develop systems that, in the background, can access it when needed. Ideally users should not be able to tell that they are getting information any differently than they would get information residing on their own computers. While this approach makes updating easier since it is done centrally, it is also likely to require expertise at the local level to integrate the CDS. In addition, obtaining consensus as to what should be included in a centralized system can be a challenge. Given the expense of knowledge management, and to some extent duplication of effort when one looks at the aggregate effort across health care facilities, it has been advocated that some sort of national repository of knowledge that can be incorporated into a variety of CDS be developed.^106,107

Clinician Motivation To Use CDS

For CDS to be effective, clinicians must be motivated to use these systems, and many features of the health care environment may decrease, rather than increase, this motivation. Even when efforts are made to engage clinicians and integrate CDS into clinician workflow, the use of CDS may still be resisted by clinicians, especially if use of CDS exacerbates the increasingly time-pressured patient care process, which may occur.

One of the challenges for CDS implementation that is also faced by the patient safety and quality improvement movements is that the culture of medicine has always emphasized individual physician autonomy. System changes are not always well-received if physicians are concerned about maintaining that autonomy, as surveys of clinicians’ views about CDS have shown.^108,109 In addition to worries about autonomy, physicians have been concerned about overreliance on an outside device, and the legal and ethical ramifications of listening to, or overriding, the CDS.

These concerns have some basis. For example, use of CDS is not currently part of the standard of care and, although the CDS systems can frequently provide useful advice, the advice is not foolproof. It should be recognized that these concerns are not new, nor are they confined to CDS. Crenner’s discussion of the history of the use of the blood pressure cuff shows that in the early part of the last century physicians were uneasy about relying on the cuff to determine a patient’s blood pressure, instead of using their palpation skills, as was the practice at the time. Over time, physicians became more comfortable with using the cuff, as long as a skilled physician was using it and interpreting its output.¹¹⁰ Today, not only are nonphysicians usually the people in the health care setting who take the patient’s blood pressure, but automated devices even allow patients to do it themselves at home. This example illustrates how new devices or systems that appear to challenge what clinicians perceive as their unique skills are likely to be resisted. Because CDS is still fairly new, many clinicians today have misconceptions about how CDS systems work and may not be interested in using it. However, over time, as CDS is used more, and the legal situation in regard to liability for its use or nonuse becomes clearer, clinicians’ resistance to CDS will lessen.¹⁰³ However, until the use of CDS is as routine as the use of the blood pressure cuff, it is important to be sensitive to resistance to using these systems.

Evaluation

Any evaluation of CDS should assess how the systems are used in practice and their impact on users. Systems that are “less than perfect” may positively impact users’ decisions, and others that perform well outside the clinical setting may not be used in such settings, or may be overridden when implemented in a clinical environment. Also, as Carayon, et al. and Osheroff, et al. have emphasized, evaluations of CDS should assess the entire work situation and all stakeholders, rather than focus solely on system performance.^18,28 Osheroff and colleagues use the acronym METRIC which stands for Measure Everything That Really Impacts Customers. The customers in CDS are in fact a diverse group of stakeholders which can include clinicians, patients, and the care delivery organization.¹⁸

There are a number of challenges in evaluating the impact of CDS. One of the reasons there are so few RCTs on the use of CDS is because an RCT is expensive and time consuming to conduct and cannot usually be undertaken without external funds. Part of the expense lies in having enough use of the system to have the power to detect an effect, especially for outcomes such as adverse drug events, which may be comparatively infrequent, hard to detect, and difficult to assess in terms of preventability. There are also very few evaluation studies outside academic medical centers. The non-RCT studies have tended to support the general results of the more rigorous trials—that CDS can be helpful to clinicians and can improve patient safety. However, for a variety of reasons, CDS is not always utilized, or is not implemented effectively, and hence the potentially positive impact on the quality of care is not always realized.

There is clearly a need for more rigorous studies of CDS, although designs other than RCTs may be appropriate. There is also a need to conduct CDS implementation studies outside of academic medical centers’ homegrown systems. The recent study by Isaac and colleagues was one of the first to study CDS alert overrides within an electronic prescribing system in multiple community settings.³⁷ AHRQ has also funded the implementation of an academic medical center’s CDS within a commercial EMR, as well as the implementation of CDS in community settings nationwide.¹⁰⁴

As stated throughout this paper, for CDS to achieve its purpose of improving quality, it must be properly designed, carefully implemented, and used when appropriate. In addition to randomized controlled studies of CDS’ impact on quality, Friedman recommends the use of what he terms “smallball evaluation,” which are systematic evaluations of processes of implementation, user satisfaction, and other factors that may affect the outcome of the intervention.¹¹¹ Similarly, Kaplan argues that there is need for qualitative evaluations that examine the user-CDS interaction and its impact on the clinician, the workflow, and other organizational processes and outcomes.³⁹ Given the challenge of designing systems so that they will be optimally used, these types of studies are sorely needed.

Factors in the Current U.S. Health Care Environment Facilitating Broader Utilization of CDS

There are a number of factors that can facilitate adoption and more extensive use of CDS. These include (1) Federal or other payer initiatives that provide incentives for CDS deployment and (2) technological developments, including more widespread use of EMRs with CDS capabilities, increased capabilities of systems, development of technologies for health care providers to share information across entities, and cheaper, faster or more flexible technology. In both of these areas in recent years, there has been movement to facilitate the adoption and use of CDS.

Payer Initiatives To Increase Incentives for Use of CDS

Insurers increasingly recognize that the current payment models do not facilitate use of CDS, but rather discourage it, by paying more for procedures than cognition and failing to tie provider payments to the quality of care provided. Recently passed legislation related to pay for performance and e-prescribing (electronic prescribing systems that usually include CDS related to drug interactions) shift payment incentives to make use of CDS more attractive.¹¹² EMRs with alerts, reminders, and standardized order sets that support pay for performance quality metrics can improve adherence to these requirements. Although health care providers who already have CDS capabilities will benefit from these new incentives, the monetary compensation may not be sufficient to motivate large-scale EMR and CDS adoption.

In addition to efforts to provide incentives for use of CDS, the members of the Leapfrog Group, an organization of Fortune 500 companies whose goal is to foster “leaps” in patient safety by incentivizing practices to improve safety, has proposed differential payment incentives for hospitals that have CPOE. With AHRQ support, Leapfrog has developed an evaluation tool to ensure that CDS within CPOE implementations are functioning properly.⁹⁹ The momentum is picking up at the Federal level to provide incentives that promote the use of health IT (such as incentives in the new economic stimulus legislation).¹¹³ Also, as recommended in a recent report from the National Research Council, health care facilities should be offered incentives to deploy health IT that provides “cognitive support for health care providers,”¹³ that is, well-designed CDS that truly support clinicians’ cognitive tasks.

Technological Developments

Even without the incentives described above, there have been increases in the numbers of EMR vendors and in the purchase of these systems by health care providers.^7,9 In addition, there have been funding and policy initiatives that are likely to lead to both improved systems and standardization across systems. These changes will lead to more “interoperable” systems that can communicate with each other. The Commission for Certification of Healthcare Information Technology (CCHIT) has developed requirements for ambulatory and inpatient systems and is beginning to develop standards for CDS.⁹⁸ In addition, standards development organizations are developing technical and functional standards for CDS.^97,100 The American Recovery and Reinvestment Act provides additional mandates related to health IT certification.¹¹³ Health care providers are now more commonly using Internet resources, such as Internet-based knowledge resources and Internet-based technologies such as service-oriented architecture (SOA), with promise to facilitate broad dissemination of CDS interventions.^105,114 The impact of these initiatives and trends is that the technological infrastructure to support the use of CDS is improving, and, with increased use, the quality goals toward which CDS systems are aimed will have a better chance of being met.

Summary

There is growing recognition that CDS, when well-designed and implemented, holds great potential to improve health care quality and possibly even increase efficiency and reduce health care costs For the potential to be realized, CDS should not be viewed as a technology or as a substitute for the clinician, but as a complex intervention requiring careful consideration of its goals, how it is delivered, and who receives it. To gain optimal benefit, clinician users need to understand its benefits and limitations, and the unique challenges of designing and implementing the different types of CDS. Those responsible for implementation need to recognize that CDS requires careful integration into the clinical workflow, which will take effort and involvement on the part of clinician users. The high frequency of failure to attend to the CDS alerts and recommendations represents a challenge for both researchers and vendors. Researchers need to address the cognitive, informatics, structural, and workflow issues that lead to less than optimal CDS design or implementation and, therefore, limited use and effectiveness. Vendors need to use the insights gained from research and development efforts to design systems that will increase, rather than decrease, clinician efficiency. Dissemination of careful evaluations of commercial CDS systems in community settings is also important for presenting the full picture of CDS design, implementation, and impact.

Fortunately the opportunities in the current environment hold promise for increased use of CDS. These include growing concerns about quality of care at the national level,⁵ calls for better cognitive support for clinicians,¹³ and incentives at the Federal level for meaningful use of health IT.¹¹³ In addition, the new generation of clinicians has trained in academic medical centers and other environments with advanced IT systems and is likely to be comfortable with technology, as will many of their patients. All of these factors are likely to lead to a more receptive environment for use of health IT. At this point in time, the appropriate decision is not whether to design and implement CDS, but how to design and implement it so that, as the Institute of Medicine report says, we “make it easy to do the right thing.”⁴

References

1. Osheroff JA, Teich JM, Middleton BF, et al. A roadmap for national action on clinical decision support. American Medical Informatics Association; 2006 June 13. Available at: http://www.amia.org/inside/initiatives/cds/. Accessed March 20, 2009.

2. Dick R, Steen E, Detmer DE. The computer-based patient record:
An essential technology for health care, revised edition. Washington, DC: The National Academies Press; 1997.

3. Chassin MR, Galvin RW. The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. JAMA 1998 Sep 16;280(11):1000-5.

4. Kohn L, Corrigan JM, Donaldson M, eds. To err is human: building a safer health system. Washington, DC: National Academy Press; 1999.

5. Institute of Medicine. Crossing the quality chasm: a new health system for the 21st Century. Washington, DC: National Academy Press; 2001.

6. Defining key information technology terms. National Alliance for Health Information Technology; 2008 April 28. Available at: http://www.nahit.org/images/pdfs/HITTermsFinalReport_051508.pdf. Accessed March 14, 2009.

7. DesRoches CM, Campbell EG, Rao SR,, et al. Electronic health records in ambulatory care—a national survey of physicians. N Engl J Med 2008 Jul;359(1):50-60.

8. Menachemi N, Saunders C, Chukmaitov A, et al. Hospital adoption of information technologies and improved patient safety: a study of 98 hospitals in Florida. J Healthc Manag 2007 Nov;52(6):398-409.

9. Hsaio C, Burt C, Rechtsteiner E, et al. Preliminary estimates of electronic medical records use by office-based physicians, 2008. Health E-Stat National Center for Health Statistics; 2008. Available at: http://www.cdc.gov/nchs/products/pubs/pubd/hestats/physicians08/physicians08.htm Accessed March 20, 2009.

10. National Priorities Partnership: priority areas and corresponding goals. National Quality Forum;

2008. Available at: http://www.qualityforum.org/about/NPP/assets/NPP_Goals_07_03_08.pdf. Accessed March 20, 2009.

11. Linder JA, Ma J, Bates DW,, et al. Electronic health record use and the quality of ambulatory care in the United States. Arch Intern Med 2007 Jul;167(13):1400-5.

12. Nebeker JR, Hoffman JM, Weir CR, et al. High rates of adverse drug events in a highly computerized hospital. Arch Intern Med 2005 May;165(10):1111-6.

13. Stead WW, Lin HS. Computational technology for effective health care: immediate steps and strategic directions. Washington, DC: The National Academies Press; 2009.

14. Glaser JP, Davenport-Ennis N, Robertson RM,, et al. AHIC April 2008 meeting: clinical decision support recommendation letter. American Health Information Community; 2008 April 22. Available at: http://www.hhs.gov/healthit/documents/m20080422/6.2_cds_recs.html. Accessed March 20, 2009.

15. Johnston D, Pan E, Walker J, et al. The value of computerized provider order entry in ambulatory settings: executive
preview. Center for Information Technology Leadership; 2003. Available at: http://www.citl.org/research/ACPOE_Executive_Preview.pdf. Accessed March 20, 2009.

16. Aspden P, Wolcott JA, Bootman JL, et al. Preventing medication errors. Washington, DC: The National Academies Press; 2007.

17. Miller RA. Medical diagnostic decision support systems—past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc 1994;1(1):8-27.

18. Osheroff JA. Improving medication use and outcomes with clinical decision support: a step-by-step guide. Chicago, IL: The Healthcare Information and Management Systems Society; 2009.

19. Sim I, Gorman P, Greenes RA, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001 Nov;8(6):527-34.

20. Payne TH. Computer decision support systems. Chest 2000 Aug;118(2 Suppl):47S-52S.

21. Berlin A, Sorani M, Sim I. A taxonomic description of computer-based clinical decision support systems. J Biomed Inform 2006 Dec;39(6):656-67.

22. Wyatt JC. Decision support systems. J R Soc Med 2000 Dec;93(12):629-33.

23. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes. JAMA 2005;293(10):1223-38.

24. Miller RA. Why the standard view is standard: people, not machines, understand patients' problems. J Med Philos 1990;15(6):581-91.

25. Miller RA, Masarie FE, Jr. The demise of the "Greek Oracle" model for medical diagnostic systems. Methods Inf Med 1990 Jan;29(1):1-2.

26. Dexter PR, Perkins SM, Maharry KS, et al. Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA 2004 Nov;292(19):2366-71.

27. Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005;83(4):691-729.

28. Carayon P, Schoofs HA, Karsh BT, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006 Dec;15 Suppl 1:i50-i58.

29. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006 May;144(10):742-52.

30. Kawamoto K, Houlihan CA, Balas EA, et al. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005 Apr;330(7494):765.

31. Johnston ME, Langton KB, Haynes RB, et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med 1994 Jan;120(2):135-42.

32. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998 Oct;280(15):1339-46.

33. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003 Jun;163(12):1409-16.

34. Eslami S, bu-Hanna A, de Keizer NF. Evaluation of outpatient computerized physician medication order entry systems: a systematic review. J Am Med Inform Assoc 2007 Jul;14(4):400-6.

35. Ammenwerth E, Schnell-Inderst P, Machan C, et al. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008 Sep;15(5):585-600.

36. Metzger J, MacDonald K. Clinical decision support for independent physician practice. California Healthcare Foundation; 2002 October. Available at: http://www.chcf.org/documents/healthit/ClinicalDecisionSupport.pdf. Accessed March 20, 2009.

37. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009 Feb;169(3):305-11.

38. AHRQ Ambulatory Safety and Quality Program. Agency for Healthcare Research and Quality; 2009. Available at: http://www.ahrq.gov/qual/ambsafety/ambsafety.htm. Accessed April 18, 2009.

39. Kaplan B. Evaluating informatics applications—clinical decision support systems literature review. Int J Med Inform 2001 Nov;64(1):15-37.

40. Trowbridge R, Weingarten S. Clinical decision support systems. Making health care safer: a critical analysis of patient safety practices. Rockville, MD: Agency for Healthcare Research and Quality; 2001. Evidence Report/Technology Assessment, No. 43 AHRQ Publication No. 01-E058. p. 589-94.

41. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996 Nov;3(6):399-409.

42. Wolfstadt J, Gurwitz J, Field T, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008 Apr;23(4):451-8.

43. Field TS, Rochon P, Lee M, et al. Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. J Am Med Inform Assoc 2009 Apr 23;M2981.

44. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med 2005 Mar;352(10):969-77.

45. Love TE, Cebul RD, Einstadter D, et al. Electronic medical record-assisted design of a cluster-randomized trial to improve diabetes care and outcomes. J Gen Intern Med 2008 Apr;23(4):383-91.

46. Meigs JB, Cagliero E, Dubey A, et al. A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project. Diabetes Care 2003 Mar;26(3):750-7.

47. Berner ES. Testing system accuracy. In: Berner ES, ed. Clinical decision support systems: theory and practice. New York: Springer-Verlag New York, Inc.; 1999. p. 61-74.

48. Ramnarayan P, Kapoor RR, Coren M, et al. Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score. J Am Med Inform Assoc 2003 Nov;10(6):563-72.

49. Ramnarayan P, Winrow A, Coren M, et al. Diagnostic omission errors in acute paediatric practice: impact of a reminder system on decision-making. BMC Med Inform Decis Mak 2006;6:37.

50. Ramnarayan P, Roberts GC, Coren M, et al. Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study. BMC Med Inform Decis Mak 2006;6:22.

51. Berner ES, Webster GD, Shugerman AA, et al. Performance of four computer-based diagnostic systems. N Engl J Med 1994 Jun;330(25):1792-6.

52. Apkon M, Mattera JA, Lin Z, et al. A randomized outpatient trial of a decision-support information technology tool. Arch Intern Med 2005 Nov;165(20):2388-94.

53. Ozdas A, Speroff T, Waitman LR, et al. Integrating "best of care" protocols into clinicians' workflow via care provider order entry: impact on quality-of-care indicators for acute myocardial infarction. J Am Med Inform Assoc 2006 Mar;13(2):188-96.

54. Weingart SN, Toth M, Sands DZ, et al. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003 Nov;163(21):2625-31.

55. Berner ES, Maisiak RS, Heudebert GR, et al. Clinician performance and prominence of diagnoses displayed by a clinical diagnostic decision support system. AMIA Annu Symp Proc 2003;76-80.

56. Goodacre S, Webster A, Morris F. Do computer generated ECG reports improve interpretation by accident and emergency senior house officers? Postgrad Med J 2001 Jul;77(909):455-7.

57. Tierney WM, Overhage JM, Murray MD, et al. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med 2003 Dec;18(12):967-76.

58. Tierney WM, Overhage JM, Murray MD, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res 2005 Apr;40(2):477-97.

59. Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc 2008 Jul;15(4):430-8.

60. Teich JM, Merchia PR, Schmiz JL, et al. Effects of computerized physician order entry on prescribing practices. Arch Intern Med 2000 Oct;160(18):2741-7.

61. Glassman PA, Belperio PP, Simon BM, et al. Exposure to automated drug alerts over time: effects on clinicians' knowledge and perceptions. Med Care 2006 Mar;44(3):250-6.

62. Matheny ME, Sequist TD, Seger AC, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc 2008 Jul;15(4):424-9.

63. Gurwitz JH, Field TS, Rochon P, et al. Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting. J Am Geriatr Soc 2008 Dec;56(12):2225-33.

64. Thaler RH, Sunstein CR. Nudge. New Haven, CT: Yale University Press; 2008.

65. van Wyk JT, van Wijk MA, Sturkenboom MC, et al. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation 2008 Jan;117(3):371-8.

66. Rosenbloom ST, Geissbuhler AJ, Dupont WD, et al. Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care. J Am Med Inform Assoc 2005 Jul;12(4):458-73.

67. van der Sijs H, Aarts J, van Gelder T, et al. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008 Jul;15(4):439-48.

68. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003 Nov;10(6):523-30.

69. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006 Jan;13(1):5-11.

70. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009 Jan;16(1):40-6.

71. Dexter PR, Perkins S, Overhage JM, et al. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001;345(13):965-70.

72. Del Fiol G, Haug PJ, Cimino JJ, et al. Effectiveness of topic-specific infobuttons: a randomized controlled trial. J Am Med Inform Assoc 2008 Nov;15(6):752-9.

73. Maviglia SM, Yoon CS, Bates DW, et al. KnowledgeLink: impact of context-sensitive information retrieval on clinicians' information needs. J Am Med Inform Assoc 2006 Jan;13(1):67-73.

74. Knowledge Management Services. Partners Healthcare Clinical Informatics Research and Development; 2009. Available at: http://www.partners.org/cird/StaffPrj.asp?cBox=CurrProj&prAb=KMSI2. Accessed March 20, 2009.

75. Reducing and preventing adverse drug events to decrease hospital costs. Rockville, MD: Agency for Healthcare Research and Quality; March 2001. Research In Action Issue 1. AHRQ Publication Number -1-0020. 2. Available at: http://www.ahrq.gov/qual/aderia/aderia.htm. Accessed March 21, 2009.

76. Kaushal R, Jha AK, Franz C, et al. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc 2006 May;13(3):261-6.

77. Kuperman GJ, Bobb A, Payne TH, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007 Jan;14(1):29-40.

78. Classen DC, Pestotnik SL, Evans RS, et al. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997 Jan;277(4):301-6.

79. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998 Jan;338(4):232-8.

80. Pestotnik SL, Classen DC, Evans RS, et al. Implementing antibiotic practice guidelines through computer-assisted decision support: clinical and financial outcomes. Ann Intern Med 1996 May;124(10):884-90.

81. Field TS, Rochon P, Lee M, et al. Costs associated with developing and implementing a computerized clinical decision support system for medication dosing for patients with renal insufficiency in the long-term care setting. J Am Med Inform Assoc 2008 Jul;15(4):466-72.

82. Poissant L, Pereira J, Tamblyn R, et al. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005 Sep;12(5):505-16.

83. Maviglia SM, Zielstorff RD, Paterno M, et al. Automating complex guidelines for chronic disease: lessons learned. J Am Med Inform Assoc 2003 Mar;10(2):154-65.

84. Lorenzi NM, Novak LL, Weiss JB, et al. Crossing the implementation chasm: a proposal for bold action. J Am Med Inform Assoc 2008 May;15(3):290-6.

85. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005 Mar;293(10):1197-203.

86. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005 Dec;116(6):1506-12.

87. Tsai TL, Fridsma DB, Gatti G. Computer decision support as a source of interpretation error: the case of electrocardiograms. J Am Med Inform Assoc 2003;10:478-83.

88. Ash JS, Sittig DF, Poon EG, et al. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4):415-23.

89. Weiner JP, Kfuri T, Chan K, et al. "e-Iatrogenesis": the most critical unintended consequence of CPOE and other HIT. J Am Med Inform Assoc 2007 May;14(3):387-8.

90. Ammenwerth E, Talmon J, Ash JS, et al. Impact of CPOE on mortality rates—contradictory findings, important messages. Methods Inf Med 2006;45(6):586-93.

91. Sittig DF, Ash JS, Zhang J, et al. Lessons from "Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system." Pediatrics 2006 Aug;118(2):797-801.

92. Del Beccaro MA, Jeffries HE, Eisenberg MA, et al. Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit. Pediatrics 2006 Jul 1;118(1):290-5.

93. Sentinel event alert: safety implementing health information and converging technologies. The Joint Commission Issue 42; 2008 December 11. Available at: http://www.jointcommission.org/SentinelEvents/SentinelEventAlert/sea_42.htm. Accessed January 11, 2009.

94. Berg M. Rationalizing medical work: decision-support techniques and medical practices. Cambridge, MA: MIT Press; 1997.

95. Karsh BT. Clinical practice improvement and redesign: how change in workflow can be supported by CDS. AHRQ White Papers. In press 2009.

96. Miller RA, Waitman LR, Chen S, et al. The anatomy of decision support during inpatient care provider order entry (CPOE): empirical observations from a decade of CPOE experience at Vanderbilt. J Biomed Inform 2005 Dec;38(6):469-85.

97. Healthcare Information Technology Standards Panel (HITSP). Healthcare Information Technology Standards Panel; 2009. Available at: www.hitsp.org. Accessed March 20, 2009.

98. Draft roadmap for expansion of health IT certification. Certification Commission for Healthcare Information Technology; 2009 January 14. Available at: http://www.cchit.org/files/Expansion/CCHITExpansionRoadmapDRAFT20090115.pdf. Accessed March 20, 2009.

99. Metzger J, Welebob E, Turisco F, et al. The Leapfrog Group's CPOE standard and evaluation tool. Patient Safety and Quality Healthcare; 2008 July. Available at: http://www.psqh.com/julaug08/cpoe.html. Accessed March 20, 2009.

100. Clinical decision support work group. Health Level Seven, Inc.: 2009. Available at: http://www.hl7.org/. Accessed March 20, 2009.

101. Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004 Nov;11(6):482-91.

102. Berner ES. Ethical and legal issues in the use of clinical decision support systems. J Healthc Inf Manag 2002;16(4):34-7.

103. Berner ES. Ethical and legal issues in the use of health information technology to improve patient safety. HEC Forum 2008 Sep;20(3):243-58.

104. Clinical decision support initiative. Agency for Healthcare Research and Quality; 2009. Available at: http://healthit.ahrq.gov/portal/server.pt?open=512&objID=654&&PageID=13665&mode=2&in_hi_userid=3882&cached=true. Accessed January 11, 2009.

105. Wright A, Sittig DF. SANDS: A service-oriented architecture for clinical decision support in a National Health Information Network. J Biomed Inform 2008 Dec;41(6):962-81.

106. Miller RA, Gardner RM, Johnson KB, et al. Clinical decision support and electronic prescribing systems: a time for responsible thought and action. J Am Med Inform Assoc 2005 Jul;12(4):403-9.

107. Greenes RA. Clinical decision support: the road ahead. Burlington, MA: Elsevier; 2007.

108. Varonen H, Kortteisto T, Kaila M, for the EBMeDS Study Group. What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians. Fam Pract 2008 Jun;25(3):162-7.

109. Sittig D, Krall M, Dykstra R, et al. A survey of factors affecting clinician acceptance of clinical decision support. BMC Med Inform Decis Mak 2006;6(1):6.

110. Crenner CW. Introduction of the blood pressure cuff into U.S. medical practice: technology and skilled practice. Ann Intern Med 1998 Mar;128(6):488-93.

111. Friedman CP. "Smallball" evaluation: a prescription for studying community-based information interventions. J Med Libr Assoc 2005 Oct;93(4 Suppl):S43-S48.

112. Medicare Improvements for Patients and Providers Act of 2008, H.R. 6331: 2008.

113. The American Recovery and Reinvestment Act of 2009, H.R.1: 2009.

114. Casebeer L, Bennett N, Kristofco R, et al. Physician Internet medical information seeking and on-line continuing education use patterns. J Contin Educ Health Prof 2002;22(1):33-42.

*Recent consensus definitions have made a distinction between EHRs (electronic health records,

records that span organizations), and EMRs (electronic medical records that contain information from a single organization).⁶ As technology develops there will likely be more development of EHRs, but since EMRs are more common today the term EMR will be used throughout this paper.