Evidence Report/Technology Assessment

Number 43

Prepared for:
Agency for Healthcare Research and Quality

U.S. Department of Health and Human Services
2101 East Jefferson Street
Rockville, MD 20852
http://www.ahrq.gov/

Contract No. 290-97-0013

Prepared by:
University of California at San Francisco (UCSF)-Stanford University
Evidence-based Practice Center

Editorial Board
Kaveh G. Shojania, M.D. (UCSF)
Bradford W. Duncan, M.D. (Stanford)
Kathryn M. McDonald, M.M. (Stanford)
Robert M. Wachter, M.D. (UCSF)

Managing Editor
Amy J. Markowitz, JD

Robert M. Wachter, MD
Project Director
Kathryn M. McDonald, MM
UCSF-Stanford EPC Coordinator

AHRQ Publication No. 01-E058

July 20, 2001 (Revised printing, indexed)

On December 6, 1999, under Public Law 106-129, the Agency for Health Care Policy and Research (AHCPR) was reauthorized and renamed the Agency for Healthcare Research and Quality (AHRQ). The law authorizes AHRQ to continue its research on the cost, quality, and outcomes of health care, and expands its role to improve patient safety and address medical errors.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

The Agency for Healthcare Research and Quality (AHRQ), formerly the Agency for Health Care Policy and Research (AHCPR), through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.

To bring the broadest range of experts into the development of evidence reports and health technology assessments, AHRQ encourages the EPCs to form partnerships and enter into collaborations with other medical and research organizations. The EPCs work with these partner organizations to ensure that the evidence reports and technology assessments they produce will become building blocks for health care quality improvement projects throughout the Nation. The reports undergo peer review prior to their release.

AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the health care system as a whole by providing important information to help improve health care quality.

We welcome written comments on this evidence report. They may be sent to: Director, Center for Practice and Technology Assessment, Agency for Healthcare Research and Quality, 6010 Executive Blvd., Suite 300, Rockville, MD 20852.

John M. Eisenberg, M.D.	Douglas B. Kamerow, M.D.
Director	Director, Center for Practice and
Agency for Healthcare	Technology Assessment
Research and Quality	Agency for Healthcare
	Research and Quality

The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services of a particular drug, device, test, treatment, or other clinical service.

Objectives: Patient safety has received increased attention in recent years, but mostly with a focus on the epidemiology of errors and adverse events, rather than on practices that reduce such events. This project aimed to collect and critically review the existing evidence on practices relevant to improving patient safety.

Search Strategy and Selection Criteria: Patient safety practices were defined as those that reduce the risk of adverse events related to exposure to medical care across a range of diagnoses or conditions. Potential patient safety practices were identified based on preliminary surveys of the literature and expert consultation. This process resulted in the identification of 79 practices for review. The practices focused primarily on hospitalized patients, but some involved nursing home or ambulatory patients. Protocols specified the inclusion criteria for studies and the structure for evaluation of the evidence regarding each practice. Pertinent studies were identified using various bibliographic databases (e.g., MEDLINE, PsycINFO, ABI/INFORM, INSPEC), targeted searches of the Internet, and communication with relevant experts.

Data Collection and Analysis: Included literature consisted of controlled observational studies, clinical trials and systematic reviews found in the peer-reviewed medical literature, relevant non-health care literature and "gray literature." For most practices, the project team required that the primary outcome consist of a clinical endpoint (i.e., some measure of morbidity or mortality) or a surrogate outcome with a clear connection to patient morbidity or mortality. This criterion was relaxed for some practices drawn from the non-health care literature. The evidence supporting each practice was summarized using a prospectively determined format. The project team then used a predefined consensus technique to rank the practices according to the strength of evidence presented in practice summaries. A separate ranking was developed for research priorities.

Main Results: Practices with the strongest supporting evidence are generally clinical interventions that decrease the risks associated with hospitalization, critical care, or surgery. Many patient safety practices drawn primarily from nonmedical fields (e.g., use of simulators, bar coding, computerized physician order entry, crew resource management) deserve additional research to elucidate their value in the health care environment. The following 11 practices were rated most highly in terms of strength of the evidence supporting more widespread implementation.

• Appropriate use of prophylaxis to prevent venous thromboembolism in patients at risk;
• Use of perioperative beta-blockers in appropriate patients to prevent perioperative morbidity and mortality;
• Use of maximum sterile barriers while placing central intravenous catheters to prevent infections;
• Appropriate use of antibiotic prophylaxis in surgical patients to prevent postoperative infections;
• Asking that patients recall and restate what they have been told during the informed consent process;
• Continuous aspiration of subglottic secretions (CASS) to prevent ventilator-associated pneumonia;
• Use of pressure relieving bedding materials to prevent pressure ulcers;
• Use of real-time ultrasound guidance during central line insertion to prevent complications;
• Patient self-management for warfarin (Coumadin^TM) to achieve appropriate outpatient anticoagulation and prevent complications;
• Appropriate provision of nutrition, with a particular emphasis on early enteral nutrition in critically ill and surgical patients; and
• Use of antibiotic-impregnated central venous catheters to prevent catheter-related infections.

Conclusions: An evidence-based approach can help identify practices that are likely to improve patient safety. Such practices target a diverse array of safety problems. Further research is needed to fill the substantial gaps in the evidentiary base, particularly with regard to the generalizability of patient safety practices heretofore tested only in limited settings and to promising practices drawn from industries outside of health care.

This document is in the public domain and may be used and reprinted without permission except those copyrighted materials noted for which further reproduction is prohibited without the specific permission of copyright holders.

Suggested Citation: Shojania KG, Duncan BW, McDonald KM, et al., eds. Making Health Care Safer: A Critical Analysis of Patient Safety Practices. Evidence Report/Technology Assessment No. 43 (Prepared by the University of California at San Francisco-Stanford Evidence-based Practice Center under Contract No. 290-97-0013), AHRQ Publication No. 01-E058, Rockville, MD: Agency for Healthcare Research and Quality. July 2001.

Patient safety has become a major concern of the general public and of policymakers at the State and Federal levels. This interest has been fueled, in part, by news coverage of individuals who were the victims of serious medical errors and by the publication in 1999 of the Institute of Medicine's (IOM's) report To Err is Human: Building a Safer Health System. In its report, IOM highlighted the risks of medical care in the United States and shocked the sensibilities of many Americans, in large part through its estimates of the magnitude of medical-errors-related deaths (44,000 to 98,000 deaths per year) and other serious adverse events. The report prompted a number of legislative and regulatory initiatives designed to document errors and begin the search for solutions. But Americans, who now wondered whether their next doctor's or hospital visit might harm rather than help them, began to demand concerted action.

Three months after publication of the IOM report, an interagency Federal government group, the Quality Interagency Coordination Task Force (QuIC), released its response, Doing What Counts for Patient Safety: Federal Actions to Reduce Medical Errors and Their Impact. That report, prepared at the President's request, both inventoried on-going Federal actions to reduce medical errors and listed more than 100 action items to be undertaken by Federal agencies.

An action promised by the Agency for Healthcare Research and Quality (AHRQ), the Federal agency leading efforts to research and promote patient safety, was "the development and dissemination of evidence-based, best safety practices to provider organizations." To initiate the work to be done in fulfilling this promise, AHRQ commissioned the University of California at San Francisco (UCSF) - Stanford University Evidence-based Practice Center (EPC) in January 2001 to review the scientific literature regarding safety improvement. To accomplish this, the EPC established an Editorial Board that oversaw development of this report by teams of content experts who served as authors.

Working closely with AHRQ and the National Forum for Quality Measurement and Reporting (the National Quality Forum, or NQF) -- a public-private partnership formed in 1999 to promote a national health care quality agenda the EPC began its work by defining a patient safety practice as a type of process or structure whose application reduces the probability of adverse events resulting from exposure to the health care system across a range of diseases and procedures.

This definition is consistent with the dominant conceptual framework in patient safety, which holds that systemic change will be far more productive in reducing medical errors than will targeting and punishing individual providers. The definition's focus on actions that cut across diseases and procedures also allowed the research team to distinguish patient safety activities from the more targeted quality improvement practices (e.g., practices designed to increase the use of beta-blockers in patients who are admitted to the hospital after having a myocardial infarction). The editors recognize, however, that this distinction is imprecise.

This evidence-based review also focuses on hospital care as a starting point because the risks associated with hospitalization are significant, the strategies for improvement are better documented there than in other health care settings, and the importance of patient trust is paramount. The report, however, also considers evidence regarding other sites of care, such as nursing homes, ambulatory care, and patient self-management.

The results of this EPC study will be used by the NQF to identify a set of proven patient safety practices that should be used by hospitals. Identification of these practices by NQF will allow patients throughout the nation to evaluate the actions their hospitals and/or health care facilities have taken to improve safety.

As is typical for evidence-based reviews, the goal was to provide a critical appraisal of the evidence on the topic. This information would then be available to others to ensure that no practice unsupported by evidence would be endorsed and that no practice substantiated by a high level of proof would lack endorsement. Readers familiar with the state of the evidence regarding quality improvement in areas of health care where this has been a research priority (e.g., cardiovascular care) may be surprised and even disappointed, by the paucity of high quality evidence in other areas of health care for many patient safety practices. One reason for this is the relative youth of the field. Just as there had been little public recognition of the risks of health care prior to the first IOM report, there has been relatively little attention paid to such risks - and strategies to mitigate them - among health professionals and researchers.

Moreover, there are a number of methodologic reasons why research in patient safety is particularly challenging. Many practices (e.g., the presence of computerized physician order entry systems, modifying nurse staffing levels) cannot be the subject of double-blind studies because their use is evident to the participants. Second, capturing all relevant outcomes, including "near misses" (such as a nurse catching an excessive dosage of a drug just before it is administered to a patient) and actual harm, is often very difficult. Third, many effective practices are multidimensional, and sorting out precisely which part of the intervention works is often quite challenging. Fourth, many of the patient safety problems that generate the most concern (wrong-site surgery, for example) are uncommon enough that demonstrating the success of a "safety practice" in a statistically meaningful manner with respect to outcomes is all but impossible.

Finally, establishing firm epidemiologic links between presumed (and accepted) causes and adverse events is critical, and frequently difficult. For instance, in studying an intuitively plausible "risk factor" for errors, such as "fatigue," analyses of errors commonly reveal the presence of fatigued providers (because many health care providers work long hours and/or late at night). The question is whether or not fatigue is over-represented among situations that lead to errors. The point is not that the problem of long work-hours should be ignored, but rather that strong epidemiologic methods need to be applied before concluding that an intuitive cause of errors is, in fact, causal.

Researchers now believe that most medical errors cannot be prevented by perfecting the technical work of individual doctors, nurses, or pharmacists. Improving patient safety often involves the coordinated efforts of multiple members of the health care team, who may adopt strategies from outside health care. The report reviews several practices whose evidence came from the domains of commercial aviation, nuclear safety, and aerospace, and the disciplines of human factors engineering and organizational theory. Such practices include root cause analysis, computerized physician order entry and decision support, automated medication dispensing systems, bar coding technology, aviation-style preoperative checklists, promoting a "culture of safety," crew resource management, the use of simulators in training, and integrating human factors theory into the design of medical devices and alarms. In reviewing these practices, the research team sought to be flexible regarding standards of evidence, and included research evidence that would not have been considered for medical interventions. For example, the randomized trial that is appropriately hailed as the "gold standard" in clinical medicine is not used in aviation, as this design would not capture all relevant information. Instead, detailed case studies and industrial engineering research approaches are utilized.

To facilitate identification and evaluation of potential patient safety practices, the Editorial Board divided the content for the project into different domains. Some cover "content areas," including traditional clinical areas such as adverse drug events, nosocomial infections, and complications of surgery, but also less traditional areas such as fatigue and information transfer. Other domains consist of practices drawn from broad (primarily nonmedical) disciplines likely to contain promising approaches to improving patient safety (e.g., information technology, human factors research, organizational theory). Once this list was created -- with significant input from patient safety experts, clinician-researchers, AHRQ, and the NQF Safe Practices Committee -- the editors selected teams of authors with expertise in the relevant subject matter and/or familiarity with the techniques of evidence-based review and technology appraisal.

The authors were given explicit instructions regarding search strategies for identifying safety practices for evaluation (including explicit inclusion and exclusion criteria) and criteria for assessing each practice's level of evidence for efficacy or effectiveness in terms of study design and study outcomes. Some safety practices did not meet the inclusion criteria because of the paucity of evidence regarding efficacy or effectiveness but were included in the report because an informed reader might reasonably expect them to be evaluated or because of the depth of public and professional interest in them. For such high profile topics (such as bar coding to prevent misidentifications), the researchers tried to fairly present the practice's background, the experience with the practice thus far, and the evidence (and gaps in the evidence) regarding the practice's value.

For each practice, authors were instructed to research the literature for information on:

• prevalence of the problem targeted by the practice;
• severity of the problem targeted by the practice;
• the current utilization of the practice;
• evidence on efficacy and/or effectiveness of the practice;
• the practice's potential for harm;
• data on cost, if available; and
• implementation issues.

The report presents the salient elements of each included study (e.g., study design, population/setting, intervention details, results), and highlights any important weaknesses and biases of these studies. Authors were not asked to formally synthesize or combine the evidence across studies (e.g., perform a meta-analysis) as part of their task.

The Editorial Board and the Advisory Panel reviewed the list of domains and practices to identify gaps in coverage. Submitted chapters were reviewed by the Editorial Board and revised by the authors, aided by feedback from the Advisory Panel. Once the content was finalized, the editors analyzed and ranked the practices using a methodology summarized below.

Because the report is essentially an anthology of a diverse and extensive group of patient safety practices with highly variable relevant evidence, synthesizing the findings was challenging, but necessary to help readers use the information. Two of the most obvious uses for this report are: 1) to inform efforts of providers and health care organizations to improve the safety of the care they provide, and 2) to inform AHRQ, other research agencies, and foundations about potential fruitful investments for their research support. Other uses of the information are likely. In fact, the National Quality Forum plans to use this report to help identify a list of patient safety practices that consumers and others should know about as they choose among the health care provider organizations to which they have access.

In an effort to assist both health care organizations interested in taking substantive actions to improve patient safety and research funders seeking to spend scarce resources wisely, AHRQ asked the EPC to rate the evidence and rank the practices by opportunity for safety improvement and by research priority. This report, therefore, contains two lists.

To create these lists, the editors aimed to separate the practices that are most promising or effective from those that are least so on a range of dimensions, without implying any ability to calibrate a finely gradated scale for those practices in between. The editors also sought to present the ratings in an organized, accessible way while highlighting the limitations inherent in their rating schema. Proper metrics for more precise comparisons (e.g., cost-effectiveness analysis) require more data than are currently available in the literature.

Three major categories of information were gathered to inform the rating exercise:

• Potential Impact of the Practice: based on prevalence and severity of the patient safety target, and current utilization of the practice;
• Strength of the Evidence Supporting the Practice: including an assessment of the relative weight of the evidence, effect size, and need for vigilance to reduce any potential negative collateral effects of the practice; and
  • Implementation: considering costs, logistical barriers, and policy issues.

For all of these data inputs into the practice ratings, the primary goal was to find the best available evidence from publications and other sources. Because the literature has not been previously organized with an eye toward addressing each of these areas, most of the estimates could be improved with further research, and some are informed by only general and somewhat speculative knowledge. In the summaries, the editors have attempted to highlight those assessments made with limited data.

The four-person editorial team independently rated each of the 79 practices using general scores (e.g., High, Medium, Low) for a number of dimensions, including those italicized in the section above. The editorial team convened for 3 days in June, 2001 to compare scores, discuss disparities, and come to consensus about ratings for each category.

In addition, each member of the team considered the totality of information on potential impact and support for a practice to score each of these factors on a 0 to 10 scale (creating a "Strength of the Evidence" list). For these ratings, the editors took the perspective of a leader of a large health care enterprise (e.g., a hospital or integrated delivery system) and asked the question, "If I wanted to improve patient safety at my institution over the next 3 years and resources were not a significant consideration, how would I grade this practice?" For this rating, the Editorial Board explicitly chose not to formally consider the difficulty or cost of implementation in the rating. Rather, the rating simply reflected the strength of the evidence regarding the effectiveness of the practice and the probable impact of its implementation on reducing adverse events related to health care exposure. If the patient safety target was rated as "High" impact and there was compelling evidence (i.e., "High" relative study strength) that a particular practice could significantly reduce (e.g., "Robust" effect size) the negative consequences of exposure to the health care system (e.g., hospital-acquired infections), raters were likely to score the practice close to 10. If the studies were less convincing, the effect size was less robust, or there was a need for a "Medium" or "High" degree of vigilance because of potential harms, then the rating would be lower.

At the same time, the editors also rated the usefulness of conducting more research on each practice, emphasizing whether there appeared to be questions that a research program might have a reasonable chance of addressing successfully (creating a "Research Priority" list). Here, they asked themselves, "If I were the leader of a large agency or foundation committed to improving patient safety, and were considering allocating funds to promote additional research, how would I grade this practice?" If there was a simple gap in the evidence that could be addressed by a research study or if the practice was multifaceted and implementation could be eased by determining the specific elements that were effective, then the research priority was high. (For this reason, some practices are highly rated on both the "Strength of the Evidence" and "Research Priority" lists.) If the area was one of high potential impact (i.e., large number of patients at risk for morbid or mortal adverse events) and a practice had been inadequately researched, then it would also receive a relatively high rating for research need. Practices might receive low research scores if they held little promise (e.g., relatively few patients are affected by the safety problem addressed by the practice or a significant body of knowledge already demonstrates the practice's lack of utility). Conversely, a practice that was clearly effective, low cost, and easy to implement would not require further research and would also receive low research scores.

In rating both the strength of the evidence and the research priority, the purpose was not to report precise 0 to 10 scores, but to develop general "zones" or practice groupings. This is important because better methods are available for making comparative ratings when the data inputs are available. The relative paucity of the evidence dissuaded the editors from using a more precise, sophisticated, but ultimately unfeasible, approach.

The following 11 patient safety practices were the most highly rated (of the 79 practices reviewed in detail in the full report and ranked in the Executive Summary Addendum, AHRQ Publication No. 01-E057b) in terms of strength of the evidence supporting more widespread implementation. Practices appear in descending order, with the most highly rated practices listed first. Because of the imprecision of the ratings, the editors did not further divide the practices, nor indicate where there were ties.

• Appropriate use of prophylaxis to prevent venous thromboembolism in patients at risk;
• Use of perioperative beta-blockers in appropriate patients to prevent perioperative morbidity and mortality;
• Use of maximum sterile barriers while placing central intravenous catheters to prevent infections;
• Appropriate use of antibiotic prophylaxis in surgical patients to prevent perioperative infections;
• Asking that patients recall and restate what they have been told during the informed consent process;
• Continuous aspiration of subglottic secretions (CASS) to prevent ventilator-associated pneumonia;
• Use of pressure relieving bedding materials to prevent pressure ulcers;
• Use of real-time ultrasound guidance during central line insertion to prevent complications;
• Patient self-management for warfarin (Coumadin^TM) to achieve appropriate outpatient anticoagulation and prevent complications;
• Appropriate provision of nutrition, with a particular emphasis on early enteral nutrition in critically ill and surgical patients; and
• Use of antibiotic-impregnated central venous catheters to prevent catheter-related infections.

This list is generally weighted toward clinical rather than organizational matters, and toward care of the very, rather than the mildly or chronically ill. Although more than a dozen practices considered were general safety practices that have been the focus of patient safety experts for decades (i.e., computerized physician order entry, simulators, creating a "culture of safety," crew resource management), most research on patient safety has focused on more clinical areas. The potential application of practices drawn from outside health care has excited the patient safety community, and many such practices have apparent validity. However, clinical research has been promoted by the significant resources applied to it through Federal, foundation, and industry support. Since this study went where the evidence took it, more clinical practices rose to the top as potentially ready for implementation.

Until recently, patient safety research has had few champions, and even fewer champions with resources to bring to bear. The recent initiatives from AHRQ and other funders are a promising shift in this historical situation, and should yield important benefits.

In terms of the research agenda for patient safety, the following 12 practices rated most highly, as follows:

• Improved perioperative glucose control to decrease perioperative infections;
• Localizing specific surgeries and procedures to high volume centers;
• Use of supplemental perioperative oxygen to decrease perioperative infections;
• Changes in nursing staffing to decrease overall hospital morbidity and mortality;
• Use of silver alloy-coated urinary catheters to prevent urinary tract infections;
• Computerized physician order entry with computerized decision support systems to decrease medication errors and adverse events primarily due to the drug ordering process;
• Limitations placed on antibiotic use to prevent hospital-acquired infections due to antibiotic-resistant organisms;
• Appropriate use of antibiotic prophylaxis in surgical patients to prevent perioperative infections;
• Appropriate use of prophylaxis to prevent venous thromboembolism in patients at risk;
• Appropriate provision of nutrition, with a particular emphasis on early enteral nutrition in critically ill and post-surgical patients;
• Use of analgesics in the patient with an acutely painful abdomen without compromising diagnostic accuracy; and
• Improved handwashing compliance (via education/behavior change; sink technology and placement; or the use of antimicrobial washing substances).

Of course, the vast majority of the 79 practices covered in this report would benefit from additional research. In particular, some practices with longstanding success outside of medicine (e.g., promoting a culture of safety) deserve further analysis, but were not explicitly ranked due to their unique nature and the present weakness of the evidentiary base in the health care literature.

This report represents a first effort to approach the field of patient safety through the lens of evidence-based medicine. Just as To Err is Human sounded a national alarm regarding patient safety and catalyzed other important commentaries regarding this vital problem, this review seeks to plant a seed for future implementation and research by organizing and evaluating the relevant literature. Although all those involved tried hard to include all relevant practices and to review all pertinent evidence, inevitably some of both were missed. Moreover, the effort to grade and rank practices, many of which have only the beginnings of an evidentiary base, was admittedly ambitious and challenging. It is hoped that this report provides a template for future clinicians, researchers, and policy makers as they extend, and inevitably improve upon, this work.

In the detailed reviews of the practices, the editors have tried to define (to the extent possible from the literature) the associated costs -- financial, operational, and political. However, these considerations were not factored into the summary ratings, nor were judgments made regarding the appropriate expenditures to improve safety. Such judgments, which involve complex tradeoffs between public dollars and private ones, and between saving lives by improving patient safety versus doing so by investing in other health care or non-health care practices, will obviously be critical. However, the public reaction to the IOM report, and the media and legislative responses that followed it, seem to indicate that Americans are highly concerned about the risks of medical errors and would welcome public and private investment to decrease them. It seems logical to infer that Americans value safety during a hospitalization just as highly as safety during a transcontinental flight.