Advances in Patient Safety: From Research to Implementation Volume 4. Programs, Tools, and ProductsAHRQ WebM&M
Online Medical Error Reporting and AnalysisRobert M. Wachter, Kaveh G. Shojania, Tracy Minichiello, Scott A. Flanders, Erin E. Hartman
AbstractThe AHRQ WebM&M Web site represents an unprecedented effort to publish illustrative cases of confidentially-reported medical errors on the Internet, accompanied by straightforward evidence-based expert commentaries. Modeled on hospital morbidity and mortality conferences, five cases are posted each month to illustrate diverse patient safety issues. The Web site has become a popular source for medical error case information and has garnered positive feedback. As of March 11, 2005, 296 cases had been submitted, and 90 had been posted on the site. Twenty-four percent of the cases appearing on AHRQ WebM&M resulted in death or permanent disability. The site had 9,767 registered users and 663 unique visitors daily; the average visitor stayed for 12 minutes. Responses to a May 2004 user survey indicated that visitors were divided almost equally between providers (half nurses and half physicians) and nonproviders with an interest in safety. Seventy-five percent of users rated the educational value of the site as "excellent"; virtually all the others rated it as "good." Similar response rates were tallied for questions regarding practical value, patient safety content, cases, commentaries, and continuing education. These results demonstrate a willingness on the part of providers to report medical errors under favorable circumstances, as well as a strong demand among health care professionals for Internet-based information pertaining to patient safety. Thus, AHRQ WebM&M represents one of the modern era's most successful experiments in patient safety reporting and education.
Hospitals, particularly teaching hospitals, have a long tradition of discussing complications of care and medical errors in a forum known as the morbidity and mortality (M&M) conference. The content of such conferences traditionally has been protected from legal disclosure to foster an environment in which providers can review their mistakes honestly and highlight general lessons learned from them. 1, 2 Although concerns over the adequacy of medicolegal protection linger, 3 the M&M conference, at its best, presents a unique opportunity for health care providers to learn from their errors.
As the focus on medical errors and patient safety expanded in the wake of the 2000 Institute of Medicine report, To Err Is Human,
4
the limitations of M&M conferences became more visible. First, relatively few nonteaching hospitals host them on a regular basis. Second, even in major teaching hospitals, such conferences often are conducted by the larger clinical departments (e.g., medicine and surgery), while the participation of physicians in other specialties (e.g., radiology, psychiatry, pediatrics) has not been as great. Third, although many errors are caused by systemic failures or breakdowns in teamwork, nurses and other nonphysician providers and hospital administrators rarely participate in M&M conferences. Finally, when M&M conferences do take place, they often miss the markeither failing to identify errors as such (more often a problem in internal medicine) or by focusing so narrowly on individual culpability that potential systems issues are not considered (more often in surgery).
5
7
The growing interest in medical errors also has led to increasing public and professional demands for error reporting. With this push has come a challenge: How can error reporting be leveraged to provide general lessons for providers and institutions? Publications such as the Joint Commission on Accreditation of Healthcare Organizations' Sentinel Event Alerts 8 highlight lessons from commonly reported errors and serve a vital function at health care institutions. Publications from regulators, however, tend to target an audience of safety and quality professionals, rather than a general audience of clinicians. And while some medical journals do occasionally feature patient safety articles (including the case-based "Quality Grand Rounds" series that we [RMW, KGS] edit for the Annals of Internal Medicine 9 ), those discussions are particularly comprehensive and the cases are generated by the editors, rather than readers. Moreover, the Annals is read mostly by internists, and is available only to paid subscribers.
In the late 1990s, the Agency for Healthcare Research and Quality (AHRQ), under the leadership of the late Dr. John Eisenberg, recognized in all of the above an opportunity to create a resource that would marry the best of the local M&M conference with a confidential national reporting system. In bringing this vision to reality, AHRQ appreciated that the World Wide Web could facilitate easy, anonymous reporting (by anyone, from anywhere) and the efficient production of a journal that could be made available worldwide at relatively low cost. This vision led to a Request for Applications (RFA) in February 2001, to "develop, implement, maintain, and assess a national Web-based morbidity and mortality conference site" under an AHRQ contract.
Our teamhospitalist physicians with a strong interest in patient safety and medical educationpartnered with the health care quality/technology company, DoctorQuality, and were awarded the contract, with a start date of September 2001. Several months later, a managing editor (EEH) was recruited to supervise the editorial office and publication of the electronic journal.
The AHRQ WebM&M (www.webmm.ahrq.gov) has since developed into a national Web-based learning program for health care providers. Modeled after M&M conferences, the site represents an unprecedented effort to publish interesting and illustrative cases of confidentially-reported medical errors on the Web, accompanied by straightforward, evidence-based expert commentaries. It also represents an important element of the national movement to promote "blame-free" medical error reporting and stimulate open discussions of patient safety among practicing physicians, educators, and trainees. In this article, we will describe the development and evolution of WebM&M, some of the key outcome measures, and a few related successes and challenges. Because WebM&M serves as both an educational vehicle and a reporting system, we hope that our experiences will impart some general lessons that can be applied across the entire field of patient safety.
Although the editors and AHRQ agreed in principal on the vision for the electronic publication, some of the RFA's original stipulations were modified after early discussions and experiences (Table 1). The original RFA, for example, called for posted cases to be limited to "near misses" (i.e., errors that are detected and corrected before causing patient harm), largely because of medicolegal concerns. These restrictions later were relaxed, and we began to include errors that may have reached the patient but caused no lasting harm (e.g., a medication error that led to an unanticipated intensive care unit [ICU] stay but ultimate recovery).
10
12
After a few months of anonymous submissions delivered through the site, the editorial leadership realized that about one-third of the submitted cases (many of them quite instructive) were being rejected because the error resulted in lasting harm or death to the patient. In light of the Web site's robust security, privacy, and anonymity protections, the selection criteria were further relaxed and the site began to host the full spectrum of medical errors, including those resulting in permanent harm.
| Vision | Challenge or opportunity | Modifications |
| The site will be easy to use and graphically pleasing. | Ensuring full compliance with Federal regulations regarding Web sites. | Major focus on graphic appeal and functionality design and testing. |
| Cases will be of "near misses." | Ensuring confidentiality (in terms of patient, providers, and institution) while hosting interesting, illustrative cases. | Loosened restriction to include "no-harm events" and, ultimately, full spectrum of errors. Major focus on ensuring confidentiality. |
| Cases accompanied by detailed root cause analyses. | Balance between patient safety lessons and desire to recruit a nonexpert readership. | Choice to use brief case presentations, relatively brief (1000 word) commentaries; avoid jargon. Hope that readers learn principles of patient safety through an approach they are comfortable with. When new terms introduced, include in a glossary to promote learning. |
| Recruit a broad audience interested, but not expert, in patient safety. | Added CME function to accompany each monthly "Spotlight Case" (with a broader analysis and a downloadable slideshow). Significant marketing effort focused on relevant specialty societies. Easy and non-intrusive site registration informs readers of new issues. Promote media coverage of site. | |
| Take advantage of the capability of the Web. | Hope to generate interactivity and a "users' community;" also use the multimedia potential. | The Web's ease of use and access has been a huge plus. We have begun to host videos (e.g., demonstrating a surgical simulation), in addition to the "Spotlight Cases" slideshowthese have been among our most popular features. Our attempt to create an active users' forum has been disappointing, with relatively few postings. |
| Generate a diverse, interesting array of cases illustrating the full range of patient safety problems. | We recognized early that too many cases focusing on medication errors, or "systems thinking," would get in the way of a growing, sustained readership. | To encourage reporting (since no academic credits or bylines are available), we pay a small honorarium to successful case submitters (through a third-party payer, to create an arms-length relationship and ensure anonymity). The number and breadth of submissions has been sufficient to fuel the site, but could be greater. |
The original vision called for the site to focus on five clinical specialties: medicine, surgery, obstetrics-gynecology, pediatrics, and psychiatry. In practice, we found that the overwhelming majority of case submissions came from the fields of medicine and surgery; fewer came from pediatrics and obstetrics-gynecology, and almost none concerned psychiatry. Certain other specialties, including emergency medicine and radiology, were also represented. Moreover, although the RFA initially targeted a physician audience (with consideration of a related nursing-targeted site to follow), we found that our readership included many nurses, pharmacists, and others. This discovery led us to broaden the specialty categories beginning with the publication of our July 2003 (fifth) issue, and the addition of one or two "swing slot" cases devoted to a variety of other medical specialties (e.g., laboratory medicine and radiology) and related topics (e.g., nursing and clinical ethics).
Cases are selected carefully to illustrate a compelling array of patient safety issues and clinical situations. The editors review each submission using criteria such as clinical interest, patient safety interest, systems focus, and novelty to select cases for publication. Members of the editorial board (including experts in clinical fields such as obstetrics-gynecology and safety disciplines such as human factors and informatics) are consulted with specific questions, for example, whether a given clinical scenario is credible or whether a particular patient safety issue is of great importance within a given specialty. One noteworthy case is selected each month for an extended learning module, named the "Spotlight Cases." Spotlight cases are chosen on the basis of their broad appeal across specialties and their excellent teaching value. Keeping in mind the importance of diverse authorship, expertise, and institutional representation, the commentary authors for each of the cases are chosen on the basis of their publication track record in the relevant clinical or patient safety domains.
We recognized that the site could be only as good as the quality of the cases we received. With that in mind, we took several steps to encourage readers to submit cases. First, we ensure the confidentiality of all case summaries and their reporters, facilities, and patients. Second, we promote case submission widely, using notices in monthly e-mail messages sent to registered users, and through our advisory panel and editorial board. Third, those who submit cases accepted for publication are paid a small honorarium. Finally, the submission process is simple, allowing users to describe the events of the case in their own words, much as they might relate them to a colleague. In contrast to many incident reporting systems, we do not ask users to categorize the type of event, the severity of injury, or to supply other details related to incident taxonomies. Our decision to structure the reporting format in this manner was based on the concern that such detailed questions would represent a significant barrier to participation. The editors have the means to contact the individual submitting a case, in the event that clarifications are needed or key details have been omitted, while at the same time preserving the submitter's anonymity.
A second key feature of the site is the quality of the commentaries. We have been pleased with the commentators and their willingness to participate, given that WebM&M does not yet offer MEDLINE![[reg]](https://webarchive.library.unt.edu/eot2008/20090830213021im_/http://www.ncbi.nlm.nih.gov/books/bookres.fcgi/system_default/reg.gif)
citation (we are working on this) and the fact that we often require a very short turnaround time. Our commentators have included many of the world's foremost authorities on patient safety research and practice. The WebM&M editors line edit the commentaries, when necessary, to achieve a consistent length, style, and level of accessibility. In turn, the commentators receive a modest honorarium as compensation for their time.
Our efforts to promote interactivity include the development of an easy-to-use "Forum," in which readers can post their own comments regarding the cases. In addition, we have made the "Spotlight" slide presentations easy to download, and encourage their dissemination. Continuing medical education (CME) credit offerings also have served to attract readers to the site, and CME usership has grown steadily since our launch. Users read the Spotlight case, complete a CME quiz, and receive an annotated review of their answers. Individuals who complete the CME module with a passing grade receive one hour of credit, offered through the University of California, San Francisco (UCSF) Office of Continuing Medical Education.
Given the new Accreditation Council for Graduate Medical Education requirements that residents must demonstrate proficiency in systems-based learning (defined as "actions that demonstrate an awareness of, and responsiveness to, the larger context and system of health"),
13
and the challenges faced by program directors attempting to document these abilities, we added the option of trainee certification for the "Spotlight Cases." AHRQ WebM&M's content is well suited to this purpose. Like CME users, medical residents read the "Spotlight Cases" and take the quiz. Those receiving a passing grade for the learning module can print out a certificate of participation for inclusion in their academic file.
The AHRQ WebM&M site first hosted case profiles on November 18, 2002, and the first complete issue was published on February 3, 2003. As of March 2005, we have published 90 commentaries in 19 issues.
We have been pleased with the quality and breadth of the submitted cases, but their volume has been relatively modest. As of March 11, 2005, we had received 296 case submissions; approximately 37.5 percent of these were accepted for publication. The cases are culled from a variety of specialties, with the majority from medicine (47 percent), surgery/anesthesia (20 percent), pediatrics (8 percent), obstetrics-gynecology (6 percent), and other specialties (19 percent). Just 1 percent of the cases came from the psychiatry field. The errors and issues described in the cases also were of a diverse nature. Among the published cases, the most common were diagnostic errors (27 percent), medication errors (25 percent), procedural complications (18 percent), and communication errors (18 percent) (Table 2). It is worth noting that the comparable percentages for diagnostic and medication errors reflect editorial decisions: we have received roughly twice as many medication error submissions, but many described the same types of errors.
Table 2. Adverse event/error types among published cases*| Delayed or missed diagnosis | 15 (27%) |
| Adverse drug event or medication error | 14 (25%) |
| Complications of a procedure | 10 (18%) |
| Problem with teamwork / communication among providers | 10 (18%) |
| Device related | 9 (16%) |
| Identification error (wrong patient, procedure, medication, test) | 9 (16%) |
| Training/learning curve issue | 9 (16%) |
| Communication: provider-patient issue | 5 (9%) |
| Difficult judgment calls | 5 (9%) |
Other![[dagger]](https://webarchive.library.unt.edu/eot2008/20090830213021im_/http://www.ncbi.nlm.nih.gov/books/bookres.fcgi/system_default/dagger.gif)
|
14 (25%) |
|
* Note: Cases include only the 55 published through our 11th issue (March 2004). The numbers don't add up to 55, because each of the 55 cases could have more than one type of error or adverse event. In fact, median was 2, with range from 1 4.Other included: staffing or other structural (4), discontinuity/transitions (3), miscellaneous others (7). | |
Conversely, the cases involving diagnostic errors have been more diverse, and thus we have accepted a higher percentage of them for publication. Near misses comprise only about 7 percent of the published cases. Twenty-four percent of the cases ended in a patient death or permanent disability. The remaining 69 percent involved intermediate degrees of severity and harm. Thirty-seven errors (67 percent) occurred in hospital, while 8 errors (14.5 percent) occurred in emergency departments (without subsequent admission); 8 errors (14.5 percent) occurred in ambulatory practices; 1 error (2 percent) took place in a skilled nursing facility; and 1 (2 percent) transpired in an undetermined setting.
As of March 11, 2005, AHRQ WebM&M has 9,767 registered users. An average of 663 unique visitors come to the site daily (20,150 each month), and each one stays for an average of 12 minutes. Combining these figures, the site has had approximately 505,000 visit "sessions," and users have spent more than 6 million minutes on WebM&M. Very few other patient safety publications or resources have achieved similar levels of usage or impact.
The anonymity of the Internet (and stringent federal guidelines prohibiting detailed collection of user information) prevents us from developing a comprehensive analysis of our readership. After tallying the extensions of e-mail addresses of registered users to given sites, we discovered that 35 percent hail from a "dot-com" domain, 23 percent from a "dot-org" domain, 15 percent from "dot-edu," 10 percent from "dot-net," and 3 percent from "dot-gov." The vast majority of our readers (91 percent) are from the United States, with the remaining (9 percent) from other countries (the most popular being Canada [2 percent], Australia [2 percent], and the United Kingdom [0.5 percent] ).
A voluntary users' survey accompanied our May 2004 issue, and 542 users completed it. Seventy-seven percent of respondents were registered users of the site, and 85 percent had visited the site more than five times. Ninety-one percent of respondents indicated they had viewed more than more commentary when visiting the site. Interestingly, nurses and physicians were almost equally represented among survey respondents (24 percent and 21 percent, respectively). Four percent of the respondents were pharmacists. Of the remaining users, 11 percent selected "health care administrator" or "manager" from a drop-down menu of professions, and another 32 percent wrote in a category (a wide-ranging list, including risk managers, policy analysts, systems engineers, and ethicists).
The "Spotlight" slides are among the most popular features on the AHRQ WebM&M site. As of our 18th issue (and our most recent count on February 28, 2005), 24,400 copies of our "Spotlight" slideshows had been downloadedan average of 1,355 per issue. Many of our readers tell us that they have used these slideshows in teaching conferences, patient safety or quality meetings, and attending rounds.
We currently have 2,200 registered CME users (a registration separate from general site registration, to further protect anonymity), and have awarded a total of 3,344 CME credit hours.
The "Forum" feature has been used infrequently (103 postings in the course of our first 18 issues, or about 1.2 postings per case, on average). The postings have been thoughtful and on point; any early concerns regarding personal attacks have not been realized. The cases that generated the greatest number of postings were "Patient Mix-Up," and "Too Tight Control" (9 postings each).
Judging by the responses to the May 2004 survey, AHRQ WebM&M users are very satisfied with the Web site. Seventy-five percent of the users rated the site's educational value as "excellent," and another 25 percent rated it as "good." Just one of the 542 survey respondents (0.2 percent) rated the educational value "fair" or "poor." Similarly, 75 percent rated the practical value as "excellent," 24 percent as "good;" only 8 of the 542 respondents (1.5 percent) rated it as "not very useful" or "not useful." Similarly positive results were seen in the ratings of the site's various content areas and functions (Figure 1).
Reader-suggested improvements or enhancements included a "lessons learned" section, continuing education credit modules for nurses and physician's assistants, an ethics forum, a printer-friendly version of the site, and an upgraded search function. However, the vast majority of the comments praised various aspects of the site (or the site as a whole).
WebM&M is the culmination of a bold AHRQ experiment. At the time of its conception, many questions were raised: Would individuals report cases of medical errors to such a public forum? Could the value of the institution-based M&M conference be migrated to a national and international platform? Would the site draw nonexpert readers, particularly clinicians and trainees? And, if early marketing efforts were successful in creating an initial "buzz" of popularity for the product, would the Web site have staying power?
The answers to all of these questions have been surprisingly positive. Individuals have been willing to report interesting and illustrative cases of medical mistakes, and their confidentially has never been compromised. Commentators have articulated important and general safety lessons (including the concepts of root cause analysis, human factors engineering, forcing functions, etc.), and have done so in an straightforward manner with a relative absence of jargon. Discussions have been "systems focused," with practical "take-home points" for providers and quality leaders alike, while honestly identifying individual error where it occurred. Readership began strong, aided by a robust marketing effort by AHRQ and others, and has continued to grow steadily into the site's second year. The background of the readership is extremely broad: approximately half of our readers are clinicians (divided almost equally between physicians and nurses) and half are nonclinicians (including administrators, researchers, and individuals working in the safety field). Some of the site's innovative features, including the "Spotlight" slideshows, videos, and CME credit modules, have proven very popular. The site has generated considerable attention, in the lay press (e.g., the Wall Street Journal) and professional media alike. A Google search conducted on February 23, 2005, for the term "WebM&M" yielded 2,250 hits, indicating the site is linked widely and referenced across the Web.
The AHRQ WebM&M has yielded some disappointments, which we are working with the Agency to address. In the future, the relatively low number and breadth of case submissions may compromise our ability to generate five fresh, interesting cases each month. The relatively low level of activity on the "Forum" demonstrates that we have not yet discovered the best means with which to engage our readers in forward-thinking, interactive dialogues. Future plans for the site may include a decrease in the number of monthly case offerings, as well as new content additions, such as point-counterpoint debates and letters to the editor, and hosting "Reader Sound-Off" instant polls as a means of stimulating more direct user engagement with the site and its content. Finally, we plan to continue using videos and other presentation tools to leverage the growing multimedia capacity of the Internet.
In summary, AHRQ WebM&M represents an ambitious and unprecedented effort to publish illustrative cases of medical errors; to elicit reviews of such cases from the top experts in their fields; to generate lively commentary and provoke thoughtful discussion on the application of evidence-based medicine to the reduction of medical errors; and to draw together a broad readership from the various disciplines that comprise the patient safety field. In addition, we sought to create a Web site that was attractive and user-friendly, and to bring credit to AHRQ for a very practical and popular addition to its critical efforts to improve the safety of patients.
Overall, the success of AHRQ WebM&M has shown that providers will report medical errors under favorable circumstances, that a strong demand exists among health care professionals for Web-based information on the topic of patient safety, and that readers from different disciplines with common interests will visit and return to a Web site that presents the information they seek in an accessible style and an easy-to-navigate format. The AHRQ WebM&M site represents one of the modern era's most successful experiments in both patient safety reporting and education.
The authors wish to thank Susan Nguyen for her invaluable administrative assistance, and our colleagues at DoctorQuality and Quantros for their technical assistance. We also wish to thank all our commentary authors, as well as the individuals who have submitted cases to the AHRQ WebM&M site. Funding for AHRQ WebM&M and this article was provided by AHRQ Contract number 290-01-0011.
University of California, San Francisco School of Medicine, San Francisco (RMW, TM, EEH). The University of Ottawa School of Medicine, Ottawa, Ontario (KGS). The University of Michigan School of Medicine, Ann Arbor, MI (SAF).
Address correspondence to: Robert M. Wachter, M.D, University of California, San Francisco, Department of Medicine, Box 0120 UCSF, San Francisco, CA 94143-0120; phone: 415-476-5632; e-mail: bobw@medicine.ucsf.edu.
1. Bosk CL. Forgive and remember: managing medical failure. Chicago, IL: University of Chicago Press; 1979.
2. Wachter RM, Shojania KG. Internal bleeding: the truth behind America's terrifying epidemic of medical mistakes. New York: Rugged Land Press; 2004.
3. Liang BA. Risks of reporting sentinel events. Health Aff (Millwood) 2000. 19(5):112-20. (PubMed)
4. Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000.
5. Pierluissi E, Fischer MA, Campbell AR, et al. Discussion of medical errors in morbidity and mortality conferences. JAMA 2003. 290(21):2838-42. (PubMed)
6. Orlander JD, Fincke BG. Morbidity and mortality conference: a survey of academic internal medicine departments. J Gen Intern Med 2003. 18(8):656-8. (PubMed) (Full Text in PMC)
7. Wanzel KR, Jamieson CG, Bohnen JM. Complications on a general surgery service: incidence and reporting. Can J Surg 2000. 43(2):113-7. (PubMed)
8. Sentinel Event Alert. Joint Commission on Accreditation of Healthcare Organizations. http://www.jcaho.org/about+us/news+letters/sentinel+event+alert/ . (Accessed 2004 Jul 9.)
9. Wachter RM, Shojania KG, Saint S, et al. Learning from our mistakes: quality grand rounds, a new case-based series on medical errors and patient safety. Ann Intern Med 2002. 136:850-2. (PubMed)
10. Barach P, Small SD. Reporting and preventing medical mishaps: lessons from non-medical near miss reporting systems. BMJ 2000. 320(7237):759-63. (PubMed) (Full Text in PMC)
11. Battles JB, Shea CE. A system of analyzing medical errors to improve GME curricula and programs. Acad Med 2001. 76(2):125-33. (PubMed)
12. McCafferty MH, Polk HC Jr. Addition of "near-miss" cases enhances a quality improvement conference. Arch Surg 2004. 139(2):216-7. (PubMed)
13. ACGME Outcome Project. The Accreditation Council for Graduate Medical Education. http://www.acgme.org/Outcome/ . (Accessed 2004 Jul 9.)