AHRQ's Patient Safety Initiative: Building Foundations, Reducing Risk (continued)

Chapter 1. AHRQ's Medical Error Reporting Demonstration Evaluation Projects: Key Issues in Identifying Threats to Patient Safety

Background

AHRQ's Medical Error Reporting Demonstration Evaluation Program was established in response to concerns about the number and impact of medical errors that occur as part of health care. In early 2000, the Senate Committee on Appropriations identified medical errors as a health care priority after the Institute of Medicine reported that these errors cause from 44,000 to 98,000 deaths a year. The Committee's Report 106-293, dated May 12, 2000, provided that the Director of AHRQ should:

Establish a competitive demonstration program for health care facilities and organizations in geographically diverse locations, including rural and urban areas, to determine the causes of medical errors. These projects should use technology, staff training, and other methods to reduce such errors; develop replicable models that minimize the frequency and severity of medical errors; develop mechanisms that encourage reporting, prompt review, and corrective action with respect to medical errors; and develop methods to minimize any additional paperwork burden on health care professionals. Health systems and providers participating in this demonstration program should utilize all available and appropriate technologies to reduce the probability of future medical errors.²

The Committee report further stated that in order to evaluate the best and most effective methods of error reporting and use of the collected data, AHRQ should:

Ensure that each of the awarded projects examines any one of three types of error reporting parameters: voluntary reporting to the Director by participating health care providers of any adverse or sentinel events, health care-related errors, or medication-related errors; required reporting to the Director by participating health care providers of any adverse or sentinel events, health care-related errors, or medication-related errors; and required reporting to the Director and (with and without) reporting to the affected patient or family member by participating health care providers of any adverse or sentinel events, health care-related errors, or medication-related errors.²

The Committee's directives on medical error reporting are consistent with IOM Recommendations 5.1 and 5.2, which address mandatory and voluntary reporting efforts.¹

Return to Contents

Sixteen Medical Error Reporting Demonstration Evaluation Grants Awarded

In response to the Committee's request, and consistent with IOM recommendations, AHRQ published the Health System Reporting, Analysis, and Safety Improvement Demonstrations Request for Applications (RFAs) in February 2001. This RFA called for proposals that would develop and evaluate various reporting systems for identifying and analyzing medical errors in States, health care systems, and/or networks of providers, consistent with the Committee's direction.

Fifty-four applications were received that conformed to the requirements of the RFA. These applications were reviewed for technical merit by a special panel that included health care system leaders and some of the world's leading experts on patient safety and patient safety data collection. Of the 54 applications received, AHRQ funded 16 that were highly rated by the review panel. In September 2001, AHRQ awarded 16 3-year error reporting demonstration evaluation grants representing approximately $23 million per year for FY 2001, 2002, and 2003.

Table 1 lists the 16 grants by research institution, principle investigator, and the State where the research institution is located.

Table 1. 16 Medical Error Reporting Demonstration Evaluation Grants Awarded

As shown in the table, the 16 grantees are located in 12 States. However, the geographic diversity represented by these demonstrations is actually broader because some grantees include sites in different States. Some even include nationwide reporting systems. For example, one of the projects (Pace) includes a network that stretches across many rural communities. Kaplan is evaluating reporting in two major urban areas in different States—New York City and Chicago. Three grantees—Layde, Osten and Ridley—are evaluating statewide reporting systems in Wisconsin, New York, and Massachusetts, respectively. Williams includes coverage of two States—Utah and Missouri. Khuri and Pronovost represent specialty-based reporting on a national scale.

Because the 16 grants are currently in the 18th month of their 36-month life cycle, final results will not be available for at least another 18 months. In some cases, early observations are available and of interest as decisions are made about modifying current reporting systems and as new databases are developed. Of note, however, is that these observations are preliminary and could potentially change as the projects are completed.

Appendix 2 contains a brief abstract of each grant.

Return to Contents

Preliminary Observations

In preparation for writing the Interim Report to the Senate Appropriations Committee, the AHRQ Patient Safety Team contacted all of the grantees via E-mail and asked them to complete a form designed to capture the three to five major accomplishments that they had achieved since they were awarded grant funds in 2001. During a pre-conference session at the 2nd Annual Patient Safety Conference, held in March 2003, the Reporting Demonstration grantees met as a group. They discussed their individual and collective accomplishments in light of the questions that AHRQ needed to address in this Interim Report. The observations that follow were culled from discussion of the evidence-based projects at the annual meeting and from the forms completed by the grantees.

Observation 1. No single data source is sufficient to gain a complete understanding of medical errors contributing to actual or potential patient injury.

From the early days of AHRQ's Patient Safety Initiative, much discussion on medical error reporting has focused on determining which type of reporting system is best. AHRQ sought applications for reporting demonstration projects that represent a full range of event reporting types and evaluate the effectiveness of various methods of collecting and using information to reduce medical errors. Interim results from the demonstration grants indicate that no single data source is sufficient to gain a complete understanding of medical errors contributing to actual or potential patient injury. Active reporting systems, surveillance through the use of medical records (paper and/or electronic), and patient safety indicators from administrative data are all needed to obtain the most complete information on medical errors and patient safety issues. Figure 1 illustrates this concept of triangulation of patient safety information from error reporting and surveillance systems.

Figure 1. Looking for Harm

Source: Battles JB, Lilford RJ. Organizing patient safety research to identify risks and hazards. Qual Saf Health Care 2003;(suppl II):ii2-ii7.

Mandatory vs. Voluntary Reporting Systems

One issue identified by grantees in the evaluation of reporting systems is whether they should be mandatory or voluntary. The advantages of mandatory and voluntary systems vary depending on the circumstances under which each is implemented. Mandatory systems that were tested by the grantees are generally believed to be acceptable for accountability which is most often associated with the disclosure and examination of serious or sentinel events where a patient is severely harmed. There is an assumption that some form of public accountability is required in cases such as this (e.g., the wrong limb was amputated, the wrong organ was removed).

Reporting systems have proven to be extremely useful in error management in a number of industries including aviation, nuclear power, and petrochemicals. The IOM cited, as a model for safety improvement, the Federal Aviation Administration's voluntary Aviation Safety Reporting System and its companion mandatory reporting of crashes. This system collects information on perceptible missteps in the process that seem to signal a threat to safety. Some may be events that result in no harm, while others might be near misses or result in only minor harm. Experts analyze the physical evidence and the information reported to determine whether an error occurred and, if so, the nature of the error.

While three grantees found mandatory reporting systems suffice for public accountability, 11 grantees found voluntary reporting systems, on the other hand, have the distinct advantage of being able to capture less severe, no harm, or near miss events that may not otherwise be reported. Voluntary, non-punitive systems may provide the best opportunity for capturing more information on more cases and provide a much richer opportunity for learning before serious consequences occur.

Mandatory disclosure may not be effective when incentives to report are misaligned with the consequences of reporting. Grantees found that if there is a desire to have individuals report medical errors, they should not be penalized for committing them.

Personnel policies that lead to employees being fired or formally reprimanded because of errors will need to be addressed if, in fact, organizations expect those involved in the delivery of health care to report the errors they commit. One of our grantees found for example, that for nurses in the State of Texas, reporting can lead to license suspension.

Types of Reporting

Of the 16 demonstration grants awarded, four evaluate systems with both mandatory and voluntary components, seven evaluate voluntary systems, two evaluate mandatory systems, and three evaluate systems that use data from other sources such as discharge or medical record data. The grantees report on different types of reporting systems, some using spontaneous active reporting while others rely on more passive surveillance systems based on record information.

1. Spontaneous Active Reporting Systems

Spontaneous active reporting systems receive reports from individuals or organizations in response to a specific medical error. They are considered active because a reporter must actually prepare a report in response to the medical error and submit it for entry into the system. Such systems can be voluntary or mandatory. Ten of the demonstration projects involve spontaneous active reporting systems. However, they vary widely as to the scope and type of data collected, as described in the following sections.

State and regional reporting systems. State and regional reporting systems have the advantage of looking at patient safety events from a defined geographical area. Statewide systems allow the examination of patient safety events from a variety of health care institutions and facilities within a given State to look for patterns within the State. Regional systems can be a subset within a State, such as the Pittsburgh area, or a defined geographical area that might cross the boundaries of several States. The information gathered from these systems allows the examination of significant numbers of events from a common geographic area. Four projects are evaluating reporting systems that can be classified as State or regional. They are shown in Table 2.

Table 2. State or Regional Reporting Systems

Large Multi-hospital Reporting Systems. Two of the reporting demonstration evaluation grants are assessing multi-hospital reporting systems. These multi-hospital systems have the advantage of looking across multiple settings of care within one or more organizations. When data are shared among organizations in the system, comparisons can be made and trends can be studied. An advantage of multi-hospital system reporting is that organizations within the system share common organizational linkages and some administrative structures. One important aspect of studying such systems is that it provides the ability to monitor how the information was used and what actions were taken to deal with the patient safety issues identified through the medical errors reporting system.

Table 3. Large Multi-Hospital Reporting Systems

Ambulatory Care System Reporting. One of the issues raised in the IOM report, To Err is Human, is that there is little information on the types of errors and events that occur in the ambulatory setting. AHRQ held that it was essential to include ambulatory care system reporting as part of the overall mix of reporting demonstration evaluation grants. This belief is based on the assumption that the types of errors, risks, and hazards that exist in the ambulatory care setting may be different from those in the inpatient setting. Two projects are evaluating systems that focus on reporting events from ambulatory, non-hospital settings. These represent new approaches to error reporting given that most reporting systems have focused on gathering hospital-based information.

Table 4. Ambulatory Care Systems

Multi-institutional Domain/Setting-specific Systems. Two of the demonstration projects are evaluating multi-institutional systems that collect data from a specific domain of medical care or from a setting-specific reporting system. These reporting systems share a common interest and activity. Additionally they look at events in their domain across different organizations for comparison and observation of common trends and problems. These reporting systems represent an opportunity to have a concentration of events reported within a single domain, such as in surgery or in the intensive care unit—a setting in which complicated care is delivered to vulnerable patients. By comparing events reported from different organizations but within a single domain, organizational differences can be observed. Participants in such systems find that they can benchmark their own unit with similar units. Domain-based systems often involve national organizations as in the case of the American College of Surgeons and the Society for Critical Care Medicine. Often, standards or strategies to eliminate errors can be developed for a discipline based on systems such as these.

Table 5. Multi-institutional Systems

2. Administrative Data Systems

Administrative data systems consist of a variety of information most often associated with billing and discharge data. Such data already exist and do not require additional efforts to collect. Two demonstration projects are using discharge data primarily generated for billing purposes. Currently, most States require hospitals to report administrative data to the State departments of health or other equivalent organizations in the State. These data have been found to be valuable sources of indicators of patient safety and helpful in identifying sources of potential harm to patients.

Table 6. Administrative Data Systems

3. Medical Records: Paper or Electronic Record Surveillance Systems

Medical records have traditionally been a major source for identifying medical errors. The medical chart audit has been widely used to identify events after the fact. Grantees found that the major limitation of the medical record is that medical error or harm information can be difficult to identify and is not always included in the chart. Furthermore, traditional chart audits are often time consuming, labor intensive, and expensive. However, two grantees found the use of electronic medical record surveillance has shown considerable promise. In addition, one grantee found a form of record that may be a valuable source of patient safety information is the closed claim malpractice insurance files. Such records may contain a rich resource of information about medical error and patient harm. Four of the demonstration projects are using electronic or data records as a source for such patient safety information.

Table 7. Medical Records

Return to Contents
Proceed to Next Section