Advances in Patient Safety: From Research to Implementation Volume 3. Implementation IssuesIdentifying Barriers to the Success of a Reporting System
Michelle L. Harper, Robert L. Helmreich
AbstractSpurred by a controversial report from the Institute of Medicine on the prevalence of medical error, To Err Is Human, the medical profession has seen an increase in event reporting systems at the international, national, and institutional levels. Aviation, with its long history of reporting systems, has shown that these systems can yield previously unknown, but safety-critical information for developing a proactive approach to managing human error. Despite many similarities between health care and aviation, event reporting systems have not been well received in health care. Studies have shown that many physicians are reluctant to participate in programs to report medical errors, and that underreporting of adverse events may be as high as 96 percent. These findings suggest that the success of a reporting system is determined by the attitudes and perceptions of frontline care providers. Therefore, prior to implementing an event reporting system, an assessment of the opinions of care providers should be conducted to identify critical barriers to reporting. The University of Texas Human Factors Research Project has developed a survey instrument designed to assess a wide array of attitudes deemed relevant to the implementation of reporting systems. This paper summarizes preliminary survey findings and recommendations for successful implementation of an event reporting system.
"The most detrimental error is failing to learn from an error." 1
The past decade has seen an increase in the implementation of event reporting systems in high-risk industries. These systems are based on the assumption that the individual at the front line is privy to a wide range of information that remains unknown to the rest of the organization. Fueled by a report from the Institute of Medicine (IOM) suggesting that human error is responsible for an annual total of 44,000 to 98,000 accidental deaths, the field of medicine has seen a large increase in development and implementation of both mandatory and voluntary medical error reporting systems. 2 Adding incentive to this movement, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) stated that the development of either mandatory or voluntary reporting systems could, through the aggregation and analysis of data, result in a reduction of medical and health care errors. 3
Aviation, with its long history of reporting systems, has shown that these systems can yield previously unknown, safety-critical information for developing a proactive approach to managing human error. Driven by success stories from aviation, the Department of Veterans Affairs (VA) signed a 3-year, $8.2 million contract with the National Aeronautics and Space Administration (NASA) to develop a reporting system to be used by care providers at the 162 VA medical centers throughout the United States. 4 The VA endorsed the NASA Aviation Safety Reporting System (ASRS) as a model for its new system's development and, as a result, developed one of the first and largest reporting systems in medicine. Prompted by these advances, many hospitals have made the development of a reporting system a top objective in their efforts to increase patient safety.
Regardless of size, financial backing, or reporting requirements, the purposes of these reporting systems are to (1) collect patient safety information by providing care providers with the means to report events or errors, and (2) enable organizations to use this information to create changes to reduce the likelihood of the reoccurrence of the error.
Although simple in principle and supported by success stories from aviation, researchers and professionals in the health care field have questioned whether the focus on implementing reporting systems will result in an increase in patient safety. Further questions have been raised about the use of aviation as an appropriate model for developing reporting systems within medicine. These questions have cited potential difficulties in establishing legal protection for health care providers and basic differences in the operating environments of medical and aviation professionals. And, on a much more practical basis, the question has been raised as to whether a care provider who is already burdened by long work days will take the time out of a busy schedule to report a mistake.
A recent study in Colorado did attempt to answer these questions and reported that, while most physicians agreed there is a need to develop a reliable means for reporting medical error, many stated a reluctance to participate in such programs.
5
Reasons for this reluctance are unclear, but potential liability issues remain prominent. Further findings from studies conducted in the United States and the United Kingdom indicate that, despite mandatory reporting requirements, the underreporting of adverse events may be as high as 96 percent
6,
7
(Figure 1).
Given the current emphasis on the implementation of reporting systems and the potential benefit such systems may have in identifying issues affecting patient safety, it is evident that a more in-depth look at issues relating to the success of these systems is needed. Evidence from aviation suggests that there exists a set of design and implementation requirements that must be established before a reporting system will be consistently used. This evidence, coupled with preliminary findings of low reporting rates both within the United States and internationally, suggests there are significant yet identifiable barriers to implementing a successful reporting system within a medical institution. We propose that critical issues surrounding the use and subsequent success of a reporting system designed to enhance patient safety may be determined by attitudes held by care providers who will be tasked with using these systems.
To identify potential barriers surrounding the use of reporting systems, the University of Texas Human Factors Research Project designed a survey to assess care providers' attitudes toward the use of both mandatory and voluntary reporting systems. Results from this survey at two University of Texas (UT) System hospitals will be discussed.
Physicians and nurses at two University of Texas System hospitals voluntarily completed a self-report survey conducted from October 2003 through March 2004. These two hospitals where chosen to participate in this study because both have an established mandatory reporting system and both agreed to participate in a program to implement a new voluntary, close-call reporting system.
Both hospitals participating in this study require care providers to use a mandatory reporting system if the care provider witnessed or contributed to an error that caused or had the potential to cause harm to a patient. Care providers are required to submit contact information with the submission of a mandatory report. Reports submitted to the mandatory reporting systems are reviewed by the Risk Management or Quality Assurance departments within each of the hospitals. At both hospitals these reports are primarily reviewed for potential hospital liability issues.
Both hospitals participating in this study also support an incident review process. Incident reviews are completed for the purpose of investigating an error that caused or had the potential to cause harm to a patient. Care providers who were involved or witnessed the event are required to attend the review process. Similar to mandatory reports, incident reviews are organized through the Risk Management or Quality Assurance departments and are conducted with a focus on the identification of root causes and potential hospital liability issues.
Both hospitals participating in this study agreed to implement a new voluntary, close-call reporting system that has been designed by the University of Texas Center of Excellence for Patient Safety Research and Practice. This reporting system is based on the voluntary submission of close-call reports from all care providers throughout the hospitals. Care providers who use this system are prompted to submit reports describing situations that could have resulted in an accident, injury, or illness, but did not, due to chance or a timely intervention. Care providers are instructed not to provide any identifying information with their reports. A team of researchers at the University of Texas Center of Excellence for Patient Safety Research and Practice reviews the reports submitted to the close-call reporting system. Care providers who use this reporting system are informed that their report will be used in the development of targeted interventions within their hospital and the information they provide may lead to the identification and implementation of best practices in quality improvement.
The UT Reporting Culture Survey was developed to identify and validate self-report behaviors and attitudes held by care providers toward the use and perceived utility of the current mandatory reporting system and the newly designed voluntary, anonymous, close-call reporting system. The survey was designed to identify a set of core issues that relate to individual care providers' experience with the existing mandatory system and attitudes individuals hold toward the use of the voluntary close-call system. The practical goal for the development of this survey was to create an instrument that could be used by medical institutions to identify potential barriers surrounding the use of the close-call reporting system as it is being developed and implemented at several University of Texas Medical System hospitals.
The survey was developed through a series of structured interviews over a period of 2 weeks. Findings from these interviews were used to derive a set of scaled and open-ended questions focused on care providers' attitudes toward the use of event reporting systems in their workplace. The resulting survey has two parts: (1) scaled and open-ended questions focusing on current experience with the mandatory reporting system and incident investigations completed at the two hospitals, and (2) scaled and open-ended questions referring to the use of the new close-call reporting system. An additional section of the survey contains demographic questions for each type of care provider.
Surveys were mailed directly to the care providers' homes, as well as distributed through internal institution mail. Follow-up reminders were sent 2 weeks after the first survey distribution. An additional survey distribution was completed 30 days after the reminder letter was sent, and a third distribution was completed 30 days later. The care provider groups surveyed included physicians, pharmacists, physician assistants, and nurses across all departments within the two participating hospitals. Findings from physicians and nurses will be discussed in this paper.
Demographics
According to statistics derived from previous survey research conducted at University of Texas Medical System hospitals, a 25
45 percent response rate is common for surveys that target nurses and physicians across all departments and through internal mailings or mailings sent to home addresses. Other industries administering mail surveys with no incentive and no follow-up or reminder letter typically get a 40 percent response rate.
8
We expected a response rate between 35 and 50 percent from the UT Reporting Culture Survey, and we achieved a response rate of 41 percent across nurses and physicians (n = 858).
More than three-fourths of respondents (81 percent) reported clinical duties as their primary type of work. A small percentage of care providers stated multiple types of work completed during their shifts, but worked with patients directly on a daily basis. Fifty percent of the physicians and 41 percent of the nurses reported overseeing other care providers in their departments or having some type of administrative duty. Eighty percent of the physicians and 47 percent of the nurses reported working weekends. Eighty-two percent of the physicians reported seeing patients at least 5 days a week. Physicians reported working the most hours per week (average of 55 hours) and reported the longest workday (average of 11.5 hours). Nurses averaged 45-hour workweeks and 9 hours per day. Fifty-six percent of all care providers who responded had been employed at the current institution for at least 5 years.
The UT Reporting Culture Survey included items regarding the current mandatory reporting systems used at the participating hospitals and the use of the new, voluntary, anonymous, close-call reporting system. Questions were worded with both negative and positive connotations to reduce response bias. Care providers were asked to rate the extent to which they agreed or disagreed with each statement on a 5-point Likert scale from "strongly agree" to "strongly disagree."
Behaviors and attitudes toward the mandatory reporting system and incident review processTo establish a baseline of attitudes that care providers held toward the current mandatory reporting system and incident review process supported within the hospitals, the survey addressed specific experiences care providers had with these programs. Table 1 displays a list of statements presented in the survey and the percent of care providers who agreed with each statement.
Table 1. Statements addressing the use and perceived effectiveness of the current mandatory reporting system| % agreeing | Statement |
| 92 | I am aware that (hospital name) has a mandatory system for reporting incidents. |
| 39 | I have submitted a report to this system. |
| 28 | I have been involved in an event that resulted in an investigation of an incident or root cause analysis at (hospital name). |
| (n = 241) | If yes: |
| 70 | Do you feel the causes of the event were identified? |
| 40 | Do you feel any positive changes resulted form this process? |
| 70 | Submitting an incident or variance report to the mandatory reporting system is an effective way to improve patient care. |
| 37 | My time is better spent immediately addressing a problem I encounter rather than submitting an incident report. |
More than 90 percent of all care providers were aware of the mandatory reporting system, although only half the nurses and less than 20 percent of the physicians reported having used the mandatory system. This finding supports previous findings derived from analysis of physicians' tendency to underreport errors.
6
Additional responses regarding care providers' experience and participation in the mandatory review of incidents or root-cause analysis investigations gave further evidence of limited endorsement of the current processes of reporting errors and reviewing incidents. Approximately 70 percent of care providers who had been involved in a review process believed such programs resulted in identification of the causes of an event. Yet only 40 percent of those care providers stated that they thought any positive changes occurred as a result of the review. When asked directly if they agreed with the statement that the current mandatory reporting system could be used to improve patient safety, care providers responded more positively
70 percent agreed with this statement. However, more than one-third of the respondents believed it would be easier to address problems directly rather than submitting a report to the mandatory reporting system.
A potential explanation for the low percentage of physicians who reported having used the mandatory reporting system could be a general perception held by physicians that the reporting of errors is an administrative task that falls under the responsibility of the nurse who is in charge of the patient at the time of the event. Unlike aviation, where reporting responsibilities are clearly defined and often discussed before the completion of a flight, the responses from these questions suggested care providers were unclear about reporting responsibilities, despite the mandatory requirements of the system for all care providers.
The second part of the survey included a series of scaled questions assessing care providers' attitudes toward the use of the new, voluntary, close-call reporting system. The following definition was used to describe a close call: "a potential error that is caught and prevented before it can affect patient care." This definition was stated as part of the survey instructions at the beginning of the survey and at the beginning of the section referring to the close-call reporting system to ensure comprehension of the statements.
Findings from the questions referring to the close-call reporting system suggest that care providers see value in reporting minor events and potential precursors that could lead to more serious incidents. Ninety-three percent of care providers felt close calls were important to address and that they have a personal and professional responsibility to address minor problems. These findings suggest that, despite low levels of belief that the current mandatory processes will produce change within the hospital, care providers remain individually motivated to address problems encountered while completing their jobs.
Similar to their opinions toward the use of mandatory reporting systems, care providers were split when asked to give their opinion of the potential benefits of a close-call reporting system. Fifty-one percent of care providers agreed that significant changes could result from an anonymous close-call reporting system. Twenty-five percent disagreed with this statement, and the remaining care providers were neutral on this topic.
In summary, it can be concluded that nurses and physicians held negative opinions about their respective institution's ability to create a positive change based on current investigation processes. But, more than half agreed that a voluntary or mandatory reporting system could result in change. The previous findings suggest that care providers maintain a professional sense of responsibility to address problems and, as members of their professions, hold the belief that part of their job is to correct problems that could result in patient harm despite busy schedules or predefined tasks. But these positive opinions remain confined to those statements that referred to what a nurse or physician can personally or immediately address and did not extend to opinions held toward the effectiveness of hospital-mandated or voluntary programs.
The following set of questions addressed potential barriers to the use of both mandatory and close-call reporting systems, including a set of questions that addressed why the current mandatory system was not used. (See Table 2 for a list of statements and percent of care providers who agreed with each statement referring to reasons why the mandatory system was not used.)
Table 2. Reasons for not using a mandatory reporting system| % agreeing | Statement |
| 28 | I have not encountered any problems or made any mistakes that would be appropriate to report. |
| 15 | I did not think my report would result in any changes being made. |
| 13 | I did not have the time. |
| 12 | I did not want anything negative to happen to the people I work with. |
| 10 | I did not want my name to be attached to a report. |
Questions addressing potential barriers to the close-call reporting system yielded evidence of care providers' fears that a report would be used against them rather than being used to address potential causes of events or to create change within the hospital. The statement referring to care providers' concern that their report would not be truly anonymous was stated as follows: "Despite not having to give my name, I would still worry I could be identified by a report I submit to an anonymous reporting system." This issue appeared to be most prominent with physicians, with 40 percent reporting agreement with this statement. Thirty percent of nurses were in agreement. Findings from this statement suggest that despite the fact that the instructions for using the voluntary system clearly stated the reports were anonymous and only to be used to report close callscare providers were informed that no reports of errors resulting in patient harm would be accepted and any identifying information removedthere was still a substantial percentage of respondents who worried their reports could somehow be used against them.
Another issue of concern regarding anonymity was the potential for the system to be used to report the behavior of other people. Both sets of care providers endorsed the statement, "I would be concerned that an anonymous reporting system would be used to report the behavior of other people." This statement was more strongly endorsed by physicians, with more than 50 percent in agreement (Figure 2).
Findings from this survey clearly point to reporter protection as being among the most critical elements influencing the use of a reporting system. An example of the impact reporter protection can have on the success of a reporting system stems from aviation. One major requirement that was adhered to during the design phase of the ASRS reporting system was an agreement from the Federal Aviation Administration (FAA) to grant pilots immunity from punishment in return for voluntary submission of reports. Providing such protection to an individual was unprecedented in any high-risk industry and left the FAA without punitive authority, yet providing this level of reporter protection has contributed to ASRS receiving more than 30,000 reports per year.
The next set of questions focused on how these barriers to reporting could be overcome. Several statements were strongly endorsed as important characteristics of a reporting system if it were to be consistently used by a care provider. (See Table 3 for a list of statements referring to potential solutions to overcoming barriers to the use of a reporting system and the percentage of care providers who agreed with each statement.)
Table 3. Overcoming barriers to reporting| % agreeing | Statement |
| 85 | Feedback needs to be given on what was being done to correct the problem. |
| 75 | Reporting system should be designed for the specific problems encountered by each type of care provider. |
| 72 | Physicians: My report was given Federal and State protection from legal discoverability. |
| 66 | It should be mandated that senior staff and management address the problems reported. |
| 60 | Nurses: My report was given protection from the Texas Board of Nurse Examiners. |
The statement regarding the importance of supplying the reporter with feedback resulted in the strongest endorsement by both sets of care providers. An average of 86 percent of nurses and 81 percent of physicians agreed with the statement, "I would be more likely to submit a report to an anonymous, close-call reporting system if I was given feedback on what was being done to correct the problem I report." Further evidence of the importance of feedback was found in the endorsement of the statement, "Senior staff or management should be mandated to address reported issues." Approximately 70 percent of nurses and physicians agreed with this statement. The strong endorsement given to these statements suggest care providers will be more likely to support a program that puts the information they submit to use.
The other statement that was strongly endorsed in this section was, "An anonymous, close-call reporting system should be designed for the specific types of problems nurses or physicians encounter." Seventy-seven percent of nurses and 73 percent of physicians agreed with this statement. This finding suggests that customization of the reporting systems could be a powerful way to overcome low reporting rates and could also aid in changing the perception of physicians that reporting errors is the responsibility of nurses.
In summary, responses to scaled questions regarding opinions held toward reporting systems suggest that despite views of institutional practices to address errors, care providers hold strong opinions of their own professional responsibilities to address errors. Findings from the survey yielded several examples of how this professional responsibility could be effectively supported. First, is it clear that a nonpunitive, trusted source of reporter protection must be established to overcome prominent fears relating to the potential misuse of reported information. Second, care providers collectively endorsed the importance of feedback, suggesting a need for structured assessment of the data contained within reports and circulation of information that is reported. And third, high percentages of care providers suggested the need to customize reporting programs based on professions, suggesting a way to overcome the current resistance of physicians to endorse these programs. It can be concluded from these findings that the critical components of a successful reporting system are a nonpunitive, customized reporting process and a focus on the distribution and continual feedback of information to care providers.
Findings from the survey results suggest that there are several issues defining care providers' opinions surrounding the use of a reporting system, whether that system is voluntary or mandatory. Two issues will be discussed in light of these findings: (1) the power of a nonpunitive process, and (2) the importance of a systemic focus. Relevance of the survey findings in reference to these two issues and further evidence from models of aviation in developing a nonpunitive and systemic-focused reporting program are discussed in the following section.
The power of a nonpunitive processThis study suggests that the development of a truly nonpunitive process could increase individuals' motivation to report errors they have made. The depth of concern about a system being truly nonpunitive was evident from the endorsements of the statement that care providers feared legal or disciplinary action as the result of the use of a reporting system. These findings are particularly relevant because care providers reported these fears despite the anonymous design of the close-call reporting system. Care providers were informed at several places in the survey that the close-call reporting system was designed to be completely anonymous and that no identifying information would be collected. They were further instructed that the new reporting system would only be used to collect information on close calls in which no harm came to the patient involved. Yet despite these statements, 40 percent of physicians and 30 percent of nurses still expressed concern over potential punitive actions as a result of using the system. Also strongly endorsed by care providers was the statement that they feared the system could be used maliciously to report the behaviors of other care providers. This finding suggests that care providers not only lack trust in the hospital administrative processes for handling sensitive data, but may also maintain a level of distrust for their coworkers.
On a more positive note, findings from both sets of care providers suggested that a guarantee of reporter protection from legal discoverability or disciplinary action could increase their motivation to use a reporting system. Evidence of the motivating power of this type of guarantee in increasing reporting rates can be found in aviation. This example demonstrates the impact that ASRS policy has played in creating a culture of reporting within aviation.
ASRS has been running since 1976 and has undergone a series of changes to the immunity and reporter protection policies it supports. The impact of these changes and the power of immunity or a nonpunitive process were demonstrated by a large increase in reporting rates that the program experienced over 8 years following enhancements in the immunity policy. 9 Changes in the immunity policy wording referred directly to the prohibited use of a report for any type of enforcement action by the FAA.
Although other issues were likely to have contributed to the increase in reporting rates for ASRS during this period, the lessons to be learned from the example set by ASRS are clear: If you want an individual to voluntarily report an error, the risk involved with the disclosure of this information must be eliminated. Although a simple lesson, even the reporting system developed by the VA, which is modeled after ASRS and housed at the same third-party location under NASA, has yet to achieve the level of immunity that demonstrated such a strong influence on establishing reliable reporting rates with ASRS. And, unfortunately, there are some indications that several groups who promote policy and develop recommendations for governing medical reporting systems may be ignoring this lesson altogether. A 2002 report by the IOM suggested that Government health care programs should make data from quality reports available to the public.
10
Evidence from this survey suggests that such a movement could cause an already distrustful group to completely lose motivation to participate in any type of reporting system.
A focus on identifying system-level factors is often stated as a primary objective of many reporting systems, regardless of the structure and methods of collecting information. As demonstrated through much of the research resulting from studies of medical error, patient harm is often a result of a long series of events. What is evident from research assessing human error within medicine and other high-risk industries is that adverse events often occur as a result of failures within multiple levels of the entire system and are rarely the result of a single error. Yet, despite this knowledge of the importance and potential impact of systemic factors, many hospitals developing reporting systems are still focusing on what type of error should be accepted into a reporting program. This focus on defining specific types of errors and the identification of who is at fault commonly supersedes the development of methodologies to protect the reporting care provider or the development of tools that tap into reporters' expertise as to why they think the event occurred. For a reporting system to focus primarily on educating care providers on what types of errors are appropriate to report sends a message that submission of the wrong type of error could result in punitive action.
Findings from this survey suggest there are two ways in which a reporting system can use a systemic focus to maintain a stable reporting rate. First, give feedback to the reporter on what is being done to reduce the chance of the error reoccurring. And second, require action be taken as a result of the information submitted. Strong endorsement of these concepts suggest that until the focus of reporting programs can be redirected to identify system-level factors and give the reporter consistent and reliable feedback on what was being done to address the reported issue, motivation to report may remain low.
One of the more optimistic findings of this survey is that a significant percentage of care providers expressed a strong professional obligation to report errors, even errors related to close calls. This finding remained strong despite busy schedules reported by care providers. Opponents of the comparisons that are made between aviation and medicine have raised the issue that individuals providing care to patients may not feel the same type of professional responsibility to maintain safe practices as is commonly found with pilots. This argument was not supported in this study; rather it was found that given immunity from punishment and a guarantee that the information reported will be used to make changes in the system, there is willingness on the part of care providers to take time out of their busy schedules to submit reports.
A special thanks to Melissa Threlkeld and Jane Geraci, M.D., for their support and promotion of this project.
Both authors are affiliated with the Human Factors Research Project at the University of Texas at Austin.
Address correspondence to: Michelle Harper, M.A., University of Texas at Austin, Department of PsychologyA8000, 108 East Dean Keeton, Austin, TX 78712; phone: 512-475-7980; e-mail: mlharper@mail.utexas.edu.
1. Reason JT. Human error. New York: Cambridge University Press; 1990.
2. Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safety health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000.
3. Joint Commission on the Accreditation of Healthcare Organizations. The measurement mandate. Oakbrook Terrace, IL: JCAHO; 1993.
4. The Patient Safety Reporting System. Program overview. http://psrs.arc.nasa.gov/program_ overview.htm . (Accessed 2004, Apr 19.)
5. Robinson AR, Hohmann KB, Rifkin JI, et al. Physician and public opinions on quality of health care and the problem of medical errors. Arch Intern Med 2002. 162(19):2186-90. Oct 28. (PubMed)
6. Vincent C, Stanhope N, Crowley-Murphy M. Reasons for not reporting adverse incidents: an empirical study. J Eval Clin Pract 1999. 5:13-21. (PubMed)
7. Barach P, Small SD. Reporting and preventing medical mishaps: lessons from non-medical near miss reporting systems. BMJ 2002. 320:759-63.
8. Visser PS, Krosnick JA, Lavrakas P. Survey research methods. In Reis HT, Judd CM, editors. Handbook of research methods in social psychology. New York: Cambridge University Press; 2000.
9. Bond LM. Aviation safety reporting program. Federal Aviation Administration, Advisory Circular 00-46B. Department of Transportation; June 1979.
10. Corrigan JM, Eden J, Smith BM. Leadership by example: coordinating government in improving health care quality. Washington, DC: National Academy Press; 2003.