Measuring Progress

It is impossible to be preoccupied with failure or to respond to system breakdowns if information is not available to measure system performance. A general theme across repeated discussions of measurement with HRO Network systems is that measuring is essential, but often does not work as planned. Missing baseline information makes progress hard to assess; excess complexity makes results difficult to understand or use; and measures that are too labor intensive are unsustainable over time.

This section identifies several general insights about effective measurement shared by systems in the Network. It also addresses issues related to several specific areas where measurement is important.

Measurement Insights

• Measure fewer things better. Multiple systems in the Network noted the common problem of having too much data. Too much information can make it harder to be truly sensitive to operations and to noticing important failures that occur within key systems. Cincinnati Children's uses a series of basic questions to ensure that it is measuring the right things, but not too many things:
  • What do we want to know?
  • How are we going to collect that information in the clinical process?
  • What are we trying to show at the end of the data collection?
    These questions reduce the tendency to measure everything that is measurable, which in systems with strong technology infrastructure can be much more than is meaningful or usable.
• Stories count and simplify. We heard as many examples of improvements stemming from a story about a problem than we did about initiatives based on data. Both are very important, but leaders noted that sometimes problems are well known and the need to collect data regarding them is irrelevant and slows the process. If there is agreement related to a problem and a way to fix it, then resources should focus on the fix, not documenting the obvious. Over time, measures become more crucial and their accuracy must be refined, but in many cases, stories are the starting place.
• Couple measures with high performance standards. Data can desensitize people to system failures. If a certain failure rate is the norm, then trending data that show no change in that failure rate can contribute to complacency. Each system we visited placed very high importance on establishing goals that were well above current levels of performance on key indicators. This approach reduces complacency and contributes to a culture in which continuous improvement is essential.

Return to Contents

Specific Measurement Areas

Many of the specific initiatives described below include descriptions of how progress was measured over time. The three examples shown here illustrate important measurement concepts: Anything can be measured and measures can be quite simple, but sometimes multiple measures are essential to track system performance.

• Measuring leadership. Jeff Selberg's discussion of leadership's role in creating a high performance culture posed several important questions useful for assessing leadership performance.
  • Are you committed to your own growth as you grow your organization? Your organization's ability to transform and improve is directly correlated to your ability as a leader to transform and improve.
  • Are you creating the environment so that the right and, most of the time, the wicked questions are asked? It is not your role to have an answer for all of the questions, but rather to create an environment where the right questions are asked and greater personal and organizational awareness are achieved. Asking these types of questions may feel risky, but the result will be greater organizational tolerance for diversity of thought.
  • Are you engaging in patient-centered versus ego-centered conversations? You must take yourself out of the center of your strategy and replace yourself with the patient to ensure that you are protecting your patients first and foremost. A great deal of self-awareness is required to know where you are in every conversation.
  • Are you embracing challenges that stretch your capacity as a leader? Your approach must be that every situation, no matter how challenging, is the perfect opportunity to learn, grow, and meet long-term objectives.
    
    While these questions are basic and the answers subjective, they reinforce the importance of assessment of all aspects of an organization's behavior, including the actions of its leaders. If they are unwilling to assess themselves, they will find it hard to create a culture where assessment is the norm.

• Measuring chemotherapy orders. Exempla made changes designed to reduce risks and improve efficiency of chemotherapy orders. The safety metrics they developed (number of abbreviations, use of standardized order sets, illegibility, etc.) were all quite simple and easy for staff to measure before and after the initiative was introduced. But these measures were combined with assessments of nurse's satisfaction with the process and changes. Exempla realized two important things. If they could not make changes that were easy to assess and that were supported by staff, the changes would not be sustainable. In other words, Exempla wanted to ensure that the processes implemented for measuring chemotherapy orders were working effectively for the staff members actually measuring the medications. Tracking both dimensions was simple, but also vital to knowing whether they were achieving their goals.
• Measuring errors and near misses. Measuring safety events is quite complex. Some systems reported experiencing increases in reported events as they worked to make their cultures more transparent and attuned to safety issues. Other systems reported instances where a large percentage of some kinds of errors (e.g., medication) were not reported. There was general agreement about several issues relating to measuring errors:
  • Measure both minor and major events so that both can be trended. In a punitive culture, both will be underreported. In a just culture, both will be reported more frequently, but major events should decline more substantially than minor ones.
  • Look for alignment between these measures and other indicators of safety. Sentara became more confident in their measures because their improvements on event measures corresponded to reduced insurance claims.
  • Consider measures that examine the ratio of major to minor safety events. Such measures may encourage reporting of small errors and allow hospitals to see whether the ratio of major to minor errors is declining over time.

While measuring too much can be unhelpful, systems have recognized that for issues such as safety, no single metric will provide a clear sense of how they are actually doing. This reluctance to simplify safety into a single indicator prevents measurements that can be useless, or potentially even dangerous to patients.

Return to Contents

Implementing Specific Improvement Initiatives

Applying HRO concepts to specific improvement initiatives is what truly matters. If the concepts cannot be used to make specific aspects of hospital care safer, higher in quality, or more efficient, then they are of no value to hospital leaders. This section highlights the breadth of applications of HRO concepts to improvement initiatives, all of which are described in more detail in the site visit summaries and case studies. Those sections reflect an important aspect of HRO thinking: that changes are often driven by several or all of the HRO concepts.

• Christiana Care applied concepts of resilience and preoccupation with failure to successfully create an EICU that provides an additional level of support to staff caring for their sickest, highest risk patients.
• Sentara's preoccupation with failure led them to notice and reduce the number of interruptions experienced by people at the medication dispensing machines. This resulted in lowering the risk of drawing the wrong medications and reducing the time lost for staff associated with required rework when medications were forgotten.
• Exempla applied Lean concepts to the challenge of improving chemotherapy orders. In a relatively short time they raised staff satisfaction with the process and reduced problems in orders that increased the risk of medication errors.
• Cincinnati Children's identified flaws in their discharge planning process that kept patients hospitalized longer than necessary and limited bed space for patients scheduled for surgery. Their initiative substantially raised the percentage of patients leaving the hospital within 4 hours of meeting their discharge goals.
• Cincinnati Children's applied a range of strategies to substantially reduce ventilator-acquired pneumonia cases among their patients. The reduction reduced patients' length of stay and freed hospital beds to care for additional patients, which also generated more revenue for the hospital.
• Working together, hospitals in the Minneapolis area agreed to standardize medication concentrations to reduce errors that could occur by staff working in facilities that used different concentrations.
• Exempla redesigned their processes for stocking and using their medication dispensing machines. The changes they made reduced inventory costs, the number of medications that the pharmacy had to send to the ICU, and the number of unused medications in the medication dispensing machine.
• Christiana Care applied the HRO concept of sensitivity to operations to prevent and more quickly detect and treat sepsis. These changes substantially lowered the impact of sepsis in their facility.
• Cincinnati Children's applied the concepts of sensitivity to operations and preoccupation with failure to recognize the need to reduce codes occurring outside the ICU. These efforts have made codes outside the ICU exceptionally rare events.
• Several systems in the Network have computerized physician order entry (CPOE) systems in place. While these systems have much promise, they sometimes have no, or even negative effects on patient safety. Cincinnati Children's deferred to the expertise of the users of the system when designing and implementing it. They rejected overly simplistic understandings of the potential risks and rolled out a system that substantially reduced the number of calls required to clarify orders and cut delivery time of the medications to the unit by over 50 percent.
• Exempla redesigned their specimen processing workstation to improve efficiency and reduce the potential for errors and rework. This process created more workspace and reduced both retesting and the need for redraws of patient specimens.
• Sentara and other systems implemented safety huddles and other processes designed to improve patient handoffs within and between units. These processes allow staff to be more sensitive to operations, understand and attend to risks confronting particular patients, and defer to the expertise of the providers who have been caring for the patient most recently.

These and the other examples described in the appendixes should provide you with a broader understanding of the potential applications of HRO concepts to the challenges you face. It is important to remember that even the most detailed explanations of these changes might omit key details, and your facility will need to adapt what others have done to make it work for you. But these examples should demonstrate that a culture built on a high reliability mindset is one that will lead to safer, better, and more efficient care for your patients.

Return to Contents

Spreading Improvements to Other Units and Facilities

No system participating in the HRO Learning Network was satisfied that their innovations and improvements had been embraced by all the units and facilities in their systems that could benefit from them. Although it would be wonderful to feature a system that has mastered the process of rapidly spreading improvements, it is unsurprising that this challenge remains unsolved. Often people fear change because it is unknown, can disrupt work patterns, and can take more time to implement. Change does not occur overnight but takes time, and these initiatives are new, so it can be difficult to implement them.

Much of this challenge relates to the need to establish and sustain a culture built on high reliability concepts. Without leadership and a culture that encourages constant reflection about system risks and opportunities for improvement, initiatives that worked elsewhere may fail. As a result, spreading improvements across a system is part of an even broader challenge: the challenge of spreading a high reliability culture across a system. Cultures change slowly, but systems in the network identified a number of suggestions for facilitating this process across units, to physicians, and across systems.

Between-Unit Spread

• Aggregating data and sharing it across the hospital has been used by several of the systems to raise awareness of key issues and to motivate other units to improve to a standard being set in other units. Some hospitals post unit performance data in public places to communicate the norm of transparency and accountability.
• Stories were regarded as key to spreading ideas. Specific ideas related to sharing stories effectively included:
  • Capture people doing good things and share those stories. These stories reinforce a culture where doing good gets as much attention as avoiding bad.
  • Talking openly about mistakes and near mistakes reinforces the message that they can occur everywhere and that they should be acknowledged when they occur. This was regarded as essential to creating a high reliability culture across the whole organization.
  • Sharing stories from and about all types of staff and from patients helps reinforce the principle of equality and teamwork.

Spread to Physicians

Every system present agreed that developing and implementing HRO concepts for staff other than physicians was much easier than doing the same thing with physicians. Although difficult, ideas for supporting spread to physicians include:

• Frame changes in ways that appeal to physicians' needs. When physicians view a change as something that will make them more efficient, they are much more likely to support it.
• Don't even try to implement changes focused on physicians without very strong executive and physician leadership. The few success stories that were shared involving physicians all occurred where strong leadership support existed.
• Begin by making successful changes that involve other staff. These successes increase the willingness of physicians to try them. One hospital in Sentara's system is introducing Red Rules for physicians, but this is still a work in progress.
• Allow physicians to violate some rules based on their clinical judgment—but only if they document the reason for the exception. Some systems felt that allowing these types of exceptions also encouraged mindfulness required to be an HRO.

Spread Improvements Across Systems

• Sharing data systemwide can be effective in creating awareness of performance differences between hospitals. If improvements are substantive and effectively measured, demand may increase for these improvements so that other hospitals can achieve similar improvements.
• Creating informal and even formal settings for peers from different facilities to network and share ideas with each other can help spread good ideas. A number of improvements that have spread in Sentara have occurred because of informal discussions between peers.
• Some systems have tried formal rollouts from one hospital to others in the system. It was not clear whether these efforts worked better than spread that occurred informally.
• Seeing where spread may be occurring informally and then supporting those efforts with well-trained staff appeared to work well. This strategy ensures that the interest in change already exists and maximizes the impact of trained staff.

Return to Contents

Using This Information

The preceding section reflects a very broad range of applications of high reliability concepts to the practical challenges faced by hospitals and their leaders. While the appendixes provide more detail about many of the concepts, there are no step-by-step detailed descriptions of exactly how to implement any of the interventions that we describe. What worked for these hospitals will not work exactly the same way for you. You and others in your facility will need to develop strategies for planning, implementing, and measuring your initiatives that match your environment and culture and adapt to your unique challenges and opportunities.

If you have read through the preceding section, we hope you now:

• Understand high reliability concepts more clearly. Although the concepts are simple, they can also be threatening. Really embracing them will require that you openly acknowledge and respond to risks your patients face and that you reject a hierarchical approach to decisionmaking in favor of one that defers to the expertise of others—even when they are less senior in the organization or from professions different from your own. To become a high reliability organization you will need to both understand these concepts and support a culture that makes their application possible.
• Learn from examples of how these concepts have been applied in hospitals. We hope you were intrigued and excited by the range of improvements that are described in this document. Some represent small and rapid changes that are likely to produce modest improvements while others are major initiatives that require extended periods of planning and considerable resources. Hospitals in the HRO Network are certainly not the only ones experimenting with ways to make their patients safer and their quality better. But the breadth of their efforts means that the examples offer something of value to every hospital leader.
• Apply HRO concepts to the most pressing needs you face. Many people who work in hospitals—even those who are leaders—sometimes feel that they lack the organizational support needed to make substantive improvements. It's clear that executive- and even board-level support are enormously valuable in becoming a high reliability organization, but it's also clear that each person has opportunities to make improvements. We suggest you consider starting with smaller initiatives that don't necessarily require extensive support from others. As you begin to model and use the HRO concepts described in this document you'll learn a great deal. You can also achieve some small successes that can lay the groundwork for bigger initiatives. Each system in the HRO Network made progress slowly and incrementally.

Return to Contents
Proceed to Next Section