You Are Here: AHRQ Home > Quality & Patient Safety > Quality Information & Improvement > Managing and Evaluating Rapid-Cycle Process Improvements as Vehicles for Hospital System Redesign (continued)

Managing and Evaluating Rapid-Cycle Process Improvements as Vehicles for Hospital System Redesign

Final Contract Report

Creation and Coordination of Operational and Evaluation Structure

Project Oversight

The internal oversight of this project has multiple levels, representing the individuals and groups actively involved in the project. These include:

Chief Executive Officer.
Executive staff.
Lean Systems Improvement Department.
Evaluation team.
Black Belts and rapid improvement event^h (RIE) teams.

The CEO has led the system transformation and meets monthly with executive staff to discuss progress and make adjustments as necessary. They also review the evaluation results at the system and value stream levels (described below). The executive staff meets with their assigned facilitator, senior facilitator, and senior financial analyst to discuss RIE scheduling, data collection, team membership and evaluation results. These meetings can range from monthly to weekly depending on the executive staff member and the scope of their system area.

The facilitators in the Lean Systems Improvement Department are becoming experts within their respective value streams and are the "primary keepers" of data and other information for their respective value streams. The senior facilitator conducts quarterly meetings with the Black Belts and facilitators as another way to ensure there is communication between system areas and among operational managers. The quarterly agenda is partially defined by the Black Belts.

The senior facilitator, senior financial analyst, and health services research director comprise the evaluation team and meet with the CEO monthly to review the metrics which are collected on a continual basis as more and more RIEs are implemented. This team developed the structure for collecting the data and formatting the information in a way that could be communicated to various levels throughout the organization.

External oversight has been provided by an external advisory committee, including representation from other industries, since 2003 during the planning phase of the project. Go to Appendix A for a list of current advisory committee members.

Evolution of Project Structure and Coordination

The structure for implementing, coordinating, and evaluating system redesign evolved as more RIEs were implemented, as more segments of the health care system were affected by process redesign and, most importantly, as a greater understanding of Lean and its application developed over the course of the project.

Initially Denver Health proposed to develop structures and processes capable of coordinating and aligning 100 projects initiated by the 50 Black Belts (2 projects per Black Belt) during 2005. Denver Health then decided to hire three industrial engineers trained in Lean to provide project support. Each of the industrial engineers was then assigned a group of Black Belts that were responsible for broad system areas: hospital, community health centers, and finance.

Because one of the important aspects of both sequencing projects and sustaining improvement is to implement progressively more complex projects utilizing progressively more sophisticated Lean tools, by June 2005 and prior to the task order start date, each of the Black Belts was also asked to complete a 5Sⁱ project as a "pre-step" to implementing rapid process improvement projects.

By July 2005 each of the Black Belts was then to propose a more sophisticated Lean project focused on process improvement using a structured template created to assist with standardizing and coordinating the multiple projects. The Black Belts were to develop these projects based on implementing Lean as part of their daily work.

At the same time, Denver Health began to work with a consultant who introduced Denver Health to the rapid improvement event (RIE) as the Kaizen^j approach to process improvement, starting with the operating room. The executive team believed that this approach would complement the daily work Lean project approach used by the Black Belts.

Critical Lesson #1: Modifying Coordination/Evaluation Structure

The original structure for coordinating and evaluating the impact of many rapid process improvement projects may need to be changed. At Denver Health, with Black Belts identifying process improvements, it was originally believed there would be minimal need for coordination and prioritization of projects within the system areas each of the Black Belts represented. However, we found that there was a significant need for coordination. Therefore, we developed the value stream map^k to provide a structure for coordinating and linking projects within each of the system areas. The structure for implementing rapid process improvement events became driven by the value stream map, although Black Belts continued to identify improvement projects within their scope of operational responsibility.

The first major change in approach was to develop system-level value stream maps defined by executive staff. They identified a set of five strategic organization components that were critical to Denver Health's survival:

Access to care.
Billing.
Outpatient flow.
Inpatient flow.
Operating room flow

As part of the planning for the value stream mapping events for each these five areas, it was important to define the scope of processes that represented each of these areas. The beginning and end points of the processes that define each system area are as follows:

The access to care value stream begins with the patient attempting to access the system to obtaining an appointment.
The billing value stream begins with the generation of a charge to the receipt of payment.
The outpatient flow begins with patient entrance into ambulatory care until leaving the clinic.
The inpatient flow begins with the patient entrance into hospital based care system until discharge.
The operating room flow begins with the decision for surgery until the discharge from post-anesthesia care unit.

In September 2005, the value stream maps were created for each of these five system areas. At least one executive staff member was assigned responsibility for each value stream map. The access and billing to the chief financial officer; outpatient flow to the executive director of the community health centers; inpatient flow to a team that include the chief nursing officer, the chief operating officer and the associate medical director; and operating room to the chief nursing officer.

RIEs were identified from each these value streams by the executive staff. The teams that created the maps included operational staff intimately involved with the processes represented by each of these five system areas. The mapping teams were selected by the CEO and executive staff; executive staff also participated in the 2½-day value stream mapping sessions. RIEs were often prioritized by beginning at the most distal component of the processes to create pull. The value stream maps were intended to assist with the identification of RIEs in a sequence that would be optimal for maximizing system flow and reducing waste.

The second major change in structure for implementing many rapid process improvement projects was the development of "Rapid Improvement Event Week." Because there were three industrial engineers, Denver Health began with three RIEs occurring simultaneously during the third week of each month. The industrial engineers acted as facilitators during RIE week, sharing with teams their expertise in Lean, and were assigned one or two of the five system areas. With three RIEs each month, two of the value streams would not participate in an RIE during any month.

Critical Lesson #2: Selection of Rapid Improvement Events

Rather than randomly choosing RIEs from disparate components and processes in the organization, the value stream maps would be key to identifying the appropriate sequence and linking of RIEs. Using value stream maps as a tool for identifying RIEs would create a "pull" approach to identifying RIEs rather than a "push" process, following the Lean philosophy to process improvement. Efficient systems create products and services that are demanded or pulled by the consumer and not pushed by the producer. Therefore, RIEs would be identified by understanding the demand on the health care system by applying the value stream mapping approach.

There were two primary reasons for starting with three RIEs a month:

Three engineers were available to facilitate RIEs; and the time that it takes to prepare, conduct, and follow up one RIE per month was a full-time position.
As Denver Health was beginning to use the RIE Week-Kaizen approach to process improvement (constant process analysis), leadership thought it was important to address any obstacles before increasing to as many as 10 RIEs per month.

The structure of RIE week is described in Table 1. (Go to Appendix B for activities for the 3-week period of pre-RIE preparation and the 3-week period of post-RIE week followup.)

Critical Lesson #3: Preparation of Rapid Improvement Event Week

Although the current State and future State maps are developed as part of RIE week, it was often necessary to identify metrics and collect data prior to RIE week so as to have a "jump start" during RIE week. One week was not enough time to ensure that appropriate and sufficient data would be collected to determine the impact of the process improvement.

All team members attend the Friday out-briefing and each participates in the presentation. All the executive staff and the CEO are present at this meeting.

Critical Lesson #4: Involvement of Executive Staff

The need for visible executive staff involvement is critical. With Black Belt-initiated projects, there was not a structure for formally reporting the results of their process improvement projects to executive staff. With the RIE-defined structure, it became clear that this component of internal communication was important to team members and to executive staff.

After 4 months of implementing three RIEs per month, Denver Health increased this number to five RIEs per month whereby each value stream would be represented. With this process change, Black Belts were asked to assist with facilitating RIEs.

Critical Lesson #5: Workload Constraints of Black Belts

Because the Black Belts were being tapped to not only participate in process improvement projects as part of their daily work but also as part of RIEs, they communicated their concerns about coordinating and implementing RIEs to executive staff and at the Black Belt quarterly meetings. After about 3 months, executive leaders realized that Black Belts were strained to add these responsibilities to their existing full-time jobs and recognized there was a need to hire full-time facilitators for each RIE scheduled during the month.

This change was also a result of a significant learning that although RIEs are defined as week-long events, the event will be disorganized and will not deliver its maximal potential without 3 weeks of advanced preparation and 3 weeks of followup. The hiring of full-time facilitators was necessary in order to make the RIE a success.

Therefore, in May 2006, with the expansion of RIEs came the need to hire more facilitators and to move the implementation, structure, and coordination from the Health Services Research Department to Denver Health operations. A Lean Systems Improvement Department was created, which includes a senior Lean facilitator that reports to the CEO and is one of the members of the evaluation team. The senior Lean facilitator manages the facilitators assigned to each of the value streams and works with each of the executive staff and facilitators to:

Schedule rapid improvement events.
Identify appropriate team members.
Ensure appropriate scope.
Define metrics.
Assure pre-data collection.
Monitor and coordinate appropriate followup.

Return to Contents

Evaluating Impact of Implementing Lean Tools and Creating Structure

The methods for evaluating the impact of many small rapid process improvement projects also expanded with the change in structure from primarily Black Belt-initiated projects to many rapid improvement events identified through value stream mapping. The evaluation of the impact of the many projects was originally focused on methods related to the evaluation of each of the Black Belt-initiated projects followed by an evaluation at the system level by monitoring metrics at that level.

Critical Lesson #6: Expansion of the Evaluation

The original proposal called for inclusion of only Black Belt projects and a system-level evaluation. The evaluation was expanded to include RIE-level metrics in June 2005 and again to add value stream metrics in February 2006. After about 6 months into implementation of the rapid improvement event and value stream mapping Lean tools, the executive staff decided to monitor metrics relevant to their operational areas of responsibility between the RIE level and the system level. Hence, a tiered set of metrics evolved, as illustrated in Figure 2.

Black Belt-Level Metrics

The evaluation of the Black Belt projects has been less formal than the evaluation of the other three levels of the redesign evaluation for the following reasons:

The selection of Black Belt projects is determined by the Black Belts and the employees in their operational area of responsibility and not identified through the more formal process of value stream mapping.
The process by which the improvement occurred may or may not have been through the use of Kaizen and the RIE approach.
The improvement may have occurred without the use of Lean tools and may have occurred though operational decisions for change.

In February 2006, the Black Belts began submitting reports to the CEO using a standard format. These reports are reviewed by the CEO and evaluation team. By October 2006, the 50 Black Belts had submitted 205 monthly reports to the CEO, with an estimated $701,697^l in new revenue or cost savings on an annual basis. Eighteen of these projects contributed to a financial impact; the other projects were focused on improving flow of a specific process or eliminating non-value added steps in a process and have not been translated into a realized financial gain.

One example of a successful Black Belt project was the application of the 5S tool. The Engineering Department reduced the number of sites with inventory from 21 to 19. This improvement translated into a cost reduction for materials management generated from space, administrative, and inventory savings of $221,346.

In addition, the paramedics initiated a project that streamlined the scheduling process, thereby reducing the need for overtime. With this reduction in waste, the paramedics reduced the cost of overtime by $432,000.

RIE-Level Metrics

The executive staff member, RIE team, and facilitators identify the RIE metrics prior to RIE week. Executive staff identified the following broad metrics that each RIE should target:

Revenue per discharge/encounter.
Costs per discharge/encounter.
Employee engagement-employee participation on RIE teams.

Therefore, each RIE should have at least one metric that is expected to have an impact on revenue or costs. All RIEs would have an impact on employee engagement because the teams are encouraged to involve employees related to each process and employees unfamiliar with the process. The more RIEs, the more employees engaged. A fourth metric was added in October 2006 for each RIE to target—a quality metric.

It is very important to clearly define the scope of the process and the process problem the RIE is to address prior to selecting metrics. This has been a challenge for RIE teams to understand and can sometime result in the selection of metrics that are not tied to measuring the impact of the expected improvement.

Critical Lesson #7: Choosing the Metrics, Collecting the Data

Identifying measures directly linked to the process improvements identified by employees can be a challenge. This became apparent during the presentation by RIE team members during out-briefing to executive staff at the conclusion of RIE week. The outcome measures were often weakly linked or not inked to the process problem statements. Therefore, additional training was provided to the Black Belts and facilitators on methods for identifying problems and their related metrics. Training was conducted by Black Belts that were successful in understanding metrics; they presented these successful projects as examples to the group. With the hiring of additional facilitators, additional training was also necessary and was provided by the Health Services Research Department.

Another important consideration relates to the degree to which the data for a metric are cumbersome to obtain. For instance, followup data were often not collected for those processes in which the metric required observation and manual recording of time for each step in a process. It is more important for data to be readily available when identifying a metric rather than to select to identify a metric that is directly linked to the process improvement but for which data are not readily available.

As noted above, a quality metric was added, and it appears that this type of metric may offer a different identification challenge for RIE teams. For example, many RIEs are not focused on a clinical improvement but may be focused on a process improvement related to reducing the number of errors in preparing a bill. It is expected that, as more examples of nonclinical quality indicators are gathered, facilitators and teams will gain a better understanding in how to identify these quality metrics.

Table 2 provides an example of the template developed by the evaluation team for monitoring the numerous metrics related to the many rapid improvement events that are implemented each month. The information on this spreadsheet is distinguished by value stream and is provided to executive staff members responsible for their respective value stream mapping area.

The assigned facilitator, senior lead facilitator and senior financial analyst review these metrics monthly with each of the responsible executive staff members. The evaluation team also reviews these metrics with the CEO on a monthly basis, highlighting relevant areas of change or of problematic no change. The facilitators funnel the metric updates to the senior finance analyst who then translates the metrics into financial impact, if possible. Once a month, the CEO and the executive staff review the overall process for implementing, coordinating, and evaluating RIEs; each quarter, they review all of the RIE metrics with the evaluation team.

Figure 3 depicts the flow of information for reviewing evaluation metrics at the RIE level. The RIE team members and facilitator are responsible for gathering the data for the metrics that have been identified for each RIE. The facilitator then reports these data to the senior financial analyst who records the information, whether or not it is a financial metric. The senior facilitator and value stream facilitator work with the senior financial analyst to ensure the data are reported timely and accurately. The senior financial analyst translates the financial indicators into a financial impact.

This team meets monthly with the executive staff representative to review metrics on all past and current RIEs for the relevant value stream. During this process the evaluation team reviews the data for methodology accuracy. The report is reviewed monthly with the CEO for all value streams. The CEO reviews the RIE-level metrics quarterly with the executive staff.

Critical Lesson #8: Improving the Structure for Defining and Tracking Metrics

The following guidelines were developed to improve the structure for defining problematic metrics and determining how long metrics should be tracked:

If the metric reflects achievement of the target and the target reflects optimal or best practice outcome, the metric is followed for 3 months of stability. If stable for 3 months, the metric will move to 3-month monitoring, then 6 months, and then yearly.
If a metric reflects achievement of target but the target was below optimal or best practice, this is discussed at executive staff/CEO level and the target is raised.
If the metric has not been achieved or was achieved and not sustained, the facilitator and executive staff member may refine the problem statement and associated metrics.

At the time of preparation of this report, 59 RIEs have been implemented, and there are several examples of RIEs that have improved process flow. At the same time, even though a rapid improvement event may remove process waste, this waste is not always readily translated into cost savings of revenue.

Table 2 also provides operational leaders with information on the financial impact RIEs are having on a particular system area. For instance, a group of RIEs may improve operating room (OR) gross charges. Some entries in Table 2 related to the OR value stream include the following:

Sustained improvement in patient receiving antibiotic within 60 minutes of incision.
Maintenance of percentages of OR cases that have started on time.
Reduction in billing errors for OR/anesthesia billing.

Note that the OR gross charges have dramatically increased. This may be in part due to the improvement in start times and billing processes.

Other system areas where there has been a marked improvement include the outpatient and access value streams, which are discussed below:

One of the largest of the eight Denver Health community health centers implemented an RIE which removed waste in the flow of clinic visits by improving the standardization of staff responsibilities related to a clinic visit. The impact of this RIE has resulted in improved patient flow through a sustained increase in the number of patient visits per provider session, also resulting in an increase in net revenue.
The access system area implemented a RIE focused on reducing the value of inpatient charges that remained self-pay after 90 days. As a safety net hospital, Denver Health provides care to many uninsured patients classified as self-pay. These patients are responsible for paying the full charge for their visit if they do not qualify for one of various subsidy and insurance programs. If patients do not qualify for an insurance program, Denver Health is not expected to receive a large percentage of reimbursement for the costs related to these services. If the paperwork to determine whether these patients qualify for these programs is not processed within 90 days of the encounter, these patients are no longer eligible. Using Lean tools to improve the process by which inpatients are qualified and enrolled into these programs, Denver Health has reduced the monthly charges that remain as self-pay after 90 days and the number of self-pay accounts created by almost 50 percent. This improvement is expected to have a positive impact on net revenue.

Value Stream-Level Metrics

The value stream metrics are intended to provide progress of the impact of RIEs on the value stream system area. The metrics for each of the value streams were selected by the executive staff responsible for each system area, in collaboration with the facilitators. This was necessary to ensure the metrics are relevant to the expected impact of the RIEs implemented and to ensure that the data are readily available and not cumbersome to obtain.

Table 3 provides a list of these metrics. Some of these metrics overlap with the RIE-level metrics and system-level metrics, and some of these metrics need refining of the definition. These metrics have changed and continue to be discussed for completeness and appropriateness. In addition, the value stream system areas may also change over time, depending on the progress of the impact of the RIEs on a particular value stream.

System-Level Metrics

Table 4 describes a dashboard of metrics that have been identified as representing the areas of redesign framework (patient safety and quality, efficiency, customer service, workforce development, and architecture/environment) and as reflecting the outcomes of value stream mapping at a global system level. Importantly these metrics are available in most health care systems and are of interest to the leadership at Denver Health and other leadership teams. Many of these metrics can also be compared to benchmark and best practice data. The baseline value for each of these metrics is included and begins in 2004. The frequency of reporting is dependent on how often the data are updated.

Broad, driving metric areas listed in Table 4 that RIEs should target include the following:

Total cost and cost per hospital discharge or outpatient encounter (metrics 2-4).
Net revenue and revenue per hospital discharge or outpatient encounter (metrics 5-7).
Employee engagement in the transformation process (metric 8).

Clearly both from Denver Health's perspective and for dissemination, financial metrics are critically important. Demonstration of improvement in financial metrics is also essential for sustainability; few if any CEOs would opt to sustain an effort which undermined the financial viability of the organization.

Another critically important factor in sustainability is the creation of a culture committed to transformation. A key driver will be employee engagement. Therefore, number of employees engaged in the transformation process was selected as one of the driving metrics. This number will include employees directly and indirectly engaged in RIE events, both as members of RIE teams and as those that are briefly consulted during the RIE week project.

These driver metrics were selected based on the need to see financial gains from the outset, the importance of involving as many employees as possible in the transformation process, and the desire to have rapid process improvement event teams focus on a few driving metrics. Even though these broad metrics have been identified as the drivers for process redesign, projects will still include other dashboard system metrics and project-specific metrics as part of the expected improvement results. Many RIE-specific metrics will roll to system-level metrics. For example, in the operating room RIE, one metric used was reduction in overtime usage, which would ultimately roll to cost per discharge and net revenue per discharge.

In interpreting the change in these metrics one must exercise caution in attributing the change directly to the implementation of Lean. In addition, many of these indicators might not move immediately as RIEs and Black Belt projects might only indirectly impact these system measures.

Also, it is a challenge to determine the direct impact of system redesign at the system level as there are many outside factors that can affect these measures. For instance, Denver Health has opened a new wing of the hospital and a new clinic, where construction began before the implementation of Lean. Therefore, one would expect patient visits and admissions to increase as capacity increased, irrespective of Lean implementation.

Net revenue. The overall financial condition of Denver Health is very positive and 2006 has been the best year for net revenue in recent history. With the Black Belt projects and RIEs, Denver Health has realized cost savings and revenue enhancements totaling $2,694,219. This amount is a conservative estimate as it represents the realized benefit to date, and is not annualized. The cost for implementing Lean during the 18-month project period is as follows:

Consultant and facilitators: $741,000.
Evaluation: $134,000.
Total cost (est.): $875,000.

Therefore the net gain to Denver Health during the 18-month project period is estimated to be $1,819,219. It is expected that the consultant costs will reduce in the next 12 months, potentially to zero. Over this time Denver Health, will have gained the internal expertise to implement and coordinate the RIEs independently. The financial benefits from the RIEs and Black Belt projects are expected to increase as more and more improvements are implemented and linked.

In addition, as the cost reductions and revenue enhancements are realized dollars, these benefits represent the minimum benefits redesign has had on the Denver Health system to date. For example, the increase in the percentage of surgery patients receiving an antibiotic in a timely manner has not been translated into a cost savings.

Patient satisfaction. Denver Health also has experienced an improvement in patient satisfaction as measured by the survey developed and administered by Press Ganey. Since 2005, Denver Health experienced an improvement in 49 of 50 survey questions. A question that is commonly used as an all inclusive indicator of satisfaction is whether the patient would recommend Denver Health to a person he or she knows. Denver Health improved from the 10th to the 54th percentile for this question for hospitals of similar bed size (300-449 beds) and structure. Although Denver Health has much room for improvement in this area, this upward movement is very promising.

Employee engagement. The number of employees engaged in health care system transformation has also been steadily increasing as represented by the Employee Engagement metric. The degree of employee engagement in the redesign process has been further evaluated through an employee questionnaire tailored to Denver Health. This employee questionnaire was distributed via the internal Web site in June 2006 and had a response rate of 61 percent.

Some of the specific questions directly linked to Lean are described in Table 5. More than one-third of the respondents (916/2,622) indicated they have participated in a rapid improvement event, which is more than the 201 that were tracked as being officially part of an RIE in 2006 (Table 4). This may be due to the fact that many employees may be affected by an RIE but not officially on the RIE team. The survey indicates that as of June 2006, the majority of the respondents have been involved in the Lean initiative and understood the Lean approach and the benefits of this approach.

The system metric table is a useful tool for monitoring changes in the health care system. Although there are many factors that can influence the changes in these metrics both internal and external to Denver Health, it provides leadership with information on the status of the values, such as reasons the numbers may change and other factors that may influence the values. For example, improvements in reporting are common and have had a significant influence on some of the system metrics. It is important to continue to monitor these metrics so that leadership can see the net impact of all of these factors, some of which they can control and some of which they cannot.

^h. The RIE utilizes a team which focuses on a particular process with the aim of reducing 50 percent of the waste. The week-long event requires prior preparation and subsequent monitoring of outcome over time.
^i. The steps to 5S are as follows: (1) Sort through all contents of an area; (2) Shine and inspect through cleaning; (3) Set in order; (4) Standardize and share information; (5) Sustain the improvements made.
^j. Kaizen is the Lean manufacturing term for continuous improvement and describes an environment in which organizations and individuals work proactively to focus on improving the manufacturing process by understanding both the current process and future goals. Denver Health chose the RIE method to Kaizen, whereby teams of employees and executives identify a process, value stream map the process, gather data, implement the improvement, and evaluate the impact of the implementation during a 1-week project time period.
^k. A value stream map depicts all the steps required to bring a product or service from the beginning of the process through to the customer (e.g., patient, provider, or staff member). The map fulfills two key functions: First, it depicts the current and future state of a product/service's production path from customer to supplier through a visual representation of every process for both material/patient and information flow; and second, it facilitates the identification of process improvement projects in optimal order.
^l. Verified by the evaluation team's senior financial analyst.

Return to Contents
Proceed to Next Section