Creation and Coordination of Operational and Evaluation Structure
Project Oversight
The internal oversight of this project has multiple levels, representing the individuals and groups actively involved in the project. These include:
- Chief Executive Officer.
- Executive staff.
- Lean Systems Improvement Department.
- Evaluation team.
- Black Belts and rapid improvement eventh (RIE) teams.
The CEO has led the system transformation and meets monthly with executive
staff to discuss progress and make adjustments as necessary. They also
review the evaluation results at the system and value stream levels (described
below). The executive staff meets with their assigned facilitator, senior
facilitator, and senior financial analyst to discuss RIE scheduling, data collection,
team membership and evaluation results. These meetings can range from
monthly to weekly depending on the executive staff member and the scope of
their system area.
The facilitators in the Lean Systems Improvement Department are becoming experts
within their respective value streams and are the "primary keepers" of
data and other information for their respective value streams. The senior
facilitator conducts quarterly meetings with the Black Belts and facilitators
as another way to ensure there is communication between system areas and among
operational managers. The quarterly agenda is partially defined by the Black Belts.
The senior facilitator, senior financial analyst, and health services research
director comprise the evaluation team and meet with the CEO monthly to review
the metrics which are collected on a continual basis as more and more RIEs
are implemented. This team developed the structure for collecting the
data and formatting the information in a way that could be communicated to
various levels throughout the organization.
External oversight has been provided by an external advisory committee, including
representation from other industries, since 2003 during the planning phase
of the project. Go to Appendix A for a list of
current advisory committee members.
Evolution of Project Structure and Coordination
The structure for implementing, coordinating, and evaluating system redesign
evolved as more RIEs were implemented, as more segments of the health care
system were affected by process redesign and, most importantly, as a greater
understanding of Lean and its application developed over the course of the project.
Initially Denver Health proposed to develop structures and processes capable
of coordinating and aligning 100 projects initiated by the 50 Black Belts (2
projects per Black Belt) during 2005. Denver Health then decided to hire three
industrial engineers trained in Lean to provide project support. Each of the
industrial engineers was then assigned a group of Black Belts that were responsible
for broad system areas: hospital, community health centers, and finance.
Because one of the important aspects of both sequencing projects and sustaining
improvement is to implement progressively more complex projects utilizing progressively
more sophisticated Lean tools, by June 2005 and prior to the task order start
date, each of the Black Belts was also asked to complete a 5Si
project as a "pre-step" to implementing rapid process improvement
projects.
By July 2005 each of the Black Belts was then to propose a more sophisticated
Lean project focused on process improvement using a structured template created
to assist with standardizing and coordinating the multiple projects. The
Black Belts were to develop these projects based on implementing Lean as part
of their daily work.
At the same time, Denver Health began to work with a consultant who introduced
Denver Health to the rapid improvement event (RIE) as the
Kaizenj approach to process improvement, starting with
the operating room. The executive team believed that this approach would complement
the daily work Lean project approach used by the Black Belts.
Critical Lesson #1: Modifying Coordination/Evaluation Structure
The original structure for coordinating and evaluating the impact of
many rapid process improvement projects may need to be changed. At
Denver Health, with Black Belts identifying process improvements, it
was originally believed there would be minimal need for coordination
and prioritization of projects within the system areas each of the
Black Belts represented. However, we found that there was a significant
need for coordination. Therefore, we developed the value stream
mapk to
provide a structure for coordinating and linking projects within each
of the system areas. The structure for implementing rapid process
improvement events became driven by the value stream map, although
Black Belts continued to identify improvement projects within their
scope of operational responsibility. |
The first major change in approach was to develop system-level value stream
maps defined by executive staff. They identified a set of five strategic organization
components that were critical to Denver Health's survival:
- Access to care.
- Billing.
- Outpatient flow.
- Inpatient flow.
- Operating room flow
As part of the planning for the value stream mapping events for each these
five areas, it was important to define the scope of processes that represented
each of these areas. The beginning and end points of the processes that define
each system area are as follows:
- The access to care value stream begins with the patient attempting to access
the system to obtaining an appointment.
- The billing value stream begins with the generation of a charge to the
receipt of payment.
- The outpatient flow begins with patient entrance into ambulatory care until
leaving the clinic.
- The inpatient flow begins with the patient entrance into hospital based
care system until discharge.
- The operating room flow begins with the decision for surgery until the
discharge from post-anesthesia care unit.
In September 2005, the value stream maps were created for each of these five
system areas. At least one executive staff member was assigned responsibility
for each value stream map. The access and billing to the chief financial
officer; outpatient flow to the executive director of the community health
centers; inpatient flow to a team that include the chief nursing officer, the
chief operating officer and the associate medical director; and operating room
to the chief nursing officer.
RIEs were identified from each these value streams by the executive staff. The
teams that created the maps included operational staff intimately involved
with the processes represented by each of these five system areas. The mapping
teams were selected by the CEO and executive staff; executive staff also participated
in the 2½-day value stream mapping sessions. RIEs were often prioritized
by beginning at the most distal component of the processes to create pull. The
value stream maps were intended to assist with the identification of RIEs in
a sequence that would be optimal for maximizing system flow and reducing waste.
The second major change in structure for implementing many rapid process improvement
projects was the development of "Rapid Improvement Event Week." Because
there were three industrial engineers, Denver Health began with three RIEs
occurring simultaneously during the third week of each month. The industrial
engineers acted as facilitators during RIE week, sharing with teams their expertise
in Lean, and were assigned one or two of the five system areas. With
three RIEs each month, two of the value streams would not participate in an
RIE during any month.
Critical Lesson #2: Selection of Rapid Improvement Events
Rather than randomly choosing RIEs from disparate components and processes
in the organization, the value stream maps would be key to identifying
the appropriate sequence and linking of RIEs. Using value stream
maps as a tool for identifying RIEs would create a "pull" approach
to identifying RIEs rather than a "push" process, following
the Lean philosophy to process improvement. Efficient systems create
products and services that are demanded or pulled by the consumer and
not pushed by the producer. Therefore, RIEs would be identified
by understanding the demand on the health care system by applying the
value stream mapping approach. |
There were two primary reasons for starting with three RIEs a month:
- Three engineers were available to facilitate RIEs; and the time that it
takes to prepare, conduct, and follow up one RIE per month was a full-time
position.
- As Denver Health was beginning to use the RIE Week-Kaizen approach to process
improvement (constant process analysis), leadership thought it was important
to address any obstacles before increasing to as many as 10 RIEs per month.
The structure of RIE week is described in Table 1. (Go
to Appendix B for activities for the 3-week period of pre-RIE preparation and the 3-week
period of post-RIE week followup.)
Critical Lesson #3: Preparation of Rapid Improvement Event
Week
Although the current State and future State maps are developed as part
of RIE week, it was often necessary
to identify metrics and collect data prior to RIE week so as to have
a "jump start" during RIE week. One week was not enough
time to ensure that appropriate and sufficient data would be collected
to determine the impact of the process improvement. |
All team members attend the Friday out-briefing and each participates in the
presentation. All the executive staff and the CEO are present at this meeting.
Critical Lesson #4: Involvement of Executive Staff
The need for visible executive staff involvement is critical. With
Black Belt-initiated projects, there was not a structure for formally
reporting the results of their process improvement projects to executive
staff. With the RIE-defined structure, it became clear that this
component of internal communication was important to team members and
to executive staff. |
After 4 months of implementing three RIEs per month, Denver Health increased
this number to five RIEs per month whereby each value stream would be
represented. With this process change, Black Belts were asked to assist with
facilitating RIEs.
Critical Lesson #5: Workload Constraints of Black Belts
Because the Black Belts were being tapped to not only participate in
process improvement projects as part of their daily work but also as
part of RIEs, they communicated their concerns about coordinating and
implementing RIEs to executive staff and at the Black Belt quarterly
meetings. After about 3 months, executive leaders realized that Black
Belts were strained to add these responsibilities to their existing full-time
jobs and recognized there was a need to hire full-time facilitators for
each RIE scheduled during the month. |
This change was also a result of a significant learning that although RIEs
are defined as week-long events, the event will be disorganized and will not
deliver its maximal potential without 3 weeks of advanced preparation and 3
weeks of followup. The hiring of full-time facilitators was necessary
in order to make the RIE a success.
Therefore, in May 2006, with the expansion of RIEs came the need to hire more
facilitators and to move the implementation, structure, and coordination from
the Health Services Research Department to Denver Health operations. A
Lean Systems Improvement Department was created, which includes a senior Lean
facilitator that reports to the CEO and is one of the members of the evaluation
team. The senior Lean facilitator manages the facilitators assigned
to each of the value streams and works with each of the executive staff and
facilitators to:
- Schedule rapid improvement events.
- Identify appropriate team members.
- Ensure appropriate scope.
- Define metrics.
- Assure pre-data collection.
- Monitor and coordinate appropriate followup.
Return to Contents
Evaluating Impact of Implementing Lean
Tools and Creating Structure
The methods for evaluating the impact of many small rapid process improvement
projects also expanded with the change in structure from primarily Black Belt-initiated
projects to many rapid improvement events identified through value stream mapping. The
evaluation of the impact of the many projects was originally focused on methods
related to the evaluation of each of the Black Belt-initiated projects followed
by an evaluation at the system level by monitoring metrics at that level.
Critical Lesson #6: Expansion of the Evaluation
The original proposal called for inclusion of only Black Belt projects
and a system-level evaluation. The evaluation was expanded to include
RIE-level metrics in June 2005 and again to add value stream metrics
in February 2006. After about 6 months into implementation of the rapid
improvement event and value stream mapping Lean tools, the executive
staff decided to monitor metrics relevant to their operational areas
of responsibility between the RIE level and the system level. Hence,
a tiered set of metrics evolved, as illustrated in Figure 2. |
Black Belt-Level Metrics
The evaluation of the Black Belt projects has been less formal than the evaluation
of the other three levels of the redesign evaluation for the following reasons:
- The selection of Black Belt projects is determined by the Black Belts and
the employees in their operational area of responsibility and not identified
through the more formal process of value stream mapping.
- The process by which the improvement occurred may or may not have been
through the use of Kaizen and the RIE approach.
- The improvement may have occurred without the use of Lean tools and may
have occurred though operational decisions for change.
In February 2006, the Black Belts began submitting reports to the CEO using
a standard format. These reports are reviewed by the CEO and evaluation
team. By October 2006, the 50 Black Belts had submitted 205 monthly reports
to the CEO, with an estimated $701,697l in
new revenue or cost savings on an annual basis. Eighteen of these projects
contributed to a financial impact; the other projects were focused on improving
flow of a specific process or eliminating non-value added steps in a process
and have not been translated into a realized financial gain.
One example of a successful Black Belt project was the application of the
5S tool. The Engineering Department reduced the number of sites with
inventory from 21 to 19. This improvement translated into a cost reduction
for materials management generated from space, administrative, and inventory
savings of $221,346.
In addition, the paramedics initiated a project that streamlined the scheduling
process, thereby reducing the need for overtime. With this reduction
in waste, the paramedics reduced the cost of overtime by $432,000.
RIE-Level Metrics
The executive staff member, RIE team, and facilitators identify the RIE metrics
prior to RIE week. Executive staff identified the following broad metrics
that each RIE should target:
- Revenue per discharge/encounter.
- Costs per discharge/encounter.
- Employee engagement-employee participation on RIE teams.
Therefore, each RIE should have at least one metric that is expected to have
an impact on revenue or costs. All RIEs would have an impact on employee
engagement because the teams are encouraged to involve employees related to
each process and employees unfamiliar with the process. The more RIEs,
the more employees engaged. A fourth metric was added in October 2006
for each RIE to target—a quality metric.
It is very important to clearly define the scope of the process and the process
problem the RIE is to address prior to selecting metrics. This has been
a challenge for RIE teams to understand and can sometime result in the selection
of metrics that are not tied to measuring the impact of the expected improvement.
Critical Lesson #7: Choosing the Metrics, Collecting the Data
Identifying measures directly linked to the process improvements identified
by employees can be a challenge. This became apparent during the
presentation by RIE team members during out-briefing to executive staff
at the conclusion of RIE week. The outcome measures were often
weakly linked or not inked to the process problem statements. Therefore,
additional training was provided to the Black Belts and facilitators
on methods for identifying problems and their related metrics. Training
was conducted by Black Belts that were successful in understanding metrics;
they presented these successful projects as examples to the group. With
the hiring of additional facilitators, additional training was also necessary
and was provided by the Health Services Research Department.
Another important consideration relates to the degree to which the data
for a metric are cumbersome to obtain. For instance, followup data
were often not collected for those processes in which the metric required
observation and manual recording of time for each step in a process. It
is more important for data to be readily available when identifying a
metric rather than to select to identify a metric that is directly linked to the process improvement
but for which data are not readily available. |
As noted above, a quality metric was added, and it appears that this type
of metric may offer a different identification challenge for RIE teams. For
example, many RIEs are not focused on a clinical improvement but may be focused
on a process improvement related to reducing the number of errors in preparing
a bill. It is expected that, as more examples of nonclinical quality
indicators are gathered, facilitators and teams will gain a better understanding
in how to identify these quality metrics.
Table 2 provides an example of the template developed by the evaluation team
for monitoring the numerous metrics related to the many rapid improvement events
that are implemented each month. The information on this spreadsheet
is distinguished by value stream and is provided to executive staff members
responsible for their respective value stream mapping area.
The assigned facilitator, senior lead facilitator and senior financial analyst
review these metrics monthly with each of the responsible executive staff members.
The evaluation team also reviews these metrics with the CEO on a monthly basis,
highlighting relevant areas of change or of problematic no change. The
facilitators funnel the metric updates to the senior finance analyst who then
translates the metrics into financial impact, if possible. Once a month, the
CEO and the executive staff review the overall process for implementing, coordinating,
and evaluating RIEs; each quarter, they review all of the RIE metrics with
the evaluation team.
Figure 3 depicts the flow of information for reviewing evaluation metrics
at the RIE level. The RIE team members and facilitator are responsible
for gathering the data for the metrics that have been identified for each RIE. The
facilitator then reports these data to the senior financial analyst who records
the information, whether or not it is a financial metric. The senior
facilitator and value stream facilitator work with the senior financial analyst
to ensure the data are reported timely and accurately. The senior financial
analyst translates the financial indicators into a financial impact.
This team meets monthly with the executive staff representative to review
metrics on all past and current RIEs for the relevant value stream. During
this process the evaluation team reviews the data for methodology accuracy. The
report is reviewed monthly with the CEO for all value streams. The CEO
reviews the RIE-level metrics quarterly with the executive staff.
Critical Lesson #8: Improving the Structure for Defining
and Tracking Metrics
The following guidelines were developed to improve the structure for
defining problematic metrics and determining how long metrics should
be tracked:
- If the metric reflects achievement of the target and the target reflects
optimal or best practice outcome, the metric is followed for 3 months
of stability. If stable for 3 months, the metric will move to
3-month monitoring, then 6 months, and then yearly.
- If a metric reflects achievement of target but the target was below
optimal or best practice, this is discussed at executive staff/CEO
level and the target is raised.
- If the metric has not been achieved or was achieved and not sustained,
the facilitator and executive staff member may refine the problem statement
and associated metrics.
|
At the time of preparation of this report, 59 RIEs have been implemented,
and there are several examples of RIEs that have improved process flow. At
the same time, even though a rapid improvement event may remove process waste,
this waste is not always readily translated into cost savings of revenue.
Table 2 also provides operational leaders with information on the financial
impact RIEs are having on a particular system area. For instance, a group
of RIEs may improve operating room (OR) gross charges. Some entries
in Table 2 related to the OR value stream include the following:
- Sustained improvement in patient receiving antibiotic within 60 minutes
of incision.
- Maintenance of percentages of OR cases that have started on time.
- Reduction in billing errors for OR/anesthesia billing.
Note that the OR gross charges have dramatically increased. This may
be in part due to the improvement in start times and billing processes.
Other system areas where there has been a marked improvement include the outpatient
and access value streams, which are discussed below:
- One of the largest of the eight Denver Health community health centers
implemented an RIE which removed waste in the flow of clinic visits by improving
the standardization of staff responsibilities related to a clinic visit. The
impact of this RIE has resulted in improved patient flow through a sustained
increase in the number of patient visits per provider session, also resulting
in an increase in net revenue.
- The access system area implemented a RIE focused on reducing the value
of inpatient charges that remained self-pay after 90 days. As a safety
net hospital, Denver Health provides care to many uninsured patients classified
as self-pay. These patients are responsible for paying the full charge
for their visit if they do not qualify for one of various subsidy and insurance
programs. If patients do not qualify for an insurance program, Denver
Health is not expected to receive a large percentage of reimbursement for
the costs related to these services. If the paperwork to determine
whether these patients qualify for these programs is not processed within
90 days of the encounter, these patients are no longer eligible. Using
Lean tools to improve the process by which inpatients are qualified and enrolled
into these programs, Denver Health has reduced the monthly charges that remain
as self-pay after 90 days and the number of self-pay accounts created by
almost 50 percent. This improvement is expected to have a positive
impact on net revenue.
Value Stream-Level Metrics
The value stream metrics are intended to provide progress of the impact of
RIEs on the value stream system area. The metrics for each of the value
streams were selected by the executive staff responsible for each system area,
in collaboration with the facilitators. This was necessary to ensure
the metrics are relevant to the expected impact of the RIEs implemented and
to ensure that the data are readily available and not cumbersome to obtain.
Table 3 provides a list of these metrics. Some of these metrics overlap
with the RIE-level metrics and system-level metrics, and some of these metrics
need refining of the definition. These metrics have changed and continue
to be discussed for completeness and appropriateness. In addition, the
value stream system areas may also change over time, depending on the progress
of the impact of the RIEs on a particular value stream.
System-Level Metrics
Table 4 describes a dashboard of metrics that have been identified as representing
the areas of redesign framework (patient safety and quality, efficiency, customer
service, workforce development, and architecture/environment) and as reflecting
the outcomes of value stream mapping at a global system level. Importantly
these metrics are available in most health care systems and are of interest
to the leadership at Denver Health and other leadership teams. Many of
these metrics can also be compared to benchmark and best practice data. The
baseline value for each of these metrics is included and begins in 2004. The
frequency of reporting is dependent on how often the data are updated.
Broad, driving metric areas listed in Table 4 that RIEs should target include
the following:
- Total cost and cost per hospital discharge or outpatient encounter (metrics
2-4).
- Net revenue and revenue per hospital discharge or outpatient encounter
(metrics 5-7).
- Employee engagement in the transformation process (metric 8).
Clearly both from Denver Health's perspective and for dissemination,
financial metrics are critically important. Demonstration of improvement
in financial metrics is also essential for sustainability; few if any CEOs
would opt to sustain an effort which undermined the financial viability of
the organization.
Another critically important factor in sustainability is the creation of a
culture committed to transformation. A key driver will be employee engagement.
Therefore, number of employees engaged in the transformation process was selected
as one of the driving metrics. This number will include employees directly
and indirectly engaged in RIE events, both as members of RIE teams and as those
that are briefly consulted during the RIE week project.
These driver metrics were selected based on the need to see financial gains
from the outset, the importance of involving as many employees as possible
in the transformation process, and the desire to have rapid process improvement
event teams focus on a few driving metrics. Even though these broad metrics
have been identified as the drivers for process redesign, projects will still
include other dashboard system metrics and project-specific metrics as part
of the expected improvement results. Many RIE-specific metrics will roll to
system-level metrics. For example, in the operating room RIE, one metric used
was reduction in overtime usage, which would ultimately roll to cost per discharge
and net revenue per discharge.
In interpreting the change in these metrics one must exercise caution in attributing
the change directly to the implementation of Lean. In addition, many
of these indicators might not move immediately as RIEs and Black Belt projects
might only indirectly impact these system measures.
Also, it is a challenge to determine the direct impact of system redesign
at the system level as there are many outside factors that can affect these
measures. For instance, Denver Health has opened a new wing of the hospital
and a new clinic, where construction began before the implementation of Lean. Therefore,
one would expect patient visits and admissions to increase as capacity
increased, irrespective of Lean implementation.
Net revenue. The overall financial condition of Denver Health
is very positive and 2006 has been the best year for net revenue in recent
history. With the Black Belt projects and RIEs, Denver Health has realized
cost savings and revenue enhancements totaling $2,694,219. This amount
is a conservative estimate as it represents the realized benefit to date, and
is not annualized. The cost for implementing Lean during the 18-month
project period is as follows:
Consultant and facilitators: $741,000.
Evaluation: $134,000.
Total cost (est.): $875,000.
Therefore the net gain to Denver Health during the 18-month project period
is estimated to be $1,819,219. It is expected that the consultant costs
will reduce in the next 12 months, potentially to zero. Over this time
Denver Health, will have gained the internal expertise to implement and coordinate
the RIEs independently. The financial benefits from the RIEs and Black
Belt projects are expected to increase as more and more improvements are implemented
and linked.
In addition, as the cost reductions and revenue enhancements are realized
dollars, these benefits represent the minimum benefits redesign has had on
the Denver Health system to date. For example, the increase in the percentage
of surgery patients receiving an antibiotic in a timely manner has not been
translated into a cost savings.
Patient satisfaction. Denver Health also has experienced
an improvement in patient satisfaction as measured by the survey developed
and administered by Press Ganey. Since 2005, Denver Health experienced
an improvement in 49 of 50 survey questions. A question that is commonly
used as an all inclusive indicator of satisfaction is whether the patient would
recommend Denver Health to a person he or she knows. Denver Health improved
from the 10th to the 54th percentile for this question for hospitals of similar
bed size (300-449 beds) and structure. Although Denver Health has much
room for improvement in this area, this upward movement is very promising.
Employee engagement. The number of employees engaged in health
care system transformation has also been steadily increasing as represented
by the Employee Engagement metric. The degree of employee engagement
in the redesign process has been further evaluated through an employee questionnaire
tailored to Denver Health. This employee questionnaire was distributed
via the internal Web site in June 2006 and had a response rate of 61 percent.
Some of the specific questions directly linked to Lean are described in Table 5. More than one-third of the respondents (916/2,622) indicated they
have participated in a rapid improvement event, which is more than the 201
that were tracked as being officially part of an RIE in 2006
(Table 4). This may be due
to the fact that many employees may be affected by an RIE but not officially
on the RIE team. The survey indicates that as of June 2006, the majority
of the respondents have been involved in the Lean initiative and understood
the Lean approach and the benefits of this approach.
The system metric table is a useful tool for monitoring changes in the health
care system. Although there are many factors that can influence the changes
in these metrics both internal and external to Denver Health, it provides leadership
with information on the status of the values, such as reasons the numbers may
change and other factors that may influence the values. For example,
improvements in reporting are common and have had a significant influence on
some of the system metrics. It is important to continue to monitor these
metrics so that leadership can see the net impact of all of these factors,
some of which they can control and some of which they cannot.
h. The RIE utilizes
a team which focuses on a particular process with the aim of reducing 50
percent of the waste. The week-long event requires prior preparation and
subsequent monitoring of outcome over time.
i. The steps to 5S are as follows: (1) Sort through all contents of an area; (2) Shine
and inspect through cleaning; (3) Set in order; (4) Standardize
and share information; (5) Sustain the improvements made.
j. Kaizen is the Lean manufacturing term for continuous improvement and describes an environment
in which organizations and individuals work proactively to focus on improving
the manufacturing process by understanding both the current process and future
goals. Denver Health chose the RIE method to Kaizen, whereby teams of
employees and executives identify a process, value stream map the process,
gather data, implement the improvement, and evaluate the impact of the
implementation during a 1-week project time period.
k. A value stream
map depicts all the steps required to bring a product or service from the beginning
of the process through to the customer (e.g., patient, provider, or staff member). The
map fulfills two key functions: First, it depicts the current and future state
of a product/service's production path from customer to supplier through
a visual representation of every process for both material/patient and information
flow; and second, it facilitates the identification of process improvement
projects in optimal order.
l. Verified by
the evaluation team's senior financial analyst.
Return to Contents
Proceed to Next Section