Chapter 8. Evaluation and Quality Improvement
Once ESI has been implemented it is important to
continue ongoing evaluation. In fact, thought
should be given to evaluation prior to
implementation, and plans should be made to
continually evaluate the system.
When evaluating
the success or failure of implementation, it is
important to remember why the triage process was
changed. The following reasons are frequently
identified as driving forces to change existing triage
processes:
- Reduction in variation of assigned triage categories.
- Decreased risk of negative outcomes due to mis-triage, particularly while patients are waiting.
- The ability to obtain more accurate data to use for administrative purposes.
- The need to move from a three-category to a five-category triage system to better "sort" the increasing number of ED patients.
- A more accurate description of patient triage levels and departmental case mix (Wuerz, Milne, Eitel, Travers & Gilboy, 2000).
The ultimate goal of ESI implementation is to
improve the triage process and accurately capture
patient acuity to optimize the safety of patients in
the waiting room by ensuring that only patients
stable to wait are selected to wait. It is also
important to clearly articulate to the ED staff what is
not a goal of ESI triage implementation. For
example, ESI triage alone cannot decrease the ED
length of stay nor improve customer satisfaction
with the ED visit.
The continued success of ESI triage is best
accomplished by including the evaluation of triage
in the overall quality improvement (QI) plan for the
emergency department. The Institute of Medicine
(IOM) defines quality as "the degree to which health
services for individuals and populations increase the
likelihood of desired health outcomes and are
consistent with current professional knowledge"
(Lohr & Schroeder, 1990, p. 707).
In the past, health
care organizations introduced quality assurance and
quality improvement activities as part of a new
management strategy called quality management
(QM). QM includes the entire process of setting
standards, collecting information, using
interventions and tools to change identified
processes, assessing outcomes, and adjusting policies
(Wagner, Groenewegen, de Bakker & van der Wal,
2001). Through QM, health care leaders analyze and
transform health care data into information that can
be used by health care providers and policymakers
to evaluate any change that has been implemented
to determine whether it contributes to overall
quality. More recently, these processes are referred to
as process improvement.
Return to Contents
ESI Triage Quality Indicators and Thresholds
In any quality improvement plan, it is important to
select meaningful indicators to monitor. Following is
a list of potential ESI quality, or process
improvement, indicators:
- Accuracy of ESI triage category rating by the triage nurse.
- Rates of under and over-triage.
- Review of all negative outcomes which occurred due to a mis-triage.
- Measurement of time from patient arrival to being seen by a physician for each ESI triage category.
- Measurement of length of stay for each ESI triage category.
- Measurement of admission rates for each ESI triage category.
- Review of patients rated ESI level 4 and 5 who are admitted to the hospital.
Specific patient populations can also be analyzed as
an indication of process improvement. For example,
if a hospital decides to increase the number of
transfer trauma patients they accept, the ED can
check to see if the number of transfer trauma
patients and their acuity level has increased.
While selecting indicators to review is critical, it is
also important to recognize specific indicators that
are not appropriate to review. For example, the
actual number of resources that were used in
providing care to the patient is NOT an appropriate
QI indicator to monitor. Resources are incorporated
in the ESI algorithm only to help the triage nurse to
differentiate among the large group of patients that
are not acutely ill. Monitoring of the number of
resources used "on the back end" will only further
increase the triage nurses' focus on counting
resources, which is not the most important
component of the algorithm.
In addition to selecting useful indicators, it is also
important that the ED management team select a
realistic threshold to meet for each indicator. All
indicators do not need to have the same threshold.
For example, when reviewing accuracy of triage
categorization, a realistic goal must be determined.
Should the triage category be correct 100 percent, 90
percent, or 80 percent of the time? Frequently a
threshold of 90 percent is selected. However, the
goals and circumstances of each department may be
unique and should be considered when selecting
each indicator and threshold. For example, the ED
management team might stipulate that, when in
doubt about a patient's triage rating, nurses err on
the side of over-triage. While this approach might
result in some patients being mis-triaged as more
acute than they actually are, it is preferable to
risking an adverse event because the patient was
triaged to a less urgent category. In this ED, the
triage accuracy threshold might be 80 percent, with
a goal to keep the under-triage rate at 20 percent of
the mis-triages.
Finally, it is also important to determine how many
triage indicators should be monitored on an
ongoing basis. It is reasonable to select one or more
indicators. The number of indicators to be
monitored will be determined by available staff
resources and the relationship of ESI indicators to
other quality indicators that are routinely
monitored. It is also possible to focus on monitoring
one aspect of triage for a period of time, and then
switch to another indicator when improvement
occurs in the previously monitored indicator.
Accuracy of triage acuity level should probably be
monitored on a continuous basis to evaluate new
triage nurses as well as monitor for trends which
may identify the need for re-education on a
particular aspect of triage.
Return to Contents
ESI Triage Data Collection
The method of collecting QI data for ESI triage
indicators can be incorporated into the data
collection process for other ED quality indicators or
data can be collected as a separate process. The
method of data collection will depend on the
indicator selected, the availability of triage experts,
and logistic issues such as accessibility to electronic
versus paper ED records. For example, if "accuracy of
triage category" is selected as a triage QI indicator, a
triage expert is needed to review the triage
categories.
This is a critical indicator to monitor
when ESI is first implemented and the actual
method of review is discussed further in the
examples below. If it is determined that the
institution wishes to measure ED length of stay or
wait times to see the physician for each ESI triage
category, it is vital to have access to electronic
information in order to successfully monitor this
indicator. Without electronic sources, these data are
cumbersome to track and manual calculations most
likely result in error.
Finally, when monitoring QI indicators, it is
important to determine how many charts must be
reviewed for each indicator and how frequently the
indicator should be reviewed (monthly, quarterly,
etc.). The selection of the appropriate number of
charts for each indicator will again depend on the
particular indicator. If wait times for each category
are reviewed, data will be most accurate when a
large percentage of cases, preferably all, are reviewed.
Evaluating the accuracy of ESI triage does not
require the review of each occurrence, but should
reflect an appropriate number of randomly selected
charts. Cases from different nurses and each shift
and day of the week should be reviewed. Ten
percent of all cases are often selected as an
"appropriate" number of cases to review. In a busy
ED, this is an unrealistic number. It is important for
each institution to consider the number of review
staff, background, and availability of time of those
who review triage indicators.
When determining the frequency of performing
triage audits, the institution should consider other
departmental QI activities and try to integrate the
review of triage indicators into the same process and
time structure.
Sharing Results and Making
Improvements
Often, 95 percent of the time and attention to QI
and process improvement activities is given to the
monitoring stage of the process. The "numbers" are
often posted somewhere and little is done to
actually improve the outcomes. The most important
component of QI is sharing the data and discussing
ways to improve the results. All staff should be
aware of the triage QI indicators, the current overall
incidence in which the threshold is met, and the
actual goal. For example, if the accuracy of the triage
category is being monitored and continues to be
reported as 60 percent, intervention is necessary.
Often education is helpful. It is also very helpful to
involve the triage nurses in data collection.
Return to Contents
Examples of ESI Triage Indicators
The emergency departments described below have
implemented ESI and a QI program. They have
provided examples of how they incorporate triage
indicators into their QI plan.
Hospital 1
At hospital 1, the accuracy of triage nurses' ESI triage
ratings is assessed on a continuous basis and
reported quarterly as one indicator of the overall ED
QI plan. This indicator has been monitored since ESI
was implemented and continues to be the only
triage indicator monitored to date. Each week, three
different nurses randomly select five charts to review
with the ED clinical nurse specialist (CNS). The
assessment team reviews many different general
documentation indicators, including the accuracy of
the ESI triage category. The CNS is the designated
triage expert and discusses each case with the staff
nurse as she reviews the records. When there is a
disagreement, cases are reported as mis-triages for
the QI report. The assessment team collects and
retypes all mis-triages as an educational tool and
includes a discussion which explains the correct
triage category and discusses why. These cases are
compiled in a handout and distributed to all staff
nurses monthly. The assessment team reviews sixty
charts monthly.
Hospital 1 has noted several distinct
advantages of the triage accuracy review:
- The ED staff nurse is intimately aware of the QI indicators and has an opportunity to reflect on her own practice.
- Staff nurses have the opportunity to discuss each case with the CNS to obtain additional insight.
- All nurses benefit from the discussion when the cases are distributed as a teaching tool.
Hospital 1, like many other EDs also has excellent
information technology resources that facilitate
quality monitoring of clinical information. The
triage acuity is entered into a large ED database. It is
possible to track time to physician evaluation for
each triage category. This can be a powerful
administrative tool.
Hospital 2
At hospital 2, several triage indicators are reviewed
on a regular basis. The ESI rating assigned by the
nurse at triage and time data are recorded in the
hospital's computer information system during the
ED visit. The electronic information is compiled for
monthly QI monitoring. Time data are reported by
ESI triage level, including the following:
- Total ED length of stay.
- Time from triage to placement in the ED bed.
- Time from triage to being seen by the ED physician.
- Time from placement in the ED bed to discharge.
The time data are used for many purposes, such as
monitoring for operational problems that lead to
increased length of stay. The time data prove useful
in addressing issues related to specific patient
populations at hospital 2's ED. For example, the
time data were tracked for psychiatric patients and
subsequently a new policy regarding psychiatric
consults was developed. The policy stipulates
response times for the crisis team to see ED
psychiatry patients and is based on ESI triage level.
Information about the number of patients triaged to
the various areas of the ED (medical urgent care,
minor trauma, pediatrics, acute) is also reported by
ESI triage level on a monthly basis. These data are
used to make operational decisions, such as the time
of day that medical urgent care and minor trauma
services are offered.
The accuracy of triage nurses' ESI ratings is reviewed
as part of the QI program at hospital 2. The initial
review took place during the first few months after
implementation of the ESI. The nurse educator
reviews a random sample of ED charts on a regular
basis to assess the accuracy of the triage nurses' ESI
ratings. Individual nurses get feedback and the
entire nursing staff hears about trends. Through this
process the nursing staff identified problems with
the heart rate criteria in the original version of the
ESI. Based on input from the nurses, the ESI heart
rate criteria were revised for ESI v. 2.
Triage ratings are also part of a QI effort at hospital
2, through a monthly peer chart review process.
Each nurse selects two random ED charts per month
and reviews many aspects of nurses' documentation,
including the ESI triage rating. The review is
forwarded to the management staff for followup
with individual nurses and issues related to accurate
ESI rating is communicated to the entire staff when
appropriate.
Another QI effort at hospital 2 is the review of all
ESI level-3 patients triaged to the medical urgent
care area. The nurse manager receives a monthly
report, compiled with electronic data from the hospital computer system, of all ESI level-3 patients
triaged to medical urgent care, and all ESI level-4
and 5 patients triaged to the ED. Though the
department has a guideline that ESI level-4 and 5
adult patients are primarily triaged to medical
urgent care or minor trauma, and ESI level-1, 2, and
3 adult patients are primarily triaged to the acute
ED, the triage nurse is allowed discretion in triaging
these patients. The ongoing review of the ESI level-3
patients sent to medical urgent care allows the
management team to review the appropriateness of
the nurses' triage decisions.
Hospital 3
At hospital 3 the manager assigns experts to review
triage categories. The manager and clinical
coordinators review charts identified by peers as
potential mis-triages. The expert group reviews the
chart and discusses it with the triage nurse. The
team of experts also spot checks charts frequently. If
a trend is noticed, the expert group will post the
case so that all staff can learn from it.
Hospital 4
At hospital 4, the manager created a log after
initiation of the ESI triage system. The triage nurse
logged the patient name, triage nurse name, triage
level and rationale and resources for each patient
triaged. The management team reviewed each chart
for triage category accuracy either while the patient
was in the department or the next day. The
management did this for the first 2 weeks and again
in 3 months.
Hospital 5
Hospital 5's strategic plan called for the hospital to
increase the number of trauma and stroke patients
they would accept from outlying hospitals. Most of
these patients were emergency department to
emergency department transfers. Many of these
patients arrived intubated and others were intubated
on arrival. The staff felt that the acuity of the ED
patient population was rising quickly. Nursing
leadership chose to look at case mix data (the
number of patients in each ESI category) for 1 year
and was able to make adjustments to staffing to
cover increases in patient acuity.
It is important for the emergency department
nursing leadership to put a simple quality
improvement plan into place. The plan needs to
generate meaningful data that can be shared with
the ED staff on a regular basis. Issues with individual
triage nurses must be promptly identified and
education provided. Larger trends must also be
rapidly identified and responded to. The members of
the ESI research team are repeatedly asked about QA
and our suggestion is to keep it simple, relevant and
meaningful.
Return to Contents
References
Lohr KN, Schroeder SA (1990). A strategy for
quality assurance in Medicare. New England Journal of
Medicine 322:707-12.
Wagner C, Groenewegen P, de Bakker D, van der
Wal G (2001). Environmental and organizational
determinants of quality management. Quality
Management in Health Care 9(4):63-76.
Wuerz R, Milne LW, Eitel DR, Travers D, Gilboy N (2000). Reliability and validity of a new five-level
triage instrument. Academic Emergency Medicine 7(3):236-42.
Note: Appendix A of this handbook includes frequently
asked questions and post-test assessment questions for
Chapters 3 through 8. These sections can be incorporated
into the ESI training course.
Return to Contents
Proceed to Next Section