Once ESI has been implemented it is important to continue ongoing evaluation. In fact, thought should be given to evaluation prior to implementation, and plans should be made to continually evaluate the system.
When evaluating the success or failure of implementation, it is important to remember why the triage process was changed. The following reasons are frequently identified as driving forces to change existing triage processes:
The ultimate goal of ESI implementation is to improve the triage process and accurately capture patient acuity to optimize the safety of patients in the waiting room by ensuring that only patients stable to wait are selected to wait. It is also important to clearly articulate to the ED staff what is not a goal of ESI triage implementation. For example, ESI triage alone cannot decrease the ED length of stay nor improve customer satisfaction with the ED visit.
The continued success of ESI triage is best accomplished by including the evaluation of triage in the overall quality improvement (QI) plan for the emergency department. The Institute of Medicine (IOM) defines quality as "the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge" (Lohr & Schroeder, 1990, p. 707).
In the past, health care organizations introduced quality assurance and quality improvement activities as part of a new management strategy called quality management (QM). QM includes the entire process of setting standards, collecting information, using interventions and tools to change identified processes, assessing outcomes, and adjusting policies (Wagner, Groenewegen, de Bakker & van der Wal, 2001). Through QM, health care leaders analyze and transform health care data into information that can be used by health care providers and policymakers to evaluate any change that has been implemented to determine whether it contributes to overall quality. More recently, these processes are referred to as process improvement.
In any quality improvement plan, it is important to select meaningful indicators to monitor. Following is a list of potential ESI quality, or process improvement, indicators:
Specific patient populations can also be analyzed as an indication of process improvement. For example, if a hospital decides to increase the number of transfer trauma patients they accept, the ED can check to see if the number of transfer trauma patients and their acuity level has increased.
While selecting indicators to review is critical, it is also important to recognize specific indicators that are not appropriate to review. For example, the actual number of resources that were used in providing care to the patient is NOT an appropriate QI indicator to monitor. Resources are incorporated in the ESI algorithm only to help the triage nurse to differentiate among the large group of patients that are not acutely ill. Monitoring of the number of resources used "on the back end" will only further increase the triage nurses' focus on counting resources, which is not the most important component of the algorithm.
In addition to selecting useful indicators, it is also important that the ED management team select a realistic threshold to meet for each indicator. All indicators do not need to have the same threshold. For example, when reviewing accuracy of triage categorization, a realistic goal must be determined. Should the triage category be correct 100 percent, 90 percent, or 80 percent of the time? Frequently a threshold of 90 percent is selected. However, the goals and circumstances of each department may be unique and should be considered when selecting each indicator and threshold. For example, the ED management team might stipulate that, when in doubt about a patient's triage rating, nurses err on the side of over-triage. While this approach might result in some patients being mis-triaged as more acute than they actually are, it is preferable to risking an adverse event because the patient was triaged to a less urgent category. In this ED, the triage accuracy threshold might be 80 percent, with a goal to keep the under-triage rate at 20 percent of the mis-triages.
Finally, it is also important to determine how many triage indicators should be monitored on an ongoing basis. It is reasonable to select one or more indicators. The number of indicators to be monitored will be determined by available staff resources and the relationship of ESI indicators to other quality indicators that are routinely monitored. It is also possible to focus on monitoring one aspect of triage for a period of time, and then switch to another indicator when improvement occurs in the previously monitored indicator.
Accuracy of triage acuity level should probably be monitored on a continuous basis to evaluate new triage nurses as well as monitor for trends which may identify the need for re-education on a particular aspect of triage.
The method of collecting QI data for ESI triage indicators can be incorporated into the data collection process for other ED quality indicators or data can be collected as a separate process. The method of data collection will depend on the indicator selected, the availability of triage experts, and logistic issues such as accessibility to electronic versus paper ED records. For example, if "accuracy of triage category" is selected as a triage QI indicator, a triage expert is needed to review the triage categories.
This is a critical indicator to monitor when ESI is first implemented and the actual method of review is discussed further in the examples below. If it is determined that the institution wishes to measure ED length of stay or wait times to see the physician for each ESI triage category, it is vital to have access to electronic information in order to successfully monitor this indicator. Without electronic sources, these data are cumbersome to track and manual calculations most likely result in error.
Finally, when monitoring QI indicators, it is important to determine how many charts must be reviewed for each indicator and how frequently the indicator should be reviewed (monthly, quarterly, etc.). The selection of the appropriate number of charts for each indicator will again depend on the particular indicator. If wait times for each category are reviewed, data will be most accurate when a large percentage of cases, preferably all, are reviewed.
Evaluating the accuracy of ESI triage does not require the review of each occurrence, but should reflect an appropriate number of randomly selected charts. Cases from different nurses and each shift and day of the week should be reviewed. Ten percent of all cases are often selected as an "appropriate" number of cases to review. In a busy ED, this is an unrealistic number. It is important for each institution to consider the number of review staff, background, and availability of time of those who review triage indicators.
When determining the frequency of performing triage audits, the institution should consider other departmental QI activities and try to integrate the review of triage indicators into the same process and time structure.
Sharing Results and Making Improvements
Often, 95 percent of the time and attention to QI and process improvement activities is given to the monitoring stage of the process. The "numbers" are often posted somewhere and little is done to actually improve the outcomes. The most important component of QI is sharing the data and discussing ways to improve the results. All staff should be aware of the triage QI indicators, the current overall incidence in which the threshold is met, and the actual goal. For example, if the accuracy of the triage category is being monitored and continues to be reported as 60 percent, intervention is necessary. Often education is helpful. It is also very helpful to involve the triage nurses in data collection.
The emergency departments described below have implemented ESI and a QI program. They have provided examples of how they incorporate triage indicators into their QI plan.
At hospital 1, the accuracy of triage nurses' ESI triage ratings is assessed on a continuous basis and reported quarterly as one indicator of the overall ED QI plan. This indicator has been monitored since ESI was implemented and continues to be the only triage indicator monitored to date. Each week, three different nurses randomly select five charts to review with the ED clinical nurse specialist (CNS). The assessment team reviews many different general documentation indicators, including the accuracy of the ESI triage category. The CNS is the designated triage expert and discusses each case with the staff nurse as she reviews the records. When there is a disagreement, cases are reported as mis-triages for the QI report. The assessment team collects and retypes all mis-triages as an educational tool and includes a discussion which explains the correct triage category and discusses why. These cases are compiled in a handout and distributed to all staff nurses monthly. The assessment team reviews sixty charts monthly.
Hospital 1 has noted several distinct advantages of the triage accuracy review:
Hospital 1, like many other EDs also has excellent information technology resources that facilitate quality monitoring of clinical information. The triage acuity is entered into a large ED database. It is possible to track time to physician evaluation for each triage category. This can be a powerful administrative tool.
At hospital 2, several triage indicators are reviewed on a regular basis. The ESI rating assigned by the nurse at triage and time data are recorded in the hospital's computer information system during the ED visit. The electronic information is compiled for monthly QI monitoring. Time data are reported by ESI triage level, including the following:
The time data are used for many purposes, such as monitoring for operational problems that lead to increased length of stay. The time data prove useful in addressing issues related to specific patient populations at hospital 2's ED. For example, the time data were tracked for psychiatric patients and subsequently a new policy regarding psychiatric consults was developed. The policy stipulates response times for the crisis team to see ED psychiatry patients and is based on ESI triage level. Information about the number of patients triaged to the various areas of the ED (medical urgent care, minor trauma, pediatrics, acute) is also reported by ESI triage level on a monthly basis. These data are used to make operational decisions, such as the time of day that medical urgent care and minor trauma services are offered.
The accuracy of triage nurses' ESI ratings is reviewed as part of the QI program at hospital 2. The initial review took place during the first few months after implementation of the ESI. The nurse educator reviews a random sample of ED charts on a regular basis to assess the accuracy of the triage nurses' ESI ratings. Individual nurses get feedback and the entire nursing staff hears about trends. Through this process the nursing staff identified problems with the heart rate criteria in the original version of the ESI. Based on input from the nurses, the ESI heart rate criteria were revised for ESI v. 2.
Triage ratings are also part of a QI effort at hospital 2, through a monthly peer chart review process. Each nurse selects two random ED charts per month and reviews many aspects of nurses' documentation, including the ESI triage rating. The review is forwarded to the management staff for followup with individual nurses and issues related to accurate ESI rating is communicated to the entire staff when appropriate.
Another QI effort at hospital 2 is the review of all ESI level-3 patients triaged to the medical urgent care area. The nurse manager receives a monthly report, compiled with electronic data from the hospital computer system, of all ESI level-3 patients triaged to medical urgent care, and all ESI level-4 and 5 patients triaged to the ED. Though the department has a guideline that ESI level-4 and 5 adult patients are primarily triaged to medical urgent care or minor trauma, and ESI level-1, 2, and 3 adult patients are primarily triaged to the acute ED, the triage nurse is allowed discretion in triaging these patients. The ongoing review of the ESI level-3 patients sent to medical urgent care allows the management team to review the appropriateness of the nurses' triage decisions.
At hospital 3 the manager assigns experts to review triage categories. The manager and clinical coordinators review charts identified by peers as potential mis-triages. The expert group reviews the chart and discusses it with the triage nurse. The team of experts also spot checks charts frequently. If a trend is noticed, the expert group will post the case so that all staff can learn from it.
At hospital 4, the manager created a log after initiation of the ESI triage system. The triage nurse logged the patient name, triage nurse name, triage level and rationale and resources for each patient triaged. The management team reviewed each chart for triage category accuracy either while the patient was in the department or the next day. The management did this for the first 2 weeks and again in 3 months.
Hospital 5's strategic plan called for the hospital to increase the number of trauma and stroke patients they would accept from outlying hospitals. Most of these patients were emergency department to emergency department transfers. Many of these patients arrived intubated and others were intubated on arrival. The staff felt that the acuity of the ED patient population was rising quickly. Nursing leadership chose to look at case mix data (the number of patients in each ESI category) for 1 year and was able to make adjustments to staffing to cover increases in patient acuity.
It is important for the emergency department nursing leadership to put a simple quality improvement plan into place. The plan needs to generate meaningful data that can be shared with the ED staff on a regular basis. Issues with individual triage nurses must be promptly identified and education provided. Larger trends must also be rapidly identified and responded to. The members of the ESI research team are repeatedly asked about QA and our suggestion is to keep it simple, relevant and meaningful.
Lohr KN, Schroeder SA (1990). A strategy for quality assurance in Medicare. New England Journal of Medicine 322:707-12.
Wagner C, Groenewegen P, de Bakker D, van der Wal G (2001). Environmental and organizational determinants of quality management. Quality Management in Health Care 9(4):63-76.
Wuerz R, Milne LW, Eitel DR, Travers D, Gilboy N (2000). Reliability and validity of a new five-level triage instrument. Academic Emergency Medicine 7(3):236-42.
Note: Appendix A of this handbook includes frequently asked questions and post-test assessment questions for Chapters 3 through 8. These sections can be incorporated into the ESI training course.