Advances in Patient Safety: From Research to Implementation Advances in Patient Safety: From Research to Implementation Volume 3. Implementation Issues

Establishing a Culture of Patient Safety Through a Low-tech Approach to Reducing Medication Errors

Steven H. Shaha, Linda Brodsky, Michael S. Leonard, Michael A. Cimino, Sandra A. McDougal, Joann M. Pilliod, Kristen E. Martin

Abstract

Transforming an organizational culture is a worthy and achievable endeavor, even when faced with limitations in funding and technology that appear as insurmountable obstacles. Equally ominous but necessary is the need to conquer commonplace problems such as medication errors. This paper will detail the means used at one hospital facility to make medication errors and their reduction a primary staff focus, and how a highly generalizable, low-tech, and cost-conscious error-reduction methodology spurred a successful shift toward an organization-wide culture of patient safety.top link

Introduction

Health care organizations struggle universally to identify and remediate safety-related challenges. Cultures of defensiveness or apathy, coupled with the cost-prohibitive nature of technological advances, lead most organizations to either ignore problems in patient safety or to simply make do until technology-based solutions are affordable and can be implemented successfully (e.g., computer-based physician order entry [CPOE] ). For many organizations, the hope is that safety will become endemic once medication orders, charts, communications, etc. become automated. As a result, most organizations are perpetuating cultures of mediocrity within which errors are commonplace and patient safety remains a distant hope.

Staff at the Center for Pediatric Quality (CPQ) at the Women and Children's Hospital of Buffalo (WCHOB) decided in the mid-1990s to transform their organizational culture into one focused on patient safety. Together they pursued a low-tech, low-cost strategy for achieving their goal. Financial resources were insufficient for implementing CPOE or similar technological solutions in the foreseeable future. Regardless of cost-related issues, however, evidence that CPOE genuinely alters the underlying culture of an organization ---transforming it to one where individual practitioners, care givers and employees are focused on safety as part of their unconscious and conscious thoughts and actions ---remains unconvincing. So the CPQ staff decided to proceed with solutions based more on people and paper than on technology.

This paper will describe how the WCHOB and its CPQ transitioned successfully to a culture of safety. This achievement was accomplished by focusing on the tangible challenge of medication prescribing errors. Why medication errors? First, the literature is replete with alarm over the need to face and eliminate medication-related errors, 1  -- 10 and, in 1996, the CPQ staff decided that it was unacceptable ---as well as unethical ---to wait idly for CPOE. Second, medication errors provide a very tangible problem through which virtually all health care workers, especially physicians, can personally identify their respective roles in these errors and contribute solutions for preventing them. Third, medication errors remain completely unnecessary and avoidable. Accordingly, the WCHOB and its CPQ staff realized that if they could successfully conquer medication errors using a low-tech, low-cost approach, it would clearly indicate that everyone involved in patient care had united in their conscious efforts and unconscious behaviors to ensure safety. In short, we would complete the transition to a culture of safety. And culture lasts beyond the project or initiative of today, so accentuating and explaining how culture can be transformed and supported by project-based proof, is more valuable to healthcare organizations than sharing successes in project work.

This is, therefore, a two-part story. The first part describes how cultural transformation became a goal and how it was achieved. The second part details how medication errors were addressed and significantly reduced. As a direct result of the CPQ's generalizable approach to cultural transformation ---enabled and substantiated through the focus on medication errors reduction ---unprecedented and statistically significant decreases in adverse events and risks were achieved, sustained, and patient safety tangibly improved, as will be shown.top link

The journey and process

The patient safety accomplishments were the result of a journey involving a series of successive steps, all consciously part of a transformational strategy.

1. The Center for Pediatric Quality: establishing a broad-based, inclusive infrastructure

Establishing the CPQ was an important first step in showing genuine commitment to investing tangibly in the culture of patient safety, using a quality-focused approach. The WCHOB implemented and refined the CPQ from 1996-2002. The CPQ avoided a common pitfall of identifying any single "flavor of the month" methodology for improvement, only to switch later to the next new fad. Rather, the CPQ staff continuously examined the best approaches, methodologies, and techniques from all industries ---new or proven ---and adopted a balance of the most relevant for each given project undertaken.

Paramount to the CPQ's success, however, was its sustained focus on being physician-led, data-driven and outcomes-focused. 11 Projects and initiatives were launched, managed, and executed by practicing physicians who invited additional physicians and nonphysicians to personally participate in projects in an attempt to achieve multidisciplinary success. 12 The key for the CPQ was the creation of a culture where physicians questioned variations in practice and outcomes; defined and examined data reflecting clinical, financial, and satisfaction outcomes; and subsequently adapted their practices using statistically rigorous findings. 13 This balanced-metrics approach ensured that no gains in one measurement area (e.g., financial) would compromise results from any other area (e.g., clinical or satisfaction).top link

2. Proving success in outcomes improvement: using the integrated outcomes approach

The CPQ launched more than 36 projects and completed 22 in the first 5 years of the initiative, all the while focusing on clinical, financial, and satisfaction outcome improvements. Physicians became increasingly engaged ---both in their numbers and in their participatory buy-in ---as their clinical challenges were studied and improved using the CPQ's methodological framework. The organization further benefited from more than $3 million in combined cost savings, as projects were executed and improvements were sustained, including, for example, supply costs, costs in direct care, and length of stay.

CPQ successes and outcome improvements reflected discipline, rigor and cadence:

The CPQ represented the WCHOB in national collaborative studies and further sought to share its outcomes improvement successes in public forums, including presentations at national and international conferences, and in submissions to peer-reviewed journals and books. The CPQ staff adopted the highest standards of academic and professional integrity and rigor so as to enhance the quality and generalizability of its undertakings, the reputation of the organization, and the credibility of its physicians and other professional employees.top link

3. Medication errors in a multi-organizational collaborative: exhaustively studying medication orders

In 1998, WCHOB joined the Children's Health Accountability Initiative (CHAI) of the Child Health Corporation of America (CHCA), as a charter member. As the CHAI was defining its mission and methodology, the CPQ suggested, designed, and thereafter took the lead on the CHAI's first multicenter project. The initiative focused on reducing medication errors without the assistance of CPOE, which was unavailable at that time within CHAI's participating organizations. The results of the project were recently published, 1 summarizing successes achieved 2 years prior to the concerns brought to light in the Institute of Medicine's 2000 report, To Err Is Human. 2

The CPQ staff designed the methodology with which medication errors were tallied uniformly at each of the nine participating facilities. All medication orders for all patient charts were reviewed on a pediatric intensive case unit for 14 consecutive days. The data collection process was conducted at the beginning of the study, and again at the end, to ascertain the effect of improvement interventions. Each order was analyzed for any type of prescribing, dispensing, administering, or documentation error, and each discovered error was documented in detail, regardless of whether or not it resulted in patient harm. Prescribing-related errors, for example, included wrong or missing dose, and route or frequency of occurrence. The impact of the errors was reported by leveraging a well-established severity scale wherein adverse drug events (ADEs) and potential ADEs (pADEs) were rated on a scale from A-I, where A represents no harm to the patient due to medication interception prior to administration, and I represents a patient death.

Each organization submitted its data to the CPQ for analysis and feedback. Each group then undertook its own interventions to remedy errors and patterns found. The CPQ analyzed the final results. More than 21,000 orders were analyzed, and organizations collectively experienced significant reductions in errors:

This multicenter initiative provided the CPQ and WCHOB with a wealth of information regarding medication-related prescribing errors, and built momentum for the desired organizational transformation with its focus on prescribing errors.top link

4. Creating tools: implementing a paper-based solution

Each CHAI organization established its own unique intervention for reducing medication errors. The WCHOB's intervention included the implementation of what became known as the "Forced Function Format Order Form" (FFF). The CPQ staff designed and implemented the FFF to conquer data-verified challenges of incomplete and incorrect orders. The FFF was not complex, but its implementation resulted in a 71 percent reduction in prescribing-related errors at WCHOB (during the CHAI study period). 2

The FFF prompted physicians for the various elements of a complete prescription order, reminding them to write orders completely and correctly. Columns were added to the existing order form template, in a move away from the horizontal-only format used almost universally throughout health care. Vertical, gray lines included headings and appeared at the top of the form, prompting the prescribers for each piece of information needed for a complete and correct order, including drug name, dose, dosing strategy (mg/kg), route, frequency, and indication (to be explained later in this report). When writing nondrug orders (e.g., physical therapy), the prescriber could simply write across the gray column markers as though they did not exist. Prescribers were introduced to the new form through a variety of educational means, including grand rounds, posters, and presentations at key physician forums.

The FFF further demonstrated to physicians that small, but appropriate, changes in prescribing behaviors can have a major impact on patient safety (as will be shown in the data discussed later). Second, it made patient safety a "personal" issue in the minds of the prescribers, reminding them that each order must be complete and correct if it is to reduce patient risk and improve collaboration with other caregivers and nonphysician workers. Third, the FFF was paper-based, and underscored the truism that prescribing can be made near error-free without computerization, while reinforcing the notion that the absence of CPOE is not a limiting factor in the conquest of medication errors and cultures can be changed without expensive technology.top link

5. Indication: attention to the "why" of prescribing

The FFF also featured a new prescription ordering element ---arguably one of the foremost innovations of the CPQ ---"Indication." The right-most column on the FFF carried the heading of Indication, where physicians were instructed to note the clinical rationale for the order, such as pneumonia or sepsis. It was the CPQ staff's belief that the "Indication" column would serve several purposes. It was intended to assist in the interpretation of handwritten orders, reducing illegibility and potential confusion by providing an interpretive context. The Indication column had an immediate impact on the interpretability of prescription orders, as demonstrated by one form passed to the inpatient pharmacy that was a complete and correct order for aminophylline, for the stated indication of sepsis. The pharmacist called the physician prescriber who corrected the order to read amphotericin. This example illustrates how the use of the Indication column prevented two errors: administration of a wrong drug and non-administration of the correct drug.

The Indication column also enabled more detailed analyses of prescribing practices and patterns. Why do prescribers order the things drugs they do? Do different prescribing patterns for the same Indication result in different outcomes or levels of efficacy? Indepth analysis of alternative rationales and variations in prescribing practices should lead to improved prescribing practices and patient outcomes.top link

6. The PEDS Cycle: framing an initiative to "personalize" safety and improvement

The WCHOB established the beginnings of its culture of safety through open recognition of the CPQ's leadership in CHAI's multicenter success and the successes achieved in medication error reduction. Most importantly, physicians within the WCHOB became aware of the importance of precision in prescription orders and the simplicity with which major improvements in patient safety could be achieved through vigilance and improved personal prescribing behaviors.

The importance of the nonphysician workers and caregivers within the organization became evermore apparent to practitioners as physicians gained a heightened sense of appreciation for key functions and people around them that apparently did not exist previously. Through the CHAI medication error study and the other projects undertaken by CPQ, the concept and practice of multidisciplinary approaches to improving outcomes and patient safety became intuitive ---rather than ritualistic, bothersome, or cumbersome ---to physicians. 12

The entire approach was codified into what the CPQ called its PEDS Cycle (Figure 1), an acronym for Process, Education, Dissemination and Surveillance:

Figure 1. The PEDS Cycle of the CPQ and WCHOB

top link

The CPQ staff did not define the PEDS Cycle in an attempt to create yet another "flavor" of approach to continuous improvement. Rather, it was done to provide the organization with a WCHOB-identifiable approach to improvement, and to motivate and engage all by "personalizing" the methodology, thus making safety part of the conscious and unconscious culture. PEDS represented the "WCHOB Patient Safety Approach," as it were, and thus underscored the importance and genuineness of safety within this specific organization and its patient-focused contributors.top link

7. Certification of prescribing skills: training and retraining all prescribers

Having established the credibility of the CPQ, and with the FFF tool in place to guide medication ordering, the next step in the CPQ's mission to maximize safety in prescribing practices involved the design and implementation of Web-based training for certifying prescription order compliance. In October 2003, following a year of testing and refinement, the WCHOB made Web-based training and certification mandatory for all prescribers. This requirement added further momentum to the push toward a culture of patient safety. Former efforts to train all physicians through orientations, presentations, posters, and other traditional means each had a positive impact, but not to the degree desired by WCHOB administrators. To meet the challenge of bringing all prescribers up to the highest standards of order completeness and correctness, while at the same time reinforcing a pervasive culture of patient safety, a Web-based training mechanism was needed.

The training Web site provided examples of well-written orders and, after examining the samples, the prescribers were required to write several interactive orders of their own, with immediate feedback. Once a prescriber completed the Web-based training, his or her results and certification were reported automatically to the chief safety officer and the CPQ. The implementation was highly successful, with nearly 90 percent participation and very little resistance on the part of the more "seasoned," less technology-savvy attending physicians.top link

8. Point of care information: PDA- based drug reference software

Following the certification process (October 2003), prescribers were issued personal digital assistants (PDAs) equipped with a pediatric-appropriate formulary application (Lexi-comp[reg]) to provide point-of-care prescribing assistance, at the CPQ's expense (in some instances the software was loaded on personal PDAs for prescribers who preferred using their own). Funding for the acquisition of the PDAs and software was made possible through the Health Patient Safety Award, which the New York State Department of Health gave the CPQ in 2003.top link

9. Implementing "no incomplete orders": a radical step for leadership

The WCHOB culture of patient safety became a reality on November 12, 2003, with the implementation of the zero tolerance policy. In a substantive show of commitment, organizational leadership unanimously backed the policy that "No incomplete orders will be accepted." The policy mandated rewrites for all incomplete orders prior to their pharmacy processing or fulfillment, whenever a pharmacy employee, a floor nurse, or a floor unit secretary determined that an order was incomplete or incorrect. Obvious exceptions dictated that urgent orders be filled first and rewritten later, as necessary.

The zero tolerance policy represented a revolutionary act on the part of the WCHOB leadership, uncommon in health care organizations. While the subsequent resistance was undeniable, it was also unavoidable because complete and correct orders begin as complete orders. Despite the potential for resistance, the administrative and physician leadership understood the need and took appropriate action, primarily in the form of personal interactions with prescribers whose orders showed consistent challenges. Cultural change occurs only when the leadership is willing to commit to it. And, while the policy on rewriting orders was met with some initial resistance, it eventually gained near-universal recognition, acceptance, and adherence.top link

10. "SPOT CHECKS": data-driven surveillance and remedial feedback for prescribers

The Safe Prescriber Order Tracking within the Children's Hospital Ensuring Comfort and Kids Safety initiative (SPOT CHECKS) is the CPQ's order tracking database. SPOT CHECKS involves a relational database into which order-level data are entered to enable the analysis and reporting of order completeness, quality, and appropriateness, as well as corresponding medication errors and pADEs. Prescriber data and the details of their orders are maintained for analysis and feedback purposes, as well as for subsequent study and organizational learning.

At present, the SPOT CHECKS data are entered by a dedicated registered nurse (RN), working side-by-side with the hospital's clinical pharmacists. A statistician and an epidemiologically prepared physician analyze the data to reveal and describe organization-wide patterns of completeness and correctness, as well as categorizing by prescriber, prescriber type (e.g., pediatric resident, attending general surgeon, etc.), and location (e.g., medical, intensive care unit, etc.). The data on individual prescribers are mailed confidentially to their homes and the combined data are presented to the hospital's organizational leaders on a bimonthly basis to show progress and areas in need of improvement.top link

Results

The results of the project show a significant increase in prescription order completeness and correctness, while medication errors have decreased significantly as a result of the cultural transformation. Analyses reported a total of 18,915 orders in the SPOT CHECKS database.

The results are best understood when the data are categorized according to the five implementation phases:

1. June/July 2003 ---Pre-implementation baseline established, including any residual impact from the CHAI project involvement: 1,225 prescription orders.

2. October 2003 ---Data reflecting the implementation of Web-based prescriber certification and prescriber-assisting PDAs: 4,448 orders.

3. November 12 --18, 2003 ---Data reflecting the impact of the zero tolerance policy requiring rewrites for all incomplete or incorrect prescriptions: 1,129 orders.

4. Post-November 18, 2003 ---Data reflecting the post-implementation phase, during which staff resistance and challenges to the revised prescription order process were overcome: 5,046 orders.

5. February 2004 ---Results of the first full round of data sharing, with analyses of order quality and personal comparatives by individual prescriber and collectively across the organization: 3,849 orders.

6. April/May 2004 ---Results of the second full round of data sharing, with analyses of order quality and personal comparatives by individual prescriber and collectively across the organization, as well as followup measurements assessing ability to maintain achieved gains: 3,218 orders.

The analyses and results shared here are limited to those considered to best illustrate the success achieved in addressing medication errors, and the related change in culture.

Impact on order completeness and correctness

Order completeness and correctness was the first measure of impact for the overall evaluation of strategy and implementation success. The initial analysis reflected the percentage of orders for which the four essential medication order elements ---drug name, dose, route, and frequency ---all were complete and correct. The data were limited to only those orders for which all four were needed (e.g., not topicals), in order to represent the most conservative estimate of impact.

The data showed that by the post-implementation phase (post-November 18, 2003) the correctness and completeness of prescription orders had improved by 1.6 percent from baseline (t = 3.22; P = .01), and by 3.0 percent (t = 4.96; P = .01) from implementation (Figure 2). The data further indicated that the long-term impacts remain favorable: The April/May 2004 data sharing round with prescribers led to a 9.0 percent improvement in order completeness over baseline (t = 18.36; P = .001). The goal remains 100 percent.top link

Figure 2. Percent of orders with complete and correct drug, dose, route, and frequency

NOTE: The scale of this graph is altered to accentuate improvements achieved only. The uppermost rate shown is not 100% and the lowermost shown is not 0%.top link

Rewrite rate

Analysis of the rate at which rewrites were needed due to initial order incompleteness reflected the degree of tangible change in prescribing behaviors, as evidenced by the ability to "get it right the first time." The number of orders for which rewrites were needed skyrocketed following the November 12, 2003, implementation of the "no incomplete orders" policy. Once the implementation was complete, however, and prescriber concerns and challenges had been addressed, a 58.5 percent reduction in rewrites was achieved (t = 18.49; P = .000 p<.001). The goal continues to be "no rewrites"top link

Dosing

This analysis most reflected the staff's preoccupation with safety and its determination to conquer a common type of medication-related error. Incorrect dosing was reduced by 32.5 percent from baseline (t = 10.06; P = .001), and 45.0 percent (t = 16.11; P = .001) from the high rate experienced during the implementation phase (Figure 3). The achieved gains were maintained during the 6 months of remeasurement. The goal remains zero dosing errors, despite the number of different medications involved and the complexity of accomplishing this goal in the pediatric unit of a tertiary care, teaching facility.top link

Figure 3. Percent of orders with incorrect dosing prior to rewrites

NOTE: The scale of this graph is altered to accentuate improvements achieved only. The uppermost rate shown is not 100% and the lowermost shown is not 0%.top link

Indication

Indication remains a major analytic power for future use within the CPQ and the WCHOB, as well as an unmined resource within health care as a whole. Results (Figure 4) showed a 180.3 percent increase in the use of the Indication column on the prescription order form, from baseline through the most recently analyzed data (t = 89.79; P = .000 p<.001). The increase in Indication column use was interpreted as a measure of the CPQ's success in establishing a culture of safety, primarily because the only evidence the prescribers were shown that Indication might be of value was anecdotal and theoretical. The goal of 100 percent Indication usage remains, and will become more achievable as Indication is analyzed and used more fruitfully.top link

Figure 4. Percent of orders with incorrect dosing prior to rewrites

NOTE: The scale of this graph is altered to accentuate improvements achieved only. The uppermost rate shown is not 100% and the lowermost shown is not 0%.top link

Impact on patient safety

The impact of any prescribing-related and safety-focused initiative must be measured in terms of its ability to prevent adverse events. Data were analyzed to discern all orders for which any incorrect element could potentially have resulted in an adverse drug event (ADE), regardless of severity, including those order for which any element of the order was wrong (e.g., dose). Potential events were defined as "potential ADEs" (pADEs). The CPQ staff adopted the use of pADEs as the measure of risk to patients, rather than the traditional reliance on ADEs within healthcare, which leans on incident reporting with its known pitfalls and inaccuracies.

As shown in Figure 5, a 36 percent decrease in pADEs was achieved from baseline through the February 2004 measurement occasion (t = 14.10; P = .001), and further gains have been since achieved, which equates to a greater than 36 percent relative reduction in risk to patients. 14 top link

Figure 5. Percent of incorrect orders leading to a potential adverse drug event (pADE)

NOTE: The scale of this graph is altered to accentuate improvements achieved only. The uppermost rate shown is not 100% and the lowermost shown is not 0%.top link

Conclusions

Implementing a culture of patient safety is a journey, not a project or even a series of projects. Success is dependent upon appropriate resourcing, planning, and a sense of determination that begins at the very top of the organization. That determination becomes real when practicing physicians are empowered and provided with the resources to achieve genuine and tangible improvements through an inclusive approach and data-driven methodology like that being used in the CPQ. Without the efforts of the CPQ staff, safety would remain a noteworthy concept ---not an ongoing effort and a cultural reality.

The conquest of medication errors provided a near-ideal focus for making safety a day-to-day, hour-to-hour imperative. Prescribing is an ever-occurring component of patient care. The focus on completeness and correctness in prescription orders created a basis from which to ensure that physicians personally understood and were engaged in the organization-wide culture of patient safety.

While there is work still to be done, and greater gains to be achieved, the success of the CPQ-driven initiative for transforming the organizational culture remains remarkable. Our goals have not been fully realize, and the power of the SPOT CHECKS database has not yet come to fruition, including the potential transformational capabilities of the Indication element. The effects of PDAs and other technologies upon the implementation of a culture of safety have yet to be fully recognized.

Despite these remaining challenges, the WCHOB now enjoys a new level of personalized caring and watchfulness among prescribers and care team members, and a genuine investment in making patient safety real and ubiquitous. The success was achieved through a long, investment-driven process. And the cumulative success story, illustrated by the tangible reduction in medication-related errors, is a closely held system of values demonstrated daily in the culture that now permeates the organization and those involved with it.top link

Acknowledgments

We are grateful for the ongoing support of the Women and Children's Hospital of Buffalo, Kaleida Health, Lexi-comp, the New York State Department of Health, the Children's Health Accountability Initiative, and all of the Center for Pediatric Quality participants.top link

Author affiliations

Center for Pediatric Quality, Women and Children's Hospital of Buffalo (SHS, LB, MSL, MAC, SAM, JMP, KEM). State University of New York at Buffalo (LB, MSL, MAC). Institute for Integrated Outcomes, Amherst NY (SHS, LB). Graduate Schools of Business and Political Science, University of Utah, Salt Lake City (SHS). Pediatric Otolaryngology, Women and Children's Hospital of Buffalo (LB), Health Resources and Services Administration (MSL). Women and Children's Health Research Foundations (MAC).

Address correspondence to: Linda Brodsky, M.D., Director, CPQ, Women and Children's Hospital of Buffalo, 219 Bryant Street, Buffalo, NY 14222; phone: 716-878-1074 (Kristen Martin); e-mail: Lbrodsky@kaleidahealth.org.top link

References

1. Cimino MA, Kirschbaum MS, Brodsky L, et al. Assessing medication prescribing errors in pediatric intensive care units. Pediatr Crit Care Med 2004. 5(2):124-32. (PubMed)

2. Kohn LT, Corrigan JM, and Donaldson MS, editors. To err is human: building a safer health system. A report of the committee on quality of health care in America, Institute of Medicine. Washington DC: National Academy Press; 2000.

3. American Academy of Pediatrics. Committee on Drugs, Committee on Hospital Care. Prevention of medication errors in the pediatric inpatient setting. Pediatrics 1998. 102(2 Pt 1):428-30. (PubMed)

4. Bates DW, Boyle DL, VanderVliet MB, et al. Relationship between medication errors and adverse drug events. J Gen Intern Med 1995. 10(4):199-205. (PubMed)

5. Cullen DJ, Sweitzer BJ, Bates DW, et al. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med 1997. 25(8):1289-97. (PubMed)

6. Lesar TS, Briceland L, Stein DS. Factors related to errors in medication prescribing. JAMA 1997. 277(4):312-17. (PubMed)

7. Lesar TS, Lomaestro BM, Pohl H. Medication-prescribing errors in a teaching hospital. A 9-year experience. Arch Intern Med 1997. 157(14):1569-76. (PubMed)

8. Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA 1995. 274(1):35-43. (PubMed)

9. Anderson BJ, Ellis JF. Common errors or drug administration in infants: costs and avoidance. Pediatric Drugs 1999. 1(2):93-107. (PubMed)

10. Draft guidelines for preventing medication errors in pediatrics. Journal of Pediatric Pharmaceutical Practice 1998;3:189  -- 202.

11. Brodsky L, Shaha SH. Integrated outcomes approach to improvement in healthcare. In: Coughlin KM, et al., editors. 2001 Medical outcomes & guidelines Sourcebook. New York: Faulkner & Gray; 2000.

12. Shaha SH. Strategic teaming across multiple disciplines. Advance for Health Information Executives 1999;3(10).

13. Shaha SH. Integrated outcomes: where CIOs need to be thinking. Health Manag Technol 1998; 9(10).

14. Guyatt G, Rennie D, Hayward R. Users' guides to the medical literature: essentials of evidence-based clinical practice. Chicago: AMA Press; 2002.