Chapter 4. Assessing Surgeon Acceptance of Risk Adjustment Models

The originally proposed approach for Chapter 4 was to solicit input from surgeons and IP nurses on the types of risk factors that should be examined. Due to delays in Institutional Review Board (IRB) approval at Intermountain for the nursing focus group, we used this opportunity (with permission from AHRQ and CDC) to repurpose the nursing focus group. Surgeon focus group inputs, as described below, were used to inform Chapter 3. Results of the nursing focus groups are being used to support adoption of the developed tools by offering greater insight into the decision to adopt and issues/challenges around implementation of the Chapter 2 surveillance tool.

Subtask 4.1. Identify a Representative Group of Active Surgeons in the United States

4.1.1. Identify surgeons to participate in a focus group and in-depth discussion

Surgeons were identified based on their project-relevant professional expertise, and were selected from multiple health care settings and systems in order to maximize the representative nature of the focus group participants. Walter L. Biffl, M.D., recruited participant surgeons through use of a national conference-based professional networking strategy. National meeting organizers for the 5th annual Academic Surgical Congress were contacted to solicit their facilitation and ensure support, as well as to secure space to conduct the focus group adjacent to the national meeting on February 3, 2010, in San Antonio, TX.

4.1.2. Identify nurses to participate in focus groups and in-depth discussion

Two focus groups were conducted. The first was scheduled for Denver, CO, and infection control and/or infection preventionist nurses regularly working in SSI surveillance were recruited from the Mile High chapter of the Association for Professionals in Infection Control and Epidemiology (APIC). The second focus group was scheduled for Salt Lake City, UT, and infection control and/or infection preventionist nurses regularly working in SSI surveillance were recruited from the APIC Infection Control Association. Participants were given a $35 honorarium for participation.

Return to Contents

Subtask 4.2. Develop a Mechanism to Ascertain Opinions and Ideas From Surgeons and IP Nurses About Current Risk-Adjustment Models and Proposed Changes

4.2.1. Develop focus group guides

Three distinct focus group guides were developed, one for use with each of the above- mentioned, homogeneous groups.

Surgeon focus group. The guide was designed to include a series of tailored, open-ended questions regarding surgeons' attitudes towards current risk adjustment models; the variables that they consider important among their patients, and the appropriateness and relative risks of those variables; and ideas on optimizing feedback to enhance performance. The guide was developed through a process of iterative review by the project team, and was reviewed by the Colorado Multiple Institutional Review Board (COMIRB) prior to use (go to Appendix N).

Nursing focus groups. We developed two nursing focus group guides with the intent to explore the decision to adopt the e-detection tool and issues around its implementation (go to Appendix O). A brief educational symposium on SSI detection and overview of our tool (developed in Chapter 2) was followed by an open discussion with infection control/prevention nurses—one group in Denver and one in Salt Lake City. Results were used to support development of an implementation manual for dissemination.

4.2.2. Secure IRB approval

IRB approval for the surgeon focus group was secured by Denver Health on January 29, 2010, from COMIRB. An IRB exemption was secured by Intermountain Healthcare on January 17, 2011, for the nursing focus groups

4.2.3. Secure Office of Management and Budget clearance or a clearance exemption

Consultation was held with AHRQ's Task Order Officer and the Office of Management and Budget (OMB) Liaison regarding the type of clearance exemption that would be required for the project. It was determined that OMB review and clearance was not required. No clinical exemption was necessary, because the clinical data used for the project was already required to be collected for other purposes. The new data collected through focus group sessions was not subject to OMB clearance requirements, as each focus group was designed to explore a topic area distinct from the other groups, and no group had more than nine participants. No further review was deemed necessary.

4.2.4. Conduct focus groups

Focus groups were conducted using a team approach, involving both a moderator and a trained qualitative researcher. The moderator promoted interaction and guided the discussion to ensure that the focus remained on the topic of interest, order was maintained, and all participants were engaged. The participation of an observing researcher in addition to the moderator allowed for an accurate record to be made without interrupting the flow of discussion.

Each focus group was documented through summary notes taken by the researcher and through audio recording of the session. Documentation included both a record of spoken responses and observation of group members' interactions. The use of redundant documentation methods to augment moderator and researcher recollection ensures the most complete set of data for analysis. Participant initials were recorded with individual responses to ensure accuracy of data analysis (i.e., not attributing a response to multiple individuals when a single individual raised the same point several times).

Surgeon focus group. Six surgeons in addition to the facilitator participated in the focus group conducted on February 3, 2010, adjunct to the 5th annual Academic Surgical Congress in San Antonio, TX. Participants were recruited by the facilitator based on their presence and involvement at the national meeting and interest in the topic under discussion. Surgeons represented multiple health system types and surgical specialties, as described in Exhibit 47.

Nursing focus groups. Five infection control nurses in addition to the facilitator and an observing notetaker participated in the first focus group conducted on February 2, 2011, in Denver. Participants were recruited by the facilitator based on their involvement with the Mile High chapter of APIC and their interest in the topic to be discussed.

Eight infection control nurses in addition to the facilitator participated in the second focus group conducted on February 25, 2011, in Salt Lake City. Participants were recruited from various hospitals from within Intermountain Healthcare.

4.2.5. Conduct followup discussions with focus group participants

No followup discussions were deemed necessary.

Return to Contents

Subtask 4.3. Compile Results of Focus Groups to Make Recommendations of Surgeon, Nurse Perspectives on Risk-adjustment Models for SSI

4.3.1. Independent analysis of focus group and discussion data

Surgeon focus group. Focus group data were analyzed through an inductive approach that used an open, heuristic coding process to identify initial topics mentioned by participants. Individual topics were further categorized, based on the number of participants who conveyed agreement with the concept being discussed. A topic was identified as a theme based on the mention of or agreement with an item by three or more individual participants.

Data were reviewed to the saturation point and discussed with the focus group facilitator and subject expert to ensure the most comprehensive identification of patterns. Topics identified as duplicative were combined into a single occurrence, and themes identified from the comprehensive topic list based on the number of individual mentions.

Nursing focus groups. Data from the first focus group were inductively analyzed by the group facilitator and content expert to elicit an understanding of current infection surveillance processes and to assess potential alteration of the standard process flow by the presence of an electronic SSI surveillance tool. A summary of key points emerging from the discussion was used to inform the second nursing focus group, centered on implementation, and conducted in Salt Lake City on February 25, 2011. Results from the second focus group informed the development of the implementation manual (Appendix R).

4.3.2. Research team meeting and consensus discussion

Results of the surgeon focus group were discussed on team calls and at the in-person meeting in October, 2010. Appendix P provides risk factor inputs from the surgeon focus group.

4.3.3. Summary paper preparation and presentation

Surgeon focus group. Draft and final versions of the results of the content analysis of focus group data were presented in report summary form to the research team, the TOO, and the Technical Experts. A copy of the final version is attached as Appendix Q. Results were used to inform the selection of common factors for Chapter 3, and to gain insight into how surgeons might use the risk adjustment tool and, thus, how dissemination and adoption might better be promoted. A limitation of these results is that the majority of focus group surgeon participants were primarily expert in general surgery. We acknowledge the possibility that additional risk factors specific to CABG or hip and knee arthroplasty procedures might have been identified in a more diverse group.

The consensus among surgeon participants was that current models for SSI risk assessment were inadequate for their needs. Risk models either inappropriately accounted for the factors they included or included an excessive number of factors, such that items of actual significance were obscured. The surgeons expressed desire for the development of new models based on specific patient factors that are identified as significant in affecting risk rates.

Surgeons agreed that infection rate assessments varied, based on items such as whether rates were determined based on process or outcome factors, what methods of documentation were used to report rates between private and public settings, whether or not rate data provided to an membership-based analysis database by participant hospitals are reflective of all populations, and whether or not there was variance in the interpretation of how risk factor measures were defined.

Likewise, risks were determined to vary with some factors to a degree such that different broad categories of risk might be considered, such as risks for emergency surgery patients versus those for elective surgery patients; risks for a patient with managed comorbidities versus those for patients with poorly managed or undocumented comorbidities; risks for patients in compliance with medical recommendations versus risks for noncompliant patients; and whether—in some cases—scheduling considerations, operation timing, or the risk to a patient that might result from a delayed operation outweighed the risk of infection resulting from the operation itself.

Finally, surgeons suggested approaches for improving risk assessment and management, such as giving provider-level feedback in a timely fashion; increasing risk awareness by drawing attention to measurement and tracking of risk factors; and intervening on one or more risk factor variables, based on a patient's risk level, instead of taking a “one size fits all” approach to risk.

Nursing focus group results. The first of two focus groups was held on February 2, 2011. The purpose of this focus group was to cultivate an in-depth understanding of the decision to adopt an electronic SSI surveillance tool. A strawman flow diagram of unassisted SSI surveillance activities for IPs was amended as part of this process (go to Exhibit 48 for the generic flow chart and Exhibit 49 for the amended flow chart). Overall, there was general acceptance and willingness to use an electronic cognitive-support tool. Themes gleaned from this focus group informed the second focus group on implementation, held in Salt Lake City on February 25, 2011.

The generalized model for continuous quality improvement in reducing SSIs is provided in Exhibit 50. Results from the focus groups, together with the outputs from Chapters 2 and 3, were used to develop an implementation manual (go to Appendix R). The manual incorporates tools and strategies developed in both Tasks 2 and 3. As such, the manual is intended to describe how Chapter 2's measures of surveillance allow you to determine if what you did as a result of Chapter 3's risk factor assessment is working or not, as depicted in Exhibit 50.

Return to Contents

Subtask 5.6. List of all Resulting or Anticipated Scientific Presentations and Publications From the Project

The broad-based dissemination of research results and tools developed through ACTION initiatives is an objective of primary importance. Throughout the duration of this project, our team has worked to identify a wide range of potential audiences and opportunities for dissemination at both local and national levels.

In addition to the user manual described in Chapter 4, two poster presentations and one oral presentation have been given at national conferences, and a draft manuscript has been developed for submission to a peer-reviewed journal. An additional manuscript is currently in the initial design phase. Each of these activities is further described below.

Poster presentations

• Surgeons' Acceptance of Surgical Site Infection Risk Adjustment Models. This poster was presented at the Society for Healthcare Epidemiology of America (SHEA), April 1-4, 2011, in Dallas, TX. SHEA's mission is the prevention and control of infections in health care settings. SHEA is dedicated to advancing the science and practice of health care epidemiology and to the prevention and control of morbidity, mortality, and costs connected to health care-associated infections. Copies of the abstract and poster are included as Appendix S.
• Performance of Two Surgical Site Infection Risk Stratification Models for Predicting Infection Risk in Publicly Reported Data from a Safety Net Hospital. This poster was presented at the Surgical Infection Society (SIS) annual conference, May 11-14, 2011, in Palm Beach, FL. The mission of the SIS is “to educate health care providers and the public about infection in surgical patients and promote research in the understanding, prevention and management of surgical infections.” Society members include physicians, nurses, allied health personnel, scientists and others with an interest in surgical infections. Copies of the abstract and poster are included as Appendix T.

Oral presentations. Improving the Measurement of Surgical Site Infection (SSI) Risk Stratification and Outcome Detection. This oral presentation was given at the 2nd Annual HAI Investigators' Meeting on September 18, 2011, which was held adjunct to the AHRQ 2011 Annual Conference in Bethesda, MD. A copy of this presentation is included as Appendix U.

Manuscripts

• “Screening for Surgical Site Infections by Applying Classification Trees to Electronic Data.” This draft manuscript, focused on the results from and methods utilized in Chapter 2 of this project, is being prepared for submission to the Journal of the American Medical Informatics Association (JAMIA). JAMIA is the premier peer-reviewed journal of the American Medical Informatics Association (AMIA), and as such focuses on biomedical and health informatics topics that encompass the full breadth of the field, including clinical care and research, translational science, implementation science, imaging, education, consumer health, public health, and policy. Over 4,000 health care professionals are affiliated with AMIA. A copy of the draft manuscript is included as Appendix V.
• A second manuscript focusing specifically on the results from and methods utilized for Chapter 3 of this project is currently in development. We anticipate finalizing this paper for submission to an open-access, peer-reviewed journal before the end of the calendar year, extending the discussion of methodological challenges encountered and solutions for overcoming these.

Implementation manual. An implementation manual was developed to provide specific instructions on running the algorithm and conducting site-specific testing of that application. An illustrated case study for adapted use of the tool is included. We learned in Chapter 4 from focus groups with IP nurses that the Chapter 2 algorithm could serve as a valuable tool to allow them to work more efficiently, by reducing the number of unnecessary chart reviews and allowing them to more effectively concentrate on prevention activities. We then culled results from Chapter 2, together with an out-of-scope case study (see the following section, Project Expansion), to enrich the opportunity to disseminate the results of our work. The implementation manual is provided as Appendix R. The target audience for the manual is IP staff working in hospital settings. The DH-led team will work with the AHRQ Office of Communications and Knowledge Transfer (OCKT) to identify relevant dissemination channels for the manual.

Project expansion.The DH-led research team conducted two relevant analyses that extend our Chapter 2 and Chapter 3 work. These are summarized in Chapter 5.

Return to Contents
Proceed to Next Section