Slide Presentation from the AHRQ 2008 Annual Conference
On September 10, 2008, Patrick S. Romano, M.D.,
M.P.H., made this presentation at the
2008 annual conference. Select to access the PowerPoint® presentation (590 KB).
Slide 1
Lessons from the AHRQ Patient Safety Indicator (PSI) Validation Pilot
Project
- AHRQ Quality Indicator (QI) Users Meeting
September 10, 2008
- Presenter: Patrick S. Romano, MD MPH; Professor, University of California (UC)
Davis
Center for Healthcare Policy and
Research
- Team: Mamatha Pancholi and
Marybeth Farquhar (AHRQ); Jeffrey Geppert and Teresa Schaaf (Battelle
Memorial Institute); Sheryl Davies and Kathryn McDonald (Stanford); Garth
Utter, Richard White, Daniel Tancredi, Patricia Zrelak, Ruth Baron, and
Banafsheh Sadeghi (UC Davis)
Slide 2
Outline
- What do we mean by "validation"?
- Goals
- Methods
- Results
- Future plans
- Implications
Slide 3
Validation
- A valid measure accurately represents the true state of the phenomenon
being measured (i.e., "free of systematic error"):
- Content (a.k.a., consensual) validity is the degree to which a measure "on
its face" adequately samples all relevant domains of the concept of interest.
- Criterion (a.k.a., concurrent) validity is the degree to which a measure
generates data that agree with data from a better ("gold standard") approach to
measuring the same characteristic.
- Predictive validity is the degree to which a measure successfully
predicts an outcome of interest.
- Construct (a.k.a., convergent/discriminant) validity is the degree to which
a measure correlates with other measures, based on a construct that is grounded
in prior literature or a sound conceptual framework.
Slide
4
Content validity established
- Modified RAND/University of California, Los Angeles (UCLA) Appropriateness Method.
- Physicians of various specialties/subspecialties, nurses, other
professionals (e.g., midwife, pharmacist) were nominated.
- Each potential indicator was assigned to one or two panels.
- All panelists rated all assigned indicators (1-9) on overall usefulness
and other dimensions.
- Pre-conference ratings and comments were collected.
- Individual ratings were returned to panelists with distribution of
ratings and other panelists' comments.
- Telephone conference call(s) focused on high-variability items and
panelists' suggestions, which were adopted only by consensus.
- Post-conference ratings and comments were collected.
- Excluded indicators rated "Unclear," "Unclear-," or "Unacceptable":
- Median score <7, OR
- At least 2 panelists rated the indicator in each of the extreme 3-point
ranges.
Slide
5
PSIs reviewed
- 48 indicators reviewed in total:
- 37 reviewed by multispecialty panel
- 15 of those reviewed by surgical panel
- 20 "accepted" based on face validity:
- 2 dropped due to operational concerns
- 17 "experimental" or promising indicators
- 11 rejected
Slide 6
Organisation for Economic Co-operation and Development (OECD) Expert panel
review of PSI usefulness and "preventability"
- Acceptable—Acceptable (-)—Unclear—Unclear (-)
- Decubitus ulcer—Complications of
anesthesia—Death in low mortality Diagnosis Related Group (DRG)—Failure to rescue
- Foreign body left in—Selected infections due to
medical care—Postop hemorrhage/hematoma—Postop physiologic/metabolic
derangement
- Iatrogenic pneumothoraxa—Postop Pulmonary Embolism (PE) or Deep Venous Thrombosis (DVT)b—Postop respiratory failure
- Postop hip fracturea—Transfusion
reaction—Postop abdominopelvic wound dehiscence
- Technical difficulty with procedure—Birth
trauma—Postop sepsis
- Obstetric trauma (all delivery types)
a Panel ratings
were based on definitions different than final definitions. For "Iatrogenic
pneumothorax," the rated denominator was restricted to patients receiving
thoracentesis or central lines; the final definition expands the
denominator to all patients (with same exclusions). For "In-hospital fracture"
panelists rated the broader Experimental indicator, which was replaced in the
Accepted set by "Postoperative hip fracture" due to operational concerns.
b Vascular
complications were rated as Unclear (-) by surgical panel;
multispecialty panel rating is shown here.
Slide
7
National Quality Forum (NQF) evidence review
- National Voluntary Consensus Standards for Hospital Care: Additional
Priorities, 2007.
- Purpose was to seek additional voluntary consensus standards for
measuring the performance of the nation's general acute care hospitals,
including:
- Morbidity and mortality measures
- Anesthesia and surgery
measures
- Measures for utilization
rates for risky or often unnecessary procedures
- Surgical volume and
mortality measures
- Readmission rates and length of stay (LOS) rates
- Pain assessment
- Pediatric asthma
- Convened Technical Assessment Panels for Patient Safety, Surgery and
Anesthesia, Pediatric, Public Reporting, Length of Stay and Readmission.
Slide
8
NQF Review Results
- QI
Name—Endorsed date
- Death
in Low-Mortality DRGs (PSI 2)—5/15/08
- Death
Among Surgical Patients w/ Treatable Complications (PSI 4)b—5/15/08
- Foreign Body Left During
Procedure-Provider level (PSI 5)a—5/15/08
- Iatrogenic
Pneumothorax-Provider level (PSI 6)b—5/15/08
- Postoperative DVT or PE (PSI 12)—7/31/08
- Postoperative
Wound Dehiscence-Provider level (PSI 14)b—5/15/08
- Accidental
Puncture or Laceration-Provider level (PSI 15)b—5/15/08
- Transfusion
Reaction-Provider level (PSI 16)a—5/15/08
a = proposed hospital-acquired conditions (HACs).
b = proposed Reporting Hospital Quality Data for Annual Payment Update (RHQDAPU).
Slide
9
Predictive Validity Established
- Predictive validity
established:
Impact of preventing a PSI on mortality, LOS, charges
NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74
- Indicator—) Mort (?)—) LOS (d)—Charge
- Postoperative septicemia—21.9—10.9—$57,000
- Selected infections due to medical
care—4.3—9.6—38,700
- Postop
abd/pelvic wound dehiscence—9.6—9.4—40,300
- Postoperative respiratory
failure—21.8—9.1—53,500
- Postoperative physiologic or metabolic
derangement—19.8—8.9—54,800
- Postoperative thromboembolism—6.6—5.4—21,700
- Postoperative hip fracture—4.5—5.2—13,400
- Iatrogenic pneumothorax—7.0—4.4—17,300
- Decubitis ulcer—7.2—4.0—10,800
- Postoperative hemorrhage/hematoma—3.0—3.9—21,400
- Accidental puncture or laceration—2.2—1.3—8,300
- Excess
mortality, LOS, and charges computed from mean values for PSI cases and matched
controls.
Slide 10
Predictive
validity established:
- Predictive validity
established:
Impact of preventing a PSI on mortality, LOS, charge
VA PTF 2001 analysis by Rivard et al., Med Care Res Rev; 65(1):67-87
- Indicator—) Mort (?)—) LOS (d)—Charge
- Postoperative septicemia—30.2—18,8—$31,264
- Selected infections due to medical
care—2.7—9.5—13,816
- Postop
abd/pelvic wound dehiscence—11.7—11.7—18,905
- Postoperative respiratory
failure—24.2—8.6—39,745
- Postoperative physiologic or metabolic
derangement
- Postoperative thromboembolism—6.1—5.5—7,205
- Postoperative hip fracture
- Iatrogenic pneumothorax—2.7—3.9—5,633
- Decubitis ulcer—6.8—5.2—6,713
- Postoperative hemorrhage/hematoma—5.1—3.9—7,863
- Accidental puncture or laceration—3.2—1.4—3,359
- Excess mortality, LOS, and charges computed
from mean values for PSI cases and matched controls.
Slide
11
AHRQ PSI Validation Pilot Goals
- Gather evidence on the criterion validity of the PSIs based on medical
record review.
- Improve guidance about how to interpret & use the data.
- Evaluate potential refinements to the specifications.
- Develop medical record abstraction tools.
- Develop mechanisms for conducting validation studies on a routine basis.
Slide
12
Positive Predictive Value
- The positive predictive value or post-test probability is
the proportion of flagged cases who actually had the event.
- The
Positive Predictive Value (PPV) can be further defined as:
- PPV = number of True Positives
divided by
number of True Positives + number of False Positives
Slide
13
PSI Validation Pilot Methods
- Retrospective cross-sectional study design
- Volunteer sample of collaborative partners:
- Facilitating organizations (e.g.,
Arizona)
- Hospital systems
- Individual hospitals
- Sampling based on administrative data
- Sampling probabilities assigned using AHRQ QI software to generate
desired sample size locally (30) and nationally (240 per indicator)
Slide
14
Data collection methods
- Each hospital identified chart abstractors.
- Training occurred via Webinars.
- Medical record abstraction tools & guidelines.
- Pretested in the
Sacramento
area.
- Targeted the ascertainment of the event, risk factors, evaluation &
treatment, and related outcomes.
- Coordinating center entered data from paper forms.
Slide 15
PSI Validation Pilot Timeline
- 10
indicators—divided into 2 phases of
5 each
- Phase I review—
- Training early 2007
- Chart review 4 month process
- 2nd Qtr 2006 through 1st Qtr 2007 (but some hospitals had to reach back as far as 4th Qtr
2005)
- Phase II review—
- Awaiting OMB approval
- Pre-pilot (6 hospitals) now under way
- Phase
III—sensitivity determination
Slide
16
PSI Validation Pilot Phases
- Phase I—Phase II:
- Accidental puncture and
laceration—Foreign body left in during procedure
- Selected infection due to
medical care—Postoperative hemorrhage or hematoma
- Postoperative pulmonary
embolism or deep vein thrombosis—Postoperative physiologic and metabolic
derangement
- Postoperative
sepsis—Postoperative respiratory failure
- Iatrogenic
pneumothorax—Postoperative wound dehiscence
Slide
17
PSI Validation Pilot Samples
- Phase I—Hospitals—Sample
- Accidental puncture and
laceration—43—240
- Iatrogenic pneumothorax—38—205
- Postoperative PE/DVT—37—155
- Selected Infection due to
Medical Care—37—189
- Postoperative Sepsis—33—164
- Overall—47—967
Slide
18
Accidental Puncture or Laceration
- N=249
- 91% (95% CI = 86-94%) PPV or true events
- 9% (n=23) were false positives:
- 7% (n=18) miscoded:
- 4 had disease-related lesions (perforated
appendix or ischemic colon, ruptured AA,
rectovesical fistula)
- 7 had a different complication (4 bleeding
due to operative conduct, 1 surgical site infection, 1 dislodged gastrostomy
tube, 1 periprosthetic fracture)
- 7 cases had no apparent event other than
normal operative/procedural conduct (intentional, rule-out)
- 2% POA
(related to an earlier episode of care):
- 7
cases had no apparent event other than normal operative/procedural conduct (intentional, rule-out)
Slide
19
Accidental Puncture or Laceration
- Characteristics of confirmed cases (N=226):
- 170 (75%) were potentially consequential
- Most were related to an abdominal or pelvic procedure:
- 51
(30%) enterotomy or other perforation of the GI tract
- 42
(25%) bladder injury
- 33
(19%) dural tear
- ; 27
(16%) vascular injury
- 132 (78%) involved a reparative procedure at the time of occurrence
- 19 (11%) required a return to the OR (one death)
Slide
20
Iatrogenic Pneumothorax
- N=205
- 78% (95% CI = 73-82%) PPV or true events that occurred during the
hospitalization.
- 11% were false positives:
- 7% (n=14) present or suspected at admission, most (n=8) transferred in
- 4% no documentation of event (miscoded), but some with suspicion (n=3)
- 11% had exclusionary diagnosis or procedure (e.g., trauma, metastatic
cancer)
Slide
21
Iatrogenic Pneumothorax
- Characteristics of confirmed cases (N=156):
- 9 (6%) transthoracic needle aspiration or biopsy
- 66 (47%) central venous catheter placement:
- Only 4
used sonographic and 7 fluoroscopic guidance
- 59 (40%) other invasive procedures on or near the neck or chest wall:
- 37
catheterization, pacemaker insertion
- 3 laparoscopic
procedures
- 8
nephrectomy/renal procedures
- 2
operations involving the spinal canal
- 9 other
procedures
- 5 (5%) mechanical ventilation
- 1 (1%) cardiopulmonary resuscitation
Slide
22
Postoperative DVT or PE
- N = 155.
- Coding perspective:
- PPV = 83% (95% CI = 73-95%)
- 17% were false positives
- 10%
(n=12) present at admission (POA)
- 7%
(n=8) no documentation of VTE (miscoded)
- Clinical perspective:
- PPV = 48% (95% CI = 33-61%)
- Additional false positives due to preoperative VTE (20%), upper
extremity DVT (9%), superficial or unspecified vein (6%)
Slide
23
Comparing PPV estimates with UHC sample for postoperative DVT/PE
- UHC Cohort (n=450)—Coding—Clinical:
- Sensitivity—80% (46-00%)—100%
- Specificity—99.5% (99.3-99.6%)—98.6%-99.2%)
- Positive Predictive Value—72% (67-79%)—44% (36-52%)
- Negative Predictive Value—99.6% (98.9-100%)—100%
- AHRQ Cohort (n=121):
- Positive Predictive Value—83% (73-95)—48% (33-61%)
Slide
24
Selected Infection Due to Medical Care
- N=191
- 61% (95% CI = 51-71%) were true events that occurred during the index
hospitalization
- 39% were false positives:
- 7% (n=14) had exclusionary diagnosis
- 20% (n=39) were present on admission, with no new infection
- 12% (n=23) had no documentation of infection
Slide 25
Selected Infection Due to Medical Care
- N=115 with new infection:
- 106 with new infection
- 9 with POA + new infection
- Majority related to central lines
(n= 53):
- 35** cases due to non-tunneled central lines (SC, IJ, Fem):
- Mean 11
days (7 days SD), range 2-35 days (n=33)
- 17 cases associated with peripherally inserted central catheter (PICC) lines
- Mean 13.4 days (7.7 days SD), range 2-35 days
- 26 cases related to other types of catheters (3
arterial lines, 4 long-term vascular access ports, 3 ET, 8 IV, 1 urinary, 2
other)s
* Difficult review—some
earlier termination by abstractors.
** 2 cases had extreme values of 101 and 374 days were excluded from
calculation.
Slide
26
Postoperative Sepsis
- N=164.
- 41% (95% CI = 28-54%) were true events that occurred during the
hospitalization.
- 59% were false positive:
- 17% had no documentation of bacteremia, septicemia, sepsis or systemic inflammatory response syndrome
(SIRS)
- 17% had infection (=14%) or sepsis (=3%) POA
- 25% did not have elective surgery
Slide 27
Summary of PPVs—Preliminary
estimates
- PSI—PPV
- Accidental
puncture or laceration—91%
- Iatrogenic
pneumothorax—88%
- Postoperative
DVT/PE—48-83%
- Selected
infections due to medical care—61%
- Postoperative
sepsis—41%
- PPVs high for NQF-endorsed indicators
Slide
28
Recognizing Limitations
- Not all data elements of interest available via chart review.
- Time constraints (minimize burden on collaborators).
- Inter-hospital variation in documentation and abstraction.
- Volunteer sample; time periods varied slightly across hospitals.
Slide
29
AHRQ QI Validation Pilot: Future Plans
- Further analysis of potential preventability.
- Evaluation of alternative ICD-9-CM specifications
- Establish ongoing infrastructure for validation.
- Estimate sensitivity of 10 indicators (including Foreign Body,
Pneumothorax, Wound Dehiscence, and Accidental Puncture and Laceration)
Slide
30
Policy Implications
- Coding changes needed to enhance specificity and PPV in some areas:
- AHRQ proposed new codes for DVT
- the Centers for Medicare and Medicaid Services (CMS) proposed new code for catheter-associated bloodstream infection
- With these changes, 3 of 5 PSIs tested in Phase 1 should have high PPV.
- These indicators have been endorsed by NQF.
- More data on sensitivity (false negatives) are needed to avoid rewarding
hospitals that underreport.
Slide
31
Acknowledgments
- AHRQ project team:
- Mamatha Pancholi & Marybeth Farquhar
- Battelle training and support team:
- Laura Puzniak & Lynne Jones
- All of the validation pilot partners!
Current as of February 2009
Internet Citation:
Lessons from the AHRQ PSI Validation Pilot Project. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version). February 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualmtg08/091008slides/Romano.htm