Slide Presentation from the AHRQ 2008 Annual Conference
On September 10, 2008, Amy Rosen, Ph.D., made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (800 KB).
Slide 1
Validating the Patient Safety Indicators (PSIs) in the Department of Veterans' Affairs (VA): a Multi-Faceted Approach
- Funding: VA Health Services Research and Development (HSR&D) Service
SDR 07-002 : 10/01/07-9/30/11
- Amy Rosen, Ph.D., Principal Investigator
Slide 2
Project Team
- Collaboration among:
- VA's HSR&D Service
- National Center for Patient Safety (NCPS)
- AHRQ (Quality Indicators [QI] team and individual investigators)
- VA and non-VA clinicians, surgical experts, nurse abstractors
- National steering committee:
- Representatives from VA Office of Quality Performance, NCPS
- Nursing Services, Surgery, Patient Care Services
- Selected members of the AHRQ QI team
- Selected Patient Safety/QI Managers and other potential end-users
Slide 3
Overall Project Goal
- Develop a validated and reliable set of patient safety measures that broadly reflect the interests of key VA stakeholders, but that are generalizable beyond the VA.
- Specific Objectives:
- Develop collaborations with key stakeholders to guide in PSI selection and validation
- Investigate the criterion validity of the PSIs by review of the VA's Electronic Medical Record (EMR)
- Identify processes and structures of care associated with individual PSIs
- Revise and improve the PSIs using multiple data sources and settings of care
- Assess the utility validity of the PSIs for QI and performance measurement
Slide 4
Goal 1: Develop Stakeholder Collaboration
- Stakeholders' meeting (Dec, 2007):
- Approved selection of PSIs
- Approved plan to validate AHRQ's Phase I/Phase II PSIs
- Reviewed field consultation interview questions:
- Recommended focus on general questions on patient safety
- Suggested less attention on specific PSIs in field consultations:
- Field consultations held to examine the validity of the PSIs, not to judge facilities' performance
- Contact with stakeholders subsequent to meeting:
- Approved final interview protocols "TO/THRU" memo to sites asking them to participate
Slide 5
Goal 2: Identify False Positives—Are Cases Flagged by the AHRQ PSIs Present in the EMR?
- Obtained national access to electronic medical record (EMR): "VistaWeb"
- Hired and trained two nurse abstractors to conduct chart abstraction
- Modified AHRQ Phase I and Phase II chart abstraction tools for VA:
- Pilot testing and clinician review
- Five tools "ready for prime time," five almost ready, five being developed de novo
- Completed validation of pulmonary embolism/deep vein thrombosis (PE/DVT)
- Currently abstracting charts for iatrogenic pneumothorax
- Currently piloting web-based application (InfoPath) for gathering and entering chart-abstracted data
Slide 6
Formatting Features of InfoPath
- This slide shows the conditional formatting features of InfoPath; these features enable more efficient chart abstraction.
- A1: Is there documentation that the patient had a post-operative or deep vein thrombosis during this admission?
- Yes:
- No
- Unable to determine
- Record excluded
- A2: Documentation of ascertainment of event If YES to A1, Describe documentation found in the medical record.
Slide 7
Hospital Selection
- Ran PSI software (v. 3.1a) on VA inpatient data (2003-2007):
- Obtained rates of individual PSIs and PSI composites
- Used 12 PSIs:
- PSIs 1-15
- Excluded PSIs 1, 5, 8
- Population:
- Sample for chart abstraction:
- 28 hospitals, 112 charts per PSI
Slide 8
Sample Selection Methodology
- Stratified population by observed and expected #s of PSIs:
- Group 1: at least 4 observed and 4 expected (n =28)
- Group 2: at least 2 observed and 2 expected (n=33)
- Group 3: at least 1 observed and 1 expected (n=18)
- Total for Groups 1-3: 79 hospitals
- Ranked 79 by AHRQ PSI composite (denominator weights):
- Chose top 3 and bottom 3 from each group
- Randomly selected from remaining hospitals within each group: group 1=4, group 2=4, group 3=2 to obtain 28 hospitals (10, 10, and 8, respectively)
- Geographic distribution and ICU severity taken into account
- Selected 6 hospitals for field consultations and ranked them based on PSI composite:
- Geographic location and size taken into account
Slide 9
Chart Abstraction
PE/DVT
- Conducted retrospective EMR review of 112 flagged cases.
- Conducted inter-rater reliability (IRR) throughout EMR review.
- 28 cases (25% of all charts) reviewed for IRR due to:
- Large numbers of exclusions
- IRR >90%
- 89% agreement rate achieved with 1st IRR, 94% with 2nd IRR
- Issues:
- Length of time to complete chart abstraction (1-1/2 hours for full record; 20 minutes for false positives)
- Problems with accessing VistaWeb
Slide 10
Technical Specifications of PE/DVT
- Numerator:
- Discharges among cases meeting the inclusion and exclusion rules for denominator.
- ICD-9-CM codes for PE/DVT in any secondary diagnosis field
- Denominator:
- All surgical discharges age 18 and older.
- Defined by specific DRGs and an ICD-9-CM code for an OR procedure
- Exclusion criteria for all cases:
- Preexisting (principal diagnosis or secondary diagnosis present on admission, if known) PE/DVT
- Procedure for interruption of vena cava the only OR procedure
- Procedure for interruption of vena cava occurs before or on the same day as first OR procedure
- Medical Diagnostic Category (MDC) 14 (pregnancy, childbirth, and puerperium)
Slide 11
Post-operative PE/DVT Validation Results
- Pie chart shows:
- True Postoperative PE/DVT: 49 cases (44%)
- Coding-Related Inaccurate Diagnosis: 24 cases (21%)
- Present on Admission: 16 cases (14%)
- Pre-Procedure Diagnosis: 13 cases (12%)
- Remote History of PE/DVT: 10 cases (9%)
- Total # of cases: 112
Slide 12
False Positives: A Comprehensive Analysis
| Classification of False Positives |
Number
of Cases |
Percentage |
| DVT/PE Present on Admission (POA) |
16 |
25.4% |
| Pre-Procedure Diagnosis of PE/DVT |
13 |
20.6% |
| Remote History of DVT or PE (>6 months) |
10 |
15.9% |
| Arterial (not venous) thrombosis* |
4 |
6.4% |
| Negative PE/DVT workup* |
4 |
6.4% |
| "Rule out PE" as cause of death* |
3 |
4.8% |
| Superficial (not deep) thrombosis or thrombophlebitis* |
3 |
4.8% |
| Miscellaneous* |
10 |
15.9% |
| Total |
63 |
100% |
* Represents coding-related inaccurate diagnosis
Slide 13
Coding-Related Inaccurate Diagnosis: Miscellaneous Category
| Classification of False Positives |
Number
of cases |
| Vein stenosis (no thrombosis) |
1 |
| PE stands for Physical Exam not Pulmonary Embolus |
1 |
| Low dose Coumadin prophylactic not therapeutic |
1 |
| Surgery done at outside hospital |
1 |
| Cerebral embolization of arteriovenous malformation (AVM) |
1 |
| Prophylactic heparin mistaken for therapeutic heparin |
1 |
| Right lower extremity (RLE) U/S ordered to r/o abscess at surgical site |
1 |
| Unknown |
3 |
| Total |
10 |
Slide 14
PE/DVT Results: Comparison of Studies
| |
Our
study |
Zhan
study |
AHRQ
study |
NSQIP
and PTF
study |
UHC
study |
| N |
112 |
20,868 |
155 |
55,682 |
1022 |
| PPV |
44% |
29% |
68% |
22% |
61% |
| Sensitivity |
-- |
68% |
-- |
66% |
-- |
Slide 15
Problems in Coding PE/DVT
- PE/DVT PSI designed as initial screen
- Accuracy of method to detect true positives using administrative data affected by:
- Standards used to assign codes for "other" or secondary conditions—> based on the Uniform Hospital Discharge Data Set (UHDDS).
- "Other" conditions: those that coexist at the time of admission, develop sequentially, affect the treatment received and/or length of stay, or affect patient care
- Definition of PE/DVT relative to:
- UHDDS coding standards,
- ICD-9-CM Official Coding Guidelines for Coding and Reporting
- Coding Clinic published by the American Hospital Association (AHA)
Slide 16
Problems in Coding PE/DVT, cont'd
- False Positive 1: chart review does not document a PE/DVT:
- Code was present on admission (POA) and meets UHDDS definition of "other" diagnosis.
- Code assigned as a current condition.
- Should have been coded as a "history of" with a V code
- It was still a "rule out" condition at the time of discharge
- Coding system issue:
- Was miscoded (superficial vein and not deep vein) due to coding invention and ICD-9-CM alphabetic index
- Coder did not identify the correct vein anatomically
- Should not have been coded at all
- False Positive 2: chart review documents a PE/DVT, but it is not a postoperative PE/DVT:
- Diagnosis of PE/DVT occurred after admission but before surgery
Slide 17
Recommendations for Improving PE/DVT
- Modify coding rules:
- Use National Surgical Quality Improvement Program (NSQIP) definitions to influence the coding rules
- Specify the circumstances when the PE/DVT should be coded and publish them in Coding Clinic and Official Guidelines
- As "current conditions" or "history of"
- Begin using POA in VA
- Explore use of "997" complication code as part of the PSI algorithm to capture post-operative PSIs
- Explore expansion of POA to include a special character denoting "POA prior to surgery"
- Undertake targeted education to help coders, researchers, and healthcare professionals understand the use of coding guidelines for "PE/DVT"
Slide 18
Objective 3
Question: Do High-Performing Facilities Have Higher Rates on Structures and Processes of Care than Lower-Performing Facilities?
- Conduct two pilot field consultations locally:
- determine feasibility and logistical problems
- test interview questions
- add/delete selected staff
- Conduct field consultations at 6 facilities:
- Perform structured interviews with selected staff
- Gather data on safety and quality
- Assess differences between sites on structures and process using qualitative methods and ratings
Slide 19
Selected Staff for Interviews
- Individual Interviews:
- Executives
- Service Chiefs
- Other Middle Managers
- Other Non-Managers
- Group Interviews:
- Surgical Service
- Medical Service
- Non-Managers
Slide 20
Interview Domains
- Organization, Structure, and Culture
- Coordination of Work and Communication
- Interface within Service
- Monitoring Quality of Care
- Quality Improvement
- General Clinical Topics
- Coding
- Technology and Equipment
- Technical Competence of Staff
- Leadership
- Interface with Other Services
- Systems Issues and Human Factors
- Staffing
- Summary Evaluation of Service Overall
Slide 21
Domain: Monitoring Quality of Care/Quality Improvement
- In your facility, what are some of the initiatives related to improving patient safety that you know about?
- On what does it focus?
- What facilitated its implementation?
- What were the implementation obstacles?
- How effective do you think it is?
- What are some of the most common adverse events that you see in your day-to-day work? Please refer to the list provided.
- What is being done now to reduce the incidence of this complication?
- What do you think would be helpful in further reducing the incidence of this?
- Is there anything not on the list we provided you that you believe is a concern?
Slide 22
Domain: Coding
- Who is involved in assigning ICD-9 and procedure codes to adverse events?
- Are physicians involved in reviewing the event codes?
- Do you think there is a concern about the accuracy of coding relating to adverse events?
- If yes: What is the concern?
Slide 23
Domain: Technology and Equipment/Technical Competence of Staff
- I am curious to hear about what problems, if any, you or others have had with the technology and/or equipment on the service.
- What problems have you had with the accessibility or availability, or both, of technology and/or equipment?
- What problems you have had with the quality or functioning, or both, of the technology and/or equipment?
- What problems, if any, have you or other staff had being properly trained to use the technology and/or equipment?
- What technology and/or equipment, if any, does not exist at your hospital that would help improve patient safety?
Slide 24
Capturing Initial Impressions
- Immediately after each pilot field consultation, each interviewer summarizes her/his:
- Impressions of each domain in a paragraph
- Overall impressions of the site
- —> in both cases giving specific examples
- Soon afterwards, all interviewers and other members of the PSI validation team meet to discuss the impressions.
- These discussions will be used to generate a protocol for capturing initial impressions for study's six field consultations.
- We may rate sites, creating examples for an "ideal" site
- We may decide to use only written impressions
Slide 25
Rating Category Possibilities
- Some numeric scale:
- NSQIP rating (1 to 9; 1=poor and 9=excellent)
- Other model rating (0 to 4)
- Some hierarchy scale:
- Poor, fair, good, very good, excellent
- Some recognition scale:
Slide 26
Example of Rating: NSQIP
| Standard |
Poor |
Fair |
Good |
Very Good |
Excellent |
| 1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
Technology and
Equipment |
|
|
|
|
|
|
√ |
|
|
Technical
Competence of
Staff |
|
|
|
|
|
|
|
|
√ |
Interface with Other
Services |
|
|
|
|
|
√ |
|
|
|
Relationship with
Affiliated Institution |
|
|
|
|
|
|
|
√ |
|
Slide 27
Initial Impressions of Pilot Sites
| Domains |
Rating* |
Evidence Narrative |
Examples |
Monitoring Quality of Care
Questions 1, 3, 4 |
|
|
|
Quality Improvement
Questions 1, 3 |
|
|
|
Leadership
Questions 2, 4 |
|
|
|
Systems Issues and Human Factors
Question 4 |
|
|
|
* Our initial rating scale: Excellent, Very good, Good, Fair, Poor
Slide 28
Next Steps (1)
- Identify False Negatives
- Use an existing "gold standard" (e.g., VA NSQIP) for 5 surgical PSIs
- Identify risk factors by estimating logistic regression models for each of the PSIs
- Use propensity score stratification to generate propensity class strata for each of the PSIs
- Use AHRQ Composite Tool to review medical records of "high-risk" cases for PSIs
- Screen EMRs of high-risk cases using keyword searches (selected "hits" will have chart review)
- Explore machine language processing as an informatics tool to search for false negatives
Slide 29
Next Steps (2)
- Examine association between explicit processes of care and individual PSIs:
- Match 1,680 flagged PSI cases with 1,680 controls (unflagged cases matched on demographic and clinical characteristics) to determine whether flagged cases are more likely to experience "process failures."
- Use propensity score methodology to perform matching; chi-square tests used to examine proportion of failure rates among cases and controls.
Slide 30
Next Steps (3)
- Revise and Improve the PSIs:
- Add additional data elements to inpatient data:
- Present-on-admission (POA) diagnoses, do-not-resuscitate (DNR) codes, selected clinical, laboratory and pharmacy data elements
- Link inpatient data with outpatient/inpatient data 30/60 days preceding index hospitalization (obtain POA diagnoses)
- Link inpatient data with outpatient/inpatient data 30/60 days following index hospitalization to evaluate whether additional PSIs are detected
- Link VA and Medicare data to examine PSI readmission in private sector
- Improve coding by implementing coding changes
- Modify PSI numerators and denominators on inclusion/exclusion criteria
- Recalculate false positives and negatives
Slide 31
THANK YOU!
Slide 32
- Amy Rosen, Ph.D.
Center for Health Quality, Outcomes & Economic Research
(VA Center of Excellence)
- Boston University Schools of Public Health and Medicine,
Departments of Health Policy and Management and Family Medicine
Current as of January 2009
Internet Citation:
Validating the Patient Safety Indicators (PSI) in the VA: A Multi-Faceted Approach. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version). January 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualmtg08/091008slides/Rosen.htm
540 Gaither Road Rockville, MD 20850