2056. Measuring Quality of
Care: Is There a Role for Peer Review?
TP Hofer, VA Center of Excellence-Ann Arbor, S Asch, VA Center
of Excellence-Los Angeles, M Hogan, VA Center of Excellence-Ann Arbor, J
Adams, Rand Corporation, EA Kerr, VA Center of Excellence-Ann Arbor
Objectives: We sought to
develop a more reliable structured implicit chart review instrument for use in
assessing the quality of care for chronic disease. We hypothesized that we would
be more successful for clinical conditions that have a substantial evidence base
for treatment in the clinical literature. We also illustrate a method of
analysis that allows one to determine the measurement questions to which the
instrument is generalizable.
Methods: As part of a
larger study comparing implicit and explicit measures of quality across 26 sites
in the VA, we developed a structured implicit review instrument for four
clinical conditions: COPD, diabetes(DM), hypertension(HTN) and acute conditions.
Twelve reviewers carried out 180 reviews of 60 medical records for the care
given in up to all four of the conditions over a 13 month period, including
inpatient and outpatient care. We estimated the reviewer, record and noise
variance for the calculation of reliabilities in a multilevel analysis.
Results: The reliability
of a single review for detecting differences in quality of hypertension care and
diabetes care across patients is .45. This estimate, which is generalizable to
the population of reviewers and patients, is substantially higher than most
prior estimates of implicit review reliability. However, for COPD and acute
care, two clinical areas where the body of evidence supporting specific
interventions is substantially less, the reliability is poor (.26 and .11
respectively). The lower reliabilities are due to decreased signal rather than
increased noise. With a careful
reliability study the reviewer effects can be removed raising the reliability
for the review of DM and HTN as high as .53.
Conclusions: It is
possible to obtain reliable ratings of chronic disease and that higher
reliability is achieved for reviews in clinical areas that have a more developed
evidence base. With a less developed evidence base the reviewers appear to have
more difficulty in making systematic distinctions in the quality of care between
patients. This analytic approach allows for generalizable conclusions about the
use of these instruments and reasons for low or high reliability.
Impact: Sampling relatively few records for structured implicit review may be a reliable alternative or complement to assessing quality of care by checklist approaches.