NRSA Workshop:

Slide Presentation by Andrew Ryan

On June 2, 2007, Andrew Ryan made a slide presentation on the utility of process performance measurement for Medicare value based purchasing at the 13th Annual National Research Service Award (NRSA) Trainees Research Conference. This is the text version of the slide presentation. Select to access the (PowerPoint® File, 230 KB).

Slide 1

The Utility of Process Performance Measurement for Medicare Value Based Purchasing

Andrew Ryan
Schneider Institute for Health Policy
The Heller School for Social Policy and Management
Brandeis University

Slide 2

Measurement of Quality

• Process performance measurement is dominated.
• Hospital Compare and CMS/Premier P4P programs have used predominately process measures.

• Why?
  • Providers are deemed to have control over performance.
  • Greater statistical power.
  • Provide "actionable" information for improvement (Birkmeyer et al. 2006; Mant 2001).
  • Fear of inappropriately labeling hospitals as "bad" due to random variation in outcomes.
  • Fear of selection from use of outcomes.

• Complicated nature of hospital care for Medicare beneficiaries may make process measures an inadequate proxy of underlying quality.

Slide 3

Interpretation of Process Measure Performance

• Utility of process performance conditional on correlation of process measures with true quality.
• Risk adjusted outcome can proxy for true quality.

Two questions:

1. Can observed process performance measures proxy for hospital outcomes?
2. Are observed process performance measures causally related to hospital outcomes?

Slide 4

Organizational Theory of Performance Measurement

• Agency theory
  • Government depends on actions of hospitals to provide care and cannot closely monitor hospital activities.

• Gaming
  • Data for performance measures are under the control of front line staff that the measures are being used to control: hardly a solution to the agency problem (Smith, 1995).

• Myths
  • Formal organizational structures (as can be assessed by performance measures) function as "myths" which organizations incorporate to gain legitimacy (Meyer and Rowen, 1977).

• Recognition of dysfunctional attributes of performance measurement has long history in management literature (Ridgeway, 1956).

Slide 5

Cardiac Care: Recent Studies on Association Between Process and Outcome Measures

• Bradley et al. 2006; Eagle et al. 2005; Fonarow et al. 2007; Granger et al. 2005; Luthi et al. 2004; Luthi et al. 2003; Peterson et al. 2006; Werner and Bradlow 2006.
• Have not conducted analyses that evaluate the causal relationship.

Slide 6

Performance Function

• Patient outcome = f(patient characteristics, facility characteristics, provider characteristics, measured processes of care, unmeasured processes of care).

• Regression model:
  Patient Outcome = β0 + β1 measured patient characteristics +
  β2 measured health care facility characteristics + β3 measured provider characteristics + Δ measured processes of care + v
  v = b5 unmeasured characteristics + u

• Correlation between model variables and v will bias coefficient estimates in cross sectional regression model.

• Unbiased estimates of effect of measured process on outcomes can be obtained from fixed effects or instrumental variables models.

Slide 7

Analysis: Frequently Estimated Model

(1) Linear probability: Pr(Mortality_ikjt) = β0 + β1 X_ikjt + β2 Z_j + β3 2005 _t + δ process _jkt + e _ikjt

Where:

• i is indexed to individuals, j is indexed to hospitals, k is indexed to condition (AMI or heart failure), t is indexed to time (2004 or 2005).
• X is a vector of individual-level variables (demographics, Elixhauser comorbidities, type of hospitalization).
• Z is a vector of hospital characteristics (bed size, average daily census, region, number of surgical operations, region, ownership).
• process is a vector of process measures (different for AMI and heart failure).
• 2005 is a dummy variable for 2005.

Slide 8

Analysis Continued

(2) FE Linear probability: Pr(Mortality_ikjt) = β0 + β1 X_ikjt + β2 2005 _t + δ process _jkt + h_j + e _ikjt

(3) FE 2SLS Linear probability: Pr(Mortality_ikjt) = β0 + β1 X_ikjt + β2 2005 _t + δ prôcess _jkt + h_j + e _ikjt

Where:

h is a vector of hospital-specific fixed effects

prôcess _jkt = ζ0 + ζ1 reporting ratio k_jt + ζ2 reporting ratio squared _kjt + ζ3 "non-included" indicators reported _kjt + u _kjt

Time varying unobserved factors may be related to process measures in specification 2. These include larger structural or procedural changes within the hospital. For this confounding to take place, unobserved variation must be correlated to changes in process performance. If this is true, then the change in unobservables may be part of a larger change in hospital behavior that is taking place, and reflected in the process performance measures. Thus, the process measures can be seen as proxying for a change in overall process. This is the intent of the measures.

In specification 3, the presentation attempts to separate the effects of unobservable changes at the hospital level from the pure effect of the change in the process measures themselves through 2SLS

Slide 9

Data

• Medicare fee-for-service claims data (2004 and 2005).
• Outcomes: 30 day mortality.
• 2005 American Hospital Association data on hospital characteristics.
• 2004-2005 Hospital Compare data
  • AMI process measures: aspirin at arrival/discharge, β blocker at arrival/discharge, and use of ACE inhibitor.
  • Heart failure: use of ACE inhibitor, assessment of left ventricular function.
• Instruments are measures of gaming:
  • Ratio of patients included in quality score to total Medicare patients.
  • Number of "non-included" process performance measures reported upon.

Slide 10

Preliminary Results

• Linear probability – process vector jointly significant in AMI and heart failure models at p < .05.
• FE linear probability – process vector not jointly significant in AMI or heart failure models at p < .05.
• FE 2SLS linear probability – process vector not jointly significant in AMI or heart failure models at p < .05.

Slide 11

Implications

• Accounting for unobserved time-varying and time-invariant effects is critical.
• AMI and heart failure process performance measures appear to serve as a proxy for lower hospital mortality rates at a point in time.
• However, process performance measures do not appear to be causally related to outcomes.

Current as of October 2007

Internet Citation:

Ryan, A. The Utility of Process Performance Measurement for Medicare Value Based Purchasing. Text Version of a Slide Presentation. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/fund/training/andryantxt.htm

Return to NRSA Trainees Research Conference Presentations
AHRQ Home Page