Slide Presentation from the AHRQ 2008 Annual Conference
On September 9, 2008, John D. Seeger, made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (610 KB).
Slide 1
Transparency in the Use of Propensity Score Methods
John D. Seeger, PharmD, DrPH
Chief Scientist, i3 Drug Safety
Adjunct Assistant Professor, Harvard School of Public Health
September 9, 2008
With thanks to: Alec Walker, Tobias Kurth, Jeanne Loughlin, Mona Eng, and Alex Cole
Slide 2
Propensity Score Analysis—When?
- Strategies to control for confounding in non-experimental pharmacoepidemiology:
- Measured confounders**
- Design**
- Analysis**
- Standardization
- Stratification
- Multivariate regression
- Unmeasured confounders*
- Unmeasured but measurable in a validation study:
- Two-stage sampling
- External adjustment
- Unmeasurable:
- Design
- Crossover designs
- Active comparison group (restriction)
- Analysis:
- Instrumental variables
- Sensitivity analysis
- Note: *Thanks to S. Schneeweiss
**Amenable to Propensity Techniques
Slide 3
Motivation
- Assume matching when comparing 2 treatments:
- For every drug user with given characteristics
- Find a comparator with identical characteristics
- Example: Male, age 45, smoker, with hypertension (HTN)...
- Matching fails:
- Age (10 categories) x
- Sex (2 categories) x
- Prior diagnoses (5 @ 2 categories each)
- Prior drug therapy (5 @ 2 categories each)
- Preceding cost of care (5 categories)
- Leads to 102,400 potential matching groups.
Slide 4
Propensity Score Collapses Exposure Predictors
The slide shows a line graph which presents the "Hypothetical Distribution of Propensity Scores." The vertical axis, number of people, goes from 0 to 1400 and the horizontal axis, propensity score, goes from 0 to 1. The results show "Initiators" beginning at 0, reaching a maximum of 400 people at 0.65, and ending at less than 100 people at 1. The results show "Non-Initiators" beginning at 300, reaching a maximum of 1199 people at 0.35, and ending at 0.
- Single value:
- Probability: subject will receive therapy vs comparator
- Removes confounding by components of the score
- Patient characteristics that favor one therapy over another
- Permits:
- Restriction
- Matching
- Stratification
- Modeling
- Weighting
Slide 5
Should Propensity Scores Always be Used?
The slide shows a line graph which presents "Logistic Regression."
- Note: Figure 1. Median percentage of bias with the logistic regression, by strength of the exposure and number of events per confounder. In the logistic regression, the bias declines as the number of events per confounder increases. Values greater than zero indicate an overestimation of the effect of the exposure on the outcome. Negative values indicate an underestimation of the effect of the exposure on the outcome.
- Note: Cepeda S, et al. Am J Epidemiol 2003;158:280-7.
- More than 8 events per covariate leads to unbiased estimates
- So propensity score favored when:
- Many more persons exposed to drug of interest than study outcomes.
- Common exposure
- Rare outcome
- Allows for richer model (more predictors) of exposure than outcome.
Slide 6
Estimate Propensity Score
- Predict treatment from baseline covariates within database
- Inclusion of predictors:
- A priori (what characteristics are used to prescribe?)
- Empiric (what differentiates initiators?)
- Generic (what patterns of healthcare predict initiation?)
- Coefficients of propensity score:
- Interpretable and Informative
Slide 7
Propensity Score Restriction
The slide shows a bell curve graph presenting the results for "Exposed subjects" and "Unexposed subjects."
- Note: *In this example subjects with low propensity scores are never exposed while subjects with high propensity scores are always exposed.
- Note: Sturmer T, et al. J Clin Epidemiol 2006;59:437-47.
Slide 8
Propensity Score Restriction
- Potential for serious adverse events from error (name confusion):
- Amaryl (glimepiride an oral hypoglycemic)
- Reminyl (galantamine for Alzheimer's disease)
- 36,816 people with AD diagnosis (14,626 Reminyl dispensings):
- 236 Amaryl recipients
- 24 Amaryl recipients in the lowest decile of the propensity score
- 13 with a single dispensing of Amaryl or no diabetes diagnoses
- 2 with no diabetes-related claims across entire claim history
- Medical record review suggested no error
- Propensity score restriction may be used as a screening method to identify unusual patterns of healthcare for closer scrutiny:
- Possible medication dispensing errors
- Others
- Confirmation requires additional data, which could be obtained through medical record review.
Slide 9
Propensity Score Distribution and Strata
The line graph presents the propensity scores on the x axis and percent on the y axis. There are two lines representing "Zopiclone" and "Temazepam."
- C-statistic equals 0.739.
Slide 10
Effect of Temazepam Relative to Zopiclone
The table presents statistical data of "Relative Risk" and "Upper and Lower 95th Percentile." The data shown is for Quintiles 1-5 and totals adjusted for propensity score and continuous propensity score.
- Transparent analysis:
- Within-stratum balance
- Stratum-specific effect estimates as well as pooled estimate
- Explicit evaluation of potential for effect measure modification
Slide 11
Matching on the Propensity Score
- Matching can be performed by:
- Standard automated case-control matching programs where the matching range is specified
- Nearest available match based on the propensity score
- Greedy matching techniques (http://www2.sas.com.proceedings/sugi26/p214-26.pdf)
Slide 12
Distribution of Propensity Score
The line graph presents propensity score on the x axis and number of persons on the y axis. The categories are "Statin Initiators" and "Non-Initiators." The results show a consistent trend for both categories, with a low propensity score for higher numbers of persons and exponentially higher propensity score as the number of persons decrease.
Slide 13
Propensity Score Distribution (After Matching)
The line graph presents propensity score on the x axis and number of persons on the y axis. The categories are "Statin Initiators" and "Non-Initiators." The data for both categories shows a sharp increase in number of persons in the lower propensity scores, between 0 and 0.05. Beyond propensity scores of 0.05, the number of persons decreases as the propensity scores increase, approaching 0 beyond a propensity score of 0.7.
Slide 14
Characteristics Before Matching
The table presents the results for "Initiators: N=4144," "Non-Initiators: N=4144," and "P-value" for various "Variables" such as Lipid-Related Labs, Different Prescription Drugs, Low-density lipoprotein (LDL) Level, Angina, Smoking, Hypertension, etc.
Slide 15
Balance Achieved by Matching
The table presents the results for "Initiators: N=2901," "Non-Initiators: N=2901," and "P-value" for various "Variables" such as Lipid-Related Labs, Different Prescription Drugs, LDL Level, Angina, Smoking, Hypertension, etc.
- Note: Matched at 0.01 Propensity Score
Slide 16
Analysis by 2X2 Table
The slide presents a "Table of mi by statin."
Slide 17
MI Outcome (After Matching)
The line graph presents months of follow up on the x axis and cumulative incidence on the y axis. The two data sets are "Statin Initiators" and "Statin Non-Initiators." Both data lines begin at approximately, 0, 0. Both show a somewhat linear increase in cumulative incidence as months of follow-up increase. The trend is stronger in the "Statin Non-Initiators" data.
- Note: HR=0.69 (0.52-0.93)
- Note: 31% (7%-48%) Risk Reduction
Slide 18
Regression Adjustment with Propensity Scores
- Regression adjustment:
- Note: An image of the equation is portrayed.
- All study participants are used
- Still a two-step approach (exposure and outcome)
- More power compared to including all covariates into the model, since degrees of freedom are gained
- However, assumes the underlying association between the score and the outcome is modeled appropriately
Slide 19
Weighting
The slide presents three separate line graphs showing the distribution of propensity score on the x axis relative to the number of people on the y axis. The lower two charts are entitled "IPTW" and "SMR" and both show nearly identical distributions of data for "Initiators" and "Non-Initiators". The "IPTW" chart shows a bell curve distribution of with a peak in number of people, approximately 1300, at a propensity score of approximately 0.45. The "SMR" chart shows a bell curve distribution of with a peak in number of people, approximately 400, at a propensity score of approximately 0.65.
Slide 20
Baseline Characteristics
Table 1. Variables included in the propensity scores, LABA ans ICS cohorts
The table presents statistical data of "LABA cohort" and "ICS cohort." The data is shown for various age categories, genders, geographic regions of health plan, Asthma-related drug dispensings (0-2 days), Asthma-related drug dispensings (3-365), and Asthma-related physician visits.
| Categories |
LABA Cohort
(18,596 patients) |
ICS Cohort
(30,520 patients) |
| n |
% |
n |
% |
| Age category, years |
10-19 |
3,757 |
20.20 |
6,543 |
21.44 |
| 20-29 |
1,559 |
8.38 |
2,225 |
7.29 |
| 30-39 |
3,237 |
17.41 |
4,965 |
16.27 |
| 40-49 |
4,341 |
23.34 |
6,631 |
21.73 |
| 50-64 |
4,550 |
24.47 |
7,626 |
24.99 |
| 65+ |
1,152 |
6.19 |
2,530 |
8.29 |
| Gender |
Male |
7,256 |
39.02 |
12,804 |
41.95 |
| Female |
11,340 |
60.98 |
17,716 |
58.05 |
| Geographic region of health plan |
Northeast |
2,176 |
11.70 |
3,714 |
12.17 |
| South/Southeast |
7,273 |
39.11 |
10,893 |
35.69 |
| Midwest |
6,502 |
34.96 |
12,065 |
39.53 |
| West |
2,616 |
14.07 |
3,790 |
12.42 |
| Asthma-related drug dispensing (0-2 days) |
Short-acting beta-agonists |
4,866 |
26.17 |
10,344 |
33.89 |
| Oral steroids |
1,998 |
10.74 |
1,217 |
3.99 |
| Injectible steroids |
322 |
1.73 |
180 |
0.59 |
| Leukotriene modifiers |
2,151 |
11.57 |
2,472 |
8.10 |
| Mast cell stabilizers |
55 |
0.30 |
211 |
0.69 |
| Xanthines |
176 |
0.95 |
353 |
1.16 |
| Omalizumab |
1 |
0.01 |
1 |
0.00 |
| Asthma-related drug dispensing (3-365) |
Short-acting beta-agonists |
13,287 |
71.45 |
21,753 |
71.27 |
| Oral steroids |
7,104 |
38.20 |
9,231 |
30.25 |
| Injectible steroids |
2,278 |
12.25 |
3,015 |
9.88 |
| Leukotriene modifiers |
4,958 |
26.66 |
6,580 |
21.56 |
| Mast cell stabilizers |
391 |
2.10 |
788 |
2.58 |
| Xanthines |
679 |
3.65 |
1,117 |
3.66 |
| Omalizumab |
2 |
0.01 |
2 |
0.01 |
| Asthma-related physician visits |
0 |
7,445 |
40.04 |
16,676 |
54.64 |
| 1 |
4,973 |
26.74 |
7,178 |
23.52 |
| 2+ |
6,178 |
33.22 |
6,666 |
21.84 |
Slide 21
Cohort Results
Incidence rates (per 1000) and hazard ratios
The table presents statistical data of "Events," "Person Years," "Incidence Rate," and "Hazard Ratio" for the following categories:
- All-cause Mortality
- Asthma-related emergency room visits
- Asthma-related hospitalizations
- Intubations
| Cohort |
Person-Incidence |
Hazard Ratio |
| Events |
Years |
Rate |
95% CI |
Adjusted* |
95% CI |
| |
All-cause Mortality |
|
| ICS Cohort |
171 |
35,886 |
4.77 |
4.18-5.41 |
1.00 |
Ref |
| LABA Cohort |
LABA Cohort |
93 |
25,801 |
3.60 |
3.01-4.28 |
0.69 |
0.50-0.95 |
| Formoterol |
4 |
924 |
4.33 |
1.48-9.88 |
0.78 |
0.28-2.14 |
| Salmeterol |
37 |
8,448 |
4.38 |
3.27-5.76 |
0.84 |
0.56-1.26 |
| Salmeterol/Fluticasone |
52 |
16,407 |
3.17 |
2.48-3.99 |
0.61 |
0.42-0.88 |
| |
Asthma-related emergency room visits |
|
| ICS Cohort |
723 |
35,339 |
20.46 |
19.24-21.74 |
1.00 |
Ref |
| LABA Cohort |
LABA Cohort |
710 |
25,002 |
28.40 |
26.69-30.19 |
1.24 |
1.09-1.42 |
| Formoterol |
30 |
892 |
33.63 |
24.31-45.34 |
1.39 |
0.96-2.02 |
| Salmeterol |
247 |
8,151 |
30.30 |
27.24-33.61 |
1.33 |
1.13-1.57 |
| Salmeterol/Fluticasone |
433 |
15,937 |
27.17 |
25.08-29.38 |
1.19 |
1.03-1.38 |
| |
Asthma-related hospitalizations |
|
| ICS Cohort |
154 |
35,818 |
4.30 |
3.75-4.91 |
1.00 |
Ref |
| LABA Cohort |
LABA Cohort |
254 |
25,578 |
9.93 |
8.93-11.01 |
1.76 |
1.36-2.27 |
| Formoterol |
10 |
913 |
10.95 |
5.95-18.50 |
1.88 |
0.97-3.64 |
| Salmeterol |
95 |
8,354 |
11.37 |
9.53-13.47 |
2.01 |
1.48-2.72 |
| Salmeterol/Fluticasone |
149 |
16,290 |
9.15 |
7.95-10.47 |
1.62 |
1.23-2.14 |
| |
Intubations |
|
| ICS Cohort |
86 |
35,852 |
2.40 |
1.99-2.87 |
1.00 |
Ref |
| LABA Cohort |
LABA Cohort |
74 |
25,762 |
2.87 |
2.35-3.48 |
1.03 |
0.69-1.52 |
| Formoterol |
4 |
922 |
4.34 |
1.48-9.90 |
1.46 |
0.52-4.10 |
| Salmeterol |
33 |
8,430 |
3.91 |
2.87-5.23 |
1.40 |
0.88-2.25 |
| Salmeterol/Fluticasone |
37 |
16,388 |
2.26 |
1.68-2.97 |
0.81 |
0.51-1.27 |
Slide 22
Are Divergent Results Possible?
The table presents statistical data for "No.," "OR*," and "95% Cl*" for the following categories:
- Crude model
- Multivariable model
- Matched on propensity score
- Regression adjusted with propensity score
- Propensity score, continuous
- Multivariable
- Propensity score, deciles
- Multivariable
- Weighted models
- IPTW*
- SMR* weighted
- Note: Kurth T, et al. Am J Epidemiol 2006;163:262-70.
| |
No. |
OR* |
95% CI* |
| Crude model |
6,269 |
3.35 |
2.28, 4.91 |
| Multivariable model† |
6,269 |
1.93 |
1.22, 3.06 |
| Matched on propensity score |
406 |
1.17 |
0.68, 2.00 |
| Regression adjusted with propensity score |
Propensity score, continuous |
6,269 |
1.53 |
0.95, 2.48 |
| Multivariable† |
6,269 |
1.85 |
1.13, 3.03 |
| Propensity score, deciles |
6,269 |
1.76 |
1.13, 2.72 |
| Multivariable† |
6,269 |
1.96 |
1.20, 3.20 |
| Regression adjusted with propensity score |
IPTW* |
6,269 |
10.77 |
2.47, 47.04 |
| SMR* weighted |
6,269 |
1.11 |
0.67, 1.84 |
Slide 23
What About Unmeasured Confounding?
- Obesity, Smoking, Exercise
Pharmacoepidemiology and Drug Safety (2008)
Published online in Wiley InterScience (www.interscience.wiley.com)
DOI: 10.1002/pds.1554
Original Report
Supplementary data collection with case-cohort analysis to address potential confounding in a cohort study of thromboembolism in oral contraceptive initiators matched on claims-based propensity scores.
P. Mona Eng ScD1*, John D. Seeger PharmD, DrPH1,2, Jeanne Loughlin MS1*, C. Robin Clifford MS1*, Sherry Mentor BA1* and Alexander M. Walker MD, DrPH1,2
1Ingenix, i3 Drug Safety, Waltham, MA, USA
2Harvard School of Public Health, Boston, MA, USA
Slide 24
Accounting for Variables had Little Effect
Table 3. Relative risks of thromboembolism comparing ethinyl estradiol/drospirenone (EE/DRSP) and other oral contraceptive (OC) initiators from case-cohort and propensity score matched cohort analyses, Ingenix Research Data Mart 2001-2004, United States.
| |
Relative risk |
95% CI* |
| Case cohort analysis |
Risk ratio |
|
| Univariate analysis |
EE/DRSP |
0.92 |
0.50, 1.69 |
| Other OC |
1 |
|
| Multivariate analysis |
EE/DRSP |
0.90 |
0.49, 1.68 |
| Other OC |
1 |
|
| Propensity score matched |
Rate ratio |
|
| Cohort analysis |
EE/DRSP |
0.92 |
0.50, 1.63 |
| Other OC |
1 |
|
Slide 25
Conclusion
- Propensity score can be useful for addressing confounding (by indication).
- Allows for rich model of exposure to be developed.
- Advantageous when number of people with a study outcome is small relative to number of exposed persons and number of potential confounders is large.
- Drug effects (particularly adverse ones)
- Consider transparency:
- When selecting propensity score
- When building propensity score
- When using propensity score
Slide 26
Thank You
John.Seeger@i3DrugSafety.com
Current as of January 2009
Internet Citation:
Transparency in the Use of Propensity Score Methods. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version). January 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualmtg08/090908slides/Seeger.htm
540 Gaither Road Rockville, MD 20850