Conference Program, Summary Papers, and Presentations

Item Response Theory (IRT) and Applications of IRT for Improving Health Outcomes Measurement

Instructor:
Steve P. Reise, PhD
Professor
UCLA Department of Psychology

This introductory workshop began with a discussion of the basics of item response theory (IRT) modeling, including discussions of defining model parameters, model assumptions, item and test characteristic and information curves, frequently-used IRT models, and differences between traditional and IRT methods. The course discussed the role of IRT modeling in health outcomes research of evaluating and developing questionnaires, identifying differential item functioning, linking instruments, developing item banks, and assessing change. Critical issues for applying this methodology in health outcomes research was discussed.

8:00 am

Welcome and Opening Remarks

Ron D. Hays, PhD
Professor, David Geffen School of Medicine at UCLA
Senior Scientist, RAND Corporation

8:20 am

Conference Charge

Joseph Lipscomb PhD
Branch Chief, Outcomes Research
National Cancer Institute

Eleanor M. Perfetto, PhD
President, Drug Information Association
The Weinberg Group, Inc.

8:40 am

The Science of Health Outcomes Measurement

Neil K. Aaronson, PhD
[Summary Paper (91 KB)] [Presentation (194 KB)]
Head, Division of Psychosocial Research & Epidemiology
The Netherlands Cancer Institute

This session reviewed the current state of the science in health outcomes measurement. What defines a "quality" or psychometrically-strong measure? What are the limitations of the instruments we use today? Are we satisfied with the analytical tools used to evaluate our measures?

9:20 am

The Traditional and Modern Approaches to Outcomes Measurement

Ronald K. Hambleton, PhD
[Summary Paper (132 KB)] [Presentation (339 KB)]
Distinguished University Professor
School of Education, University of Massachusetts at Amherst

This session defined and contrasted both the traditional approach (i.e., CTT) and the modern measurement (i.e., IRT) approach for data analysis and scoring. Also, it discussed the limitations and strengths of these methods in health outcomes measurement.

10:10 am

Refreshment Break

10:40 am

Building and Revising Outcome Measures: Evaluating Item and Scale Functioning with the IRT model

Bryce B. Reeve, PhD
[Summary Paper (208 KB)] [Presentation (244 KB)]
Psychometrician, Outcomes Research
National Cancer Institute

Maria Orlando, PhD
[Summary Paper (198 KB)]
Behavioral Scientist
RAND Corporation

This session discussed how IRT may be used to evaluate both the properties of the items and the scale and to inform questionnaire revisions. What added information do we learn about the data from IRT modeling?

11:40 am

Assessing Measurement Equivalence Across Populations: Differential Item Functioning (DIF)

Jeanne A. Teresi, EdD, PhD
[Summary Paper (230 KB)] [Presentation (134 KB)]
Senior Research Associate
Columbia University

Leo S. Morales, MD, PhD
[Summary Paper (364 KB)] [Presentation (1.3 MB)]
Assistant Professor, David Geffen School of Medicine at UCLA
Behavioral Scientist, RAND Corporation

John A. Fleishman, PhD
[Summary Paper (239 KB)] [Presentation (702 KB)]
Senior Social Scientist, Center for Financing Access and Cost Trends
Agency for Healthcare Research and Quality (AHRQ)

This session discussed the importance for evaluating DIF for determining measurement equivalence of item content across instruments administered to different populations (differing by race, age, gender, etc.) or instruments administered in different languages. How will DIF assessment benefit health disparities research?

1:00 PM

Luncheon and Review Posters

2:20 PM

Comparing or Combining Scores from Multiple Instruments: Instrument Linking (Equating)

Neil J. Dorans, PhD
[Summary Paper (130 KB)] [Presentation (92 KB)]
Principal Measurement Statistician in the Research & Development Division
Educational Testing Service

This presentation addressed several issues associated with the linking of health outcomes. What is meant by outcomes linking and equating? How does equating differ from other types of linking? What are common data collection designs used to capture data for outcomes linking. What are some of the standard statistical procedures used to link outcomes directly? What assumptions do they make? What role does IRT play in linking outcomes? What assumptions do IRT methods make? Illustrations of different kinds of linkages were provided.

3:00 PM

Developing Tailored Instruments: Item Banking and Computerized Adaptive Assessment

Chih-Hung Chang, PhD
[Summary Paper (181 KB)] [Presentation (348 KB)]
Assistant Professor
Northwestern University

Jakob B. Bjorner, MD, PhD
[Summary Paper (406 KB)] [Presentation (136 KB)]
Deputy Chief Science Officer
QualityMetric Incorporated

David Thissen, PhD
[Summary Paper (170 KB)] [Presentation (184 KB)]
Professor, Quantitative Program, Department of Psychology
University of North Carolina at Chapel Hill

This session discussed the critical role that IRT plays to develop the item bank and why CTT methods are inappropriate for such a task. What challenges/decisions will we have to make in choosing both the appropriate IRT model and the constructs we want to measure? How can we balance the assumptions of the unidimensional IRT models with the multifaceted nature of the constructs measured in health outcomes research?

4:45 PM

Reception and Poster Session

8:00 am

The IRT Data Analysis Project: In parallel sessions, two research teams demonstrated the strengths and challenges of applying IRT modeling for evaluating data in health outcomes research and behavioral science. The sessions provided detailed examples of the logic and flow of analyses, the tackling of technical problems, and the interpretation of results. Examples of input and output files along with analysis summaries were provided to the audience.

10:00 am

Refreshment Break

10:30 am

The Future of Outcomes Measurement: Item Banking, Tailored Short-Forms, and Computerized-Adaptive Assessment

David Cella, PhD
[Summary Paper (235 KB)] [Presentation (1.2 MB)]
Professor of Psychiatry and Behavioral Science, Northwestern University Medical School
Director, Center on Outcomes Research and Education

John E. Ware, PhD
[Presentation (1.06 MB)]
Chief Executive Officer and Chief Science Officer
QualityMetric Incorporated

This session discussed how computerized-adaptive testing (CAT) will revolutionize the way we measure outcomes through the development of tailored short forms and computerized-adaptive tests. The speakers presented results from their own work and discussed how item banking (short forms and CATs) will benefit the measurement of patient-reported outcomes in observational studies, clinical trials, and clinical practice.

12:00 PM

Luncheon

1:15 PM

Critical Issues for Developing and Maintaining Item Banks and CATs
Bringing the Concept into Reality: The Idea of a National Item Bank

Colleen A. McHorney, PhD
[Presentation (1.8 MB)]
Professor, Indiana University School of Medicine
Senior Scientist, Regenstrief Institute, Inc.

Jeff A. Sloan, PhD
[Presentation (1.7 MB)]
Lead Statistician and Chair of Quality of Life Research Committee
Mayo Clinic, Rochester

Dennis A. Revicki, PhD
[Summary Paper (80 KB)] [Presentation (30 KB)]
Director - Center for Health Outcomes Research & Sr. Research Leader, MEDTAP Int.
Adjunct Professor, Department of Psychiatry, Georgetown University Medical School.

Lawrence J. Fine, MD, DrPH
[Presentation (184 KB)]
Medical Advisor, Office of Behavioral and Social Sciences Research
National Institutes of Health

Robert T. O'Neill, PhD
[Summary Paper (198 KB)] [Presentation (291 KB)]
Director, Office of Biometrics,
Center for Drug Evaluation and Research, FDA

Laurie B. Burke, RPh, MPH
[Presentation (636 KB)]
Director, Study Endpoints and Label Development
Office of New Drugs, Center for Drug Evaluation and Research, FDA

This session discussed the feasibility for developing and implementing item banks and CATs in observational and clinical settings. What critical issues should we address and how can we overcome the barriers to issues like proprietary rights? What are the costs and amount of work that are required to create these products? What role should government (e.g., NIH, FDA) and private entities play in this effort?

3:15 PM

Refreshment Break

3:30 PM

Next Steps in Health Outcomes and Behavioral Science

Ron D. Hays, PhD
[Summary Paper (101 KB)] [Presentation (25 KB)]
Professor, David Geffen School of Medicine at UCLA
Senior Scientist, RAND Corporation

Margaret Rothman, PhD
[Presentation (15 KB)]
Executive Director, Health Economics
Johnson & Johnson

Joseph Lipscomb PhD
[Summary Paper (167 KB)] [Presentation (67 KB)]
Branch Chief, Outcomes Research
National Cancer Institute

Peter M. Fayers, PhD
[Summary Paper (120 KB)] [Presentation (82 KB)]
Professor of Medical Statistics
University of Aberdeen Medical School

For both health outcomes and behavioral researchers, speakers made recommendations for how their field of study will benefit by incorporating the methods discussed over the past two days. What demonstrations are necessary to explore the potential benefits of item banking and CATs to health outcomes measurement? What role should academia, industry, and government play to push this arena forward?