Joint Fellowship Training Program
Mentor:
Brandon D. Gallas, Ph.D.
Organizational Affiliation and Position:
Mathematician, Division of Imaging and Applied Mathematics (DIAM), Office of Science and Engineering Laboratories (OSEL), Center for Devices and Radiological Health (CDRH), FDA
Email:
brandon.gallas@fda.hhs.gov
Phone:
301-443-5020 ext 144
Running Title of Program:
Understanding and Evaluating Reader Variability and Reader Agreement
Research Project Summary:
The interpretation of medical images by doctors (readers) is a highly complex process with the potential for a large amount of variability from reader to reader. The performance of the reader is evaluated in clinical studies that ask a set of readers to score a set of images (cases) as either normal or diseased (the binary task), or to score the cases on a range from normal to diseased (the ROC task, receiver operating characteristic). While the mean performance is the summary statistic of interest, it is meaningless without knowing the uncertainty of the statistic. Therefore, we design analyses for multi-reader, multi-case (MRMC) clinical study designs. An MRMC analysis estimates the key components of variance: a case component, a reader component, and an interaction between the two.
The analyses we design must accommodate the broad range of clinical study designs and supporting truth that appear in submissions to the FDA. One common study design is the fully-crossed study design, where every reader reads every case; another is the doctor-patient study design, where each doctor reads their own cases. Both kinds of studies have appeared in submissions to the FDA, as well as hybrid designs that fall somewhere in between. Regarding truth, when the true patient status is unknown, an analysis of reader agreement can be performed. While these analyses cannot measure performance on an absolute scale, they can uncover the key component of variance that can be used to scale pivotal studies to measure performance.
Proposed project for IOTF fellow:
The fellow will analyze data from a sub-study of the Atypical Squamous Cells of Undetermined Significance (ASCUS) Low-Grade Squamous Intraepithelial Lesion (LSIL) Triage Study (ALTS), in which twenty-one readers evaluated images of the cervix of enrolled participants. Readers identified and drew boundaries around any acetowhite lesions. For each lesion, they scored key features of each lesion: color, margins, the vascular features of punctuation and mosaicism. The reader was also asked to provide a global diagnosis for the whole cervix based on the worst area evaluated.
The dataset is large (939 women) and has two unique pieces. The first piece, or "pilot study" dataset, contains the scores of 21 readers reading the same 20 images (fully-crossed data). The second piece, or "pivotal study" dataset, contains the scores for the remaining cases randomly distributed among the evaluators such that each evaluator had a set of 112 cases and all cases were read by at least two evaluators.
The Fellow will employ MRMC analysis methods already derived and collaborate with the mentor on the design of new methods. The Fellow will design a set of hypotheses for the pivotal study dataset given the pilot study dataset. One goal of this work is to formalize the process of sizing future clinical MRMC trials. Another goal is to understand the perception of the key lesion features, and translate the understanding into protocols to standardize and improve colposcopy.
Other datasets will also be provided to the Fellow with the goal of being to advance the field of MRMC analysis. These include datasets for cancer detection in the breast, chest, and colon. Additionally, the Fellow will be trained on how to use Monte Carlo methods (simulation of datasets) to validate and probe the statistical analyses.
Regulatory Activity:
Training courses will be offered to the Fellow on the premarket and postmarket functions of CDRH, the importance of risk management in evaluating the safety of new medical products, how to conduct meetings, technical writing, and writing for sponsors. The Fellow will gain experience with the device approval process used in CDRH through exposure to actual submissions of imaging devices, computer-aided diagnostic (CAD) devices, and others. The fellow will assist in all aspects: the planning of trials, the review process, meeting with sponsors, writing letters to sponsors, and preparing/attending panel meetings.
Reference(s):
B. D. Gallas, "One-shot estimate of MRMC Variance: AUC." Acad Radiol, 13(3), 353-362 (2006).
