BIOCHANGE
2008 PILOT
Benchmarking
Pilot Study of
Change Measurement
Algorithms & CAD Tools
Organized by the
National
Report Pilot Results by May
16 July 16, 2008
Introduction
The development of biomedical imaging change analysis algorithms and computer assisted diagnosis (CAD) tools capable of reliably measuring small changes in volume of lesions found using CT or other imaging modalities would be of great value in assessing response to therapy and in clinical cancer studies. At a 2006 NIST Workshop Imaging as a Biomarker: Standards for Change Measurements in Therapy, a broad group of industry leaders identified the need for improved measurements of the physical quantities underlying biomedical imagery, for better protocols, and for reliable methods for assessing algorithm/CAD performance. NIST has experience in these areas.
With regard to benchmarking,
NIST’s Information Technology Laboratory has experience with performance evaluation
of software and algorithms for text retrieval, biometrics, motion image
quality, face recognition, speech, and text retrieval. The evaluation of
biomedical change analysis algorithms presents very real challenges in
developing suitable assessment methods. We expect the use of benchmarking to
contribute to algorithm improvement.
In comparison with previous
benchmarking work at NIST, the distinctive element of the current study is the
lack of available ground truth for much of the clinical imagery data. In the
present case, lung CT derived from the medical diagnosis of lung cancer and the
assessment of the response to therapy is available, but in many cases the size
and the change in size of lung nodules and lesions are not reliably known. The Reference Image Database to Evaluate
Response (or RIDER) White
Paper discusses approaches to estimating the uncertainty associated with
the measurement of change. In the Pilot study we will examine such methods and
their applicability to the algorithms and CAD tools submitted to Biochange
2008.
While the Pilot study
contains a small number of CT pairs, it is our expectation that even a small
data set will provide insight into measuring performance of multiple algorithms
and CAD tools for volume change in lung CT. Because we lack access to reliable
ground truth, the study will consider the effectiveness of various performance
measures and methodologies near the zero-change point. The study will include
cases with no clinical change, in order to estimate the local variance and bias
of each change detector. The Pilot should provide guidance in conducting future
larger studies.
Goals
Develop
methodology & performance measures to validate change analysis using
publicly available lung CT data.
Benchmarking elements
·
Participants
provide algorithms or CAD tools that measure change in lung nodules.
·
NIST gathers
pairs of CT scans from various sources: RIDER (Reference Image Database to
Evaluate Response) clinical data from NCI, phantom collections from FDA, and
possibly other clinical data. The Pilot includes pairs of scans with “No
Change” conditions.
·
With industry
participation, NIST defines evaluation methodology and provides code
samples for data access. The design is intended to assure low barriers to
entry. The Experiment Description is available in XML format and also as a
spreadsheet. The evaluation has two stages, Training and Validation
How to Participate in
Biochange 2008 Pilot
·
Download and read
the Biochange2008
Protocol and submit a Statement
of Interest to NIST.
·
Download the CT
scan data from the NIST FTP site and from the National Cancer Institute Archive
(NCIA).
·
Train your
algorithm or clinical CAD tool using the Training
Experiment Description and Training
Data.
·
Run your
change analysis algorithm or commercial CAD tool in your lab on the Validation
data set.
·
Report your
change results in one of the required formats and provide the signed
Participation Agreement to NIST by May 16 July 16, 2008. Optionally,
Participants may submit their Training stage data by April 21 for preliminary
analysis and discussion at the Lung Cancer Workshop.
·
NIST will analyze
the results as described in the Biochange2008
Project Plan. NIST will provide Participants with individual analysis of
their results. We will publish the results of the evaluation, without
identifying individual scores by Participant.
Contact:
Dr.
Charles Fenimore, National Institute of Standards and Technology (NIST)
fenimore@nist.gov, phone 301-975-2428.
Privacy Policy/Security
Notice/ Accessibility Statement
NIST is an agency of the
U.S. Department of Commerce
Site created on March 21, 2008
Last updated: Friday, May 1, 2008