January 8, 1997 Meeting Summary
Table of Contents
I. Record of Attendance
II. Welcome and Announcements
III. Topics
CLIA Update
Centers for Disease Control and Prevention
Health Care Financing Administration
Cytology Proficiency Testing
Status of CLIA Research Studies
CLIA Research Update
Calculation of Bias and Precision Tolerance Limits Using the Variation of Test
Distributions Association Between HIV Testing Accuracy and the Use of external
Quality Control Samples Laboratory Medicine Sentinel Monitoring Network
IV. Public Comments
V. Concluding Remarks
VI. The Addenda
Record of Attendance
Committee Members Ex Officio Members
Dr. David Baines Dr. Kaiser Aziz, FDA
Dr. Regina Benjamin Dr. Carlyn Collins, CDC
Dr. Thomas Bonfiglio Ms. Judith Yost, HCFA
Dr. Lemuel Bowie
Dr. Mary Burritt
Dr. Ronald Cada Liaison Representatives
Dr. Patricia Charache Dr. Fred Lasky (HIMA)
Dr. Susanne Gollin
Dr. Verlin Janzen
Dr. Aliza Lifshitz
Dr. Bereneice Madison
Ms. Diana Mass
Ms. Deborah McHugh
Dr. Glenda Price
Ms. Sharon Radford
Mr. Elliott Segal
Centers for Disease Control and Prevention
Dr. Rex Astles Mr. Darshan Singh
Ms. Rosemary Bakes-Martin Mr. Gregory Smothers
Ms. Carol Bigelow Dr. Steve Steindel
Dr. Joe Boone Dr. Roger Taylor
Ms. Gail Bosley Ms. Julie Wasil
Ms. Diane Bosse Ms. Rhonda Whalen
Ms. Cheryl Coble Dr. Laurina Williams
Ms. Deborah Coker
Ms. Crystal Frazier
Ms. MariBeth Gagnon
Ms. Sharon Granade
Dr. Thomas Hearn
Dr. Ed Holmes
Dr. John Krolak
Dr. Harvey Lipman
Dr. James Lange
Dr. Adam Manasterski
Ms. Anne O'Connor
Dr. John Ridderhof
Dr. Eunice Rosner
Dr. Sharahan Shahangian
Ms. Marianne Simon
Clinical Laboratory Improvement Advisory Committee
The Secretary of Health and Human Services is authorized under Section
353 of the Public Health Service Act, as amended, to establish standards
to assure consistent, accurate, and reliable test results by all clinical
laboratories in the United States. The Secretary is authorized under
Section 222 to establish advisory committees.
The Clinical Laboratory Improvement Advisory Committee (CLIAC) was
chartered in February 1992 to provide scientific and technical advice and
guidance to the Secretary and the Assistant Secretary for Health
regarding the need for, and the nature of, revisions to the standards
under which clinical laboratories are regulated; the impact on medical
and laboratory practice of proposed revisions to the standards; and the
modification of the standards to accommodate technological advances.
The Committee consists of 20 members, including the Chair. Members are
selected by the Secretary from authorities knowledgeable in the fields of
microbiology, immunology, chemistry, hematology, pathology, and
representatives of medical technology, public health, clinical practice,
and consumers. In addition, CLIAC includes three ex officio members, or
designees: the Director, Centers for Disease Control and Prevention; the
Commissioner, Food and Drug Administration; the Administrator, Health
Care Financing Administration; and such additional officers of the U.S.
Government that the Secretary deems are necessary for the Committee
to effectively carry out its functions. CLIAC will also include a non-voting
liaison representative who is a member of the Health Industry
Manufacturers Association and such other non-voting liaison
representatives that the Secretary deems are necessary for the
Committee to effectively carry out its functions.
Due to the diversity of its membership, CLIAC is at times divided in the
guidance and advice it offers to the Secretary. Even when all CLIAC
members agree on a specific recommendation, the Secretary may not
follow their advice due to other overriding concerns. Thus, while some of
the actions recommended by CLIAC may eventually result in changes to
the law, the reader should not infer that all of the advisory committee's
recommendations will be automatically accepted and acted upon by
the Secretary.
Welcome and Announcements
The meeting was called to order by CLIAC Chairman Dr. Morton Schwartz. The
committee members made self-introductions and disclosure statements of their
relevant financial interests as they relate to the topics to be discussed during
the CLIAC meeting.
Note: An orientation for new CLIAC members was held prior to the meeting (See
Addendum A).
Presentations and Committee Discussion
CLINICAL LABORATORY IMPROVEMENT Addenda B-C
AMENDMENTS (CLIA) UPDATE/CDC
Dr. Carlyn Collins, Director of the Division of Laboratory Systems, CDC,
discussed CLIAC's potential involvement in updating the CLIA regulations
applicable to genetics testing. Dr. Collins is an informal participant
on the National Institutes of Health Task Force on Genetic Testing that
is examining the ethical, legal and social implications of genetics testing.
Dr. Collins anticipates that the task force will recommend the establishment
of a CLIAC subcommittee to make recommendations to revise the CLIA regulations
to ensure the quality of genetics testing. She indicated that CDC is working
to establish a genetics subcommittee which will include CLIAC members and
others with expertise in genetics testing.
Then, Dr. Collins briefly described the regulations development and publication
process and provided an update on the status of various CLIA regulations. At
this time, there are no projected publication dates for the regulation extending
the date for the phase in provisions pertaining to quality control and personnel,
the CLIA fee schedule revision notice, and the final waiver regulation (see
addendum B). Dr. Collins noted that initial work has begun to develop the final,
final regulation which will incorporate changes in proficiency testing (PT) and
quality control (QC).
Rosemary Bakes-Martin briefly reviewed issues presented at the public QC
meeting in Atlanta in September 1996. Many CLIAC members attended this
meeting and heard presentations by representatives from manufacturers and
other interested parties. Ms. Bakes-Martin reviewed CDC's approach for revising
the CLIA QC requirements in the final, final regulation to address new
technology (see addendum C). Ms. Bakes-Martin reminded the Committee that
the QC phase in requirements expired in September 1996. She noted said that
the main QC issue to be resolved prior to publication of the final, final
regulation is alternative methods for performing QC.
Committee Discussion
Dr. Schwartz asked if the Health Care Financing Administration (HCFA) inspectors
are aware that the phase in requirements expired in September. Ms. Yost
replied that the inspectors are aware but are not enforcing the requirement for
high complexity laboratory directors with a doctoral degree to be board
certified by September 1, 1996, and are continuing to allow laboratories using
commercial, non-modified, moderate complexity tests to meet less stringent QC
requirements. One committee member asked for clarification of QC changes
planned for the final, final regulation. Ms. Bakes-Martin explained that the QC
model (discussed at previous CLIAC meetings) would allow laboratories to use
non-traditional controls to assess the potential for error in the test system
with each run and use traditional controls to monitor all components of the
testing process (test system, environment, and operator) over time. This
flexibility would make the regulations more appropriate for "new
technology" test systems that have internal controls which check for
errors in the test system. Another committee member asked if the CDC would
provide a summary of presentations and comments from the September QC meeting.
Dr. Collins explained that there is no meeting summary, but copies of each
presentation are available. QC information provided at the meeting will be
used to develop the final, final regulations.
CLIA UPDATE/HCFA Addenda D-I
Ms. Judy Yost, of the HCFA, presented a brief status report on CLIA
implementation and referred to a summary of the types of deficiencies cited
during first and second cycle inspections of all laboratories vs. inspections of
physician office laboratories (POLs) (see addendum D). Noting that the majority
of laboratories surveyed by HCFA are POLs, Ms. Yost summarized the number of
POLs by CLIA application type and the number of application type 1 POLs
(laboratories that hold a regular certificate) by certificate fee schedule based
on volume and scope of services (see addendum E).
Ms. Yost then reviewed the process for surveying cytology laboratories (see
addendum F). Since the late 1980s, HCFA has contracted with the American
Society of Cytotechnologists (ASCT) to inspect these laboratories using
cytotechnologists and pathologists. The contractor uses the HCFA survey forms
and guidelines to evaluate compliance with the CLIA regulations. An ASCT
survey may be initiated at the request of a State agency or HCFA regional office
or as a result of a complaint. Ms. Yost described the laboratory selection by
HCFA, noting that HCFA follows up on all laboratory complaints. She reviewed
the contractor survey process, emphasizing that the surveys are outcome-
oriented with an educational focus. At a minimum, 0.1% of the laboratory's
volume or 100 slides are rescreened by cytotechnologists, with discrepancies
reviewed by a pathologist. The rescreened slides include negative and positive
GYN and non-GYN smears. If a problem is suggested by the rescreen results,
the sample would be expanded to determine whether the laboratory is
correctly reporting cytology results. HCFA reviews personnel qualifications,
follows up on correction of deficiencies, and takes any necessary enforcement
action.
Ms. Yost noted that there are about 3,000 cytology laboratories (most of these
are accredited). Then, she summarized the results from 109 cytology laboratories
surveyed onsite by the contractor in 1995/96 (see addendum G). Included in
the 109 were 40 laboratories in which complaints had been filed. Survey
findings substantiated complaints in 13 of the 40 laboratories, and HCFA
initiated enforcement actions against 4 laboratories. The other laboratories
either have corrected, or are in the process of correcting, identified
deficiencies.
Ms. Yost discussed the most frequently cited standard and condition level
deficiencies in cytology laboratories in FY 95-96 (see addendum H). Noting that
condition level deficiencies impact patient outcomes more than standard level
deficiencies, Ms. Yost reported that 41 of 109 laboratories (38%) surveyed in
1995/96 had condition level deficiencies, and cytology QC is the most
frequently cited condition level deficiency. When compared to the three
previous years, the actual number of laboratories with condition level
deficiencies has increased, while the percent has decreased (see addendum I).
In conclusion, Ms. Yost noted that the contractor cytology survey process has
become more streamlined and has been educational for the laboratories and
for HCFA.
Committee Discussion
The Chairman expressed concern that determining that a slide had been
misread may be subjective, depending on the surveyor. Another committee
member commented that there is no standard cytology accuracy rate against
which a laboratory may compare itself. The Chairman referred to Q-Probes
1993: Pap Smear Rescreening Data Analysis and Critique which reported a false
negative rate of 5.6% for focused rescreens, "using a narrow definition, and
12.5%, using a broad definition." The report also concluded that "poorer
performance was associated with higher Pap smear volume, higher number of
cytotechnologist FTEs, and high ratios of Pap smears per cytotechnologist and
per cytopathologist." One committee member asked for clarification on
focused rescreening. Several committee members expressed concern about
the cytology laboratory image problem and one member commented that the
format HCFA uses to report survey results may magnify deficiencies and
contribute to the image problem. Another committee member was concerned
that sample selection for rescreening might skew the statistics. Other committee
members indicated that there is much public interest in the automated cytology
systems and one member asked if HCFA's statistics included any laboratories
that use automated systems. Ms. Yost noted that the incidence of cervical
cancer had not declined, but pointed out that the laboratory detection of
abnormal conditions has improved. She said that one reason for increased
detection is that focused rescreens are performed in cytology, as required by
the CLIA law. Ms. Yost said that determining that a slide had been misread is
not based on the opinion of a single cytotechnologist on the survey team, but is
determined by a cytotechnologist and a pathologist. She also explained that
although HCFA previously cited a deficiency by referencing each applicable
regulation, HCFA now cites the deficiency only once, by referencing the
regulation most likely to affect patient care. Ms.Yost noted that HCFA would
like input on this process. Ms. Yost acknowledged the concern about sample
selection for rescreening during the survey. She also said that automated
systems have only recently been in laboratory use and are not included in the
statistics.
CYTOLOGY PT Addendum J-K
Ms. Rhonda Whalen of CDC reviewed the requirements in the CLIA law and
regulations concerning cytology PT (see addendum J). She explained the basis
for the lawsuit filed by Consumer Federation of America/Public Citizen against
the Department of Health and Human Services (HHS) related to cytology PT and
the decision of the district court that cytology PT slides should be examined at
the maximum rate allowed for examining patient slides. HHS appealed the
district court ruling and complied with the court order by publishing a proposed
rate change for examining PT slides in November 1995. HHS solicited public
comments to the proposed rate change and on computer-based PT programs
(CLIAC previously recommended the development of computer technology to
test locator and interpretive skills), since implementation of glass slide PT had
not been achieved. CDC received more than 750 comment letters. Almost all the
commenters disagreed with the proposed rate change and more than half the
letters commented on computer-based PT. The appellate court upheld the
CLIA PT rate, but directed HHS to publish the rationale supporting the time frame
for examining PT slides. HHS will publish a notice withdrawing the proposal to
revise the PT rate and plans to revise the regulations to allow approval of
computer-based PT programs.
Ms. MariBeth Gagnon of CDC reviewed the background for the CDC
development of a prototype computer-based PT program (see addendum K).
After a 1993 Request for Proposal to develop a PT program using glass slides
received no response, CDC awarded three cooperative agreements to
develop and pilot test model computer-based cytology PT programs. Based on
the input from the cooperative agreements, CDC developed a computer-
based PT program. Ms. Gagnon then described the features of the CDC model
program and gave a demonstration for the CLIAC.
Committee Discussion
A committee member commented that some patients want their Pap smears
examined by a laboratory that uses an instrument to assist in QC performance.
Another member felt that some of the manufacturers' advertisements or claims
are beyond the instruments' capabilities. He suggested that routine manual
rescreens of all slides would detect more positives at a lower cost than
automated methods. The Chairman asked about the relevance of developing
a computer PT program which would more closely resemble microscopic
examinations. Dr. Collins indicated that participant response to computer
images in the pilot study was positive. Another CLIAC member commented that
technology in histopathology and cytology is moving away from oculars and
that, in the future, normal working conditions will involve computer screens.
One CLIAC member asked about hardware requirements to administer the
CDC prototype program. Dr. Collins indicated that nationally available
computer training centers could be used to administer the cytology PT. Dr.
Schwartz noted that cytology PT does not evaluate sample collection. Another
CLIAC member commented that computer images would provide more
uniform testing than glass slides. Dr. Schwartz recognized Ms. Theresa Somrak
[co-investigator on the American Society of Clinical Pathologists (ASCP)/CDC
cooperative agreement] who stated that ASCP believes that marginal
practitioners may not be reliably identified by means of a 10-slide proficiency
test. She emphasized the importance of criterion-based validity instead of face
validity for the development of cytology competency tests.
STATUS OF CLIA RESEARCH STUDIES
CLIA Research Update Addendum L
Dr. Thomas L. Hearn, Chief of the Laboratory Practice Assessment Branch in the
Division of Laboratory Systems at CDC, provided background information for
three presentations on the CLIA research studies, which are part of CDC's
broader laboratory practice research agenda referred to as EQLPS (Evaluation
of Quality in Laboratory Practices and Standards). The CLIA research studies
focus on aspects of PT, quality assurance (QA) and QC, personnel, extent to
which errors in testing affect diagnosis and treatment of patients, and the
identification of where errors occur in the testing process. These and other
EQLPS studies are being carried out not as a single linked study but as
individual discrete studies each within a fairly narrow circumspect perspective.
CDC is doing the studies through competitive government procurement processes
(contracts and cooperative agreements), through professional collaborations,
and by collaboration among CDC and HCFA staff. Within the EQLPS program
there are more than 30 studies, some of which are now almost completed.
Three of these studies are being presented today. They were chosen because
they are at stages where there are important findings to share and because
they represent the diversity in the approaches used for carrying out EQLPS
research in terms of the questions asked and strategies used for doing the work.
As individual studies are completed, they will be published in peer-reviewed
journals.
Calculation of Bias and Precision Tolerance Limits Addendum M
Using the Variation of Test Distributions
Dr. George Klee reported on work being carried out at Mayo Clinic Laboratories
which demonstrated the critical need for defining analytical precision and
accuracy goals. Laboratory research has often focused on the analytical
precision goals for testing and Dr. Klee has investigated the utility of defining
analytical accuracy or bias goals. Small shifts in the accuracy of calcium,
thyroid stimulating hormone, and prostate specific antigen tests were shown to
have potentially adverse effects on the diagnosis and treatment of large groups
of patients. Improved detection and prevention of these small, but significant
shifts, will require better QC and QA approaches and may also be dependent
upon having more accurate and robust test systems.
Association Between HIV Testing Accuracy and Addendum N
the Use of External QC Samples
Drs. Rex Astles and Harvey Lipman of the CDC, using results from a qualitative
test, HIV antibody testing data from CDC's Model Performance Evaluation
Program (MPEP), investigated the association between HIV testing accuracy
and the use of either in-house or commercially obtained QC material. After
considering potentially confounding independent variables [such as the kit used
for testing, the type of laboratory (e.g., hospital, public health), the number
of specimens routinely tested, requiring a minimum degree for testing personnel,
and requiring retro viral specific training], the only variable (other than the
use of QC materials) found to be independently associated with the error rate was
the type of test kit used for analysis. After controlling for all significant
confounders, the use of QC materials was associated with a strongly significant
(p=0.0002) 31% reduction in the error rate on MPEP samples. Further statistical
analyses of the data revealed that systematic errors occurred in both
laboratories that did and did not use QC. However, laboratories that did not use
QC had a systematic error rate that was more than twice as great as laboratories
that used QC, meaning that laboratories that did not use QC were much more likely
to have made two or more errors in each panel of six MPEP samples than laboratories
that used QC.
Laboratory Medicine Sentinel Monitoring Network Addendum O
Ms. Kathy LaBeau, Project Director for a cooperative agreement between the
Washington State Department of Health and CDC, reported on laboratory
practices in a group of 266 laboratories which voluntarily participate in a
sentinel network organized in the Pacific Northwest. Information about
laboratory practices was collected through five mailed surveys and additional
issues of general interest were discussed in two regional focus group meetings.
Information about QA practices, access to testing, reasons for recent increases
or decreases in testing workload, analysis of common problems that result in
laboratories issuing corrected reports, and personnel issues were addressed in
the surveys. The sentinel network, although not a statistically representative
sample of the laboratory population, serves as a useful monitoring mechanism
for identifying general problems in laboratory medicine and serves as a resource
for laboratories to obtain additional information about testing practices.
Committee Discussion on CLIA Research Studies
The presenters responded to several requests for clarification on study design
and findings from CLIAC members.
In response to questions about the expense of establishing bias goals and the
number of observations needed, Dr. Klee reported that the cost and effort
involved in maintaining bias goals are dependent on several factors including
access to sufficient patient values and the distribution of the population of
test results around clinical decision points. One committee member reported PT
failures when his laboratory requested lot numbers of QC materials with specific
values. Dr. Klee noted that a potential downside to minimizing bias shifts is
that a few laboratories may actually be penalized in some PT events if the
majority of laboratories in their "peer" grading group are not as
diligent and use reagents which are less reliable.
Commenting on the MPEP data on HIV antibody testing, one committee
member felt strongly that the choice of test kit could be the most important
variable associated with the error rate. He suggested that the kits used by the
study participants may not be representative of kits used by other laboratories.
The Chairman asked how the study controlled for the quality of the participating
laboratories. Dr. Lipman responded that the study controlled for testing volume,
personnel degree and training requirements, and the results were unaffected
by these variables.
With respect to the sentinel network, one committee member asked if
laboratories which perform only waived testing or have stopped testing are
included. Ms LaBeau responded that only laboratories performing moderate or
high complexity testing are included in the network. Therefore, data from the
Pacific Northwest network do not reflect the experiences of laboratories that
formerly may have performed such testing but either are currently performing
only waived tests or may have stopped testing completely. Ms. Yost
commented that many tests performed by POLs are now waived. Commenting
on access to testing, another committee member suggested that any studies to
estimate the cost to patients of not having testing available at their health
care site should include transportation, babysitting costs, etc.
PUBLIC COMMENTS Addenda P-Q
1.William G. Cooper, QC Manager of Bio-Rad Laboratories, summarized the
results from a December 1996 survey of QC practices and preferences in U.S.
clinical laboratories selected from the Medical Laboratory Observer mailing list.
This survey was designed and conducted without any involvement or input from
CDC. Based on the survey results, Mr. Cooper submitted written
recommendations to CDC and CLIAC concerning possible revisions of the CLIA
regulations (see addendum P).
2.Theresa Somrak, Director of Cytopathology Education Consortium
Activities, recommended that a cytology PT program consist of either mailed
challenges in a variety of formats, including glass slides and photomicrographs,
or computer-based images and should evaluate the performance of the
laboratory. She suggested that laboratory directors could use education-based
exercises provided by professional organizations, such as ASCP and the College
of American Pathologists, to evaluate cytology personnel (see addendum Q).
3.Toni Casey, I-Stat Corporation, asked about the status of Accurate and
Precise Technology (APT) testing and if there would be another open meeting
for manufacturers to discuss QC. Chairman Schwartz considered the questions
out of order, but indicated Ms. Casey's remarks could be presented to CLIAC as
a public comment at a later date.
4.Don Parker, Director of Clinical Trials at Bayer Corporation, suggested that
the waiver review process be presented to CLIAC, to clarify the steps that could
be taken if a waiver application is not approved.
Committee Discussion
One CLIAC member asked about the status of waiver applications and how to
obtain comments submitted in response to CDC's request for data and
information on QC practices following the September 1996 QC meeting. Ms.
Bakes-Martin replied that CDC has denied no waiver requests thus far and said
that several applications are under review. She also pointed out that CDC is
evaluating waiver applications using guidelines which were published in the
waiver proposed rule and that publishing a final regulation on the waiver
process is a high priority. Ms. Bakes-Martin explained that significant QC
comments might be presented to CLIAC in the future and noted that copies of
comments submitted following the QC meeting could be obtained through a
Freedom of Information Act request.
Another committee member asked about the time line for the results of the
CDC contract to evaluate cytology PT which includes: (1) an assessment of
the CDC computer-based PT program by cytotechnologists and pathologists;
and (2) comparison of glass slide PT and computer-based PT to rescreening
results. Ms. Gagnon said that the contractor performing these studies should
have data available this summer.
A CLIAC member inquired about the mechanism for getting items on the CLIAC
agenda. Dr. Schwartz replied that agenda items may be submitted by
committee members or the public to any CLIAC member, but are usually
submitted to Dr. Ed Baker, CLIAC Executive Secretary.
CONCLUDING REMARKS
Dr. Schwartz announced that the dates for the next CLIAC meeting will be
September 11-12, 1997, and then adjourned the meeting.
|