CLINICAL UTILITY

Question 8. Is the test qualitative or quantitative?

The DNA tests for both factor V Leiden (FVL) and prothrombin G20210A mutation (PRO) are qualitative (e.g., a specific mutation is reported as present or absent).

CLINICAL UTILITY

Question 9: How often is a test positive when a mutation is present?
Question 10: How often is the test negative when a mutation is not present?

Definitions
Analytic performance is summarized by the sensitivity and specificity of the detection system.  Analytic sensitivity is the proportion of positive test results, when a detectable mutation is present (i.e., the test is designed to detect that specific mutation).  The analytic sensitivity may also be called the analytic detection rate. 
Analytic specificity is the proportion of negative test results when no detectable mutation is present.  Analytic specificity can also be expressed in terms of the analytic false positive rate.  This would be the proportion of positive test results when no detectable mutations are present (1-analytic specificity).

Optimal source(s) of data
Basing analytic performance estimates on external proficiency testing has drawbacks, including:

• mixing of clinical and research laboratories
• few challenges
• reporting summary results in ways that do not allow a straightforward computation of analytic sensitivity and specificity
• challenges that do not represent the ‘mix’ of genotypes expected in a screening program (e.g., too few negative tests).

Future analyses should be aimed at providing reliable method- and, possibly, mutation-specific analytic performance estimates.  One approach for collecting such data might include the following steps:

• An independent body [such as the College of American Pathologists, American College of Medical Genetics, Food and Drug Administration or the Coriell Institute of Medical Research (Camden, NJ)] would develop a standard set of samples, most of which would be randomly selected from the general population.  Included in the standard set, however, would also be additional, less common genotypes
• The sample set would then be available for method validation.  Correct genotypes would be arrived at by consensus, or, if disagreements emerged, by a reference method (e.g., sequencing).  The current validation practice of having a laboratory (or manufacturer) run a series of samples with unknown genotype is inadequate, since there is no ‘gold standard’ with which to compare.  For example, how can a laboratory running an unknown sample determine whether a positive finding is a true, or a false, positive? 
• Ideally, this blinded sample set would be available to manufacturers as part of the pre-market approval process, with the understanding that multiple laboratories using these commercial reagents would be asked by the manufacturer to analyze portions the sample set independently.  This initial assay validation process is distinct from assay control samples that are discussed later (Question 13).

Appropriate sample size for determining analytic sensitivity and specificity has been discussed in detail in an earlier ACCE review (Prenatal Cystic Fibrosis Carrier Screening).  In brief, a target sensitivity (or specificity) can be chosen, along with an acceptable lower limit (assumed to be the lower 95 percent confidence interval).  Given these targets, the number of necessary samples can be derived.  For example, if a laboratory chose a target specificity of 98 percent and wanted to rule out a specificity of 90 percent, it would need to correctly identify at least 49 of 50 known negative samples (estimated using the binomial distribution).  When the estimates approach 100 percent and include relatively tight confidence intervals, it may not be economically feasible for individual laboratories to create the data.  However, this could be attained by a consortium of laboratories using the same methodology, or by a manufacturer that forms a consortium of laboratories using its reagents.   All of these analyses could be done using a 2x2 table, and all rates could be accompanied by 95 percent confidence intervals (CI).

The present analysis, which uses the ACMG/CAP data, initially examines the rates of these two types of errors independently, by chromosome (e.g., the results on one chromosome are counted separately from the results reported for the other). 

Error rates for the ACMG/CAP external proficiency testing scheme  Table 2-1 shows the number of alleles tested and the results from the ACMG/CAP Molecular Genetics Survey from 1999 to 2003 for factor V Leiden.  Overall, 0.5 percent (95 percent CI 0.3% to 0.6%) of the FVL alleles were incorrectly identified.  For all data between 1999 and 2003, 7039 of 7072 chromosomes were correctly identified (99.5%, 95 percent CI 99.4% to 99.7%).  Table 2-2 shows the number of alleles tested and the results from the ACMG/CAP Molecular Genetics Survey from 1999 to 2003 for prothrombin G20210A mutation (PRO).  Overall, 0.5 percent (95 percent CI 0.3% to 0.6%) of the PRO alleles were incorrectly identified.  For all data between 1999 and 2003, 6063 of 6092 chromosomes were correctly identified (99.5%, 95 percent CI 99.4% to 99.7%).  Appendix 1 contains a complete listing of the sample challenges, the responses, and the types of errors (e.g., false positive).

Tables 2-3 and 2-4 make use of the ACMG/CAP external proficiency testing data (Appendix 1) to compute the analytic sensitivity and specificity for factor V Leiden and prothrombin G20210A mutations.

Table 2-3.  Analytic Performance for Identifying Factor V Leiden Mutations According to Data from the ACMG/CAP Molecular Genetics Survey

Table 2-4.  Analytic Performance for Identifying Prothrombin G20210A Mutations According to Data from the ACMG/CAP Molecular Genetics Survey

Sensitivity and specificity by person rather than by chromosome
It is possible to compute analytic sensitivity and specificity according to whether a person’s genotype has been correctly classified, rather than whether an individual chromosome has been correctly classified.  That is, the genotype is correct or incorrect when detectable mutations are present (analytic sensitivity), or the genotype is correct or incorrect when no detectable mutations are present (analytic specificity).  Tables 2-5 and 2-6 show the results of this analytic approach, stratified by the year that proficiency testing results were obtained. Overall error rates of 0.8 percent (95% CI 0.5-1.2%) and 0.8 percent (95% CI 0.5-1.1%) were found for factor V Leiden and prothrombin G20210A mutation testing, respectively, for testing in US laboratories.

Table 2-5.  Analytic Performance for Identifying Factor V Leiden Mutations Based on the ACMG/CAP Molecular Genetics Survey, Classified According to Whether a Person’s Genotype is Correctly Identified

Table 2-6.  Analytic Performance for Identifying Prothrombin G20210A Mutations Based on the ACMG/CAP Molecular Genetics Survey, Classified According to Whether a Person’s Genotype is Correctly Identified

The National External Quality Assessment Schemes (NEQAS) includes data from available from the United Kingdom and Europe.  Data for the Factor V Leiden/Molecular Genetics of Thrombophilia External Quality Assessment Programme are listed in Tables 2-7 and 2-8.

Table 2-7.  Analytic Performance for Identifying Factor V Leiden Mutations based on the NEQAS Factor V Leiden External Quality Assessment Programme, Classified According to Whether the Genotype is Correctly Identified

Table 2-8.  Analytic Performance for Identifying Prothrombin G20210A Mutations Based on the NEQAS Molecular Genetics of Thrombophilia External Quality Assessment Programme, Classified According to Whether the Genotype is Correctly Identified

As can be seen, the two quality assurance programs give similar results (overall error rate of 0.8 percent for factor V Leiden and prothrombin G20210A mutation testing in the U.S., and 0.9 and 0.6, respectively, in the UK and Europe).

References

1. ACMG/CAP Molecular Genetics Survey Sets (1999, 2000, 2001, 2002, 2003) College of American Pathologists, Northfield, IL.

Appendix 1. Data used to calculate analytic sensitivity and specificity

Tables 2-9 through 2-13 summarize the factor V Leiden external proficiency testing results obtained by the American College of Medical Genetics and the College of American Pathologists (ACMG/CAP).  Samples with known genotypes have been distributed to participants since 1999.  The first columns of the tables contain the distribution label (99 MGL-11 indicates the 11th DNA sample distributed as part of the Molecular Genetics Laboratory survey in 1999).  The second columns contain number of participating laboratories, followed by the genotype of the sample.  The number of laboratories reporting specific genotypes is then provided, along with a tabulation of their ‘correct’ and ‘incorrect’ responses.  The tables also contain the denominator for calculating the analytic sensitivity and specificity in a box, along with the yearly (and summary) totals.

Table 2-9.  Computations for the 1999 ACMG/CAP Proficiency Testing Surveys: factor V Leiden (R506Q mutation)

Table 2-10.  Computations for the 2000 ACMG/CAP Proficiency Testing Surveys: factor V Leiden (R506Q mutation)

Table 2-11.  Computations for the 2001 ACMG/CAP Proficiency Testing Surveys: factor V Leiden (R506Q mutation)

Tables 2-14 through 2-18 summarize the prothrombin G20210A mutation external proficiency testing results obtained by the American College of Medical Genetics and the College of American Pathologists (ACMG/CAP).  Samples with known genotypes have been distributed to participants since 1999.  The first column of the tables contain the distribution label (99 MGL-21 indicates the 21st DNA sample distributed as part of the Molecular Genetics Laboratory survey in 1999).  The second columns contain number of participating laboratories, followed by the genotype of the sample.  The number of laboratories reporting specific genotypes is then provided, along with a tabulation of their ‘correct’ and ‘incorrect’ responses.  The tables also contain the denominator for calculating the analytic sensitivity and specificity in a box, along with the yearly (and summary) totals.

Table 2-18.  Computations for the 2003 ACMG/CAP Proficiency Testing Surveys: Prothrombin G20210A Mutation

ANALYTIC VALIDITY

Question 11: Is an internal quality control program defined and externally monitored?

Definition
Internal quality control is a set of laboratory procedures designed to ensure that the test method is working properly.  An internal quality control program includes documentation that high standards are being practiced to ensure that:

• reagents used in all aspects of genetic testing are of high quality to allow successful test completion,
• all equipment is properly calibrated and maintained,
• good laboratory practices are being applied at every level of genetic testing.  To the extent possible, all steps of the testing process must be controlled.

Quality control procedures
Techniques that are used for analyzing DNA for factor V Leiden and prothrombin G20210A mutations are the same as those used for other molecular testing.  These techniques are widely applied and well understood.  As a result, it has been possible to design and publish generic internal quality control procedures, which many molecular laboratories already have in place.  Table 2-19 lists published guidelines that, among other topics, describe reagent quality control, equipment calibration and maintenance, education of the technical staff, and other internal quality control procedures.  The purpose of the quality control procedures is to rigorously control all steps of the DNA testing process to minimize the potential for test failure.  Given that the internal procedures for establishing and maintaining good laboratory practice are readily available (Neumaier et al., 1998), the important next step will be to encourage, assist, and require laboratories to apply and document appropriate quality control procedures.

Table 2-19.  Guidelines, Recommendations, and Checklists that Address Internal Quality Control Issues and Requirements.

External monitoring
All clinical laboratories performing genetic testing must comply with general regulations under the Clinical Laboratory Improvement Amendments (CLIA) and a CLIA certification should be considered the minimum acceptable level of external monitoring.  One shortcoming of having only a CLIA certification is that CLIA inspectors often have less experience in evaluating genetic testing laboratories than  other certifying organizations.  CLIA is in the process of upgrading its regulations regarding genetic testing.  The Task Force on Genetic Testing concluded that the current CLIA requirements are insufficient to ensure quality of molecular genetic testing.  Laboratories certified by CAP or by New York State Health Department will have undergone a more rigorous external monitoring that requires specific procedures and documentation.

References:

1. Holtzman NA, Watson MS.  1997.  Promoting Safe and Effective Genetic Testing in the United States.  Final report of the Task Force on Genetic Testing.  http://www.nhgri.nih.gov /ELSI/TFGT_final/, pp. 1-72.
   
2. Neumaier M, Braun A, Wagener N.  1998.  Fundamentals of quality assessment of molecular amplification methods in clinical diagnosis.  Clin Chem  44:12-26.

ANALYTIC VALIDITY

Question 12: Have repeated measurements been made on specimens?

Measurements made on the same specimen in different laboratories
Multiple laboratories have made repeated measurements on the same specimen, utilizing a variety of technologies.  A collaborative external proficiency testing program, jointly administered by the American College of Medical Genetics and the College of American Pathologists (ACMG/CAP) provides up to six factor V Leiden and prothrombin G20210A mutation DNA challenges each year, along with a summary report of the results.  An earlier section in Analytic Validity (Questions 9 and 10) provides more details about the results of this program.  In summary, the between-laboratory replication of a single specimen’s genotype for factor V Leiden is between 98.9 percent and 99.9 percent and for prothrombin G20210A mutation is between 93.7 percent and 99.9 percent (see Tables 2-1 and 2-2).

Measurements made repeatedly on the same sample within a laboratory
It is common practice for repeated measurements to be made on the same specimen (a control specimen) within a laboratory.  For each assay, a positive control is usually included for testing.  This internal documentation will remain within the laboratory but will be available for on-site inspections by certifying agencies.  Thus, one avenue for collection of these data would again be to use laboratory survey instruments. This type of quality control information is not currently accessible for this review.

ANALYTIC VALIDITY

Question 13: What is the within- and between-laboratory precision?

This question is not applicable to factor V Leiden and prothrombin G20210A mutation analysis, since such testing is qualitative.  This question is only relevant to quantitative measurements.

ANALYTIC VALIDITY

Question 14: If appropriate, how is confirmatory testing performed?

Definition
Confirmatory testing is performed to ensure that the initially positive test result is correct.

Importance of confirmatory testing
The analytic specificity is currently estimated to be 99.7 percent for factor V Leiden and 99.8 percent for prothrombin G20210A mutation (Question 10).  It is important, therefore, to determine how often ‘false positive’ results will be identified upon confirmatory testing.  If the error is due to clerical or laboratory sample mix-up, simple retesting of an additional aliquot may be sufficient to identify and correct the error.  Given that proficiency testing in Europe found 90 percent of the errors to be of this type (Dequeker and Cassiman, 2000), confirmatory testing can be expected to eliminate many of the false positive results.  This issue is dealt with in more detail under Clinical Validity (Questions 21 and 22).

In the thrombosis clinic at the University of Vermont Medical School, confirmatory testing for factor V Leiden is not done. At Leiden University Medical Center’s clinical laboratory, the genotype of factor V Leiden is determined by PCR, and a random sample is retested (personal communication Carla Vossen, Astrid van Hylckama Vlieg).

At times, testing for activated Protein C (APC) resistance may be used as a substitute for DNA testing, or as a confirmatory test once a mutation has been found.  The factor V Leiden mutation leads to a decreased response of plasma to the anticoagulant action of activated Protein C, so-called APC resistance.  Several methods for the detection of APC resistance have been developed including, a partial thromboplastin time-based test.  However, APC resistance is not caused exclusively by the factor V Leiden mutation.  Recently de Visser et al (1999) described an increased risk of venous thrombosis due to APC resistance in the absence of the factor V Leiden mutation. In this situation, DNA analysis can be performed to identify cases with factor V Leiden (Bertina, 1994).

Gap in Knowledge:  Performance of Confirmatory Testing
Little or no information has been found on the application of confirmatory testing to identify false positive test results in a clinical setting.  According to proficiency testing data, these false positive results should occur and might be identified as part of routine confirmatory testing of individuals found to be positive for factor V Leiden or prothrombin G20210A mutations.

1. Dequeker E, Cassiman J.  2000.  Genetic Proficiency testing in diagnostic laboratories – quality control is the message.  Am J Hum Genet  67:A274.
   
2. Bertina RM, Koeleman BP, Koster T, Rosendaal FR, Dirven RJ, de Ronde H, et al.  1994. Mutation in blood coagulation factor V associated with resistance to activated protein C.  Nature 369:64-67.
   
3. de Visser MCH, Rosendaal FR, Bertina RM. 1999. A reduced sensitivity for activated protein C in the absence of factor V Leiden increases the risk of venous thrombosis.  Blood  93:1271-1276.

ANALYTIC VALIDITY

Question 15: What types of patient samples have been tested?

Factor V Leiden and prothrombin G20210A mutation analysis has been successfully performed in a variety of specimens using available methodologies. 

Testing can be performed on:

• whole blood (purified DNA and lysates),
• buccal lysates (cheekbrush, swab and mouthwash)

Blood samples are the most reliable method of collecting large amounts of high quality DNA, but a trained phlebotomist is needed, thereby increasing costs and requiring that specimens be collected at a medical facility.  Buccal cells obtained by scraping, brushing or mouthwash yield adequate amounts of DNA for screening purposes (Doherty et al., 1996; Loader et al., 1996; Witt et al., 1996; Grody et al., 1997).  This technique can be used to collect samples at the physician’s office or at home.  Buccal samples have the disadvantage of less DNA, higher failure rates, and less documentation of chain of custody.  Buccal lysates can be frozen and stored for years and still be tested successfully (Bradley et al., 1998).  A comparison of test results from blood and buccal mouthwash samples showed consistent results (Baty et al., 1998).  In an informal survey of commercial laboratories offering factor V Leiden and prothrombin G20210A mutation testing, all accepted both blood and buccal specimens (W Allan, personal communication).

1. Baty D, Terron KA, Mechan D, Harris A, Pippard MJ, Goudie D.  1998.  Development of a multiplex ARMS test for mutations in the HFE gene associated with hereditary hemochromatosis.  J Clin Pathol  51:73-74.
   
2. Bradley LA, Johnson DD, Palomaki GE, Haddow JE, Robertson NH, Ferrie RM. 1998. Hereditary haemochromatosis mutation frequencies in the general population.  J Med Screen  5:34-36.
   
3. Doherty RA, Palomaki GE, Kloza EM, Erickson JL, and Haddow JE.  1996.  Couple-based prenatal screening for cystic fibrosis in primary care settings.  Prenat Diagn  16:307-404.
   
4. Grody WW, Dunkel-Schetter C, Tatsugawa ZH, Fox MA, Fang CY, Cantor RM, et al.  1997.  PCR-based screening for cystic fibrosis carrier mutations in an ethnically diverse pregnant population.  Am J Hum Genet  60:935-947.
   
5. Loader S, Caldwell P, Kozyra A, Lenenkron JC, Boehm CD, Kazazian HH, et al.  1996.  Cystic fibrosis carrier population screening in the primary care setting.  Am J Hum Genet  59:234-247.
   
6. Witt DR, Schaefer C, Hallam P, Wi S, Blumberg B, Fishbach A, et al.  1996.  Cystic fibrosis heterozygote screening in 5,161 pregnant women.  Am J Hum Genet  58:823-835.

ANALYTIC VALIDITY

Question 16: How often does the test fail to give a useable result?

Test ‘failures’ in the pre-analytic phase of testing
In the pre-analytic phase, it may be determined that the sample is not suitable for testing because specific clinical criteria are not met, or because the sample is considered inadequate.  While programs often monitor pre-analytic test cancellation rates as part of an overall quality assurance plan, these events are usually not considered a laboratory or methodologic ‘failure’.  Table 2-20 lists criteria commonly used for deciding whether to reject a sample in the pre-analytic phase.

Table 2-20.  Common Pre-analytic Criteria for Rejecting a Sample Submitted for
factor V Leiden or Prothrombin G20210A Testing

Test failures during the analytic phase of testing
Failures of individual samples or assays occur when preset quality control standards are not met and test results are not reportable.  Failures can arise for a number of reasons such as improperly processed samples, problems with component reagents, or equipment malfunction.  Many assay failures within the clinical molecular genetic laboratory are due to operator error.  Automation and programs to properly train laboratory personnel can avoid most of these problems.  Only a few medical technology programs, however, currently provide adequate molecular components in their programs.  Documentation of failures and subsequent corrective action is required by regulatory agencies such as CLIA and CAP.  Unfortunately, failure rates and other information on assay robustness are often not published as part of pilot trials or method evaluations.  Available data suggest, however, that repeating the analysis of an individual sample or assay run can often yield a satisfactory result.

An irretrievable assay failure occurs when an apparently suitable specimen is submitted and approved for testing, but the assay yields a result that is clinically uninterpretable.  Failures of this type are most often related to the quality of the original sample.  Procedural problems during specimen processing and DNA extraction can also be responsible.  Success rates for obtaining clinically interpretable results are close to 100 percent for blood samples.  Buccal samples have a somewhat lower success rate as a result of poor sampling (inadequate number of cells), sample contamination, desiccation (exposure to extreme heat), or inadequate sensitivity of the testing methodology to account for the lower concentration and quality of the sample.

Test failures in the pot-analytic phase of testing
Post-analytic failures, such as incorrect or inadequately interpreted results are considered separately from analytic test failures, as part of a review of overall quality assurance in the Clinical Utility Section (Question 34).

Gap in Knowledge: 
Overall, and method-specific failure rates- Clinical laboratories are required to document test failures, as described above.  For this reason, this type of information should be readily available from laboratories participating in external proficiency testing administered by the ACMG/CAP.  Gathering this information could be accomplished though the use of a supplemental question attached to a routine distribution or, alternatively, the data could be collected via an externally funded, independent project.

ANALYTIC VALIDITY

Question 17: How similar are results obtained in different laboratories?

Comparing results from different laboratories using the same or similar methodologies
The only potential source of data for evaluating differences in factor V Leiden or prothrombin G20210A mutation tests result from multiple laboratories using the same (or a similar) method would be derived from external proficiency testing.  However, the relatively small number of participants and the relatively large number of methods (Table 2-21) preclude obtaining meaningful method-specific analyses.  Even if available, such comparisons might be complicated, because laboratories in the same methodological category may be using different commercial or in-house reagent components and protocols.  For example, although three laboratories might be grouped under the ARMS™ methodology, one might use a prepared kit, a second might use commercially prepared ASRs (analyte specific reagents), and the third might use in-house reagents.  Each may also be targeting a different set of mutations.  All of these factors would make the comparison nearly equivalent to comparing different methodologies.  To help in comparing methodologies, the ACMG/CAP MGL Survey Reports might consider stratifying results into broad methodological categories.

Analytic methodologies used for factor V Leiden and prothrombin G20210A mutation analysis
Table 2-21 lists categories of methodologies that are used to detect factor V Leiden and prothrombin G20210A mutations by laboratories participating in proficiency testing programs in the United States (ACMG/CAP MGL Survey), along with the proportions using each method.  Because many laboratories utilize “home brew” assays, these categories are not homogeneous.

Table 2-21.  Testing Methods Utilized by 189 US Laboratories Performing factor V Leiden Mutation Analysis and 181 US Laboratories Performing Prothrombin G20210A Mutation Analysis According to External Surveys for 2003

References

1. ACMG/CAP Molecular Genetics Survey Sets (1999, 2000, 2001, 2002, 2003)  College of American Pathologists, Northfield, IL.