Centers for Disease Control and Prevention Centers for Disease Control and Prevention CDC Home Search CDC CDC Health Topics A-Z site search
National Office of Public Health Genomics
Centers for Disease Control and Prevention
Office of Genomics and Disease Prevention
Site Search

HuGENet Publications
Assessment of cumulative evidence on genetic associations: interim guidelines
John PA Ioannidis1–3,*, Paolo Boffetta4, Julian Little5, Thomas R O’Brien6, Andre G Uitterlinden7, Paolo Vineis8, David J Balding8 Anand Chokkalingam9, Siobhan M Dolan10, W Dana Flanders11, Julian PT Higgins12, Mark I McCarthy13,14, David H McDermott15, Grier P Page16, Timothy R Rebbeck174, Daniela Seminara18 and Muin J Khoury19
International Journal of Epidemiology 2008; 37(1):120-132
line

TABLE 1: Considerations for epidemiologic credibility in the assessment of cumulative evidence on genetic associations

Criteria
Categories
Proposed operationalization
Amount of evidence

A: Large-scale evidence

B: Moderate amount of evidence

C: Little evidence

 Thresholds may be defined based on sample size, power or false-discovery rate considerations. The frequency of the genetic variant of interest should be accounted for. As a simple rule, we suggest that category A requires a sample size over 1000 (total number in cases and controls assuming 1:1 ratio) evaluated in the least common genetic group of interest; B corresponds to a sample size of 100–1000 evaluated in this group, and C corresponds to a sample size <100 evaluated in this group (see ‘Discussion’ section in the text and Table 2 for further elaboration).a
 
Replication

A: Extensive replication including at least one well-conducted meta-analysis with little between-study inconsistency

B: Well-conducted meta-analysis with some methodological limitations or moderate between-study inconsistency

C: No association; no independent replication; failed replication; scattered studies; flawed meta-analysis or large inconsistency

 Between-study inconsistency entails statistical considerations (e.g. defined by metrics such as I2, where values of 50% and above are considered large and values of 25–50% are considered moderate inconsistency) and also epidemiological considerations for the similarity/standardization or at least harmonization of phenotyping, genotyping and analytical models across studies. See ‘Discussion’ section in the text for the threshold (statistical or others) required for claiming replication under different circumstances (e.g. with or without including the discovery data in situations with massive testing of polymorphisms).
 
Protection from bias

A: Bias, if at all present, could affect the magnitude but probably not the presence of the association

B: No obvious bias that may affect the presence of the association but there is considerable missing information on the generation of evidence

C: Considerable potential for or demonstrable bias that can affect even the presence or absence of the association

 A prerequisite for A is that the bias due to phenotype measurement, genotype measurement, confounding (population stratification) and selective reporting (for meta-analyses) can be appraised as not being high (as shown in detail in Table 3) plus there is no other demonstrable bias in any other aspect of the design, analysis or accumulation of the evidence that could invalidate the presence of the proposed association. In category B, although no strong biases are visible, there is no such assurance that major sources of bias have been minimized or accounted for because information is missing on how phenotyping, genotyping and confounding have been handled. Given that occult bias can never be ruled out completely, note that even in category A, we use the qualifier ‘probably’.

(a) For example, if the association pertains to the presence of homozygosity for a common variant and if the frequency of homozygosity is 3%, then category A amount of evidence requires over 30,000 subjects and category B between 3,000 and 30,000. The sample size refers to subjects when genotype contrasts are used, and to alleles when alleles are contrasted.

<< back to publication

 

TABLE 2: Power calculations for associations with nminor = 1000 for various ORs and various frequencies of the minor genetic group (f minor)6

OR
f minor
Power for {alpha} = 0.05
Power for {alpha} = 10–7
1.10
0.01
0.32
<0.001
1.20
0.01
0.82
0.007
1.30
0.01
0.98
0.12
1.50
0.01
1.00
0.83
2.00
0.01
1.00
1.00
5.00
0.01
1.00
1.00
1.10
0.05
0.31
<0.001
1.20
0.05
0.80
0.006
1.30
0.05
0.98
0.09
1.50
0.05
1.00
0.78
2.00
0.05
1.00
1.00
5.00
0.05
1.00
1.00
1.10
0.10
0.30
<0.001
1.20
0.10
0.78
0.005
1.30
0.10
0.97
0.74
1.50
0.10
1.00
1.00
2.00
0.10
1.00
1.00
5.00
0.10
1.00
1.00
1.10
0.25
0.25
<0.001
1.20
0.25
0.69
0.002
1.30
0.25
0.94
0.04
1.50
0.25
1.00
0.52
2.00
0.25
1.00
1.00
5.00
0.25
1.00
1.00
1.10
0.50
0.18
<0.001
1.20
0.50
0.51
<0.001
1.30
0.50
0.81
0.006
1.50
0.50
0.99
0.15
2.00
0.50
1.00
0.96
5.00
0.50
1.00
1.00

(a) All calculations assume the same number of cases and controls; results are relatively robust to modest deviations in the allocation ratio. The minor genetic group is the smallest of the two groups contrasted and may have been selected based on genotype or allele considerations.

<< back to publication

 

TABLE 3: Typical biases and their typical impact on associations depending on the status of the evidence

Likelihood of bias to invalidate
an observed association
Biases
Status of the evidence
Small OR <1.15
Typical OR 1.15–1.8
Large OR >1.8
Bias in phenotype definition Not reported what was done
Unknown
Unknown
Unknown
Unclear phenotype definitions
Possible/High
Possible/High
Possible/High
Clear widely agreed definitions of phenotypes
Low/None
Low/None
Low/None
Efforts for retrospective harmonization
Possible/High
Low
Low/None
Prospective standardization of phenotypes
Low/None
Low/None
Low/None
Bias in genotyping Not reported what was done
Unknown
Unknown
Unknown
No quality control checks
Possible/High
Low
Low
Appropriate quality control checks
Low
Low
Low/None
Population stratification Not reported what was done
Unknown
Unknown
Unknown
Nothing donea
Possible/High
Possible/High
Possible/High
Same descent groupb
Possible/High
Low
Low/None
Adjustment for reported descent
Possible/High
Low
Low/None
Family-based design
Low/None
Low/None
Low/None
Genomic control, PCA or similar method
Low/None
Low/None
Low/None
Selective reporting biases Meta-analysis of published data
Possible/High
Possible
Possible
Retrospective efforts to include unpublished data
Possible/High
Possible
Possible
Meta-analysis within consortium
Low/None
Low/None
Low/None

Category decreases from A to B, if the ‘Unknown’ are considered to be a major issue for the appraisal of the evidence. Any ‘Possible/high’ item confers category C status. ‘Possible’ (selective reporting biases for non-consortium/prospective meta-analysis) does not necessarily decrease the category grade (from A to C); this may need to be appraised separately in each field and may be facilitated by using tests for selective reporting biases (tests for small-study effects and excess of significant studies), although probably no test has high sensitivity and specificity for such biases. Clear demonstrable biases in other aspects of the design, conduct and analysis of the evidence (besides the four aspects considered in this table) also result in shift to category C for protection from bias.

(a) Including groups of clearly different descent without consideration to this diversity. (b) The ethnic population structure may need to be considered also on a case-by-case basis.

OR, odds ratio; PCA, principal component analysis.

<< back to publication

 

TABLE 4: Variation in the volume of human genome epidemiology evidence for selected diseases, 2001–66

Disease
Papers
Genes studied
Meta-analyses (HuGE reviews)
Investigators
Consortia
Type 2 diabetes
2246
555
56 (7)
9320
Established
Breast cancer
1020
275
24 (3)
4499
Established
Osteoporosisb
503
128
11 (2)
2041
Established
Pre-term birthc
176
112
1 (1)
855
Emerging
Childhood leukaemia
102
76
1 (1)
668
Emerging

(a) Data from the HuGE published literature database run November 27, 2006; does not include data on genome-wide associations that started appearing for some of these phenotypes (e.g. type 2 diabetes or breast cancer) in early 2007.
(b) Includes studies on bone mineral density.
(c) Includes studies on gestational age.

<< back to publication

 

TABLE 5: Considerations for assessment of clinical and public health relevance and importance of genetic associations

Magnitude of effect
    Effect size
    Frequency of genetic variant in population
Clinical and public health importance
    Type of phenotype: biological, endophenoype and hard clinical outcome
    Disease burden: incidence, severity and mortality
    Interaction with identified modifiable environmental exposures
    Potential to prevent disease through intervention (e.g. through Mendelian randomization insights)

<< back to publication

 
Page last reviewed: March 20, 2008 (archived document)
Content Source: National Office of Public Health Genomics