Comments and suggestions regarding this document should be submitted by February 22, 2000, to Dockets Management Branch (HFA-305), Food and Drug Administration, 5630 Fishers Lane, rm 1061, Rockville, MD 20852. All comments should be identified with the docket number 99D-5424.
For questions concerning the content of the document contact Sharon Ross at 202-205-4168.
Additional copies of this guidance document are available upon written request from the Office of Special Nutritionals (HFS-450), Food and Drug Administration, 200 C Street SW, Washington, DC 20204, by calling 202-205-4168, by faxing a request to 202-205-5295, or from the Internet at http://www.cfsan.fda.gov/~dms/guidance.html#lab.
U. S. Department of Health and Human Services
Food and Drug Administration
Center for Food Safety and Applied Nutrition
December 22, 1999
I. BACKGROUND INFORMATION
II. SCIENTIFIC REVIEW OF HEALTH CLAIMS
A. Identifying Data for Review
B. Performing Reliable Measurements
C. Evaluating Individual Studies
D. Evaluating the Totality of the Evidence
E. Assessing Significant Scientific Agreement
III. REFERENCES
Figure 1. Type of Study Evaluated During Health Claim Review
Figure 2. Schema for Assessing Strength and Consistency of Significant Scientific
Agreement
This guidance has been prepared by the Office of Special Nutritionals in the Center for Food Safety and Applied Nutrition at the Food and Drug Administration (FDA), based on the report of the FDA Food Advisory Committee (FAC) Working Group on Significant Scientific Agreement. This guidance represents the agency's current thinking on the meaning of the significant scientific agreement standard in section 403(r)(3) of the Federal Food, Drug, and Cosmetic Act (21 U.S.C. § 343(r)(3)) and 21 CFR § 101.14(c). It is being issued as level 1 guidance for immediate implementation in accordance with FDA's good guidance practices (62 FR 8961, February 27, 1997). The guidance document does not create or confer any rights for or on any person and does not operate to bind FDA or the public. An alternative approach may be used if such approach satisfies the requirements of the applicable statute, regulations or both.
This guidance document addresses the significant scientific agreement standard, which FDA uses to evaluate the scientific evidence supporting health claim petitions about the relationship between a food substance and a disease or health-related condition. The guidance document describes the scientific review approach FDA has taken in previous health claim reviews and incorporates the recommendations of the FDA FAC Working Group on Significant Scientific Agreement. This approach is used by FDA scientists in their review of health claims and should be considered as guidance by those compiling health claim petitions. The scientific principles described in this document should also be useful to those designing studies to support health claim petitions.
FDA's determination on significant scientific agreement represents the agency's best judgment as to whether qualified experts would likely agree that the scientific evidence supports the substance/disease relationship that is the subject of a proposed health claim. The significant scientific agreement standard is intended to be a strong standard that provides a high level of confidence in the validity of a substance/disease relationship. Significant scientific agreement means that the validity of the relationship is not likely to be reversed by new and evolving science, although the exact nature of the relationship may need to be refined. Application of the significant scientific agreement standard is intended to be objective, in relying upon a body of sound and relevant scientific data; flexible, in recognizing the variability in the amount and type of data needed to support the validity of different substance/disease relationships; and responsive, in recognizing the need to re-evaluate data over time as research questions and experimental approaches are refined. Significant scientific agreement does not require a consensus or agreement based on unanimous and incontrovertible scientific opinion. However, on the continuum of scientific discovery that extends from emerging evidence to consensus, it represents an area on the continuum that lies closer to the latter than to the former.
Before significant scientific agreement can be assessed, a number of sequential threshold questions are addressed in the review of the scientific evidence:
The assessment of significant scientific agreement then derives from the conclusion that there is a sufficient body of sound, relevant scientific evidence that shows consistency across different studies and among different researchers and permits the key determination of whether a change in the dietary intake of the substance will result in a change in a disease endpoint.
The specific topics addressed in this guidance document are: identifying data for review, performing reliable measurements, evaluating individual studies, evaluating the totality of the evidence, and assessing significant scientific agreement. Other aspects of and requirements for the health claim authorization process are described in the Code of Federal Regulations, in 21 CFR § 101.14 and 21 CFR § 101.70.
Major considerations in the scientific review process for health claims are highlighted in bold-face type. For each step in the process, details of the issues that should be considered are provided. Explanatory comment, illustrative discussion points, and examples of application of criteria or requirements, as demonstrated by past health claim authorization reviews, are provided in italics.
The Nutrition Labeling and Education Act of 1990 (NLEA) was designed to give consumers more scientifically valid information about the foods they eat (1). Among other provisions, NLEA authorized FDA to allow statements that describe the relationship between a nutrient and a disease or health-related condition to appear in the labeling of foods, including dietary supplements. Such statements about substance/disease relationships are known as "health claims." FDA has defined the term "substance" by regulation as a specific food or component of food. An authorized health claim may be used on both conventional foods and dietary supplements, assuming that the substance in the product and the product itself meet the appropriate standards. Health claims are directed to the general population or designated subgroups (e.g., the elderly) and are intended to assist the consumer in maintaining healthful dietary practices.
When FDA decides whether to authorize a health claim, it evaluates, among other considerations, whether the evidence supporting the relationship that is the subject of the claim meets the significant scientific agreement standard. This standard derives from 21 U.S.C. § 343(r)(3)(B)(i), which provides that FDA shall authorize a health claim to be used on conventional foods if the agency "determines, based on the totality of the publicly available scientific evidence (including evidence from well-designed studies conducted in a manner which is consistent with generally recognized scientific procedures and principles), that there is significant scientific agreement, among experts qualified by scientific training and experience to evaluate such claims, that the claim is supported by such evidence." This scientific standard applies to conventional food health claims by statute; FDA applied the same standard to dietary supplement health claims by regulation. See 21 CFR § 101.14(c).
The NLEA identified 10 substance/disease relationships for initial consideration(1). Of these, significant scientific agreement was determined to exist for eight of the relationships, and health claims describing these relationships on food labels were authorized in 1993. The legislation also permits any interested person to petition FDA to issue a regulation regarding a health claim. Additional health claims have been authorized in response to such petitions.(1)
Since NLEA was enacted, several groups have evaluated the health claim authorization process, including the interpretation of significant scientific agreement. One of these evaluations was a 2-year Keystone Center dialogue among representatives from academia, industry, consumer groups, and government. The dialogue and resulting report affirmed the principles and approach FDA had been using to authorize health claims(2). The Commission on Dietary Supplement Labels examined the health claim authorization process for dietary supplements and also generally expressed agreement with FDA's approach in its report (3). Following the Keystone dialogue, the FDA FAC convened a number of working groups in 1996 to address issues raised and recommendations made during the dialogue. The FAC Working Group on Significant Scientific Agreement was charged with developing a guide for preparing health claim petitions. In response to the recent decision of the United States Court of Appeals for the District of Columbia Circuit in Pearson v. Shalala, 164 F.3d 650 (D.C. Cir. 1999), which required FDA to clarify the meaning of significant scientific agreement, the focus of the FAC Working Group shifted to the scientific review of data for health claims and the interpretation of the significant scientific agreement standard. The final report of the FAC Working Group on Significant Scientific Agreement, entitled "Interpretation of Significant Scientific Agreement in the Review of Health Claims," was made public during the FAC meeting of June 24-25, 1999. (See http://vm.cfsan.fda.gov/~dms/facssa.html for a copy of the Working Group's report.) Following additional comment by the FAC, FDA adopted the recommendations proposed by the Working Group on Significant Scientific Agreement. This guidance document is based on the FAC Working Group report but has been expanded and edited to clarify and more fully explain some topics. The guidance represents the agency's current thinking on the meaning of significant scientific agreement in 21 U.S.C. § 343(r)(3)(B)(i) and 21 CFR § 101.14(c).
Footnote:
The scientific review process FDA uses to evaluate health claims is comprehensive and focuses first on review of individual studies. After identifying relevant, good quality studies and assessing their strengths and weaknesses, the agency conducts a more comprehensive review based on the body of evidence as a whole. Considerations in the scientific review of health claims are detailed below.
The standard of scientific validity for a health claim includes two components: 1) that the totality of the publicly available evidence supports the substance/disease relationship that is the subject of the claim, and 2) that there is significant scientific agreement among qualified experts that the relationship is valid.
FDA's evaluation of the evidence supporting a health claim is based on the totality of publicly available data. Because of the limitations of the various research methods that can be used to study substance/disease relationships, it is not possible to specify the type or number of studies needed to support a health claim. In addition, each relationship involves a unique set of confounders (see discussion below) and measurement issues.
Sound, relevant science in research design and measurement -- to ensure that research, in fact, provides the answers to the questions that need to be addressed concerning the relationship -- drives the decision to authorize health claims, not the specific type or number of studies. This point is illustrated graphically in Figure 1, which shows the number and nature of the human studies evaluated in determining the validity of certain of the initial health claims evaluated during the 1990-1992 review and claims for which petitions were submitted. The number and types of studies considered varied greatly among authorized claims.
In addition to limitations imposed by available research methods, another limitation frequently encountered is the dependence on publicly available data derived from studies that were not specifically designed or conducted for the purpose of supporting a health claim. Thus, in the agency's review of health claims, the usefulness, relevance, and generalizability of such studies to the health claim under consideration are carefully evaluated, especially in terms of specification and measurement of the substance and disease whose relationship is the subject of the claim.
The first step in preparing or reviewing a health claim petition is to identify all relevant studies.
The types of studies considered in a health claim review include human studies and frequently also include "pre-clinical" evidence, e.g., in vitro laboratory investigations and other mechanistic studies. Studies of humans can be divided into two types: interventional studies and observational studies.
In an interventional study, the investigator controls whether the subjects receive an exposure or an intervention whereas in an observational study, the investigator does not have control over the exposure or the intervention. In general, interventional studies provide the strongest evidence for an effect.
Regardless of the inherent strengths and weaknesses of a study design, the overall quality and relevance of each individual study is paramount in assessing its contribution to the weight of the evidence for the proposed substance/disease relationship.
The "gold standard" of interventional studies is the randomized controlled clinical trial.
In a randomized controlled trial, subjects similar to each other are randomly assigned either to receive the intervention or not to receive the intervention. As a result, subjects who are most likely to have a favorable outcome independent of any intervention are not preferentially selected to receive the intervention being studied (selection bias). Bias may be further reduced if the researcher who assesses the outcome does not know which subjects received the intervention (blinding). Randomized controlled clinical trials are not an absolute requirement to demonstrate significant scientific agreement in all cases, but are considered the most persuasive and given the most weight. A single large, well-conducted and controlled clinical trial could provide sufficient evidence to establish a substance/disease relationship, provided that there is a supporting body of evidence from observational or mechanistic studies.
Interventional studies for foods may differ from those for drugs. Unlike drug studies, food interventional trials may have additional confounders secondary to using a food substance as the intervention (see discussion below). In addition, it may not be possible to use a placebo control group for food studies, and subjects in such studies may not be blinded to the intervention. As a result of the greater likelihood for confounders and bias, interventional studies with foods may generate data that have less certainty than data from drug interventional studies.
Although interventional studies are the most reliable category of studies for determining cause-and-effect relationships, generalizing from selected populations often presents serious problems in the interpretation of such studies. Furthermore, in some cases, such as with cancers of different sites, interventional dietary studies are not feasible because diseases with lower frequency of occurrence, such as rare forms of cancer, require very large study samples to detect an effect. Moreover, there frequently are long delays from dietary exposure to onset of disease, often 20 to 30 years. Therefore, the scientific evidence supporting a substance/disease relationship may have to be derived wholly or in part from observational studies.
There is no universally valid method for weighing categories of observational studies. However, in general, observational studies include, in descending order of persuasiveness, cohort (longitudinal) studies, case-control studies, cross-sectional studies, uncontrolled case series or cohort studies, time-series studies, ecological or cross-population studies, descriptive epidemiology, and case reports.
Observational studies may be prospective or retrospective. In prospective studies, investigators recruit subjects and observe them prior to the occurrence of the outcome. In retrospective studies, investigators review the records of subjects and interview subjects after the outcome has occurred. Retrospective studies are usually considered to be more vulnerable to recall bias (error that occurs when subjects are asked to remember past behaviors) and measurement error but are less likely to suffer from the subject selection bias that may occur in prospective studies.
A common weakness of observational studies is the limited ability to ascertain the actual food or nutrient intake for the population studied. Observational data are also generally restricted to identifying associations between food substances and health outcomes, and often do not provide a sufficient basis for determining whether a substance/disease association reflects a causal rather than a coincidental relationship.
"Research synthesis" studies, including meta-analyses, may be useful as supporting evidence for a health claim, but any role beyond this function is as yet unresolved.
The appropriateness of research synthesis studies to establish substance/disease relationships is not known. This is especially true when observational data are entered into meta-analyses. Discussions on the topic have been published (4-7), and there are on-going efforts to identify criteria and critical factors to consider in both conducting and using such analyses, but standardization of this methodology is still emerging. Therefore, in general, such analyses serve as supporting evidence rather than as primary evidence. To date, while meta-analyses have been reviewed as part of the health claim authorization process, no health claims have been authorized on the basis of meta-analysis studies alone.
Although human studies are weighted most heavily in the evaluation of evidence on a substance/disease relationship, data from animal model and in vitro (laboratory) studies also can be used to support a substance/disease relationship.
Lacking any data from human studies, animal and in vitro studies alone would not adequately support a health claim. Although both types of studies permit greater control over variables, such as diet and genetics, and permit more aggressive intervention, each suffers from the uncertainties of extrapolating to physiological effects in humans. However, these studies can be useful in providing information on the mechanism of action and specificity of a food substance and the process that causes a disease or health-related condition. Animal and in vitro studies should be considered when there are problems designing interventional studies or in the absence of an appropriate biomarker. If such studies are used, they are subjected to the same kind of assessment as the human studies. In the case of animal studies, the consistency of the demonstrated association between a substance and the disease or health-related condition is important when considering whether evidence from such studies supports a health claim. Thus, the strongest animal evidence would be based on data derived from studies on appropriate animal models, on data that have been reproduced in different laboratories, and on data that give a statistically significant dose-response relationship.
Appropriate measurement, of both the substance and the disease or health-related condition, is a key factor in the review of data for health claims.
Assessing the effects of diet on human health is limited by a variety of measurement issues: the use of biomarkers, the difficulty of identifying and measuring the food substance that provides the effect, the difficulty of accurately measuring dietary intake, and the difficulty of distinguishing the effects of diet on a disease from those of other variables, such as weight change, physical activity, or environmental factors.
Because a number of the diseases associated with dietary factors are diseases that develop over a period of many years (chronic diseases), a person may not show outward signs or symptoms of a disease at a particular stage of the illness even though that person has the disease. For example, individuals may have deposits of fat and other material accumulating in the arteries to their hearts (atherosclerotic coronary heart disease) and not experience any symptoms until years later when they suffer a heart attack. Therefore, scientists seek to identify "biomarkers" (intermediate or surrogate endpoint markers) for the presence or risk of disease.
A biomarker is a measurement of a variable related to a disease that may serve as an indicator or predictor of that disease. Biomarkers are parameters from which the presence or risk of a disease can be inferred, rather than being a measure of the disease itself. In conducting a health claim review, FDA does not rely on a change in a biomarker as a measurement of the effect of a dietary factor on a disease unless there is evidence that altering the parameter can affect the risk of developing that disease or health-related condition. This is the case for serum cholesterol in that high levels are generally accepted as a predictor of risk for coronary heart disease, and there is evidence that decreasing high serum cholesterol can decrease that risk. Therefore, the evaluation of whether decreasing the intake of dietary fat reduces the risk of developing heart disease took into account many studies that assessed changes in serum cholesterol, specifically LDL-cholesterol, rather than the development of heart disease per se. For the existing authorized health claims, acceptable biomarkers are LDL-cholesterol levels for coronary heart disease, measures of bone mass for osteoporosis, and measures of blood pressure for hypertension.
The measurement of a food substance centers on the following questions: 1) What was measured? and 2) How does the measured substance relate to the substance that is the subject of the health claim?
Studies that examine dietary components often focus on the intake of the substance of interest as part of a food or a total diet, or may infer intake as part of post-hoc evaluations of the data. Therefore, isolating the effect of the substance can be a critical consideration in authorizing a health claim. Common difficulties involve separating the effect of the food substance from the food itself, or the use of measures that reflect heterogeneous or poorly defined food substances. Without evidence that the substance, rather than the overall diet or specific foods in the diet, is responsible for the benefit, the linkage between the substance and the disease cannot be established.
FDA applied this principle during evaluations of the initial 10 substance/disease relationships in 1990-1992. In the case of claims related to omega-3 fatty acids, fiber, and antioxidant vitamins, there was considerable measurement overlap between the food containing the substance and the substance itself, or there were concomitant changes in other dietary components. Fiber was poorly defined and/or a heterogeneous mixture as measured in research available at the time of the initial health claim review. For example, as noted during the health claim review for fiber and heart disease, the objective of the protocols of many studies was to evaluate the effectiveness of relatively large amounts of a single type of food or fiber source rich in soluble fiber (e.g., baked beans), rather than to examine total soluble dietary fiber intakes or to specifically identify the chemical and physical characteristics of soluble fiber that are most effective in lowering blood cholesterol levels. Thus, the effects could not be attributed to the fiber. Moreover, in some studies large amounts of foods (e.g., 1-2 cups of baked beans) were added to diets; these dietary changes were often accompanied by lower calorie intakes with resultant weight loss, which has an independent impact on the risk of developing heart disease.
Measurement issues generally focus on substances in food, but the same principles apply when the substance of interest is itself a food. While a single food can be the subject of a health claim, existing experience is that the subject is more likely to be a group of foods, such as fruits, vegetables, and grains, which have been associated with a reduced risk of heart disease and of cancer. This identification, and consequently measurement, of a food group is, in turn, most likely to occur because it is not possible to identify and, therefore, measure a particular component of these foods that is responsible for the benefit. Nonetheless, in theory, it is possible that a unique combination of nutrients or other substances in a single food could be the subject of a health claim. To date, this has not occurred.
In determining whether a substance that is the subject of a claim has been measured appropriately, it is important to evaluate critically the method of assessment of dietary intake. Each method has its strengths and weaknesses. No one method is adequate for every purpose.
Dietary intake assessment methods include food records, 24-hour recalls, and diet histories. Food records are based on the premise that food weights provide an accurate estimation of food intake. Subjects weigh the foods they consume and record those values. The 24-hour recall method requires that subjects describe which foods and how much of each food they consumed during the prior 24-hour period. Diet histories use questionnaires or interviewers to estimate the typical diet of subjects over a certain period of time. For a more detailed description of these methods and their strengths and weaknesses, see Diet and Health (8). Some common problems that weaken confidence in the assessment of dietary intake may be noted. For example, a single 24-hour recall is generally regarded as an inadequate method for assessing the usual intake of a nutrient or other food substance by an individual, although it may be useful for assessing mean intake of a group. A diet history taken by a food frequency questionnaire that contains a limited number of items is inadequate for assessing intake of a specific nutrient if the major food sources of the nutrient in the population studied are not included in the questionnaire. Finally, accurate estimation of the intake of a nutrient or other food substance derived from any type of intake assessment is also dependent on the availability of valid and complete food composition databases for the nutrient or other substance of interest.
Scientific studies provide the means to identify which effects on a disease or health-related condition result from the consumption of a particular food substance and which effects are the products of other factors. Evaluating the conclusions of a study requires an assessment of both the design and conduct of the study, as well as the methods used to interpret the data obtained from the study. Appropriate control of potential confounding factors, by eliminating as many as possible in interventional studies and by adjusting for them with appropriate data analysis techniques in observational studies, is needed if studies are to contribute substantively to the weight of evidence in support of a substance/disease relationship.
The evaluation of study design, protocol, measurement, and statistical issues for individual studies serves as the starting point from which FDA determines the overall strengths and weaknesses of the data and assesses the weight of the evidence.
FDA's review of individual studies on substance/disease relationships generally follows the approaches outlined in the Guide to Clinical Preventive Services (9) and Diet and Health (8).
The persuasiveness of a study depends on the quality of the study.
Evaluation of the quality of individual studies on substance/disease relationships
begins with a consideration of the inherent strengths and weaknesses of various study
designs. The three most important measures of the quality of a study are design, conduct,
and analysis and interpretation.
Certain study designs tend to be more persuasive because they are less subject to bias and measurement error. As noted earlier, retrospective studies are usually considered to be more vulnerable to recall bias and measurement error but are less likely to suffer from the subject selection bias that may occur in prospective studies. Different degrees of persuasiveness may also be assigned within classes of studies, depending on the particular assessments made. For example, case-control studies in which higher or lower serum levels of a nutrient or metabolite are found in cases versus controls will generally be less persuasive in establishing a substance/disease relationship than similar studies that assess an antecedent behavior (such as dietary intake), despite the potential for recall bias, because such studies cannot distinguish whether the high or low serum level of the nutrient was a contributing cause or a consequence of the disease.
The susceptibility of research data to bias and confounding depends on several factors, including the methods used to choose subjects and to measure outcomes, the use of a comparison (control) group, and whether the study was conducted retrospectively or prospectively. Confounders are factors that are associated with the disease in question and the intervention, and that prevent the measured outcome from being attributed unequivocally to the intervention.
Several aspects of substance/disease relationships may give rise to confounders. Foods are rarely composed of a simple mixture of chemical constituents. The addition of a nutrient to a diet, or an increase in total daily intake of that nutrient, may have unintended effects. The added nutrient may displace other nutrients in the diet. Therefore, it may be difficult to ascertain whether the health outcome is the result of the added nutrient or the related changes on the original diet. For example, weight loss was a confounder in a number of studies used to support a claim that lowering of dietary saturated fat intake and resultant decreases in serum LDL-cholesterol led to a reduced risk of coronary heart disease. Diets low in fat can result in a lower calorie intake and, in turn, weight loss. Since weight loss per se can reduce levels of LDL-cholesterol, the benefit in those studies could not be attributed to the lack of the food substance (saturated fat), but may have been related to lower calorie intake. Nonetheless, sufficient studies that did control for such related factors were available and there was adequate evidence to establish a relationship between diets low in saturated fat and cholesterol and reduced risk of heart disease. Other potential confounders include variability in the quantity or quality of the food substance being administered.
Criteria that are considered in assessing the quality of individual studies of substance/disease relationships include the following:
Were the questions to be answered by the study clearly described at the outset?
Was the methodology used in the study clearly described and appropriate for answering the questions posed by the study?
Was the duration of the study intervention or follow-up period sufficient to detect an effect on the outcome of interest?
Were potential confounding factors identified, assessed, and/or controlled?
Was subject attrition (subjects leaving the study before the study is completed) assessed, explained, and reasonable?
Was the sample size large enough to provide sufficient statistical power to detect a significant effect? (If the study is underpowered, it may be impossible to conclude that the absence of an effect is not due to chance.)
Was the study population representative (for factors such as age, gender distribution, race, socioeconomic status, geographic location, family history, health status, and motivation) of the population to which the health claim will be targeted?
Were criteria for inclusion and exclusion of study subjects clearly stated and appropriate?
Were recruitment procedures that minimized selection bias used?
For controlled interventions, were subjects randomized? If matching was employed to assign the subjects to control and treatment groups, were appropriate demographic characteristics and other variables used for the matching? Was randomization successful in producing similar control and intervention groups?
Were analytical methodology and quality control procedures to assess dietary intake adequate?
Was the dietary intervention or exposure well defined and appropriately measured? (See discussion above.)
For intervention studies, was an appropriate level of intake (i.e., the level hypothesized to be effective) for the food substance of interest planned, monitored, and achieved?
Were the background diets to which the test substance was added, or the control and interventional diets, adequately described, measured, and suitable?
Was a "lead-in" period employed for dietary interventions? (Because changes in the diet may induce compensatory metabolic changes, the effect of an intervention should be measured after stabilization has occurred, i.e., a lead-in period.)
In studies with cross-over designs, was there an appropriate "wash-out" period (period during which subjects do not receive an intervention) between dietary treatments? (Lack of a sufficient wash-out period between interventions may lead to confusion as to which intervention produced the health outcome.)
Were the form and setting of the intervention representative of the "real world?"
Were other possible concurrent changes in diet or health-related behavior (weight loss, exercise, alcohol intake, smoking cessation) during the study that could account for the outcome identified, assessed, and/or controlled?
Were the disease outcomes well defined and appropriately measured? If biomarkers (intermediate or surrogate endpoint markers) were measured, has their relevance to disease outcomes been validated?
Were efforts made to detect harmful as well as beneficial effects? (For example, increasing the consumption of some food substances may increase the risk of a chronic disease, and extracting or concentrating some food substances may render them injurious to health.)
Were appropriate statistical analyses applied to the data?
Was "statistical significance" interpreted appropriately? (For example, differences that are not statistically significant should be described as not demonstrating a difference rather than as showing a trend.)
Were relative and absolute effects distinguished?
As part of the review process, FDA creates a summary of the scientific evidence to help organize and guide its comprehensive review. FDA recommends that health claim petitions include a summary of the evidence describing the individual studies in table form. Such summaries help speed agency review of the petition.
Evaluating the totality of the evidence means evaluating whether it permits the key determination of whether a change in the dietary intake of the substance will result in a change in a disease endpoint.
After identifying relevant, good quality studies and assessing and summarizing their
strengths and weaknesses, FDA conducts a more comprehensive review based on the body of
evidence as a whole. Petitioners should be sure that the conclusions the petition draws
regarding the association between nutritional exposures or interventions and outcomes are
objectively based on the totality of the evidence, and that interpretations are limited to
the research conducted, without inappropriate extrapolations beyond the available
evidence.
A classic set of reviews that demonstrate an appropriate process for evaluating
substance/disease relationships is the work conducted by The Task Force on The Evidence
Relating Six Dietary Factors to the Nation's Health (10). Its approach incorporated the
standard principle that the strength of evidence associating a nutritional exposure with a
health outcome depends not only on the quality of the individual studies but on the
overall "grade" or assessment of the evidence taken together, the number of studies,
consistency of results, and the magnitude of effects.
The strength of evidence that exposure to a particular food substance is associated with a health outcome depends on several factors.
The design category and the quality of the research methodology should be considered together. Various coding and scoring schemes have been devised to systematize this process. The U.S. Preventive Services Task Force's grading system assigns a letter code to rate the quality of the evidence (9). Other groups have developed systems that score a study quantitatively, assigning points for different aspects of design quality and performance (11). However, although both study design codes and quantitative scores are appropriate for rating individual studies, they do not adequately describe the evidence as a whole. For example, these methods do not capture the number of studies or consistency of findings. At present, a universally applicable system for evaluation of the evidence as a whole is not available.
The number of studies required to be persuasive is often inversely related to the overall class of evidence available. Simply counting the studies with positive results without regard for their individual quality is an inadequate approach to assessing the overall strength of the evidence.
Conflicting results do not disprove an association (because elements of the study design may account for the lack of an effect in negative studies) but do tend to weaken confidence in the strength of the association. In general, the greater the consistency, the more likely the significant scientific agreement standard will be met. However, repetition of a poorly designed study does not add to the consistency or quality of the evidence.
Evidence of an association does not, however, prove cause and effect. An association of variables only indicates that they occur together but not that one causes the other. Therefore, another step in the process of a health claim review is to determine the strength of the evidence for a causal relationship.
A causal relationship exists when data show that the consumption of a substance increases or decreases the probability of developing or not developing a particular disease or health-related condition. Causality can be best established by interventional data, particularly from randomized, controlled clinical trials, that show that altering the intake of an appropriately identified and measured substance results in a change in a valid measure of a disease or health-related condition. In the absence of such data, a causal relationship may be inferred based on observational and mechanistic data through strength of association, consistency of association, independence of association, dose-response relationship, temporal relationship, effect of dechallenge, specificity, and explanation of a pathogenic mechanism or a protective effect against such a mechanism (biological plausibility). Although these features strengthen the claim that a substance contributes to a certain health outcome, they do not prove that eating more or less of the substance will produce a clinically meaningful outcome. In many cases (for example, if the intake of the substance has not been or cannot be assessed adequately in available observational studies because it has not been commonly consumed or its intake cannot be assessed independently of other substances), controlled clinical trials are necessary to establish the validity of a substance/disease relationship.
In assessing whether the totality of the evidence supports the substance/disease relationship that is the subject of the claim, FDA asks two questions:
Assessing significant scientific agreement relies on judging the extent of agreement among qualified experts.
Significant scientific agreement refers to the extent of agreement among qualified experts in the field. In the process of scientific discovery, significant scientific agreement occurs well after the stage of emerging science, where data and information permit an inference, but before the point of unanimous agreement within the relevant scientific community that the inference is valid. The significant scientific agreement standard is met when the validity of the relationship is not likely to be reversed by new and evolving science, although the exact nature of the relationship may need to be refined over time. Significant scientific agreement can be achieved when the validity of a substance/disease relationship is supported by the conclusions of federal government scientific bodies; conclusions of independent, expert bodies may also be relevant. When such conclusions are not available (for instance, if the data supporting a proposed health claim are relatively new and have not yet been reviewed by an independent, expert panel or body), a compelling and relevant body of evidence may nonetheless cause the agency to conclude that significant scientific agreement exists.
Although significant scientific agreement is not consensus in the sense of unanimity, it represents considerably more than an initial body of emerging evidence. Because each situation may differ with the nature of the claimed substance/disease relationship, it is necessary to consider both the extent of agreement and the nature of the disagreement on a case-by-case basis. If scientific agreement were to be assessed under arbitrary quantitative or rigidly defined criteria, the resulting inflexibility could cause some valid claims to be disallowed where the disagreement, while present, is not persuasive.
In order for qualified experts to reach an informed opinion regarding the claim, the
data and information that pertain to the claim must be available to the relevant
scientific community.
The usual mechanism to show that the evidence is available to qualified experts is that
the data and information are published in peer-reviewed scientific journals. Abstracts
generally provide insufficient information for review; however, not all the data need be
published. FDA reviews information that is not published as long as that information is
placed in the public domain at the time the agency takes action on a health claim
petition. The value of an expert's opinion will be limited if he/she did not have access
to all the evidence.
Significant scientific agreement cannot be reached without a strong, relevant, and consistent body of evidence on which experts in the field may base a conclusion that a substance/disease relationship exists. There is considerable potential for incorrect conclusions if only preliminary evidence (emerging science) is available for review.
This is best illustrated by the body of evidence for the association between beta-carotene and cancer risk. At the time of FDA's health claim review, no results from relevant clinical trials had been reported. However, human epidemiological studies were available, as well as laboratory data for mechanistic theories on how beta-carotene might provide a risk reduction effect. While there was strong evidence that high intakes of fruits and vegetables rich in carotenoids were associated with a reduced risk of developing cancer, it was unclear whether the component(s) of fruits and vegetables responsible for reducing the effect were beta-carotene, other carotenoids, or some other compound(s). However, animal studies strongly pointed to a positive effect of beta-carotene in lowering the frequency and severity of experimental cancer induced in animals. The review concluded, nonetheless, that existing evidence was inconclusive and significant scientific agreement did not exist; the animal studies could not be applied directly to humans because the type and amount of carcinogen exposure in the experimental conditions were not similar to human exposure. Subsequently, the decision was further supported when a randomized, controlled trial in Finland tested the ability of antioxidant vitamins, including beta-carotene, to prevent the development of lung cancer in high-risk Finnish men with a history of smoking (12). The unexpected outcome was a significant increase in the rate of lung cancer among the beta-carotene supplemented group.
Figure 2 provides a graphical representation of the interplay of considerations that contribute to determining whether the significant scientific agreement standard for a substance/disease relationship has been met. It illustrates the manner in which evaluations of the various types and amounts of data that may exist for a substance/disease relationship are combined to assess the overall strength and consistency of the scientific evidence. The schema also demonstrates that the significant scientific agreement standard is one that is objective, flexible, and responsive by illustrating the variety of combinations of data from different types of good quality studies that may give rise to a body of evidence sufficient to establish the validity of a substance/disease relationship.
In determining whether there is significant scientific agreement, FDA takes into account the viewpoints of qualified experts outside the agency, if evaluations by such experts have been conducted and are publicly available. For example, FDA will take into account:
FDA accords the greatest weight to the conclusions of federal government scientific bodies, especially when the evidence for the validity of a substance/disease relationship has been judged by such a body to be sufficient to justify dietary recommendations to the public. Although reviews by individual outside experts are considered in assessing significant scientific agreement, evidence from such reviews alone would not necessarily support a conclusion that the standard has been met, especially if the conclusions of such reviews were not supported by available assessments of the same body of evidence from federal scientific bodies, expert panels or independent, expert bodies. Reviews by outside experts or expert panels are most useful when there is a reasonable basis to conclude that they represent the larger group of qualified experts in the field. Most importantly, the relevance of an outside expert review depends on whether the evidence examined applies to the claim in terms of considerations such as specification and measurement of the substance and the disease or health-related condition.
1. Public Law 101-553, 104 Stat. 2353 codified at 21 USC § 343 (1994). Nutrition Labeling and Education Act. November 8, 1990.
2. The Keystone National Policy Dialogue on Food, Nutrition, and Health: Final Report. Keystone, CO: Keystone Press, 1996.
3. Commission on Dietary Supplement Labels. Commission of Dietary Supplement Labels Report to the President, Congress, and the Secretary of the Department of Health and Human Services. Washington, DC: Office of Disease Prevention and Health Promotion, DHHS, 1997.
4. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N Engl J Med 1987;316:450-455.
5. Sacks HS, Berrier J, Reitman D, Pagano D, Chalmers TC. Meta-analysis of randomized controlled trials: an update. In: Balder WC, Mosteller F, eds. Medical Uses of Statistics, 2nd ed. pp 427-442. Boston, MA: NEJM Books, 1992.
6. Sacks HS. Meta-analyses of clinical trials. In: Perman JA, Rey J, eds. Clinical Trials in Infant Nutrition, Nestle Nutrition Workshop Series, Vol 40, pp 85-99. Philadelphia, PA: Vevey/Lippincott-Raven Publishers, 1998.
7. Hasselblad V, Mosteller F, Littenberg B, Chalmers TC, Hunick MG, Turner JA, et al. A survey of current problems in meta-analysis. Discussion from the Agency for Health Care Policy and Research Inter-PORT Work Group on Literature Review/Meta-Analysis. Med Care 1995;33:202-220.
8. National Research Council. Diet and Health: Implications for Reducing Chronic Disease Risk. Washington, DC: National Academy Press, 1989.
9. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. Report of the US Preventive Services Task Force: Guide to Clinical Preventive Services. 2nd ed. Washington, DC: Office of Public Health and Science, April, 1989.
10. Ahrens EH, Connor WE, eds. Symposium: Report of the Task Force on the Evidence Relating Six Dietary Factors to the Nation's Health. Am J Clin Nutr 1979;23(suppl):2621-2748.
11. Mohar D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials: current issues and future directions. Int J Technol Assess Health Care 1996;12:125-208.
12. Albanes D, Heinonen OP, Huttunen JK, Taylor PR, Virtamo J, Edwards BK, Haapakoski J, Rautalahti M, Hartman AM, Palmgren J, et al. Effects of alpha-tocopherol and beta-carotene supplements on cancer incidence in the Alpha-Tocopherol Beta-Carotene Cancer Prevention Study. Am J Clin Nutr 1995;62(6 Suppl):1427S-1430S.
This document was issued on December 22, 1999.
For more recent information on Dietary Supplements
See
http://www.cfsan.fda.gov/~dms/supplmnt.html
Hypertext updated by ear/kwg 2000-NOV-22