|
July 10, 2003
Contains Nonbinding Recommendations
Comments and suggestions regarding this document may be submitted at any time. Submit comments to the Division of Dockets Management (HFA-305), Food and Drug Administration, 5630 Fishers Lane, rm. 1061, Rockville, MD 20852. All comments should be identified with the docket number listed in the notice of availability that publishes in the Federal Register.
For questions regarding this draft document contact the Center for Food Safety and Applied Nutrition (CFSAN) at 301-436-1450.
Additional copies are available from:
Office of Nutritional Products, Labeling, and Dietary Supplements
Division of Nutrition Programs and Labeling HFS-800
Center for Food Safety and Applied Nutrition
Food and Drug Administration
5100 Paint Branch Parkway
College Park, MD 20740
(Tel) 301-436-1450
http://www.cfsan.fda.gov/guidance.html
U.S. Department of Health and Human Services
Food and Drug Administration
Center for Food Safety and Applied Nutrition (CFSAN)
July 2003
| This guidance represents the Food and Drug Administration's (FDA's) current thinking on this topic. It does not create or confer any rights for or on any person and does not operate to bind FDA or the public. You can use an alternative approach if the approach satisfies the requirements of the applicable statutes and regulations. If you want to discuss an alternative approach, contact the FDA staff responsible for implementing this guidance. If you cannot identify the appropriate FDA staff, call the appropriate number listed on the title page of this guidance. |
TABLE OF CONTENTS
This guidance is intended to notify the public of the Food and Drug Administration's (FDA) interim evidence-based ranking system that is a process designed to lay a foundation for a more detailed system to be used permanently. This guidance describes a process that FDA intends to use, on an interim basis, to evaluate and rank the scientific evidence in support of a substance/disease relationship that is the subject of a qualified health claim until the agency can promulgate regulations under notice-and-comment rulemaking. Based on this process, the agency will categorize the qualified health claim into one of three levels (i.e., a "B", "C", or "D" level). This guidance does not apply to unqualified health claims, which must meet the "Significant Scientific Agreement" (SSA) standard.(2)
FDA's guidance documents, including this guidance, do not establish legally enforceable responsibilities. Instead, guidances describe the Agency's current thinking on a topic and should be viewed only as recommendations, unless specific regulatory or statutory requirements are cited. The use of the word should in Agency guidances means that something is suggested or recommended, but not required.
This interim ranking system provides criteria to rank scientific evidence relevant to substance/disease relationships that are the subject of qualified health claims. It outlines the major concepts the agency intends to consider in guiding the scientific evaluation.
The primary purpose of this guidance is to provide petitioners with a description of the major points the agency intends to consider in evaluating supporting scientific data.
What is an Evidence-based Rating System?
An evidence-based rating system is a science-based systematic evaluation of the strength of the evidence behind a statement. In the case of health claims, it would rate the strength of the evidence behind a proposed substance/disease relationship. A large number of evidence-based rating systems are currently in use today by physicians, dietitians and other health professionals.(3) FDA has tentatively chosen to model its evidence-based rating system on that of the Institute for Clinical Systems Improvement (ICSI)(4) as adapted by the American Dietetic Association(5) with modifications specific to FDA. In making this tentative decision, FDA relied on criteria for evaluating evidence-based rating systems as reviewed and critiqued by the Agency for Healthcare Research and Quality (AHRQ).(3) FDA also found the modifications from the American Dietetic Association to be particularly useful as they considered diet and health relationships, whereas other groups focused on drug and treatment applications.
How are "Rate" and "Rank" Used in this System?
The terms "rate" and "rank" are not used interchangeably to describe this system. The evaluation process involves three separate rating systems: (1) a rating for study design; (2) a rating for study quality; and (3) a rating for the strength of the entire body of evidence. Considering all classifications from the three rating systems, a final rank of the scientific evidence in support of a health claim would be assigned.
What are the Parts of an Evidence-based Rating System?
In order to evaluate the level of scientific support for a proposed substance/disease relationship, the agency intends to follow a six-part procedure.
Each part of the evidence-based rating system is described below:
Define the substance(6) /disease relationship
A proposed relationship between a substance and a disease or health-related condition is identified. If relevant, the subgroups within the general population, for which the relationship is targeted are identified. The relationship forms the basis for selecting relevant studies and for evaluating the quality of the selected studies.
Collect and submit all relevant studies
All relevant studies (both favorable and unfavorable) to the relationship to be tested (as defined above in C.1.) are collected and submitted. The evaluation of the proposed relationship relies primarily on human studies.
Classify, and therefore rate, each study as to type of study
Each study would be characterized as a study design type.(7) By categorizing the study, it automatically receives an initial study "rating" based on the type of experimental design, which is independent of the quality of the study. The rating of study design is based on the principle of minimizing bias.(8) Only primary reports of data collection are rated. Reports that synthesize or reflect collections of primary reports are not considered part of the rating system although they may provide useful background information.
Rate each study for quality
Each study would be reviewed independently and assigned a quality factor of +, Ø, - or N/A. The basis for the assignment of the quality factor is discussed below. (9)
(+) means the report has adequately addressed issues of scientific quality such as inclusion/exclusion, bias, generalizability, and data collection and analysis.
(Ø) means some uncertainties exist as to whether the report has adequately addressed issues of scientific quality such as inclusion/exclusion, bias, generalizability, and data collection and analysis.
(-) means the report has not adequately addressed issues of scientific quality such as inclusion/exclusion, bias, generalizability, and data collection and analysis.
N/A means the report is not a primary reference, therefore the quality has not been assessed, and such a reference is not considered as part of the body of evidence on which the final ranking is based. Examples of non-primary references are review articles and meta analyses.
Rate the strength of the total body of evidence
The studies are considered collectively across the evidence base in order to rate the strength of the body of evidence. The rating system is based on three factors: quantity, consistency, and relevance to disease risk reduction in the general population or target subgroup. These three factors and the final "rank" for the strength of the evidence for the "relationship" are described below.
Rating the body of evidence for quantity, consistency, and relevance to disease risk reduction in the general population or target subgroup.
Quantity. Considers the number of studies, the total number of individuals studied and the generalizability of the findings to the target population.
(***) means the number of studies and the number of individuals tested (from all studies of design types one and two that are of high quality (+) combined) are sufficiently large to comfortably generalize to the target population.
(**) means there are a sufficient number of studies and individuals tested from study design type three and higher (i.e., study design types one and two) of at least moderate quality (Ø) but uncertainties remain as to generalizability to the target population.
(*) means that the number of studies and the number of individuals tested is insufficient to generalize to the target population.
Consistency. Considers whether studies with both similar and different designs report similar findings.
(***) means a sufficient number of studies of design types one and two that are of high quality (+) have consistent results. Any inconsistencies should be explained satisfactorily.
(**) means there is a moderate consistency across all study levels.
(*) means that the results of studies are inconsistent.
Relevance to Disease Risk Reduction in the General Population or Target Subgroup. Considers whether or not the magnitude of the risk-reduction effect in the target population is physiologically meaningful and achievable in the general US population or a subgroup of the US general population under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim.
(***) means that the magnitude of the effect observed in studies of design types one and two that are of high quality (+) is physiologically meaningful and achievable under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim.
(**) means there is some suggestion from studies of design type three and higher (i.e., study design types one and two) and of moderate (Ø) to high (+) quality that the effect will be physiologically meaningful, and achievable under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim but uncertainties remain.
(*) means that the magnitude of the effect in the studies is not likely to be physiologically meaningful or achievable under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim.
Ranking the Strength of the Evidence for a Health Claim
The first level, or highest rank of scientific evidence to support the substance/disease relationship meets the "Significant Scientific Agreement among qualified experts" standard. (For the purpose of this guidance, the first level rank is only used as a reference point. In all other respects it is outside the scope of this guidance.)
This level reflects a high level of comfort(10) among qualified scientists that the claimed substance/disease relationship is scientifically valid. In general, the first level ranked relationship would be considered to have a very low probability of significant new data overturning the conclusion that the relationship is valid or significantly changing the nature of the relationship. It would have high consistency with conclusions of authoritative bodies. The relationship would be based on relevant, high quality studies of mostly study design types one and two, and sufficient numbers of individuals would be tested to result in a high degree of confidence that results are relevant to the target population. Studies of different design would almost always result in similar findings, and the benefit would be physiologically meaningful and achievable under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim.
The second level rank of scientific evidence to support the substance/disease relationship is the highest level for a qualified health claim, and represents a moderate/good level of comfort among qualified scientists that the claimed relationship is scientifically valid. Qualified experts would rank the relationship as "promising," but not definitive. The claim would be based on relevant, high to moderate quality studies of study design type three and higher (i.e., design types one and two) and sufficient numbers of individuals would be tested to result in a moderate degree of confidence that results could be extrapolated to the target population. Studies of similar or different design would generally result in similar findings and the benefit would reasonably be considered to be physiologically meaningful and achievable under intake and use conditions that are appropriate for such conventional human food and dietary supplements that would be the subject of the claim. (Note: The term "moderate/good" for the second level rank may seem ungenerous. This terminology derives from historical data evaluated by the National Academy of Sciences(11) that indicated that over time many diet/disease relationships that met this level of evidence were not necessarily sustained.)
The third level rank of scientific evidence to support the substance/disease relationship is the middle level for a qualified health claim and represents a low level of comfort among qualified scientists that the claimed relationship is scientifically valid. It would have low consistency with statements from authoritative bodies or be ranked as "low" in terms of scientific support by qualified scientists. The relationship would be based mostly on moderate to low quality studies of study design type three, and insufficient numbers of individuals would be tested, resulting in a low degree of confidence that results could be extrapolated to the target population. Studies of different design would generally result in similar findings but uncertainties would exist. Uncertainties would also exist as to whether the benefit would be considered physiologically meaningful and achievable under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim.
The fourth level, or the lowest rank of scientific evidence to support the claimed substance/disease relationship, is the lowest level for a qualified health claim and represents an extremely low level of comfort among qualified scientists that the claimed relationship is scientifically valid. It would have very low consistency with conclusions of authoritative bodies or be ranked very low by qualified scientists. The relationship would be based mostly on moderate to low quality studies of study design type three and insufficient numbers of individuals would be tested, resulting in a very low degree of confidence that results could be extrapolated to the target population. Studies of different design would generally result in similar findings but uncertainties would exist. There could be considerable uncertainty as to whether or not the benefit would be considered physiologically meaningful or achievable under intake and use conditions that are appropriate for such conventional human food and human dietary supplements that would be the subject of the claim. This level requires at least some credible evidence to support the relationship. There cannot be a strong body of evidence against the claim (e.g., a study or studies of high persuasiveness, quality and relevance that do not detect an effect). If that is the case, such evidence provides a sound basis for concluding that the claim is not valid.
If the scientific evidence to support the substance/disease relationship is below that described as the fourth level (see above) no claim will be appropriate.
The result of the evidence-based rating system will be a statement describing the nature of the evidence and the rationale for linking a substance to a disease/health-related condition with a ranking as to the strength of the scientific evidence in support of that relationship. The process for arriving at the rank of the evidence to support the substance/disease relationship is illustrated in Table 1. The rank will be supported by:
A clear and transparent demonstration of which research studies were evaluated to provide the rank.
Evidence tables showing the rigor of the evaluation.
Step One. A proposed relationship between a substance and a disease or health-related condition is identified. |
| Step Two. Individual studies are identified that are pertinent to the substance/disease relationship. |
| Step Three. Individual studies are classified according to study design type. Different design types are graded higher than others, based on their ability to minimize bias. Thus assignment of a study design automatically provides a rating. |
| Step Four. Individual studies are assigned a designator of +, Ø, -, or N/A to reflect the study quality. (The general criteria for quality determination are described in this guidance). |
| Step Five. The strength of the scientific evidence in support of the substance/disease relationship is given a rank. This rank is determined taking into account the quantity, consistency, and relevance to disease risk reduction of the aggregate of the studies. |
| Step Six. The rank is reported. |
What Resource Materials are Available?
1. Internet-based Resource Materials
Agency for Healthcare Research and Quality (at http://www.ahrq.gov)
American Dietetic Association (at http://www.eatright.org/)
Canadian Task Force on Preventive Health Care (at http://www.ctfphc.org/)
Center for Evidence Based Medicine (at http://www.cebm.utoronto.ca)
Cochrane Collaboration/Cochrane Reviews (at http://www.cochrane.org)
Evidence-based Practice Internet Resources (at http://www-hsl.mcmaster.ca/ebm/)
Federal Judicial Center (at http://www.fjc.gov)
Federal Trade Commission (at http://www.ftc.gov)
FDA Food Advisory Committee. See Report of the FDA Food Advisory Committee Emerging Science Working Group at http://www.cfsan.fda.gov/~dms/faclaims.html
FDA Guidance for Industry: Significant Scientific Agreement in the Review of Health Claims for Conventional Foods and Dietary Supplements; Availability (64 FR 71794; December 22, 1999) (see http://www.cfsan.fda.gov/guidance.html)
Health Canada. Since their June 2000 publication of the proposed standards for health claims, proposals on two approaches to regulating health claims on foods have been published. The two approaches are: generic authorization and product-specific authorization (see http://canada.ca).
National Coordination Centre for Health Technology Assessment (at http://www.ncchta.org/main.htm)
National Guideline Clearinghouse (at http://www.guideline.gov)
National Health and Medical Research Council (at http://www.health.gov.au/nhmrc/)
National Health Service Centre for Reviews and Dissemination (http://www.york.ac.uk/inst/crd/).
National Heart, Blood, and Lung Institute (specific information available at http://www.nhlbi.nih.gov/health/public/lung/)
New Zealand Guidelines Group (at http://www.nzgg.org.nz/)
Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isn't (see http://www.cebm.net/ebm_is_isnt.asp)
Scottish Intercollegiate Guidelines Network (at http://www.sign.ac.uk/)
Other Resource Materials
Ahrens, E.H., Jr. Symposium. The evidence relating six dietary factors to the nation's health: consensus statement. Introduction. Am. J. Clin. Nutr. 32:2627-2631, 1979.
Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996;276:637-39.
Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De Aguero L, Truman B, Hopkins D, Mullen PD, Thompson RS, Woolf SH, Carande-Kuis VG, Anderson A, Hinman AR, McQueen DV, Teutsch SM, Harris JR. Developing an evidence-based Guide to Community Preventive Services - methods. The Task Force on Community Preventive Services. Am J Prev Med 2000;18:35-43.
Chalmers TC, Smith H Jr, Blackburn B et al. A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981;2:31-49.
Clarke M., Oxman AD. Cochrane Reviewer's Handbook 4.0. The Cochrane Collaboration; 1999.
Cook DJ, Sackett DL, Spitzer WO. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. J Clin Epidemiol. 1995;48:167-171.
Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomized and non-randomized studies of health care interventions. J Epidemiol Community Health. 1998;52:377-384.
Fahey T, Hyde C, Milne R, Thorogood M. The type and quality of randomized controlled trials (RCTs) published in UK public health journals. J Public Health Med.1995;17:469-474.
Falk, M. Model for a third-party review of the evidence substantiating food and dietary supplement claims. J Nutr 131:2219-2223, 2001.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121:11-21.
Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000;355:103-106.
Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-based medicine working group. JAMA. 1993; 270:2598-2601.
Guyatt GH, Haynes RB, Jaeschke RZ, Cood DJ, Green L, Naylor CD, Wilson MC, Richardson WS. Users' Guides to the Medical Literature: XXV. Evidence-based medicine: principles for applying the Users' Guides to patient care. Evidence-Based Medicine Working Group. JAMA. 2000;284:1290-1296.
Harbour R, Miller J. A new system [Scottish Intercollegiate Guidelines Network (SIGN)] for grading recommendations in evidence based guidelines. BMJ. 2001;323:334-336.
Harris RP, Helfand M, Woolf SH et al. Current methods of the U.S. Preventive Services Task Force: A review of the process. Am J Prev Med. 2001;20:21-35.
Hibble, A, Kanka, D, Pencheon, D, and Pooles, F. Guidelines in general practice: the new Tower of Babel? British Medical Journal 317:862-863, 1998.
Institute of Medicine. Guidelines for clinical practice: from development to use. Washington DC: National Academy Press, 1992.
Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1-12.
Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054-1060.
Kaye DH, Freedman DA. Reference Guide on Statistics. In: Reference Manual on Scientific Evidence, Federal Judicial Center, 2000.
Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol. 1986;4:942-951.
Lohr KN, Carey TS. Assessing 'best evidence': issues in grading the quality of studies for systematic reviews. Joint Commission J Qual Improvement. 1999;25:470-479.
National Health and Medical Research Council (NHMRC). How to use the evidence: assessment and application of scientific evidence. Canberra, Australia: NHMRC,2000.
Nowak R. Problems in clinical trials go far beyond misconduct, 264 Science 1538, 1994.
Porter C, Matel JL. Are we making decisions based on evidence? JADA. 1998;98:404-407.
Reisch JS, Tyson JE, Mize SG. Aid to the evaluation of therapeutic studies. Pediatrics. 1989;84:815-827.
Schulz KF. Subverting randomization in controlled trials, 274 JAMA 1456, 1995.
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408-412.
Spilker, B. Guide to Clinical Trials. Raven Press, NY, 1991. Chapter 103, Systems to Evaluate Published Data.
Splett P. Developing and validating evidence-based guides for practice: a tool kit for dietetics professionals. Chicago: The American Dietetic Association, 2000.
Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Systematic reviews of trials and other studies. Health Technol Assess. 1998;2:1-276.
Truman BI, Smith-Akin CK, Hinman AR, Gebbie KM, Brownson R, Novic LF, Lawrence RD, Pappaioanou M, Fielding J, Evans CA, Guerra FA, Vogel-Taylor M, Mahan CS, Fullilove M, Zaza S. Developing the Guide to Community Preventive Services - overview and rationale. The Task Force on Community Preventive Services. Am J Prev Med 2000;18:18-26.
West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, Lux L. Systems to Rate the Strength of Scientific Evidence. AHRQ Publication No. 02-E016, 2002.
Zaza S, Wright-De Aguero LK, Briss PA, Truman BL, Hopkins DP, Hennessy MH, Sosin DM, Anderson L, Carande-Kulis VG, Teutsch SM, Pappaioanou M. Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med 2000;18:44-74.
(1)This guidance has been prepared by the Center for Food Safety and Applied Nutrition (CFSAN) at the U.S. Food and Drug Administration.
(2)FDA uses the term, "unqualified health claim," to refer to health claims that are or could be authorized under the Nutritional Labeling and Education Act of 1990 (NLEA) and regulations promulgated under that act, including 21 CFR 101.70.
(3)Examples of evidence-based rating systems are described and evaluated in: Agency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment. Number 47, 2002. The Healthcare Research And Quality Act of 1999, Part B, Title IX, Section 911(a) mandated that the Agency for Healthcare Research and Quality (AHRQ), in collaboration with experts from the public and private sectors, identify methods or systems to assess health care research results, particularly "methods or systems to rate the strength of the scientific evidence underlying health care practice, recommendations in the research literature, and technology assessments."
(4)Greer N, Mosser G, Logan G, Wagstrom Halaas G. A practical approach to evidence grading. Jt Comm. J Qual Improv. 2000; 26:700-712.
(5) The ICSI system has been adapted by the American Dietetic Association (ADA) for their evidence-based dietetics practice and, thus, the ADA modifications have addressed many of the diet/disease relationships that are also of interest to FDA. See: Myers EF, Pritchett E, Johnson EQ. Evidence-based practice guides vs. protocols: what's the difference? JADA. 2001;101:1085-1090.
(6)As defined in 21 CFR 101.14 (a)(2), the term "substance" means a specific food or component of food, regardless of whether the food is in conventional food form or a dietary suppleme nt that includes vitamins, minerals, herbs, or other similar nutritional substances.
(7)This rating system for type of study design is based on that described in Greer et al., 2000, with modifications.
(8)For example, randomization minimizes bias in that the groups are likely to be comparable except for the treatment. That is why inferences based on randomized experiments are considered more secure than inferences based on observational studies (from Kaye DH and Freedman DA. Reference Guide on Statistics. In: Reference Manual on Scientific Evidence, Federal Judicial Center, 2000.).
(9)Additional specific, detailed criteria, based on the above noted general principles, will be evaluated for usefulness during this interim period.
(10) The use of the phrase "level of comfort" is mentioned in rulemaking that established the general requirements for health claims (21 CFR 101.14), which published in the Federal Register (58 FR 2478 at 2506); January 6, 1993)
(11)"Evolution of Evidence for Selected Nutrient and Disease Relationships". Committee on Examination of the Evolving Science for Dietary Supplements. Food and Nutrition Board, Institute of Medicine. National Academy Press, Washington, D.C., 2002
This document supercedes
Qualified Health Claims in the Labeling of Conventional Foods and Dietary Supplements
(December 18, 2002)
