Strength of Study Design
The various types of study design are described below in descending order of strength:
-
Randomized controlled clinical trials.
-
Double-blinded.
-
Nonblinded treatment delivery.
The randomized, double-blinded controlled clinical trial (1i)
is the gold standard of study design. To achieve this ranking, the study
allocation must be blinded to the physician both before and after
the randomization and the treatment assignment take place. This design provides
protection from allocation bias by the investigator and from bias in
assessment of outcomes by both the investigator and the patient.
Unfortunately, most clinical trials in oncology cannot be double-blinded
after treatment allocation because procedures or toxic effects often vary
substantially among study allocations in ways that are obvious to both
the health care professional and the patient. In most cases, however, it
should be possible to blind the investigator and the patient until the
randomization has been made. If blinding of the therapy delivered cannot
be accomplished, a rank of 1ii is assigned.
Meta-analyses of randomized studies offer a quantitative synthesis of
previously conducted studies. The strength of evidence from a
meta-analysis is based on the quality of the conduct of individual
studies. Moreover, meta-analyses can magnify small systematic errors in
individual studies. A study comparing the results of single large randomized
trials to those of meta-analyses of smaller trials published earlier on
the same topics showed only fair agreement (kappa statistic = 0.35). Outcomes
of the large randomized controlled trials were not predicted accurately by
the meta-analysis 35% of the time.[1,2] Meta-analyses performed by different
investigators to address the same clinical issue can reach contradictory
conclusions.[2] Therefore, meta-analyses of randomized studies are placed in
the same category of strength of evidence as are randomized studies, not at a
higher level.
Subset analyses of randomized studies are subject to errors inherent in
multiplicity (i.e., statistically significant results to be expected as
a result of random variation of measured effects in multiple subsets).
Therefore, subset analyses do not represent the same strength of evidence
as the overall analysis of a randomized trial as designed unless explicit
prospective hypotheses are made for the analyzed subset. Otherwise,
subset analyses should be placed in the next lower category of study design
(nonrandomized controlled clinical trials).
-
Nonrandomized controlled clinical trials.
This category includes trials in which treatment allocation was
made by birth date, chart number, day of clinic appointment, bed
availability, or any other strategy that would make the allocation known
to the investigator before informed consent is obtained from the patient.
An imbalance can occur in treatment
allocation under such circumstances. For the reasons given above, subset
analyses within randomized trials often fall into this category of
evidence.
-
Case series.
-
Population-based, consecutive series.
-
Consecutive cases (not population-based).
-
Nonconsecutive cases.
These clinical experiences are the weakest form of study
design, but they may be the only available or practical information in
support of a therapeutic strategy, especially in the case of rare
diseases or when the evolution of the therapy predates the common use of
randomized study designs in medical practice. They may also provide the only
practical design when treatments in study arms are radically different
(e.g., amputation vs. limb-sparing surgery). Nevertheless, these
experiences do not have internal controls and must therefore look to
outside experiences for comparison. This always raises the issues of
patient selection and comparability with other populations. In order of
generalizability to other populations are population-based series,
nonpopulation-based but consecutive series, and nonconsecutive cases.
References
-
LeLorier J, Grégoire G, Benhaddad A, et al.: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 337 (8): 536-42, 1997.
[PUBMED Abstract]
-
Bailar JC 3rd: The promise and problems of meta-analysis. N Engl J Med 337 (8): 559-61, 1997.
[PUBMED Abstract]
Back to Top
< Previous Section | Next Section > |