Disposition of Comments
Project ID: CANT1106
Overview of Peer and Public Review Comments and their Disposition
Each comment submitted by peer and public reviewers has been carefully considered by the authors of the report, and has been responded to within two separate tables. At the outset, additionally, we would like to summarize the main themes raised by respondents and provide an overview response.
The majority of comments agreed with the primary conclusions of the report, that: systematic reviews are not always a feasible and/or reliable way to answer questions regarding clinical effectiveness of new therapies; data relevant to efficacy of any given drug vary greatly in quantity and quality, making it difficult to combine data or compare results across studies, and therefore difficult to draw conclusions; timeliness, in fields where the evidence is rapidly emerging, poses a major challenge when using traditional systematic review methodology, and; in the absence of better sources or methodologies, it is important to work with the existing approach to evidence synthesis/review (i.e., the compendia) so that clinicians have some source, albeit imperfect, to guide them in clinical decision-making. These comments are appreciated, as they demonstrate a general agreement that the current system of evidence synthesis and review warrants improvement. This recognition, in itself, is an important first step toward change.
A second category of comments expressed the concern that results of the 19 systematic reviews forming the body of this technology assessment should not be used to make either reimbursement or clinical decisions. A critical outcome of this work was a realistic appraisal of the feasibility of current methods of evidence review in the present-day research environment. The 19 separate systematic reviews comprising this report could be viewed as sub-studies which, taken together, powerfully illustrate the challenges and obstacles inherent in attempting to perform an up-to-date and comprehensive review of the evidence in a field where the research pace is rapid, studies vary considerably in design and quality, and clinical decisions hinging on their results are urgent. In order to conduct the systematic reviews, we needed to establish an arbitrary cut-off date for study inclusion (the last searches were conducted on September 14, 2007). As a result, the report is admittedly neither complete nor entirely up-to-date; of necessity, it had to omit studies that were conducted, published, or indexed in MEDLINE® after the cut-off date. Thus, we do not offer the results in this report as definitive for use as the basis of decision-making. Instead, the report is intended to inform a national discussion regarding how best to evaluate and synthesize data so that decision-makers have access to timely evidence that has been reviewed for quality.
A third category of responses addressed the evidence we included, and our manner of presenting the results of the 19 systematic reviews. Reviewers suggested numerous additional references and studies for inclusion in the systematic reviews. We reviewed each of the 146 suggested citations and identified five that met inclusion criteria but had not been included in our reviews. We evaluated all five references and determined that including them in the revised report would not alter our conclusions and would add little or no informative evidence to the report. Finally, some respondents suggested that we present the evidence in the format of more traditional summary evidence tables. We understand the desire for a familiar structure; however, the disparate nature of the data in many cases made it impossible to combine them, as required to create a comprehensive table.
Select for Table 1: Invited Peer Reviewer Comments (PDF file, 213 KB). Plugin Software Help.
Select for Table 2: Public Review Comments (PDF file, 886 KB). Plugin Software Help.
Contents
Table 1: Invited Peer Reviewer Comments
Table 2: Public Review Comments
Table 1: Invited Peer Reviewer Comments
| Reviewer1 |
Section2 |
Reviewer Comments2 |
Author Response3 |
| 1 |
General |
The inclusion criteria for the technology assessment are (1) FDA approved targeted agents, (2) marketed in January 2007 or before, (3) with compendia-listed indications other than the FDA-approved indication in December 2006 (there is a discrepancy - p. 16 states December however p. 10 states January 2007). This resulted in the inclusion of eight targeted therapy drugs associated with 19 off-label indications. |
December 2006 is the correct date. We have revised the report accordingly (p. 6). |
| 1 |
General |
In order to be included, studies had to be conducted in humans and include survival, disease-free survival, tumor response, quality of life, or adverse events. Abstracts from ASCO and ASH were reviewed but not included in the evidence tables. The quality of the studies was evaluated using the quality assessment criteria from NICE.
|
Please note that the mention in the draft report of "quality assessment criteria from NICE" was misleading. We actually used the quality assessment criteria described in a 2003 British report on imatinib mesylate for unresectable and/or metastatic gastrointestinal stromal tumors (GIST); these criteria were, in turn, drawn from a 2001 methodological publication of the British National Health Service Centre for Reviews and Dissemination. We have clarified this in the revised report (pp. 12-13).
|
| 1 |
General |
The scope, inclusion criteria for conditions, and inclusion criteria for studies all seem appropriate. The one question I have concerns the breadth of the definition for "Indication." Indication is defined as the use of the drug for a specific disease entity (Table 1). For many indications, this adequately captures the use of the drug for that disease. However, in some cases, there are multiple different treatment situations within a disease such that defining "indication" as the disease in which the drug is used is too broad and makes it difficult to draw conclusions about the data (i.e. alemtuzumab and NHL). Even when there are not multiple indications for a disease, the new agent may have a more narrow indication than all patients with the condition - for example, the evidence may be limited to patients who have failed other available treatments. If the compendia-specified indications are at the disease level and/or coverage decisions cannot be made at a more specific level than the disease entity, then the current broader classification may be more appropriate; however, this should be addressed explicitly in the report.
|
We appreciate the reviewer's comment that "indication" is not specifically defined. We identified eligible drug-disease combinations based on each compendium's unique use and definition of "indication." However, we agree completely with the reviewer's concern about the breadth of the definition of the concept of an "indication" in oncology. This is a major issue in oncology that is rapidly changing - sometimes indications are broad (drug x for cancer y), and other times they are very narrowly specified by point in disease trajectory or disease characteristics (e.g., drug x for relapsed or refractory disease y after 2 prior lines of treatment, or drug x for disease y when biomarker z is present). To address this and reflect the reviewer's astute assessment, we have inserted a paragraph at the end of the section on "Inclusion criteria for targeted therapies" (p. 6) in the revised the report which reads: "The concept of a disease-specific indication for a drug is fluid. For some drug-disease indications the relationship represents a one-to-one relationship. For others, there are multiple different treatment situations within a disease category such that defining "indication" as the disease in which the drug is used is too broad and makes it difficult to draw conclusions about the data (e.g., alemtuzumab for NHL). Conversely, the indication may need to be sharply narrowed to reflect appropriate use of a drug only in certain times in a disease trajectory (e.g., for refractory disease after prior specified therapy, or in the setting of a positive biomarker). For this report, we used the drug-disease indication as specified within the compendia, since that reflects the indication considered for reimbursement purposes."
|
| 1 |
Appendix tables |
The tables that report the data abstracted from these studies in the Appendix for each drug/disease are comprehensive and present the results clearly. |
Thank you. |
| 1 |
Table 5 (pp. 25-30) |
Table 5 presents the overall summary of the systematic evidence review. This summary is presented as a "Summary Discussion." While succinct, it is challenging to draw any conclusions from these summaries regarding the overall state of the evidence supporting off-label indications. A table that presents the following information for each drug/indication might be more illustrative:
- Number of compendia that list the indication
- Number of other treatments for the indication (both FDA approved and off label listed)
- Number of studies
- Total number of patients across all studies
- Benefit reported in studies (PR, CR, DFS, OS etc)
- "Best" reported benefit (e.g. 60% CR)
- Summary measure of study quality (e.g. number of studies that are rated "good")
- Number of patients included in AE assessments
- % of AEs resulting in death; % severe or life-threatening AEs
- Prevalence of disease
How does it compare to other treatment options (e.g. favorable, unfavorable, unable to determine) |
We understand the desire for a more traditional and definitive table format from which conclusions can be more easily drawn. However, the nature of the data - their heterogeneity, varied quality, differing metrics, and in many cases small numbers - makes it nearly impossible to construct such a table with integrity. |
| 1 |
Page 24, Table 4 |
Minor comments:
Typo p. 24, Table 4 - "dug/disease" instead of "drug/disease" line 14 and last line. |
Thank you for spotting this; we have fixed these typos (p. 15). |
| 2 |
General |
The Duke Evidence-Based Practice Center has written a Technology Assessment Report on the Evidence Regarding Off-Label Indications for Targeted Therapies Used in Cancer Treatments dated August 26, 2009 (project ID: CANT1106) as prepared for the Technology Assessment Program at the Agency for Healthcare Research and Quality (AHRQ). The Coverage and Analysis Group at the Centers for Medicare and Medicaid Services (AMS) requested the report (contract number HHSA 290-02-0025). The technology assessment included 8 targeted therapy drugs and 19 respective off-label indications including alemtuzumab, bevacizumab, bortezomib, cetuximab, erlotinib, gefitinib, imatinib, and rituximab. The assessment also acknowledges the FDA-approved indications of the included drugs as of January 2007.
The report clearly describes the inclusion criteria, search criteria, data abstraction, quality assessment, and data synthesis, which were instrumental in the creation of comprehensive summary tables in the results section. The references are exhaustive including 500 citations. An appendix, "Results of Systematic Literature Review for Specific Drugs/ Disease Combinations", comprises the bulk of the document, and includes a concise composition of background, methods, results, discussion, and helpful summary tables for each agent and each respective off-label indication. As such, the systemic literature review is comprehensive, balanced, and offers appropriate analyses of use based on the level of available evidence. In Chapter 3, the authors stress the difficulty in composing technology assessment for agents that are in rapid and continuous development, creating an ongoing evolutionary body of evidence in the literature. The authors importantly emphasize that "the "moving target" nature of evidence calls into question the feasibility of a time bounded, evidence review process based on systematic review, in an environment where the evidence pool is continuously expanding. The literature that was identified was inclusive through September 2007. |
Thank you for this concise summary of the report's contents.
|
| 2 |
Chapter 4 |
The most important section in the body of the report is Chapter 4, Discussion. The authors provide examples of the rapid literature evolution between 2006 and 2009. Compendia are acknowledged as helpful tools enlisting off-label indications. A concern, however, was the degree of variability in study data quality, including the poor quality of the evidence despite in some cases the accumulation of extensive data sets. It is acknowledged however that disparity and quality can represent the rarity of a particular disease entity and the limitations of attempting to conduct clinical trials for every disease permutation. The report provides examples of how even in the setting of limited data, an off-label indication may be appropriate in clinical decision making, especially for tumors that are rare, carry significant mortality and have presence of receptors or other targets justifying the use of a particular biologic. In addition, there are diseases with long survival times and evidence must be abstracted by the clinician in the absence of survival data. In the discussion of the relatively poor level of evidence ascertained in this report, the authors stress that when dealing with potentially life threatening illnesses, clinicians and patients may view the body of evidence in a different light because of the gravity of the clinical situation. The authors mention that there are significant challenges in the utilization of current methods of evidence review, particularly rapid evolution of therapeutic strategies yielding high output of data. In a brief discussion of the need for new models to evaluate evidence, the authors cite the potential importance for comparative effectiveness within a learning healthcare system model. Based on the analyses in this current technology assessment, new models could address the importance of timely ongoing adverse events, data reporting, and the limitations of a literature composed of phase II studies which may suggest benefits, but is not comparative in terms of efficacy or tolerability. |
Thank you for this cogent description of situations in which clinical decisions must be made despite limited evidence and for the clarification that true comparative evaluation requires either studies specifically designed for comparative effectiveness research or an added level of synthesis focused on that comparison. |
| 2 |
Page 38 (concluding paragraph of draft report) |
The authors conclude with three critical limitations, which it would appear will limit the applicability of this assessment. These include the inability to make definitive conclusions about efficacy or safety of the agents because of the variability and quality and quantity of data; the risk of publication bias since negative results are often under reported compared to publications with positive results; and the literature review ends as of September 2007. Regardless of the limitations, this report has the potential to guide future research strategies and the authors offer valuable insight in terms of the potential challenges for future technology assessment, which should be taken into consideration. It would have been helpful if the authors had placed more emphasis on the need for much more extensive work including infrastructure, funding, and the overall importance of directing clinical strategies to include human tumor biology assessments that could inform the development of new targeted drug strategies and help determine more appropriate use of available agents for subsets of patients. As is stressed in at least some of the comparative effectiveness research position papers, the understanding of human biological characteristics to drive clinical decision and treatment strategies will be critically important to deliver the right treatment to the right patient at the right time. Enrichment trial designs based on human tumor biology, for example, could move us away from the current empiric design approaches and such are likely to be much more focused and more likely to create higher levels of evidence, even in phase II trial design. Such would lead to more confidence in the indications for off label use for targeted therapies. |
This technology assessment was not intended as a position paper and therefore avoids making statements that seek to direct research or clinical strategies.
Thank you for expressing this interesting direction (use of tumor biology assessment to enrich trial design). |
| 3 |
Page 8 |
[This is] not how bev [bevacizumab] works, it binds the ligand. |
We changed "tyrosine kinase inhibitors block the intracellular signals responsible for uncontrolled cancer growth" to "tyrosine kinase inhibitors locate and bind to specific proteins, thereby causing a desirable effect, such as inhibiting vascular endothelial growth factor" (p. 5). |
| 3 |
Page 9 |
If you are going to cite proteosome inhibitors, then why would you not also include the taxanes? They have a cellular target as well. This definition is always hard to find just the right line, the line is fine. |
We appreciate, and agree with, the comment that there is no clear consensus regarding what constitutes a targeted therapy drug. The focus of this report is on emerging targeted therapies. The list of drugs that we considered to fall within the scope of this report was discussed with, and approved by, AHRQ and CMS. This decision was also informed by an article published in 2004 (Segota E, Bukowski RM. The promise of targeted therapy: cancer drugs become more specific. Cleveland Clinic Journal of Medicine 2004; 71( 7):551-60). Taxanes were not among the therapies included in the final agreed-upon list.
|
| 3 |
Page 10 |
What about use outside of even compendia listings? |
Agents are likely used for non-FDA-approved indications even when they are not listed in the compendia. These uses, however, were outside the scope of this report. |
| 3 |
Page 12 |
It is important to note how non-standardized the decisions are in these various compendia. |
Thank you for this suggestion. We have added the following sentence on p. 7: "It should be noted that there is no standardized approach across the various compendia to determining which off-label indications should be included." |
| 3 |
Page 14, Table 1 |
I was surprised to see bev [bevacizumab] and erb [Erbitux® = cetuximab] in pancreas cancer, am I reading correctly that these were listed on one or more compendia? |
Each of the targeted therapies considered in the report was listed in a monograph in at least one of the pre-specified compendia, with a statement that could be interpreted as supporting the use of the particular drug for the particular indication listed at the time this project started, including bevacizumab and cetuximab for the treatment of pancreatic cancer. Some of these compendia recommendations were later changed, so that some drug/disease indications may no longer be supported in a monograph. |
| 3 |
Page 15, Table 2 |
You list lines of therapy in some but not in others. I think most define line of therapy in their indications. |
We have updated Table 2 on p. 9 to include the line of therapy for each drug. |
1 Peer reviewers are not listed in alphabetical order.
2 Page and line numbers refer to the draft report.
3 Page and line numbers refer to the final report.
Current as of August 2010
Internet Citation:
Report on the Evidence Regarding Off-Label Indications for Targeted Therapies used in Cancer Treatment. Disposition of Comments. August 2010. Rockville, MD: Agency for Healthcare Research and Quality. http://www.ahrq.gov/clinic/ta/comments/targettherapy/
|
540 Gaither Road Rockville, MD 20850 