Note from the National Guideline Clearinghouse (NGC): The National Institute for Health and Clinical Excellence (NICE) commissioned an independent academic centre to perform an assessment of the manufacturer's submission on the technology considered in this appraisal and prepare an Evidence Review Group (ERG) report. The assessment report for this technology appraisal was prepared by the School of Health and Related Research (ScHARR), University of Sheffield (see the "Availability of Companion Documents" field).
Clinical Effectiveness
Description and Critique of Manufacturers Approach to Validity Assessment
A completed table recording decisions regarding trial quality assessment was not in the manufacturer's submission (MS). A completed validity assessment form for the three trials, provided at the request of the ERG, is reproduced in Table 5 of the ERG report (see the "Availability of Companion Documents" field).
The critical appraisal of the trials conducted in the MS is based on the full details of the trials, as reported in the MS, rather than the published details. For example, data about efforts to protect blinding (e.g., database lock) were not reported in the published papers. There are also three issues with the submitted critical appraisal.
Firstly, Table 5 in the ERG report (see the "Availability of Companion Documents" field) states that each patient received "twice daily subcutaneous injections", but the published papers of the RE-NOVATE and RE-MODEL trials state that only a single daily subcutaneous injection was given. Secondly, the dosing regimens described for the RE-NOVATE and RE-MODEL trials are not reported accurately. Finally, the MS reports on efforts to ensure blinding, but does not report if any of these studies assessed the success of blinding, as required by point 11 on the CONSORT checklist (http://www.consort-statement.org/). The assessment of the ERG is that they did not.
The validity assessment tool used in the MS is not referenced and the questions are not entirely adequate. The trials included were all non-inferiority trials, and an appropriate validity assessment tool is available for assessing the quality of such trials (see Table 6 of the ERG report [see the "Availability of Companion Documents" field]). The tool used in the MS appears to be appropriate for assessing superiority trials only.
The results of the validity assessment of the ERG are reported in Table 7 of the ERG report (see the "Availability of Companion Documents" field).
The overall methodological quality of the included trials was good, but a more appropriate validity assessment tool was available and could have been used in the MS. The relevant extension of the CONSORT statement regarding the reporting of non-inferiority trials was available both at the time of the publication of the RE-NOVATE and RE-MODEL trials and for validity assessment of the trials included in the MS.
Describe and Critique the Statistical Approach Used
The MS contained a series of meta-analyses. It reported relative risks (RR) for fixed effects models of the 2 pivotal trials combined (RE-NOVATE and RE-MOBILIZE) and all three trials combined, and a random effects model for all three trials combined. At the request of the ERG, the manufacturer also provided:
- A random effects model meta-analysis of the 2 pivotal trials combined
- Risk differences (RD; absolute risk reductions) in both fixed and random effects models for the two pivotal trials, and all three trials combined
- Fixed and random effects models for both RR and RD for the two total knee replacement (TKR) trials combined (RE-MODEL and RE-MOBILIZE)
A pooled analysis of RD for the two pivotal trials was reported in the MS. The rationale for presenting and pooling individual patient data was not reported. The analyses themselves appear to have been reproduced from a source external to the MS and were only performed on the secondary efficacy outcome (no explanation for this was given). The statistical methods of pooling were not made explicit in the MS.
Sensitivity analyses presenting best and worst case scenarios were also performed, imputing no events for missing trial data, or an event for each piece of missing data, respectively, as well as a pooled analysis of all three trials using a fixed effects model only. The rationale for pooling the 3 trials in this way, with a fixed effects model only, was not given.
The ERG also notes that the pooling of data is viewed as inadequate for the assessment of efficacy. A pooled analysis focuses on treatment groups rather than on studies, ignores validity of the comparisons and is subject to bias termed 'Simpson's paradox' in probability. A more satisfactory statistical technique for combining the results from two or more separate studies is meta-analysis. All efficacy and safety meta-analyses requested by the ERG were provided by the manufacturer.
Summary Statement
The manufacturer's search strategy was adequately reported but limited, although the submission appears to contain all of the relevant head-head RCTs. Processes and validation of study screening and data extraction were not reported in full, and the validity assessment tool used was not entirely appropriate or adequate, although the application of a more appropriate tool did not greatly alter judgments on the overall quality of the included trials. The outcomes selected were relevant and appropriate. Statistical methods were explicitly described for the meta-analyses and all required meta-analyses were performed. Pooled analyses were also reported, although they were not described fully and may be inappropriate.
Refer to Section 4 of the ERG report (see the "Availability of Companion Documents" field) for more information on methods used to analyze the evidence.
Economic Evaluation
Model Validation
The MS describes internal pre-specified quality control checks of all input data and programming and external validation by a panel of clinical experts.
The ERG are not aware of any further trials or models against which the manufacturer's model could be validated.
Critique of Approach Used
The decision tree/state transition model which the manufacturer used is considered to be appropriate for the economic analysis.
Refer to Section 5 of the ERG report (see the "Availability of Companion Documents" field) for more information on economic evaluation.