Section11:Pre-Study & In-Study Acceptance Criteria

From Assay Guidance Wiki

Different methods of quality control are available and routinely used in analytical methods. It is important that the methods used for assessment of method performance are suitable for the intended purpose.

The use of QC methods used in PK assays (eg 4-6-x) and clinical diagnostics (confidence limits) may both be applicable. It is up to the laboratory performing the analysis to choose the most relevant method to use and justify it scientifically based on statistical and clinical criteria. This will be critical when using 4-6-x in order to assign an appropriate value to ‘x’.

Shah et al (1990) proposed the 4-6-X rule for in-study validation phase that has become popular and widely used. This rule states that 4 out of the total 6 samples should be within X% of the nominal/reference value, and at least one out of the two samples at each level must be within X% of the reference value. The choice of X is specified a priori based on the intended use and purpose of the assay, and it was set at 20% by Shah et al. DeSilva et al (2003) proposed the following criteria for pre-study and in-study validation phase of ligand-binding assays for assessing pharmacokinetics of macromolecules.

It should be noted that the acceptance criteria for biomarker assays will depend heavily on the intended use of the assay and should ideally be based on physiological variability as well. According to the criteria listed in this table, X is set at 30% for in-study validation, and the total error is set to be within 30% for the pre-study validation, along with the 20% limits for each component of total error (bias and precision). The pre-study criteria (Total Error < X%) and the in-study criteria (4-6-X rule) are not entirely consistent because it does not take into account total error estimate variability and the consequent decision error rates. Thus the uncertainty in these estimates will depend on the magnitude of the errors and the number of measurements, and will in turn impact the level of decision error rates (Kringle, 1994). The appropriate value of X in 4-6-X can be determined based on the variability of the total error estimates in pre-study validation. When it is feasible to use more QC samples in each run, 8-12-X or 10-15-X will have much better statistical outcomes than the 4-6-X criteria. In addition, the use of control charts as described by Westgard or tolerance limits based on pre-study validation data may be considered when possible.

The concept of total error as the primary parameter, and with bias and precision as additional constraints is very useful. This is because total error has a more practical and intuitive appeal as it relates specifically to our primary question of interest about the assay; How far are my observed test results from the reference/nominal value? Since this is the primary practical question in the minds of most laboratory scientists, the criteria on the assay performance for the in-study phase is defined with respect to this question. Therefore the primary criteria for the pre-study phase are also defined with respect to this question, that is, the total error.

Given that this total error approach is very intuitive and practical, it is important to consider a rule that will provide better consistency between our expected performance for the assay to the in-study and pre-study validation criteria.

Consideration of Physiological Variation for Acceptance Criteria

One of the most important considerations for defining the performance criteria of most biomarker methods is the physiological variability in the study population of interest. That is, in order to determine whether a biomarker method is ‘fit-for-purpose’, we should determine whether it is capable of distinguishing changes that are statistically significant based on the intra-subject and inter-subject variation. The term “subject” here may refer to animal or human. For example, an assay with 50% total error during pre-study validation may still be adequate for detecting a 2-fold treatment in a clinical trial for a certain acceptable sample size. Thus whenever possible, the acceptance criteria for pre-study validation should be based on physiological variation in the study. An example of the use of intra-subject and inter-subject variation for defining the pre-study acceptance criteria can be found in http://www.westgard.com/guest17.htm.

When the relevant physiological data (say, treated patients of interest) are not available during the assay validation phase, then healthy donor samples should be used to estimate the intra and inter subject variation, and hence the desired specifications on the pre-study assay validation. This can be updated at a later time when there is access to the relevant patient data. If access to healthy donor samples is also not feasible, then other flexible biological rationale should be considered and updated periodically as more information become available over time. In the absence of physiological data or other biological rationale, the acceptance criteria for pre-study validation should not be strictly defined. Instead, only the performance characteristics from pre-study validation such as the bias, precision and total error should be reported. Any decision regarding the acceptance of the assay (pre-study acceptance criteria) and consequently the determination of the dynamic range (LQL, UQL) should be put on hold until adequate information related to the physiological data become available.

Assessment of analytical batches/runs in terms of acceptance (in-study validation) needs to take into account of the study need. Setting critical acceptance criteria a priori may not be appropriate (or even possible) to take into account all possible outcomes in the analytical phase – especially since the values seen in the incurred samples may not be what is expected or predicted. This is especially the case in new or novel BM’s as opposed to those where historical information in normal and diseased populations is available.

It is advised that when constructing batches for analysis, ALL levels of QC’s are analyzed at each QC interval. For example, a batch of 96-well microtiter plates may include 3 sets of QC’s at start, middle and end of the plate, and all QC’s (Low,Medium,High) are assayed at all three intervals. This will help in the assessment of method performance and batch acceptance for incurred samples.

In studies with large numbers of samples, assessment of method performance between batches may help before rejecting data. For example, it is of no value to reject batches when large numbers of high concentration QC’s fail but where the low and medium QC’s are good AND when all the study sample results are in the low to medium range. Here the positioning of the high QC based on expectation before the analysis of incurred samples has been flawed – but it does not necessarily make the study sample results.