Section3:Production Monitoring

From Assay Guidance Wiki

Production assays can be monitored in two basic ways: running control (reference) compounds and retrospective studies of compounds that have repeat evaluations that accumulate as part of the normal SAR process. Of the two methods, running control compounds is definitely better as it allows problems to be identified prospectively and corrected, whereas retrospective studies are limited to verification of past activity, be it acceptable or unacceptable. However, retrospective studies can be useful supplements, especially when conducted prior to important milestones such as Program Sanction where demonstration of “valid biological assays” is a requirement. Below are comments on the setup/selection of controls and the analysis of retrospective studies, and the use of bridging studies to verify that changes to assay protocols have no effect on the assay results.

Control Compounds

Key assays in a project and assays where problems are suspected should have two control compounds, a primary and a secondary (this is referred to as Close Monitoring). All other assays should have at least a primary control (Regular Monitoring). Both compounds need to be run once per run, unless plate variability is suspected. In that case the primary control compound needs to be run once per plate. The purpose of the primary control is to ensure that there isn’t any “assay drift”, i.e. that the same compound has a stable Ki/Kb/EC50 over time, and that the assay reproducibility (MSR) is stable over time. The purpose of the secondary control is to examine the stability of results over a dose-range. If problems do develop, then it is important to examine whether the entire dose-range is equally affected (a small problem) or whether the dose-range is differentially affected (a big problem). Also, two controls permit direct calculation of both the within-run and overall MSR’s, and a check that the MSR is consistent over a range of potencies.

The activity of the primary control should be at or near the most potent compound available, and ideally should be the Lead compound. There should also be sufficient stock of a single lot of the compound so that it can be run on a continuous basis for some period of time. Since the control compound is supposed to be representative of the test compounds, it should receive the same sample handling as all the test compounds, and not be specifically prepared and added to the assay outside of normal test compound procedures.

For Close Monitoring, the secondary control should be >100 fold less potent then the primary control. Otherwise it has the same requirements as the primary control. As the SAR develops the potency traditionally improves. So when the “best” compounds are more than 100-fold more potent than the primary control then select a new primary control. If the assay has a secondary control then the old primary control becomes the new secondary control, and the existing secondary control is dropped. If there is no secondary control then it is suggested to run both primary controls over the first 6 runs of the new primary control.

A scatter plot for control compound log-Ki/Kb/EC50 versus run date should be updated after every run and checked for problems. For assays with two control compounds the difference in log-Ki/Kb/EC50 versus run date should be plotted, and for agonist and non-competitive antagonist assays the efficacy versus run date should also be plotted. Outlier runs and trends either up or down (assay drift) should be checked visually, and problems investigated and corrected as they occur. Outlier runs should be repeated.

After 6 runs compute the overall MSR of the assay based on the control compounds according to formula, Equation3, where s is the standard deviation of the log-Ki/Kb/EC50 values. This MSR is the total or overall MSR (whereas the one computed in a test-retest study encompasses only the within-run variability), and should be less than or equal to 7.5. This standard comes from practical experience obtained thus far with assays in the company, and not theoretical statistical considerations. Note that this is a minimum standard that all assays should meet, and in practice chemistry requirements may indicate a smaller MSR (as low as 2-3) is required for some or all assays. The Project/Program Team should discuss this issue with a statistician to set appropriate MSR’s for their assays.

After each run, a running MSR plot should be maintained (i.e. computed from the last 6 runs) and checked to ensure the continued good reproducibility of the assay.

Examples

Example 1 illustrates results for an assay with a single control. The left panel shows the potency versus run date scatter plot, the right panel the moving MSR chart. The MSR points are based on the last 6 runs of the assay, i.e. the first point is computed using runs 1-6, the second point uses runs 2-7, etc. The Mean Summary section indicates the highest/lowest/last IC50’s in the period were 22.63, 4.42 and 11.25 uM respectively (chart units are in nM). The overall average was 10.17 uM. The potency has no apparent temporal trends, and no unusual observations. The right panel shows the trends in MSR over time, which appears to increase until mid Feb-2002, and then decrease. However, the magnitude of the increase trends is quite small and well within the variation of an estimate based on a sample of size 6. The highest/lowest/latest MSR’s are 6.8, 2.7 and 2.7 respectively. The overall MSR is 4.4, which is not the average of the 6-run MSR’s but instead is a single estimate derived using the entire sample (18 data points in this case). This is a stable assay with moderate assay variation (3 < MSR < 5).

Mean Summary		MSR Summary
High	22630.00	High	6.8
Low	4420.00	Low	2.7
Overall	10172.76	Overall	4.4
Current	11250.00	Current	2.7

Example 1. Potency, MSR Chart, and Summary Statistics for an Assay with One Control

Example 2 illustrates an assay with two controls. In the left panel the red and blue lines represent the two compounds, and are positioned using the left axis. The green line is the potency ratio between the two compounds and is positioned using the right axis. The right panel shows the moving MSR values both within run and overall. The Overall-Overall MSR is the value to be reported. The within-run MSR’s are only for comparison backwards to the test-retest study results, and for times when compounds are compared within the same run of an assay. As with example 1, there are no apparent temporal problems, i.e. this is a stable assay with an overall MSR of 2.0. This assay is less variable than the assay in example 1.

Mean Summary				MSR Summary
	Ref 1	Ref 2	Ratio		WR	Overall
High	505.26	10.84	0.02	High	1.9	2.1
Low	324.19	3.42	0.01	Low	1.6	1.7
Overall	409.51	5.53	0.01	Overall	1.8	2.0
Current	462.99	7.21	0.02	Current	1.9	2.0

Example 2. Potency, MSR Chart, and Summary Statistics for an Assay with Two Controls

Examples 3 and 4 illustrate problems with a shift in compound potency. Example 3 illustrates a steady degradation in potency over time, whereas Example 4 illustrates a more sudden shift in potency at a particular point in time. In Example 3 the assay variability appears to be shrinking, while in Example 4 it appears to be stationary. Repetitive freeze-thaw cycles of a compound may cause a slow degradation in potency whereas a change in lot of a key assay ingredient may result in a sudden potency shift. In both cases it is important to identify the cause and correct it as soon as possible.

Example 3. Potency and MSR Chart Illustrating Assay Drift

Example 4. Potency and MSR Chart Illustrating Sudden Change In Potency

Example 5 illustrates an assay with stable potency, but in June the assay variability increased. The moving MSR was stable around 3, but after June increased to over 10, and remained there. This also is most likely caused by a change in the assay process around that time. Again it is important to identify and correct the cause as soon as possible. Note however that a single outlier will cause the MSR chart to increase for the next 6 runs, and so it usually takes more time to correctly distinguish a change in assay variability from a single outlier result.

Example 5. Potency and MSR chart for change in assay variability

Retrospective Studies

During the course of project/program development numerous compounds are repeatedly evaluated and stored in archival databases. This data can be mined to examine the reproducibility of assay results. This work should always be done by a statistician as the repeated compounds are not a random selection of all compounds, and may be biased with respect to time of evaluation, potency, structure and “assayability” (the latter term is meant to reflect conditions such as solubility, quenching, stickiness to plastic and other practical problems). In spite of these potential problems retrospective studies can be a very useful exercise, particularly in establishing the acceptability of older assays that have never been formally evaluated for reproducibility. In addition, the MSR can be examined over various subsets such as potency range, structure and run date to check that the control compound MSR’s are representative of the test compounds with respect to potency range, structure and run date.

Bridging Studies

If a key aspect of an assay changes, such as an equipment change or lot of a reagent, then a test-retest study should be conducted to verify equivalence of the two protocols. A judgment should be made on a case-by-case basis of whether the full protocol outlined in Section II.B needs to be made, or only a single run under old and new conditions (i.e. one might do just Step 4 of the procedure, or one might do both Steps 3 and 4 depending upon the severity of the protocol change). Also in cases of specific modifications such as replacing equipment for a particular step in the assay an experiment can be designed to validate that the replacement is equivalent to the original in the conduct of that step of the assay.

Dimethylsulfoxide: biological compatibility and compound storage Dimethylsulfoxide (DMSO) is a universal solvent for all compounds tested in high, medium and low throughput screens (HTS, MTS and LTS). Compounds are initially dissolved in 100% DMSO and further diluted into water and assay buffers in subsequent dilutions for screening and IC50 or Ki determinations. It is extremely important that the DMSO compatibility of biological reagents such as enzymes, receptors, protein/peptide reagents and cells be established to ensure that the screening assays are not adversely affected. In general, the final DMSO concentrations in cell-based assays are <0.2% and are <1% in biochemical assays. It is highly recommended that the tolerable DMSO concentration be determined individually for each validated assay.

DMSO is also used as a cryoprotectant in the freezing of cell cultures at ATCC. The product is cell culture grade and has been tested to ensure cell viability. Each lot is also tested for the absence of bacteria, fungi, and endotoxin.

When solubilized compounds are stored in DMSO, it is important to understand the stability of these compounds under various storage conditions and freeze-thaw cycles. A detailed study of these effects was published recently (1). It is believed that the degradation of DMSO solubilized compounds is mainly due to moisture absorbed from the air. This can happen during frequent freeze-thaw cycles of compounds stored frozen in DMSO, or frequent exposure to air during repeated access for biological testing (cherry-picking).

Recommended storage conditions for DMSO solubilized compounds: