Analysis of Datasets Containing No Biological Information

To examine the classification accuracy of various methods, and therefore the significance of each method, on datasets with no biological information, 30 artificial datasets with random peak intensities have been created.  All datasets contain 300 “peaks” and have the same number of Cases and Controls (30, 42, 60, 90, 150, and 300 of each category).  For each number of Cases and Controls, five independent datasets are available for download and analysis. Descriptions of the structure of the datasets as well as the procedure used in their generation are available.  Below is a summary of the results for distinguishing Cases from Controls for all 30 datasets using the BioMarker Development Kit (BMDK), a decision tree (DT), and a medoid classification algorithm (MCA).  Specific results for each dataset using these three classification methods are also available.

Table 1 lists the best classification results obtained by the BMDK suite of programs for each dataset. The second and third columns list the number of Cases and Controls in each dataset and the fourth column lists the number of putative biomarkers identified by the 10 methods currently employed in BMDK.  The final three columns list the quality of the best classifier after an exhaustive examination of all sets of one, two, and three putative biomarkers using a distance-dependent 6-nearest neighbor algorithm, respectively.  Since this classifier has the ability to return a classification of “Undetermined”, the quality of this classifier is the sum of the sensitivity and specificity minus the percentage of samples that are “Undetermined”.  Since an “Undetermined” classification may be caused by an incomplete coverage of feature space by the available samples, which should be avoided, the results in Table 1 have the added restriction that %undetermined cannot exceed 5% for any classifier.  With this restriction, the fifth dataset containing 30 Cases and 30 Controls (Set 30_5a) could not find a single valid 3-peak classifier.  These results use a standard Euclidean distance to determine the distance between samples in feature space since none of the other three available distance metrics were able to identify a valid 3-peak classifier for any of the 30 datasets.

If the minimum required accuracy of a classifier is a sensitivity and specificity of 85%, the minimum acceptable quality score is 165, assuming that %undetermined=5%.  None of the results presented in Table 1 reach this minimum level of accuracy.  This result is expected since the datasets are constructed to contain no information.  It is also expected that the quality of the best classifier should decrease as the number of Cases and Controls increases.  While this is generally true, the 1-peak classifier for the fifth dataset containing 60 Cases and 60 Controls (Set 60_5a) produced an anomalously high quality of 151.7 (sensitivity=78.3%, specificity=73.3%, %undetermined=0.0%).  An intensity plot for this peak is shown in Figure 1.  The left column shows the intensities for all Cases, while the intensities for all Controls are in the left column. It is clear from this figure that the accuracy of the peak is caused by a high density of intensities for one category in a region with a low density of intensities for the other.  The only way to justify this peak as a useful classifier is to assume that each category is composed of several States and that each State is represented by a specific range of intensities for this peak.  This division of the individuals in each category into multiple States would have to be biologically verified before this classifier should be used; even though its quality is still well below the minimum threshold.<

The classification results using a decision tree (DT) are listed in Table 2.  This table lists the quality (sum of sensitivity and specificity) of the best and 200^th best decision tree across four runs for each dataset.  Two of the runs convert a decision node into a terminal node if it contains at most 1% of either the Cases or Controls, while the other two runs increases this criteria to 4%.  All four runs use a different seed to the random number generator.  For the five datasets with 30 Cases and 30 Controls, 18 of the 20 runs identify at least 200 decision trees with an average sensitivity and specificity of 95% or higher. All runs identified 200 decision trees with an average sensitivity and specificity above 91% and one run identified at least 200 decision trees with a sensitivity and specificity of 100%.

Using and average sensitivity and specificity of 85% as a minimum required accuracy for a “useful” classifier, Table 2 shows that this level is regularly achieved if the dataset contains 60 Cases and 60 Controls or less.  If there are 90 Cases and 90 Controls, an average sensitivity and specificity of 83.3% is achieved, which is slightly below the required accuracy.  For the largest dataset (300 Cases and 300 Controls) the average sensitivity and specificity is below 70% but still much higher than the BMDK results in Table 1.

The classification accuracies (sum of sensitivity and specificity) using the mediod classification algorithm (MCA) are listed in Table 3.  For each dataset, two runs are performed when five, six or seven of the 300 features are used in the classifier.  The first examines all Cases and then all Controls while the second reverses this order.  All six runs for a given dataset use a different seed to the random number generator and the number of Case-cells or Control-cells is not allowed to exceed two-thirds of the number of Cases or Controls, respectively.  This would allow up to one-third of the samples to effectively be treated as a testing set while still maintaining full coverage of the fingerprint patterns.

If the dataset contains only 30 Cases and 30 Controls, all runs found at least one classifier that produced a sensitivity and specificity of 100% without requiring more than 20 Cases and 20 Controls for complete coverage.  Two of the 6-feature runs and four of the 7-feature runs produced a final population where at least 200 unique classifiers produced a sensitivity and specificity of 100%.  For the largest dataset (300 Cases and 300 Controls) one 5-feature classifier was identified that had an average sensitivity and specificity of 85%, the other nine runs found a classifier with an average sensitivity and specificity of 83.5% or higher.  All 6-feature and 7-feature runs produced a final population that contains at least 200 unique classifiers with an average sensitivity and specificity above 85%; one 7-feature run found a classifier with an average sensitivity and specificity above 90.1%.

Conclusions

While it has been argued [Pet-03] that accurate classification of a testing set must imply some underlying biological principle, the results presented here clearly shows that this is not true for fingerprint-based classifiers; especially MCA classifiers.  Hundreds of 7-feature MCA classifiers produce an average sensitivity and specificity above 90% for datasets with 150 Cases, 150 Controls, and only 300 features (Table 3) even though the datasets are constructed to contain no biological information. A chance [Ran-05a, Ran-05b] fitting of the data is highly possible since fingerprint-based classifiers appear to be overly flexible.  Increasing the number of samples in the dataset generally decreases the quality of the best classifier if it is a chance fit, but increasing the number of features in the dataset will increase this quality.  For fingerprint-based classifiers, the quality should improve if more features are used in the classifier; while the results in Table 1 suggest that increasing the number of features from two to three does not make much of an improvement, and may actually produce poorer results.

The bottom line is that fingerprint-based classifiers should not be used to analyze a dataset.  Problems of chance, coverage, uniqueness, and significance result in concluding that fingerprint-based classifiers are not generalizable to the underlying population. 

Figure 1:  Intensities for the 60 cases (left column) and 60 controls (right column) for the peak that yielded a quality score of 151.7 (sensitivity=78.3%, specificity=73.3%, undetermined=0.0%) in the dataset of random peak intensities.

Table 1: Classification accuracy (sum of the sensitivity and specificity minus the percent undetermined) using between one and three peaks from the list of NPB putative biomarkers identified by BMDK in a distance-dependent 6-nearest neighbor classifier using absolute differences in the peak intensities and requiring that the %Undetermined is no more than 5%.

Table 2: Classification accuracy (sum of the sensitivity and specificity) using the decision tree algorithm for the 1^st and 200^th best classifier as a function of the number of Cases and Controls.^(a)

^(a)A decision node was converted to a terminal node if it contained at most 1% of the samples from either category (1% runs) or if it contained at most 4% of the samples from either category (4% runs). All four runs used different seeds to the random number generator that controlled the Evolutionary Programming search.

Supplemental Table 6: Classification accuracy (sum of the sensitivity and specificity) using the medoid classification algorithm for the 1st and 200th best classifier as a function of the number of cases and controls from two runs using five, six, and seven peaks with random intensities.^(a)

^(a)Run 1 examined all cases and then controls, while Run 2 examined the controls and then the cases.

(Last updated 5/2/07)