Recent Modifications in ANOVA Programs
Document Actions
December, 2005
We have recently made some modifications to 3dANOVA/3dANOVA2/3dANOVA3 that might cause changes in results compared to that with the old version. The effect of the changes will depend on your experiment design and the degree to which the design departures from sphericity. Manuals will be modified to reflect these changes in the near future.
In a nutshell, it is likely to see some significantly inflated t-statistics (i.e., lower p value) for output from options -amean/-bmean in the old version of 3dANOVA2/3dANOVA3 and that from options -acontr/-bcontr whose coefficients don't sum up to 0, but other results should be more or less comparable to that from the new version of the programs.
(1) Contrast testing
In all the three old ANOVA programs, options -contr, -diff, and -mean are consistent in the sense the underlying formulas are essentially the same; that is, -contr is basically good enough to handle all post hoc tests without resorting to -diff and -mean, which are just for convenient and intuitive usage.
However, in its strict sense a contrast is a comparison among several terms, which necessitates the weights for the terms adding up to 0. There are two issues involved in contrast testing with options -contr/-acontr/-bcontr in the old ANOVA programs:
The first one is very severe in old 3dANOVA2/3dANOVA3 with general linear tests when the weights don't add up to 0. This is not really about sphericity violation per se, instead the corresponding old t statistics are not appropriate. As the pooled (averaged) variances were employed in the relevant statistics in the old version of the programs, the expectation of the denominator squared is smaller than the variance of the numerator, invalidating the very definition of t statistic and causing inflated statistics. The reason this happens is that cross-subject variability is not accounted for when those coefficients don't cancel each other. The invalidation of the t statistics when coefficients don't sum up to 0 is not explicitly described in the manuals of the two programs, and occasionally such a general usage was indeed exploited by users. But the more troublesome situation is that simple effect testing with the common options -amean/-bmean in 3dANOVA2/3dANOVA3 is just a special case of general linear testing, as we will discuss it more in (2).
The situation with 3dANOVA in this regard is usually not a big concern due to its usage as a one-way between-subject analysis: cross-subject variability is not directly involved in the analysis, and pooled variance is more or less a legitimate estimate for the groups.
The other issue with contrast testing in the old version of the three programs is mostly very minor with the sphericity assumption: loosely speaking, the pair-wise correlation of a within-subject factor with more than two levels is homogeneous among all the factor levels, and the cross-group variances of a between-subject factor are identical. In most designs, this assumption is roughly met. However, the statistics adopted in the old version of the three programs are vulnerable when sphericity departure is significant. More discussion can be found here.
The modifications in the new version of the three programs solve both issues with one straightforward strategy: Any effect/contrast involving a portion of the data is tested exclusively based on that portion. Changes were made to avoid the severe violation of t-statistic definition and to render the aforementioned tests immune to sphericity violation. The changes may also lead to different set of degrees of freedom. Nevertheless, the consistency across the three options of -contr, -diff, and -mean are still preserved under this strategy in the new version of the three programs: one formula fits for all. However, when the coefficients in -contr don't add up to 0, the tests in the new programs are soundproof. It is expected to see big differences in the following cases compared to the old 3dANOVA2/3 in general linear testing with -acontr/-bcontr with coefficients not adding up to 0.
Here is the comparision of a contrast (-acontr 1 -1) of two conditions between the old (upper) and new (lower) version of 3dANOVA2 (courtesy of Dr. Stephen Rao's lab at Medical College of Wisconsin). The difference is subtle:
The three options -contr, -diff, and -mean in 3dANOVA have been modified to align up with the same philosophy. However, as it is used mainly for one-way between-subject analysis (multiple-sample test or generalized two-sample t test), no dramatic difference of results between the old and new 3dANOVA is expected.
(2) Simple effect testing
As explained above, the formulas for group effect testing (-amean in 3dANOVA2 -type 3, -amean and -bmean in 3dANOVA3 -type 4 or 5) were too liberal and generous, and not appropriate in the sense that cross-subject variability was not considered in the involving denominators, leading to potential significant inflation of t values for those tests. As a result, this would render more or/and stronger activation regions than what it ought to be, and again it is expected to see some extent of discrepancies in simple effect testing with -amean/-bmean in the old 3dANOVA2/3dANOVA3. Detailed discussion with more technical terms and new formulas can be found here.
Here is the comparision of a simple effect (-amean 1) between the old (upper) and new (lower) version of 3dANOVA2 (courtesy of Dr. Stephen Rao's lab at Medical College of Wisconsin). The difference is significant in this case:
(3) Second-order contrast testing
A new set of new options -aBcontr, -Abcontr, -aBdiff, -Abdiff, and -abmean (last three coming soon) now allow the user to run second-order contrast testing in 3dANOVA3.
Note: F tests for main effects and interactions, and general linear tests with more than two terms are currently not immune to sphericity violation in all ANOVA programs.
