Whole-Brain Thresholding & the Multiple Comparisons Problem

This re-runs the entire analysis 200 times on data with no activation anywhere, and reports how often each method reported at least one activation. That proportion is the family-wise error rate. A method that controls FWER at 0.05 should land near 5%; uncorrected thresholding will not come close.

Read these numbers with their error bars. Two hundred runs puts a standard error of roughly 1.5 percentage points on each figure, so a method showing 3% and one showing 7% have not been distinguished, and you should not conclude that cluster inference beat Bonferroni because it came in a point lower. What the audit can settle, several times over, is the gap between the corrected methods and the uncorrected one, which is not a couple of points but an order of magnitude.

Four ways to threshold a map, and what each one promises:

1. Uncorrected thresholding applies a single-test threshold to every voxel independently. With a couple of thousand voxels in the search space and α = 0.001, you should expect a couple of false voxels in every map you produce, from pure noise, forever. Switch the truth to Pure Null, where nothing whatsoever is active, and look at the uncorrected panel: there is almost always something there, and with a little spatial smoothness it will look like a plausible blob in a plausible place. This is the logic of the dead salmon result of Bennett and colleagues, who ran a standard uncorrected analysis on a post-mortem Atlantic salmon and recovered a cluster of “activation” in its brain cavity. The salmon was not the point; the threshold was.

2. Bonferroni correction divides α by the number of tests, which controls the family-wise error rate exactly. It is also the most conservative option available, and it assumes the tests are independent. Neighbouring voxels in an fMRI map are emphatically not independent: they share signal through smoothing and through the vasculature, so the effective number of independent tests is far smaller than the voxel count. Bonferroni therefore corrects for tests you did not really perform, and the price is paid in the weak focus, which it will usually miss entirely. Raise the noise smoothness slider and Bonferroni gets relatively worse, because the effective number of independent tests falls further while the correction stays the same.

3. False Discovery Rate (Benjamini–Hochberg) changes the question. Instead of controlling the probability of any false positive, it controls the expected proportion of your declared activations that are false. Sort the m p-values ascending, find the largest i with p_(i) ≤ iq/m, and reject everything up to it. The left-hand plot shows exactly that crossing. FDR is adaptive: when there is a lot of real signal the threshold relaxes, and when there is none it becomes almost as strict as Bonferroni. That is why its measured FWER in the audit below sits near the nominal level on null data while still being far more sensitive than Bonferroni when signal is present. It is a different guarantee, not a weaker version of the same one: accepting an FDR of 0.05 means accepting that about one in twenty of the voxels you report is wrong.

4. Cluster-extent inference gives up on individual voxels. It applies a lenient cluster-defining threshold, then asks whether any resulting blob is bigger than blobs that arise by chance. The null distribution of the largest cluster is not something you can look up, because it depends on the smoothness of your data, so this demo estimates it by Monte Carlo: 200 analyses of pure noise at the current smoothness, the largest cluster recorded each time, and the 95th percentile taken as the threshold. The right-hand histogram is that distribution. Cluster inference is usually the most sensitive of the three valid methods for spatially extended effects, and it is the standard choice in the field, but note what it buys you and what it does not: a significant cluster means something in this blob is real, not this voxel is real. You cannot legitimately point at the peak.

It also has a blind spot worth knowing about, and the three foci in this simulation are sized to expose it. Cluster inference matches uncorrected thresholding on the broad focus, detecting it about 89% of the time against Bonferroni's 62%, because extent is exactly what it rewards. On the tight focus it is the worst of all four methods, finding it around 5% of the time where FDR manages 24%, because a genuinely focal effect never assembles enough suprathreshold voxels to clear the extent threshold no matter how strong it is. The method that wins on one kind of signal loses on the other, so "which correction is best" has no answer independent of what you expect to find.

5. Why the cluster-defining threshold matters so much. Drop the CDT to p < 0.01 and watch the calibrated extent threshold jump, because lenient thresholds produce sprawling noise clusters. Eklund, Nichols and Knutsson showed in 2016 that the parametric random field theory approximations used by the major packages become badly anti-conservative at exactly this setting, with measured family-wise error rates reaching 70% where 5% was claimed, because the theory assumes a spatial autocorrelation shape that real data does not have. The Monte Carlo calibration used here does not make that assumption, which is the same reasoning behind the shift toward permutation-based inference.

6. What this demo simplifies. One slice rather than a volume, so m is in the thousands instead of the hundreds of thousands, which makes Bonferroni look far kinder than it is in practice. The noise is stationary, with the same smoothness everywhere, whereas real data is smoother in some tissue than others, and the residual autocorrelation modelled in the GLM Studio is absent here. Every one of those simplifications makes the corrections look better than they are, and the uncorrected map still fails.