Big Data Neuroimaging · Population Cohort Sizes, Modest Effect Sizes, The Winner's Curse & Out-of-Sample Replication
Set \(N = 35\) and run the Monte Carlo audit. Notice how the few studies that reach \(p < 0.05\) report \(r \approx 0.45\) (a 4.5× inflation of the true \(r = 0.10\) effect), and replication rate is under 15%. Now increase \(N \ge 2,000\) to see inflation disappear and replication climb above 90%.
Large-scale consortia (e.g. Human Connectome Project, Adolescent Brain Cognitive Development, UK Biobank) aggregating tens of thousands of participants established that univariate brain-behavior correlations rarely exceed \(|r| \approx 0.15\). Complex behavioral traits (intelligence, depression, executive function) are distributed across millions of synapses and distributed neural networks, meaning no single brain voxel or structural connection accounts for more than 1–2% of variance.
When an underpowered study (\(N = 30\)) searches for a small effect (\(r = 0.10\)), statistical power is typically below 8%. For an observed estimate to clear the conventional two-tailed threshold of \(p < 0.05\), it requires \(|r_{\text{obs}}| > 0.33\). Thus, the only studies that get published are those where thermal noise, sampling variability, and chance alignment happen to align in the same direction, generating massive effect size inflation.
When researchers attempt to replicate an inflated small-sample discovery in a new independent cohort, the chance noise disappears. The expected replication effect size regresses to the true population value (\(r \approx 0.10\)). Because the replication cohort is also underpowered, it fails to achieve significance (\(p > 0.05\)) over 85% of the time, creating an apparent failure to replicate.