Mahmood Ahmad
Tahir Heart Institute
mahmood.ahmad2@nhs.net

The Fragility Atlas: Multiverse Analysis of 403 Cochrane Meta-Analyses Reveals 58% Are Sensitive to Analyst Choices

How robust are Cochrane meta-analysis conclusions when analysts make different but equally reasonable methodological choices? We applied multiverse analysis to 403 Cochrane systematic reviews from the Pairwise70 dataset, encompassing 394,569 individual meta-analytic specifications across five analytical dimensions. Each review was re-analysed across seven variance estimators, three confidence interval methods, three publication bias corrections, and leave-one-out subsets, then classified by agreement with the standard DerSimonian-Laird reference conclusion. The mean robustness score was 66.4 percent (median 66.7 percent, IQR 56.5 to 87.2 percent), with 58 percent of all reviews classified as Fragile or Unstable. Publication bias correction was the single most influential dimension, explaining ten times more variance (eta-squared 0.374) than variance estimator choice (eta-squared 0.036). These findings suggest that most Cochrane conclusions are substantially sensitive to defensible analytical alternatives, particularly the decision whether to adjust for publication bias. This analysis is limited to binary agreement classification and cannot capture the magnitude of effect size changes across specifications.

Outside Notes

Type: methods
Primary estimand: Robustness score (percent agreement across specifications)
App: Fragility Atlas v1.0
Data: Pairwise70 dataset: 403 Cochrane reviews, 394,569 specifications
Code: https://github.com/mahmood726-cyber/fragility-atlas
Version: 1.0
Certainty: high
Validation: DRAFT

References

1. Walsh M, Srinathan SK, McAuley DF, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. J Clin Epidemiol. 2014;67(6):622-628.
2. Atal I, Porcher R, Boutron I, Ravaud P. The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses. J Clin Epidemiol. 2019;111:32-40.
3. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. 2nd ed. Wiley; 2021.