Mahmood Ahmad
Tahir Heart Institute
mahmood.ahmad2@nhs.net

Outcome Reporting Bias Risk in 403 Cochrane Meta-Analyses: An Excess Significance Approach

What is the prevalence of statistical patterns consistent with outcome reporting bias across Cochrane systematic reviews? We applied excess significance testing and three complementary indicators to 403 Cochrane reviews from the Pairwise70 dataset, each containing at least three primary studies. A composite scoring system combining excess significance, heterogeneity, outlier ratio, and precision asymmetry classified reviews into Low, Moderate, and High risk categories. The prevalence of High risk was 16.1% (65 of 403, 95% CI 12.6-20.1%), with mean excess significance of 2.83 and mean I-squared of 70.3% compared to 14.8% in Low risk reviews. Spearman correlation between excess significance and heterogeneity was 0.36, confirming that reviews with more significant results than expected showed greater between-study variability. These findings suggest that one in six Cochrane reviews exhibits statistical signatures warranting closer scrutiny of pre-registration fidelity and outcome switching. A limitation is that statistical proxy indicators cannot distinguish outcome reporting bias from publication bias or other selective reporting mechanisms.

Outside Notes

Type: meta-research
Primary estimand: ORB risk prevalence
App: ORB Detector v1.0
Data: 403 Cochrane reviews from Pairwise70 dataset (k >= 3)
Code: https://github.com/mahmood726-cyber/outcome-reporting-bias
Version: 1.0
Validation: Author reviewed draft

References

1. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634.
2. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56(2):455-463.
3. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. 2nd ed. Wiley; 2021.