Mahmood Ahmad
Tahir Heart Institute
mahmood.ahmad2@nhs.net

The Bias Fingerprint: Eight Methods Disagree in One of Seven Cochrane Meta-Analyses

When eight publication-bias methods are applied to the same meta-analysis, how often do they reach the same conclusion? We applied four detection tests and four correction methods to 307 Cochrane reviews with at least five studies from the Pairwise70 dataset. Each review received a bias fingerprint summarising which tests flagged bias and how corrections shifted the pooled estimate, classified as Clean, Suspected, Confirmed, or Discordant. Only 54 reviews (17.6%) were Clean while 42 were Discordant, giving a discordance rate of 13.7% and a median risk-ratio shift below 0.05 (95% CI 0.02 to 0.08). Agreement ranged from 98% between PET-PEESE and limit meta-analysis to 73% between trim-and-fill and regression corrections, with mean relative effect shift of 1.01. In roughly one in seven Cochrane reviews, the apparent direction of the pooled conclusion depends on which bias method is chosen. These methods detect statistical asymmetry rather than bias itself and cannot separate publication bias from clinical or methodological heterogeneity.

Outside Notes

Type: methods
Primary estimand: Discordance rate
App: BiasForensics v1.0
Data: Pairwise70 dataset (307 Cochrane reviews, k>=5)
Code: https://github.com/mahmood726-cyber/bias-forensics
Version: 1.0
Validation: Author reviewed draft

References

1. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634.
2. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56(2):455-463.
3. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. 2nd ed. Wiley; 2021.