Mahmood Ahmad
Tahir Heart Institute
mahmood.ahmad2@nhs.net

The Cochrane Quality Concordance: Six Orthogonal Dimensions of Evidence Reliability

Are six dimensions of evidence quality in systematic reviews correlated or orthogonal when measured simultaneously across Cochrane meta-analyses? We profiled 403 Cochrane reviews from Pairwise70 across fragility, publication bias, prediction gap, outcome reporting bias, reproducibility, and study overlap. Spearman rank correlations and composite quality grades were computed across all fifteen pairwise dimension combinations, with significance assessed after Bonferroni correction. The median composite score was 69.2 (95% CI 67.5-70.9), with the strongest pairwise correlation being publication bias versus outcome reporting bias at rho equals 0.468, confirming non-redundancy. Only 43 reviews earned a top-tier quality grade, while 30.5% exhibited simultaneous fragility and publication bias, a double jeopardy pattern invisible to single-dimension assessment tools. Evidence quality in systematic reviews is multidimensional and largely orthogonal, meaning that passing one quality check provides little assurance about performance on others. Nonetheless, this framework is limited to six computationally derived dimensions and does not incorporate clinical domain expertise or individual patient-level outcome data.

Outside Notes

Type: methods
Primary estimand: Mean absolute pairwise correlation
App: EvidenceQuality v1.0
Data: Pairwise70 dataset (403 Cochrane reviews, 6 quality dimensions)
Code: https://github.com/mahmood726-cyber/evidence-quality-concordance
Version: 1.0
Validation: DRAFT

References

1. Walsh M, Srinathan SK, McAuley DF, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. J Clin Epidemiol. 2014;67(6):622-628.
2. Atal I, Porcher R, Boutron I, Ravaud P. The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses. J Clin Epidemiol. 2019;111:32-40.
3. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. 2nd ed. Wiley; 2021.