Spec-Collapse Atlas

Correct inference for multiverse / many-analyst meta-analysis

A multiverse runs many specifications on one dataset. Summarising them by inverse-variance pooling — the usual move — treats correlated re-analyses as independent studies, collapsing the confidence interval and manufacturing robustness the data do not support. The weighted-likelihood interval accounts for both within- and between-specification uncertainty and restores calibrated coverage.

Try it on a real dataset

Pick a built-in meta-analysis. This tool runs the full 36-specification grid (3 τ² estimators × 2 CI methods × 3 outlier rules × raw/trim-and-fill) with the scaled-t weighted-likelihood interval — the same methodology as the corpus analysis below.

Across 473 Cochrane meta-analyses

Pairwise70 corpus, k≥3. Distribution of the interval-width ratio (naive IV-RE pool ÷ calibrated weighted-likelihood). Values well below 1 mean the naive summary understates uncertainty.

Why the naive pool fails — Monte-Carlo coverage under the null

Data simulated with no true effect (μ=0). A calibrated 95% interval should cover 0 about 95% of the time.

τ² (true)kIV-RE coverage WL coverageIV-RE Type-I

Nominal coverage is 0.95; nominal Type-I is 0.05. The naive pool's false-positive rate runs 70–81%.

Method: weighted-likelihood is a mixture of the per-specification likelihoods (uniform weights), each a scaled-t with df = k−1; its variance equals mean within-spec variance plus between-spec variance, so it is never narrower than a single specification. The corpus grid is 36 specs (3 τ² × 2 CI × 3 outlier × raw/trim-and-fill); the false-robustness rate is 52–55% across uniform / REML-only / HKSJ-only / AIC weighting. Engine validated against metafor's published dat.bcg result (REML τ²=0.3132, estimate=−0.7145) and REML reproduced two independent ways; trim-and-fill matches the fragility-atlas implementation to 1e−7. Corpus ingested via the fragility-atlas loader. Code & tests: GitHub. Offline single-file dashboard — no external dependencies.