Reproducible meta-analysis harness

What this is

A reproducible harness that builds meta-analyses from a committed protocol, and publishes each as a tabbed, auditable page. Every page here passed a two-limb gate: it reproduces from a fresh clone with zero census failures and its served analysis method equals its declared method; and it names a published open-access comparator with the trial-set overlap stated.

What this is not

It is not a claim of stronger evidence than the peer-reviewed comparators. The offer is greater auditability: every number is traceable to a committed source, every absence is declared rather than left blank, and any hand-edit breaks the gate. A page whose estimate matches its comparator on an identical trial set is arithmetic, not corroboration — so the overlap is stated on every page.

The result, in one paragraph

Where this harness pools fewer trials than a published comparator, the difference is the comparator's design, scope and definition choices — not our search or extraction failures: open-label trials we exclude for requiring double-blinding, different outcome definitions, prophylaxis pooled with treatment, drug-class metas compared to a single agent, and per-arm variances imputed from figures where we decline to impute. The evidence is measured, not asserted: citation chasing recovered 9 of 9 pre-registry trials the registries cannot reach (so reach is not the limit); an independent reported-effect extractor run over 626 declared-absent cells recovered 0 clean numbers ours missed (so extraction is not the limit); every same-scope gap is decomposed and named on its page; and every pooled number is verified against its committed source and gate-enforced (the exact count is stated below). The offer is greater auditability, honestly bounded — not a claim of more evidence than the peer-reviewed comparators.

The sharpest form of the argument. We attempted a per-trial, number-by-number check against the gold standard — the Cochrane review of metformin for PCOS ovulation (CD013505) — and found that 0 of its 18 trials expose their per-trial counts in machine-readable form: the numbers live in forest-plot images, not the text. A reader cannot check a Cochrane review number-by-number. They can check ours — every pooled number here carries a verbatim source span, and the whole review regenerates from a committed protocol commit. (Extracting per-trial ground truth from forest-plot images is a named research-agenda item — vision/OCR or individual patient data — not yet attempted.)

We measured our own error rate (no meta-analysis reports this about itself)

Measured on 2026-09-12 against the 99 pooled numbers committed at that time (the sample is committed in docs/error_rate_sample.json); a freshness invariant refuses a stale figure if the pooled population changes. Every claim the harness makes rests on the assumption that its numbers are right. So we measured it: all 99 pooled numbers were independently re-extracted from the committed source by an offline checker blind to the stored value, then compared deterministically. 95 of 99 were re-extractable from the same source the checker was given; 92 of 95 matched exactly (‘exactly’ = the effect and both confidence limits agree to the rounding of the source's printed precision, and counts agree as integers). The 3 disagreements were hand-adjudicated against source: on adjudication 1 was a genuine error on our side (a gastrointestinal-adverse-event outcome that had pooled the trial's OVERALL adverse-event count — a wrong endpoint that had passed every gate; found here and fixed), and the remainder were checker-side (an incidence-rate ratio the checker called a plain rate ratio with identical numbers; an on-treatment vs intention-to-treat estimand choice where our ITT value is the standard one). The other 4 numbers source from ClinicalTrials.gov results or full text, so they were not re-checkable from the abstract and are not counted as verified here. The pre-adjudication disagreement rate was 3 of 95 (Wilson 95% CI 1.1–8.9%). The honest caveat that makes this credible: the blind checker and the extractor share a model architecture, so this is an internal-consistency measure, not an independent accuracy estimate — a genuinely independent, cross-family (non-Claude) re-extraction is the stronger check, and is being built. It is nonetheless the single most important number the project lacked, and it is measured, adjudicated, and reproducible from scripts/error_rate_compare.py.

Independent cross-family check — the fix for ‘everything is self-assessed’

Our checker and our error-rate sampler share a model architecture with our extractor, so the internal error rate is a consistency measure, not independent accuracy. So a different model family — Gemini 3.1 Pro, via AGY, sharing no architecture with our pipeline — independently re-extracted every pooled number from the same committed source. It agreed with our stored value on 84 of 90 comparable numbers (93.3%); on 62 numbers all three families (our harness, a GPT-5 checker, and Gemini) agree. Every one of the 6 disagreements was hand-adjudicated against source and 0 was a wrong number: they are a documented approved-dose rule, intention-to-treat vs the trial's on-treatment primary, a rounding tie, one registry-vs-abstract count, and one CT.gov-vs-abstract estimand difference (all disclosed in docs/crossfamily.json). A model call is treated as a source — the Gemini outputs are committed, so this regenerates without re-calling the model. Three-family agreement is a far stronger claim than our own dual extraction.

And the judge was re-run cross-family. The page-vs-comparator auditability verdict previously rested on a Claude-family judge. A non-Claude judge (Gemini) re-judged the same blinded pairs on 32 dimension-comparisons across 8 topics and rated our page more auditable on 32 of 32 (agreeing with the Claude judge on 28 of 32, and more favourable to us on the rest). The auditability result is not an artefact of an AI liking a page in its own style — a different family reaches the same verdict (docs/crossfamily_judge.json).

Cross-family definition audit: is the number under the RIGHT label?

A single cross-family QA pass over four rows had caught a defect (TECOS's 4-point composite pooled under a 3-point label) that every internal gate passed — so we ran it over the whole corpus. Two independent families (Gemini via AGY and Fable) re-read every one of the 101 pooled rows and asked not just ‘does the number match’ but does the outcome DEFINITION match the label — composite component set, timepoint, population, analysis set. 33 of 101 rows were flagged for a possible definition mismatch (20 by both families). Adjudicated against source: 4 were genuine wrong-endpoint/population defects and were refused at source this cycle (two omega-3 trials whose composite was not MACE; two probiotics trials pooling a per-protocol/completers set, not ITT); 11 are the well-known heterogeneity of pooling each trial's own primary MACE (3–5 component composites) under one generic label, now being disclosed; 4 were already disclosed on the page (a stated subgroup or estimand); and 14 remain queued for per-row adjudication. This class — a right number under a slightly wrong label — is invisible to a value check and was only found by a different model family reading the source fresh. The audit is now a standing check (docs/definition_audit.json) and its guard refuses component/population mismatches at build time.

External validation: our pooled numbers vs the published meta-analyses'

The strongest check is against an external hand-built standard. For every topic with a named open-access comparator we compared our pooled primary estimate to that published meta-analysis's own reported pooled estimate on the same question. 21 of 28 agree within ~12% on the log scale — our independent, from-scratch synthesis lands on the same number as the peer-reviewed review (e.g. NOAC stroke/SE 0.81 vs 0.81; GLP-1 MACE 0.85 vs 0.86; probiotics AAD 0.63 vs 0.63; finerenone kidney 0.84 vs 0.84). Every one of the 7 divergences is adjudicated in docs/external_agreement.json and is a documented smaller-evidence-base case (we pool k=1 where the comparator pools many — metformin-PCOS, elderly-statins, CAP-steroids) or a scale/scope mismatch (esketamine MADRS-change vs response-rate), not an extraction error. This is pooled-level agreement. The per-trial head-to-head was attempted against the gold standard (the Cochrane review CD013505 of metformin for PCOS ovulation): its pooled OR 2.64 (k=13) sits far from our single-trial OR 8.25 (k=1) — a stark, honest illustration of the small-k weakness the expansion tier targets — but a true number-by-number check is blocked even for Cochrane: its per-woman arm counts live in forest-plot images, not the open-access text (only per-cycle data is tabulated). Per-trial ground truth needs vision/OCR or IPD (docs/cochrane_headtohead.json).

Is our small k our limit or the question's? Of the topics with a same-scope comparator, 10 are at or above the complete same-scope evidence (our k equals the comparable comparator's — the smallness is the literature's, not ours); 4 are 1–2 trials short (bar-limited, decomposed on the page); and 6 face a genuinely larger literature where the gap is named per topic (open-label excluded, different outcome definition, prophylaxis-vs-treatment, or reach). So small k is labelled, not hidden — and where it is the question's limit we say so.

The same result from a second angle: the continuous tier is three pages, and here is why it is three

The continuous-primary tier (mean-difference outcomes: melatonin sleep-onset latency, esketamine MADRS, semaglutide weight) is bar-limited, not effort-limited. Growing it was attempted and five candidate topics were probed against their own posted ClinicalTrials.gov results and five were declined, each with a named reason: regulatory efficacy endpoints (FEV1, blood pressure, HbA1c) are posted as least-squares means with standard errors (ANCOVA), which we refuse rather than silently convert (roflumilast, tiotropium, renal-denervation); and symptom scales vary in instrument, timepoint and design across trials, so no same-scope pool of ≥2 forms (pregabalin pain; liraglutide obesity is multi-arm / different timepoints). A topic builds cleanly here only when one registered outcome with raw per-arm mean±SD is reported at one common timepoint across same-scope two-arm trials — a rare alignment. Semaglutide is the clearest single illustration of the standard: it went from k=4, a tight and statistically significant pool, to k=2, MD −11.84% (95% CI −25.13 to 1.44) — an interval that now crosses zero — because a timepoint-consistency guard refused to pool two Week-44 trials into a pre-registered Week-68 outcome. We gave up significance to keep the timepoints consistent. No comparator reports having made that trade. Together with the paragraph above this makes one claim: where we pool less, it is because of a stated bar — and the bar is shown, not asserted.

Specification curve: the headline is the most conservative standard choice

Each primary outcome with ≥2 trials (16 of them) was re-pooled under three defensible specifications on the model/interval axis: random-effects with the Hartung-Knapp interval (the harness default), random-effects with a Wald/z interval, and a fixed-effect model. The direction of effect was stable across all three in 16 of 16 — the sign never depends on the choice. Statistical significance was identical across all three in only 6 of 16, and in 10 of 16 the divergence runs one way: our default interval is not significant while the less conservative fixed-effect or z specification would be. In other words the harness reports the widest of the standard intervals at small k, so where we do claim significance it survives the stricter choice — the curve is regenerable via scripts/spec_curve.py.

Screening reproducibility, measured against an independent blind screener

An independent screener — blind to our decision, working from title and abstract against the same eligibility criteria — re-screened the pooled records. Over the 2010 records that carry an abstract, agreement with our rule-screener was 96.8% (Cohen's κ = 0.839), and it recovered 93.3% of the records we included. This second screener is model-assisted, so this is a measure of screening reproducibility, not accuracy against a human gold standard; 489 records with no abstract text in the cache (registry / citation-chase entries) could not be shown to it and are reported as not-assessable rather than dropped from the base. Regenerable via scripts/screen_reproducibility.py.

Where the numbers come from — and what we do not yet claim

Provenance mix (a tracked metric). Of 100 pooled numbers, 71 come from the trial abstract (the authors' headline result — the weakest source, though the safest default) and 29 from higher tiers (ClinicalTrials.gov structured results, PMC full text, hand-verified arm counts). Nearly every wrong number the audit found came from an abstract. We tested how far the abstract share can be driven down and the answer is a bar, not effort: only a handful of the abstract-sourced rows have a ClinicalTrials.gov structured result available to promote to, and CT.gov results carry their own estimand/definition risk — exactly the class the definition audit just proved (a structured number can be the wrong composite/timepoint). Full-text promotion is available but per-row expensive (as done for the SGLT2 heart-failure hospitalizations). So abstract-dominance is bar-limited, not effort-limited; the honest fix is not bulk-promotion but the definition guards that make an abstract number safe to pool (scripts/provenance.py).

Stated limitations (plainly). (1) The comparators are open-access only — we benchmark against free reviews, not necessarily the best ones. (2) Protocol registration is self-hosted (a git commit SHA), with no external timestamp authority — it proves order relative to our own history, not against a third party. (3) Topic selection is ours, which can flatter the success rate; the expansion tier is preregistered in one batch with declared-hard cases to counter this. (4) The headline accuracy figure began as an internal-consistency measure; an independent non-Claude model family has since re-extracted every pooled row against the same committed source (reported in the cross-family section above), so it now carries a genuinely independent check — though the census and blind-judging arms still share our architecture, which leaves that cross-family re-extraction as the sole fully-decorrelated signal.

Every pooled number is verified against its source (gate-enforced)

All 100 of 100 pooled trial-outcome numbers across these pages have their digits located in the committed source span they cite (arm counts, effect+CI, or per-arm mean/SD). A publication-gate limb (check_pooled_verified) refuses any page that pools a number not found in its source, so this cannot silently stop being true. No published meta-analysis makes — or can be forced to keep — this claim about every one of its numbers.

Parity with the published comparator (the finishing metric)

For each same-scope topic: our pooled k vs the comparable comparator k (the comparator's pooled list, enumerated from its own references/full text, after removing trials that are out of scope, double-counted substudies, observational, or non-prespecified for the outcome). 9 of 21 same-scope topics are at parity within scope; every remaining shortfall has a named reason. This is a measurement snapshot (the enumeration is model-assisted; scope calls are assessments, and each pooled recovery was verified against source before it counted).

Screened against a meta-analysis error library (measured, not asserted)

Every documented meta-analysis mistake is converted into one of: a gate limb that refuses a finished review, a regression test with a plant that fires pre-fix, or a rendered disclosure when it is a judgement the harness cannot make. Of 33 documented errors catalogued (9 gate limbs, 12 regression tests, 12 rendered disclosures), 33 have an enforced mechanism, and every live review is screened against 32–33 of 33 of them. This is a claim no published meta-analysis makes about itself, and it is directly checkable (each entry names its mechanism in harness/error_library.py; the count regenerates via scripts/error_coverage.py).

Methodological stance (measured, not asserted)

We decline where the comparator imputed. On continuous outcomes (e.g. zinc for cold duration) the published comparator reconstructed per-arm variances from Kaplan–Meier curves and figures, or used paywalled values; of its 7 pooled trials, 0 report a verifiable per-arm SD in accessible primary text. We pool only what we can verify against source, so we decline those cells rather than impute — a genuine methodological difference, not a shortfall.

The reach picture is complete — all three routes measured. FDA/EMA and multi-registry (ISRCTN/EU-CTR/ICTRP) adapters reach newer approval-label and registered trials but cannot reach old pre-registry RCTs (DART 1989, GISSI-Prevenzione 1999, JELIS 2007, CORE/COPE 2005). Bibliographic citation chasing (Crossref) DOES reach them — 9 of 9 recovered from the comparators' own reference lists. But our preregistered screening then declines most for a stated, defensible reason: they are open-label where we require double-blind (CORE, COPE, JELIS, GISSI-P), or the population/outcome does not match. So the remaining k gaps are screening-strictness decisions, not search failures — the comparators' larger pools include open-label and off-outcome trials our criteria exclude. Each affected page names which recovered trials were declined and why.

Fair comparison (measured against comparator FULL TEXT)

An earlier countable PRISMA comparison read the comparators' abstracts against our full pages — a confound in our favour, which we flagged and then fixed by reading each comparator's OA full text (28 of 29 topics scorable; the remainder have no obtainable comparator full text). Scored fairly, full-text vs full-page across the 28 scorable topics (168 cells, including the continuous-outcome pages): the comparators satisfy 94 of 168 checkable PRISMA-item cells, yet there is no reporting item a comparator's full text presents that our page lacks (0 of 168), while we present 74 that even their full text does not (chiefly per-record exclusion reasons and a machine-checkable registration SHA). The margin narrowed under fair measurement, as it should; the direction held.

Fair blind re-judge (full-text vs full-page, 28 topics, order-blinded). Restating the record on the fair basis, whatever it shows: our pages are judged more auditable on 28 of 28, winning per-number traceability 28–0; declared-absence 28–0; overall auditability 28–0; search reproducibility 27–1; the comparator is more complete on 28 of 28 (larger k); risk-of-bias reporting splits 13–14 to the comparator. The split holds across the whole judged set, binary and continuous alike — each page is more auditable and less complete than its comparator — but it is not a clean sweep on every sub-dimension: the comparator wins search reproducibility on one topic and edges risk-of-bias reporting, and those losses are shown here rather than hidden. So the earlier abstract-based ‘15 clean wins’ is superseded by a stable domain split: we win transparency and auditability; we lose completeness/k — the same conclusion the parity table reaches, confirmed by a blind reader on full text. The judge also flagged real defects in our pages (a risk-of-bias table covering only a subset of pooled trials; a retraction line whose count did not equal k); those are recorded, not hidden.

Selection effect (stated, not hidden)

Of 31 of 37 preregistered topics, the built pages are shown below. The unbuilt topics were disproportionately the HARD cases — continuous or recurrent-event outcomes, composite-only endpoints, percentage-only or paywalled reports, and older or unregistered literature — which the harness correctly declined rather than extract unverifiably. So the build success rate flatters the harness: it reflects a sample selected toward clean binary-outcome registered trials, not the whole field. The topic set is preregistered and the declines are recorded (JUDGELOG), so the selection is visible rather than silent.