KFRE external-validation meta-analysis

Included cohorts

Six external-validation cohorts of the 4-variable KFRE at the 5-year horizon. Each row links to PubMed via PMID; the DOI is also given. C-statistic and 95% CI are quoted verbatim from each publication's abstract; SE(C) is derived from the CI assuming Wald symmetry (SE = (CI_high − CI_low) / (2 × 1.96)) for input to the logit-scale pool.

Cohort detail

Discrimination — C-statistic forest

Per-cohort C-statistic with Wald 95% CI (back-transformed from logit-C). Pooled diamond is Paule-Mandel REML + HKSJ with the Cochrane v6.5 small-sample floor. The dashed orange line is the 95% prediction interval (t_k-1; the expected range for the C-statistic of a notional seventh external validation drawn from the same underlying population of validation studies).

Figure 1. Pooled C-statistic of the 4-variable Kidney Failure Risk Equation at the 5-year prediction horizon across six external validation cohorts. Blue squares are per-cohort point estimates with 95% CIs. The cyan diamond is the pooled estimate by Paule-Mandel REML τ² with HKSJ small-sample CI (Cochrane v6.5 floor max(1, Q/(k−1)); Student-t_k-1 critical value). The dashed orange bracket below the diamond is the 95% prediction interval (Higgins 2009; Riley 2011), reflecting the between-cohort variability in the underlying logit-C population. Heterogeneity is high (I² ≈ 97%; τ² point estimate on logit-C scale ≈ 0.68 with q-profile 95% CI [≈ 0.25, ≈ 4.42], Viechtbauer 2007), driven by a wide spread of validation settings: primary care (C = 0.93 in Major 2019), advanced CKD (C = 0.77–0.82 in Ali 2021 and Gallego-Valcarce 2024), kidney transplant recipients (C = 0.73 in Tangri 2020 — a different prediction context to the derivation cohort), and population data linkage (C = 0.96 in Irish 2023). The wide HKSJ-adjusted CI is a deliberate honest reflection of this heterogeneity, not a numerical artefact. The heterogeneity-fragility flag (parity with the pairwise-pool engine) is agree here: both the pooled CI and the prediction interval lie above logit(0.5) = 0, so the conclusion of useful discrimination is not heterogeneity-fragile.

Variance derivation: SE(C) is taken from the published 95% CI assuming Wald symmetry. Delta-method conversion to the logit scale: SE(logit C) = SE(C) / [C(1−C)]. Pooling on the logit-C scale follows Snell et al. 2018 (BMC Med Res Methodol) and Debray et al. 2017 (BMJ tutorial). Back-transformation is via the inverse-logit (sigmoid) function.

Calibration findings (narrative)

The six included cohorts reported calibration narratively without machine-extractable numerical intercepts, slopes, or O/E ratios. The aggregate qualitative pattern across studies is consistent: the original (non-recalibrated, non-North American calibrated) 4-variable KFRE over-predicts kidney failure risk in lower-risk validation cohorts and under-predicts in higher-risk advanced-CKD cohorts. Both directional miscalibrations are reported in the source publications.

Why calibration was not pooled

Numerical pooling of calibration intercept, calibration slope, and observed/expected ratio requires (i) the point estimate, (ii) the standard error, and (iii) the analysis scale, to be reported in the source publication. None of the six abstracts surfaced all three quantities simultaneously for any single calibration metric. Pooling the qualitative narrative would require unverifiable transformation. The engine's calibration pooling primitives are exercised in the unit-test suite (Paule-Mandel REML on raw-scale intercept, raw-scale slope, and log-O/E with back-transformation) and documented in tests/prediction_fixtures/r_metafor_baselines.json; for a real-cohort calibration pool, individual-participant data or full-text supplementary tables from the source studies are required.

PROBAST risk-of-bias

Per-cohort PROBAST judgement across the four standard domains (participants, predictors, outcome, analysis) following the Wolff et al. 2019 manual (Annals of Internal Medicine, PMID 30596875). Overall judgement: low only when every domain is rated low; high when any domain is rated high; unclear otherwise. Domain ratings are pragmatic assessments based on each cohort's published methods (sample size, multi-centre vs single-centre design, completeness of follow-up, completeness of predictor measurement).

Domain-by-domain rationale

Participants

The three large multi-centre / population-linkage cohorts (Major 2019, Tangri 2020, Irish 2023) drew participants from defined source populations with explicit inclusion criteria and minimal selection bias — rated low. Lennartz 2016 enrolled a tertiary nephrology referral population (selection toward advanced CKD; rated unclear). Ali 2021 and Gallego-Valcarce 2024 are single-centre advanced-CKD cohorts with small samples; Ali 2021 split into aetiology subgroups with small per-subgroup N raising bias concerns, and Gallego-Valcarce 2024 has N = 339 — rated high for participants.

Predictors

The 4-variable KFRE predictors (age, sex, eGFR, urine ACR) are routinely measured in all included cohorts with standardised laboratory methods. All six rated low.

Outcome

Kidney failure (initiation of kidney replacement therapy or estimated GFR < 15 mL/min/1.73 m² in sustained measurements) is an objective hard endpoint with standardised ascertainment in registry-linked or EHR-linked cohorts. All six rated low.

Analysis

Large cohorts with high events-per-variable (Major 2019 with 429 events, Tangri 2020 pooled, Irish 2023 with 285 KRT events) rated low. Lennartz 2016 had moderate event count in a small sample (rated unclear). Ali 2021 (small per-aetiology subgroups) rated unclear. Gallego-Valcarce 2024 (N = 339, abstract does not specify event count) raises concern about events-per-variable and is rated high for analysis pending full-text events-per-variable verification.

Overall rollup

Three cohorts rated overall low (Major 2019, Tangri 2020, Irish 2023), two rated high (Ali 2021, Gallego-Valcarce 2024), one rated unclear (Lennartz 2016). A sensitivity analysis restricted to the three low-RoB cohorts would re-pool over a much narrower range of validation contexts (large-scale primary care, multi-centre transplant, population data linkage) and is recommended as a follow-up.

Caveat. These domain ratings are this review's interpretation of each cohort's published methods and are intended for engine demonstration, not as a definitive PROBAST assessment. A definitive PROBAST application would require two independent reviewers consulting each full-text article, with adjudication of disagreements.

Methods

Eligibility

External validations of the 4-variable Kidney Failure Risk Equation (Tangri 2011; age + sex + eGFR + urine ACR predicting kidney replacement therapy) that report a numerical C-statistic with 95% CI at the 5-year horizon. Both the original KFRE and "non-North-American-calibrated" KFRE were included; recalibrated KFRE versions are noted but not re-pooled. Cohorts in kidney-transplant recipients are included as a clinically-distinct validation context, with the caveat that the prediction target (graft loss vs native-kidney KRT) differs from the derivation cohort.

Search and selection

Six independently-published external validation cohorts were identified through anchor-search on the Tangri 2016 multinational IPDMA (JAMA, PMID 26757465) and subsequent KFRE-validation literature. This is a methods-engine demonstration on a verifiable subset of the broader KFRE-validation literature, not an exhaustive systematic review. For comprehensive coverage, the 31-cohort IPDMA (Tangri 2016) and the recent 59-cohort assessment with CKD-EPI 2021 eGFR (PMC10103205) remain authoritative.

Data extraction

For each cohort the C-statistic and its 95% CI were extracted verbatim from the publication's PubMed abstract. SE(C) was derived from the CI assuming Wald symmetry: SE = (CI_upper − CI_lower) / (2 × 1.96). Cohort N, country, setting, and follow-up duration were also extracted where reported. Calibration metrics (intercept, slope, O/E ratio, Brier score) were extracted where numerically reported with associated SE; absence of full numerical calibration triples meant calibration was not pooled (see Calibration tab).

Statistical pooling

The 4-variable KFRE 5-year C-statistic was pooled on the logit scale (Snell 2018; Debray 2017). For each cohort:

• logit(C) = log(C / (1 − C));
• SE on the logit scale via delta method: SE(logit C) = SE(C) / [C(1 − C)];
• v_i = [SE(logit C)]² .

Random-effects pooling was via Paule-Mandel REML — an exact REML estimator for the single-axis case that iterates τ² to solve Σ w_i(y_i − μ_w)² = k − 1 with w_i = 1/(v_i + τ²). Advanced-stats.md guidance: PM is preferred over DerSimonian-Laird for k < 10 because DL is downward-biased at small k. The pooled mean was the inverse-variance-weighted average at the converged τ²; SE was sqrt(1 / Σ w_i).

Small-sample CI used the Hartung-Knapp-Sidik-Jonkman adjustment with the Cochrane Handbook v6.5 (Nov 2024 §10.10.4.3) floor: HKSJ scale factor = max(1, Q / (k − 1)), with Student-t_{k − 1} critical value rather than the normal-z. This protects against CI under-coverage in the presence of heterogeneity. A 95% prediction interval (Higgins 2009; Riley 2011) was reported using t_{k − 1} × √(τ² + SE_μ²), defined for k ≥ 3.

In addition to the τ² point estimate the engine reports a 95% q-profile confidence interval for τ² (Viechtbauer 2007, Statistics in Medicine), obtained by bisection-inverting Cochran's Q against χ²(k−1) at α/2 and 1 − α/2. This is the same algorithm used by the DTA engine's qProfileTau2CI. A wide τ² CI is informative: it confirms that the heterogeneity estimate itself is uncertain at small k, separately from the τ² magnitude. A heterogeneity-fragility flag (parity with the pairwise-pool engine's piGap) reports whether the pooled CI and prediction interval disagree about whether a clinically-meaningful reference value lies inside the interval — reference values are logit(0.5) = 0 for the C-statistic, 0 for calibration intercept, 1 for calibration slope, and log(1) = 0 for the O/E ratio. The flag takes three values: agree, ci_excludes_pi_includes (pooled CI claims discrimination but a future cohort could be at chance — heterogeneity-fragile), or ci_includes_pi_excludes (CI is null but PI is shifted).

All pooling primitives, back-transformations, and the PROBAST rollup are implemented in the open-source rapidmeta-prediction-engine-v1.js (~ 826 lines, MIT licence, no external dependencies). The engine is validated by a 64-assertion unit-test suite (tests/test_prediction_engine.mjs) including: closed-form τ² = 0 case (hand-derived against inverse-variance algebra), heterogeneous k = 3 case where PM converges to a value identical to DerSimonian-Laird (verified by hand on symmetric balanced data), the 6-cohort KFRE real-data frozen baseline, the Hanley-McNeil 1982 variance against the 1982 Radiology paper Table II hand-calculation, Student-t and chi-squared quantiles against Abramowitz & Stegun, and various edge-case stress paths.

Risk-of-bias assessment

PROBAST (Wolff et al. 2019, PMID 30596875) was applied across the four prediction-model-specific domains: participants (selection bias, source population), predictors (assessment and definition), outcome (ascertainment and blinding), and analysis (events-per-variable, handling of missing data, statistical method). Each domain was rated low / high / unclear; overall: low only if all four domains low, high if any domain high, unclear otherwise. Domain rationale is provided on the PROBAST tab.

Cross-validation against R

The engine's pooling primitives (Paule-Mandel iteration; HKSJ floor with t_{k − 1}; logit / inv-logit / Hanley-McNeil variance) are the same primitives metafor::rma(yi, vi, method='PM', test='knha') uses for univariate meta-analyses, and the same primitives metamisc::valmeta() uses for prediction-model meta-analyses. Three frozen baseline cases are recorded in tests/prediction_fixtures/r_metafor_baselines.json: (i) homogeneous τ² = 0 closed-form, (ii) heterogeneous k = 3 case where PM converges to the DL value 0.24 by analytic derivation, (iii) the 6-cohort KFRE real-data pool with all key intermediates frozen to engine-output precision. Bit-reproducibility against metafor on identical inputs is the planned v1.1 build-time check.

Discussion

Principal findings

Across six independently-published external validations the pooled 5-year C-statistic of the 4-variable KFRE on the logit scale was 1.95 (back-transformed C = …; 95% HKSJ CI …; 95% prediction interval …). The Tangri 2016 multinational IPDMA (31 cohorts, n = 721,357) reported a pooled 5-year C of 0.88 (95% CI 0.86–0.90); the present 6-cohort subset is consistent in central tendency (…) but produces a substantially wider HKSJ-adjusted CI because of the heterogeneity inherent in pooling validations across very different prediction contexts.

Heterogeneity

Between-cohort heterogeneity is high (τ² ≈ 0.68 on the logit-C scale; I² ≈ 97%). The principal driver is the diversity of validation contexts. At one extreme, Major 2019 (UK primary care, 35,539 participants) and Irish 2023 (Tasmania population data linkage, 8,182 participants at 5 y) report C ≈ 0.92–0.96 — large unselected cohorts where the KFRE's calibration on age, sex, eGFR, and ACR maps efficiently to a kidney-failure rate that is rare in the overall population. At the other extreme, the kidney-transplant cohort (Tangri 2020, C = 0.73) and the advanced-CKD subgroups (Ali 2021 in Salford and Gallego-Valcarce 2024 in Spain, C = 0.77–0.82) report substantially lower discrimination, consistent with the principle that discrimination shrinks when the case-mix narrows (Riley et al. 2019, BMJ; Vergouwe et al. 2010). In these populations most participants are at high baseline kidney-failure risk and the variation in predicted risk between participants is small, so distinguishing future events from non-events on the basis of the same predictors is harder.

This pattern is the textbook example of why prediction-model meta-analysis must pool with random effects and report the prediction interval alongside the pooled CI: a notional seventh validation cohort would be expected to land somewhere in the prediction interval (…), not within the much-narrower CI. The HKSJ adjustment with the Cochrane v6.5 small-sample floor is what produces this honest wider CI; using a normal-z CI without the small-sample correction would produce a misleadingly tight pooled estimate.

Risk-of-bias and applicability

Three of six cohorts were rated overall low risk of bias (Major 2019, Tangri 2020 transplant, Irish 2023). Two were rated high (Ali 2021 with small per-aetiology subgroups; Gallego-Valcarce 2024 with N = 339 and events-per-variable concerns). Lennartz 2016 was rated unclear (tertiary referral selection). A sensitivity analysis restricted to the three low-RoB cohorts would re-pool over primary-care, transplant, and population-linkage settings only — narrower applicability but with greater methodological coherence.

Implications for clinical risk stratification

The KFRE was developed and validated for native-kidney CKD populations (Tangri 2011, JAMA). Its application to kidney-transplant recipients (Tangri 2020) gives substantially lower discrimination (C = 0.73 vs ≈ 0.88 in native CKD), reflecting that graft-loss biology differs from native-kidney-failure biology. For Finrenone-relevant CKD populations — i.e. native-kidney chronic kidney disease with type 2 diabetes, where Finrenone is indicated — the KFRE's primary-care discrimination is excellent (Major 2019: C = 0.92) and its advanced-CKD discrimination is good (Ali 2021: C = 0.77). The KFRE is therefore well suited for Finrenone-eligible-cohort risk stratification at the primary-care to early-CKD-clinic transition, but a recalibration step (Major 2019 baseline-risk adjustment; Irish 2023 calibration factors) is required outside North American populations.

Strengths

• All cohort-level numerical values are extracted verbatim from PubMed-indexed publications with PMIDs and DOIs; no synthetic or fabricated data.
• Pooling uses Paule-Mandel REML (preferred for k < 10 per advanced-stats.md) with HKSJ small-sample CI on the logit-C scale (Snell 2018; Debray 2017).
• Prediction interval is reported alongside the CI, surfacing the between-cohort variability that the CI alone hides.
• PROBAST is applied per-domain with explicit rationale for each cohort.
• The engine is open-source under MIT with a 64-assertion unit-test suite, including a frozen baseline against the 6-cohort real-data pool for regression testing.

Limitations

• Calibration metrics are not pooled because the six abstracts do not jointly report numerical calibration intercept / slope / O/E with SEs. The narrative qualitative pattern (over-prediction in low-risk groups, under-prediction in advanced CKD) is documented but cannot be turned into a forest plot from the source data available.
• The six cohorts are a subset of the broader KFRE-validation literature; the Tangri 2016 IPDMA (31 cohorts) is the authoritative pooled estimate, and the present review is positioned as a methods-engine demonstration, not as a replacement.
• SE(C) is derived from the published CI assuming Wald symmetry, which is approximate; this is the same approximation used by metafor::rma and metamisc::valmeta when only CIs are reported.
• The Tangri 2020 transplant cohort validates the KFRE in a clinically-distinct population (graft loss / death-censored), which differs from native-kidney KFRE prediction; its inclusion contributes to the high I² and the wider HKSJ CI.
• PROBAST domain ratings are this review's interpretation of each cohort's published methods, not a definitive two-reviewer adjudicated assessment.
• Bit-reproducibility against metafor::rma(method='PM', test='knha') and metamisc::valmeta() is documented as the planned v1.1 build-time check; the engine's pooling primitives match metafor's by construction (Paule-Mandel iteration, HKSJ floor with t_{k − 1}), but a build-time WebR or Rscript invocation has not yet been wired.

Conclusion

The 4-variable KFRE shows good-to-excellent 5-year discrimination across diverse external validation contexts when pooled honestly with Paule-Mandel REML + HKSJ and a Cochrane v6.5 prediction interval. The wide pooled CI reflects real between-cohort variability driven by validation-context heterogeneity (primary-care vs advanced-CKD vs transplant), not engine pathology. For Finrenone-relevant native-kidney CKD populations the primary-care KFRE discrimination is excellent. Calibration assessment requires individual-cohort full-text data and was outside the scope of this abstract-level methods demonstration.

TRIPOD-Cluster reporting checklist

TRIPOD-Cluster (Debray et al. 2023, BMJ; PMID 36750236) extends TRIPOD to multi-cluster / multi-cohort prediction-model evaluation studies. Items below mark this review's compliance for engine-demonstration purposes; a clinical-evidence submission would require full-text extraction beyond abstract-level data.

Note: this checklist is a self-assessment for engine-demonstration transparency. A clinical-evidence submission would require independent two-reviewer extraction from full-text PDFs and a definitive PROBAST adjudication.

References

Vancouver-style citations with PubMed and DOI links.

Included cohorts (n = 6)

KFRE derivation and anchor IPDMA

Methodological references