Writing > The Hidden Data
Have you not heard of the trials that were hidden,
the data that was buried,
the papers that told only half the truth?
the data that was buried,
the papers that told only half the truth?
The Tamiflu Scandal
COCHRANE COLLABORATION, 2009-2014
Governments stockpiled $9 billion worth of Tamiflu to fight pandemic flu.
But when Cochrane researchers tried to verify the drug's benefits, they found that 60% of trial data had never been published.
After a 5-year battle, they obtained the hidden data. The conclusion changed: Tamiflu shortened symptoms by less than a day and did not prevent complications.
$9 billion spent on evidence that was never fully disclosed.
But when Cochrane researchers tried to verify the drug's benefits, they found that 60% of trial data had never been published.
After a 5-year battle, they obtained the hidden data. The conclusion changed: Tamiflu shortened symptoms by less than a day and did not prevent complications.
$9 billion spent on evidence that was never fully disclosed.
Jefferson T et al. Cochrane Database Syst Rev. 2014;4:CD008965
Why We Write Meta-Analyses
The Purpose of Synthesis
Individual Studies
↓
Problem?
Small samplesLow power
Conflicting resultsWhich to believe?
Publication biasMissing negatives
↓
Meta-AnalysisSystematic synthesis
↓
More precise estimate + Bias detection
The Tamiflu Transparency Campaign
2009-2014 | COCHRANE COLLABORATION vs. ROCHE
For years, governments worldwide stockpiled Tamiflu (oseltamivir) at a cost of billions of dollars, based on manufacturer claims that it prevented flu complications and hospitalizations.
When Cochrane reviewers requested the full trial data to verify these claims, Roche refused for 5 years, citing "confidentiality." The company had conducted 10 treatment trials, but only 2 were fully published.
After relentless pressure, Clinical Study Reports were finally released in 2014. The picture changed dramatically: Tamiflu shortened symptoms by less than a day and showed no evidence of preventing hospitalizations or serious complications.
When Cochrane reviewers requested the full trial data to verify these claims, Roche refused for 5 years, citing "confidentiality." The company had conducted 10 treatment trials, but only 2 were fully published.
After relentless pressure, Clinical Study Reports were finally released in 2014. The picture changed dramatically: Tamiflu shortened symptoms by less than a day and showed no evidence of preventing hospitalizations or serious complications.
THE LESSON
A meta-analysis is only as good as the data it can access. Hidden trials can make ineffective treatments look effective—and cost billions in misspent resources.
"And the company knew,
and the regulators knew,
but the published papers did not tell—
and billions were spent on a half-truth."
and the regulators knew,
but the published papers did not tell—
and billions were spent on a half-truth."
This is why we write meta-analyses—to find the whole truth.
If you wish your work to be trusted,
you must enter into a covenant with your readers.
That covenant has a name:
PRISMA.
you must enter into a covenant with your readers.
That covenant has a name:
PRISMA.
PRISMA 2020
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
27
Checklist items
2020
Updated version
50K+
Citations
THE COVENANT
PRISMA is not bureaucracy. It is a promise to your readers
that you have done the work transparently and completely.
The Seven Sections
1
Title
Identify as systematic review, meta-analysis, or both
2
Abstract
Structured summary of the entire review
3
Introduction
Rationale and objectives with PICO
4
Methods
Protocol, search, selection, data, bias, synthesis
5
Results
Flow diagram, characteristics, risk of bias, synthesis results
6
Discussion
Summary, limitations, interpretation, implications
7
Other
Registration, funding, conflicts of interest
The PRISMA Revolution
2009-PRESENT | TRANSFORMING SYSTEMATIC REVIEW REPORTING
Before PRISMA (2009), systematic review reporting was chaotic. Some reviews didn't report their search strategies at all. Others omitted risk of bias assessments. Many failed to explain why studies were excluded. Readers couldn't judge quality—they had to trust blindly.
PRISMA's 27-item checklist changed everything. It required authors to document every step: the full search strategy, selection criteria, extraction methods, and synthesis decisions.
Today, over 10,000 journals endorse PRISMA. What was once exceptional transparency became the expected standard.
PRISMA's 27-item checklist changed everything. It required authors to document every step: the full search strategy, selection criteria, extraction methods, and synthesis decisions.
Today, over 10,000 journals endorse PRISMA. What was once exceptional transparency became the expected standard.
THE LESSON
A simple checklist transformed an entire field. Transparent reporting went from exception to norm—proof that standards matter.
"PRISMA is the covenant between author and reader:
I will show you everything—
how I searched, what I found, what I excluded, why.
So you may judge my work, and trust—or question—my conclusions."
I will show you everything—
how I searched, what I found, what I excluded, why.
So you may judge my work, and trust—or question—my conclusions."
Have you not seen the reviewer
who changed the outcome after seeing the data,
who moved the goalposts until the results looked right?
who changed the outcome after seeing the data,
who moved the goalposts until the results looked right?
The Retracted Meta-Analyses
MULTIPLE JOURNALS, 2010-2023
Researchers found that many retracted meta-analyses had no pre-registered protocol.
Without a protocol, reviewers could:
• Change inclusion criteria after seeing results
• Switch primary outcomes to show significance
• Add or remove studies to change the conclusion
The protocol is your pre-commitment device— it prevents you from fooling yourself.
Without a protocol, reviewers could:
• Change inclusion criteria after seeing results
• Switch primary outcomes to show significance
• Add or remove studies to change the conclusion
The protocol is your pre-commitment device— it prevents you from fooling yourself.
Defined outcome switching: PROSPERO registration prevents bias
Protocol Registration Decision Tree
Where to Register Your Protocol
New Systematic Review
↓
Type of Review?
Health/Medical
PROSPEROprospero.york.ac.uk
Any Field
OSFosf.io/registries
Cochrane
Cochrane LibraryIntegrated protocol
↓
Registration ID in PaperCite in Methods
What the Protocol Must Contain
Essential Protocol Elements
1. Research question (PICO format)
2. Eligibility criteria (inclusion/exclusion)
3. Information sources and search strategy
4. Study selection process
5. Data extraction items
6. Risk of bias assessment tool
7. Primary and secondary outcomes
8. Synthesis methods (meta-analysis plan)
9. Subgroup and sensitivity analyses
"Write the protocol before you see the data.
Lock it in a public registry.
Then follow it—or explain why you deviated.
This is how you prove you did not cheat."
Lock it in a public registry.
Then follow it—or explain why you deviated.
This is how you prove you did not cheat."
The title is the first promise you make.
It must tell the reader:
What you studied, how you studied it, and what kind of study it is.
It must tell the reader:
What you studied, how you studied it, and what kind of study it is.
The Anatomy of a Title
PRISMA Title Requirements
Title Must Include
↓
PopulationWho was studied
InterventionWhat was done
OutcomeWhat was measured
↓
+ "Systematic Review" or "Meta-Analysis"
Good vs. Bad Titles
❌ BAD TITLE
"A Review of Diabetes Treatment"
Problems: No population specified, no intervention, no outcome, doesn't say systematic review
Problems: No population specified, no intervention, no outcome, doesn't say systematic review
✓ GOOD TITLE
"Efficacy of SGLT2 Inhibitors on Cardiovascular Mortality in Adults with Type 2 Diabetes: A Systematic Review and Meta-Analysis"
Population, intervention, outcome, and study type all clear
Population, intervention, outcome, and study type all clear
"The title is your first word to the reader.
Make it complete. Make it honest.
Tell them exactly what they will find within."
Make it complete. Make it honest.
Tell them exactly what they will find within."
Most readers will only read your abstract.
If the abstract lies, or omits, or misleads—
most readers will never know.
If the abstract lies, or omits, or misleads—
most readers will never know.
The Spin Problem
BOUTRON ET AL., 2010
Researchers analyzed 72 RCTs with non-significant primary outcomes.
They found that 40% of abstracts contained "spin"— reporting that focused on secondary outcomes, subgroups, or within-group changes to make results appear more favorable than they were.
The abstract told a different story than the data.
They found that 40% of abstracts contained "spin"— reporting that focused on secondary outcomes, subgroups, or within-group changes to make results appear more favorable than they were.
The abstract told a different story than the data.
Boutron I et al. JAMA. 2010;303:2058-2064
Structured Abstract Elements
PRISMA Abstract Checklist
□ Background and objectives
□ Eligibility criteria
□ Information sources
□ Risk of bias assessment
□ Synthesis methods
□ Results (# studies, # participants, effect estimate with CI)
□ Limitations
□ Conclusions and implications
□ Registration number
What happens when the abstract tells a different story than the paper itself?
REAL DATA
Pitkin et al. (1999, BMJ) examined structured abstracts in six major journals and found that 18-68% of abstracts contained data inconsistent with the full article. Deficiencies ranged from numerical errors to conclusions not supported by the reported results.
The Misleading Abstract: Pitkin 1999
Your meta-analysis found a non-significant primary outcome (RR 0.92, 95% CI 0.78-1.09). How do you write the abstract?
PATH A: Spin the Abstract
Emphasize a significant secondary outcome and use language like "trend toward benefit"
↓
Readers who only see the abstract form a misleadingly positive impression of the treatment
OUTCOME: Misleading clinical decisions
PATH B: Report Faithfully
State the non-significant primary result clearly and note secondary outcomes as exploratory
↓
Readers get an accurate summary; those who need details read the full paper
OUTCOME: Evidence integrity preserved
THE REVELATION
Most readers never get past the abstract. If the abstract misleads, the full paper's honesty cannot undo the damage.
"Do not spin. Do not hide.
If the primary outcome was null, say so.
The abstract must be a faithful mirror—
not a flattering portrait."
If the primary outcome was null, say so.
The abstract must be a faithful mirror—
not a flattering portrait."
A vague question yields vague answers.
Before you search, before you write—
you must know exactly what you seek.
Before you search, before you write—
you must know exactly what you seek.
The PICO Framework
Structuring the Research Question
Research Question
↓
PPopulation
Who?
IIntervention
What treatment?
CComparator
Vs. what?
OOutcome
What measured?
PICO Example
TRANSFORMING A VAGUE QUESTION
Vague: "Does exercise help depression?"
PICO:
• P: Adults diagnosed with major depressive disorder
• I: Supervised aerobic exercise (≥3x/week for ≥8 weeks)
• C: Usual care or waitlist control
• O: Change in depression score (HAM-D or BDI)
Now you know exactly what to search for.
PICO:
• P: Adults diagnosed with major depressive disorder
• I: Supervised aerobic exercise (≥3x/week for ≥8 weeks)
• C: Usual care or waitlist control
• O: Change in depression score (HAM-D or BDI)
Now you know exactly what to search for.
"Define your question with precision.
Who are the patients? What is the treatment?
What is the comparator? What will you measure?
PICO is the map before the journey."
Who are the patients? What is the treatment?
What is the comparator? What will you measure?
PICO is the map before the journey."
Have you not heard of the meta-analysis
that searched only one database,
missed half the evidence,
and drew the wrong conclusion?
that searched only one database,
missed half the evidence,
and drew the wrong conclusion?
The Search Strategy Decision Tree
Where to Search
Comprehensive Search
↓
Minimum Databases
MEDLINEPubMed
EmbaseEuropean focus
CENTRALCochrane trials
↓
Plus Additional Sources
Trial registriesClinicalTrials.gov
Grey literatureTheses, reports
Reference listsBackward citation
Documenting the Search
WHAT TO REPORT
• Full search strategy for at least one database (appendix)
• Date of search for each database
• Any limits (language, date, publication type)
• Hand-searching (journals, conference proceedings)
• Contact with authors for unpublished data
• Date of search for each database
• Any limits (language, date, publication type)
• Hand-searching (journals, conference proceedings)
• Contact with authors for unpublished data
THE REPRODUCIBILITY TEST
Another researcher should be able to replicate your search exactly
and find the same number of records.
The Cochrane Search Strategy Discovery
2003 | COCHRANE METHODOLOGY REVIEW
Cochrane researchers asked a simple question: What would happen if systematic reviewers only searched MEDLINE?
The answer was alarming. They would have missed 30% of included studies—including some that changed the meta-analysis conclusions entirely.
One striking example: an anti-depressant meta-analysis showed benefit when based on MEDLINE alone, but no benefit when all sources were included. The missing studies were smaller, negative trials indexed in specialty databases like EMBASE and PsycINFO.
The answer was alarming. They would have missed 30% of included studies—including some that changed the meta-analysis conclusions entirely.
One striking example: an anti-depressant meta-analysis showed benefit when based on MEDLINE alone, but no benefit when all sources were included. The missing studies were smaller, negative trials indexed in specialty databases like EMBASE and PsycINFO.
THE LESSON
Single-database searches can systematically miss negative trials. The studies not in MEDLINE may be the very studies that change your conclusion.
"Search wide. Search deep.
Document every database, every date, every term.
The evidence you miss may be the evidence that matters most."
Document every database, every date, every term.
The evidence you miss may be the evidence that matters most."
From thousands of records, you must choose.
But choose by what rule?
And who will check your choices?
But choose by what rule?
And who will check your choices?
The PRISMA Flow Diagram
PRISMA 2020 Flow Diagram
IDENTIFICATION
n = 3,847
Records from databases
↓
Duplicates removed (n = 892)
SCREENING
n = 2,955
Titles/abstracts screened
↓
Excluded (n = 2,680)
ELIGIBILITY
n = 275
Full-text assessed
↓
Excluded with reasons (n = 247)
INCLUDED
n = 28
Studies in synthesis
Selection Process Decision Tree
Who Selects? How?
Study Selection
↓
Two independent reviewersGold standard
↓
Disagreement?
ConsensusDiscussion
Third reviewerArbiter
One reviewer onlyAcknowledge limitation
REPORT AGREEMENT
Calculate and report inter-rater agreement (kappa statistic).
Low agreement suggests unclear criteria.
Can a post-hoc subgroup analysis from a single trial reshape an entire field for a decade?
REAL DATA
The Women's Health Initiative (WHI, 2002) found that HRT increased cardiovascular risk overall. But post-hoc subgroup analysis suggested women aged <60 or within 10 years of menopause might benefit, while older women were harmed. This "timing hypothesis" fueled years of debate and further studies.
The HRT Timing Hypothesis: WHI 2002
Your meta-analysis of HRT shows harm overall, but an exploratory subgroup suggests benefit in younger women. How do you write this?
PATH A: Overstate the Subgroup
Headline the age subgroup result as if it were the primary finding
↓
Clinicians prescribe HRT based on exploratory, underpowered subgroup; the finding may not replicate
OUTCOME: Premature clinical change
PATH B: Report Honestly
Present overall result as primary; label subgroup as exploratory and pre-specified or post-hoc
↓
Readers understand the hypothesis needs confirmation; future trials can be designed to test it
OUTCOME: Responsible hypothesis generation
THE REVELATION
Subgroup analyses generate hypotheses, not conclusions. Always label them as exploratory and report the overall result first.
"Every exclusion must have a reason.
Every reason must be documented.
Two pairs of eyes are better than one—
for what one misses, the other may catch."
Every reason must be documented.
Two pairs of eyes are better than one—
for what one misses, the other may catch."
Have you not seen the meta-analysis
that pooled good studies with bad,
and called the average truth?
that pooled good studies with bad,
and called the average truth?
The Danger of Ignoring Bias
THE ANTIDEPRESSANT SCANDAL
Turner et al. (2008) obtained FDA data on 74 antidepressant trials.
In the published literature: 94% of trials were positive.
In the FDA database: only 51% were positive.
The published meta-analyses had pooled selectively reported data. The effect size was inflated by 32%.
In the published literature: 94% of trials were positive.
In the FDA database: only 51% were positive.
The published meta-analyses had pooled selectively reported data. The effect size was inflated by 32%.
Turner EH et al. N Engl J Med. 2008;358:252-260
Risk of Bias Assessment Tools
Which Tool to Use?
Study Design
↓
RCTs
RoB 2Cochrane tool
Non-randomized
ROBINS-IInterventions
DTA studies
QUADAS-2Diagnostic
Observational
NOSNewcastle-Ottawa
RoB 2 Domains
Cochrane Risk of Bias 2.0 for RCTs
D1 Randomization process
D2 Deviations from intended interventions
D3 Missing outcome data
D4 Measurement of the outcome
D5 Selection of the reported result
JUDGMENT OPTIONS
Each domain: Low risk /
Some concerns /
High risk
"A meta-analysis of biased studies
yields a biased conclusion—
with a narrower confidence interval.
You have made the lie more precise."
yields a biased conclusion—
with a narrower confidence interval.
You have made the lie more precise."
From each study, you must extract the numbers.
Extract wrong, and your whole analysis
is built on sand.
Extract wrong, and your whole analysis
is built on sand.
Data Extraction Form
Essential Data Items
1 Study identifiers (author, year, country)
2 Study design and setting
3 Participant characteristics (n, age, sex, severity)
4 Intervention details (dose, duration, delivery)
5 Comparator details
6 Outcome definitions and measurement
7 Results (means, SDs, events, sample sizes)
8 Follow-up duration and loss to follow-up
9 Funding source and conflicts of interest
Extraction Decision Tree
Handling Missing Data
Data Not Reported
↓
What to Do?
First
Contact authorsEmail for data
If no response
Calculate/imputeDocument method
If impossible
Exclude from MAInclude in narrative
The Rosiglitazone Data Extraction Error
2007 | NEW ENGLAND JOURNAL OF MEDICINE
Nissen and Wolski's 2007 meta-analysis of rosiglitazone (Avandia) found a 43% increased risk of heart attack. The finding triggered FDA warnings and caused prescriptions to collapse worldwide.
But later scrutiny revealed complications. Some effect estimates had been extracted from secondary publications rather than primary trial reports. Small differences in how events were counted—extracted from different sources—meaningfully changed the results.
The meta-analysis was influential and largely correct, but the controversy highlighted how small extraction decisions can have billion-dollar consequences. Merck's competing drug gained market share; GSK faced massive litigation.
But later scrutiny revealed complications. Some effect estimates had been extracted from secondary publications rather than primary trial reports. Small differences in how events were counted—extracted from different sources—meaningfully changed the results.
The meta-analysis was influential and largely correct, but the controversy highlighted how small extraction decisions can have billion-dollar consequences. Merck's competing drug gained market share; GSK faced massive litigation.
THE LESSON
Always extract from primary sources. Document every choice. A small discrepancy in extracted numbers can change regulatory decisions and market fortunes.
"Extract in duplicate. Check each number.
One digit wrong can change the conclusion.
The extraction form is your ledger—
keep it meticulous, keep it true."
One digit wrong can change the conclusion.
The extraction form is your ledger—
keep it meticulous, keep it true."
The effect size is the heart of your meta-analysis.
Choose the wrong measure,
and your pooled estimate will be meaningless.
Choose the wrong measure,
and your pooled estimate will be meaningless.
Effect Measure Decision Tree
Choosing the Right Effect Size
Outcome Type
↓
Binary
RR, OR, RDEvents/no events
Continuous
Same scale?
YesMD (mean diff)
NoSMD (Hedge's g)
Time-to-event
HRHazard ratio
Common Effect Measures
RR
Risk Ratio
Multiplicative
Multiplicative
OR
Odds Ratio
Case-control
Case-control
MD
Mean Diff
Same units
Same units
SMD
Std Mean Diff
Different scales
Different scales
THE PRINCIPLE
The effect measure must be comparable across studies.
If studies used different scales, standardize.
Can a trial that transforms global practice still have serious limitations?
REAL DATA
The RECOVERY trial (2020) demonstrated that dexamethasone reduced 28-day mortality in hospitalized COVID-19 patients requiring oxygen: RR 0.83, 95% CI 0.75-0.93. Yet the trial was open-label (no blinding), conducted predominantly in UK hospitals, and the control group received usual care (which varied).
The RECOVERY Trial: 2020
Your meta-analysis includes RECOVERY as the dominant study. How do you handle limitations of an otherwise landmark trial?
PATH A: Minimize Limitations
Downplay open-label design and geographic concentration; focus on the striking mortality benefit
↓
Readers cannot judge generalizability to other settings; potential detection bias is obscured
OUTCOME: Incomplete evidence appraisal
PATH B: Honest Limitations
Acknowledge open-label design and geographic limitations while clearly stating the mortality benefit
↓
Readers understand both the strength of the finding and where uncertainty remains
OUTCOME: Trustworthy, balanced reporting
THE REVELATION
Even groundbreaking trials have limitations. Acknowledging them does not undermine the findings; it builds reader trust and guides future research.
"Choose the measure that fits the data.
Risk ratios for common outcomes, odds ratios for rare.
Standardize when scales differ.
The wrong measure pools apples with oranges."
Risk ratios for common outcomes, odds ratios for rare.
Standardize when scales differ.
The wrong measure pools apples with oranges."
Have you not seen the forest plot
where studies pointed in opposite directions,
yet the diamond declared a single truth?
where studies pointed in opposite directions,
yet the diamond declared a single truth?
Fixed vs. Random Effects
Which Model to Use?
Meta-Analysis Model
↓
Assumption About True Effect?
One true effect
Fixed EffectAll studies estimate same θ
↓
Rarely appropriateVery similar studies
Effects vary
Random EffectsDistribution of θᵢ
↓
Usually preferredMore conservative
When NOT to Pool
Do Not Meta-Analyze If...
✗ Studies are too heterogeneous (clinical or methodological)
✗ Outcomes are defined differently
✗ Populations are fundamentally different
✗ Risk of bias is too high across studies
✗ Publication bias is severe
THE WISDOM
Sometimes the most honest conclusion is:
"These studies should not be pooled."
What happens when a methodological critique of a Cochrane review escalates into an organizational crisis?
REAL DATA
In 2018, Peter Gøtzsche and colleagues published a critique of the Cochrane HPV vaccine review, arguing it had excluded key trials and used inappropriate inclusion criteria. The Cochrane review had included 26 studies with over 73,000 women. The dispute became a governance crisis, culminating in Gøtzsche's expulsion from Cochrane's board.
The Cochrane HPV Controversy: 2018
You receive a methodological critique of your published synthesis arguing you should have included additional studies. How do you respond?
PATH A: Dismiss the Critique
Defend the original approach without engaging with the specific methodological points raised
↓
Public trust erodes; the dispute becomes personal rather than scientific; the evidence base is not improved
OUTCOME: Polarization and lost credibility
PATH B: Engage Transparently
Conduct sensitivity analyses incorporating the suggested studies; publish a transparent response showing if conclusions change
↓
The evidence is strengthened; methodological discourse advances the field; trust is maintained
OUTCOME: Science self-corrects publicly
THE REVELATION
Methodological critique is how science improves. Responding with data, not defensiveness, strengthens both the review and the field.
"Do not pool for the sake of pooling.
A meta-analysis of incompatible studies
is not synthesis—it is confusion.
Know when to say: these cannot be combined."
A meta-analysis of incompatible studies
is not synthesis—it is confusion.
Know when to say: these cannot be combined."
When studies disagree,
the disagreement itself is data.
Do not hide it. Explain it.
the disagreement itself is data.
Do not hide it. Explain it.
Heterogeneity Measures
Q
Cochran's Q
Significance test
Significance test
I²
Inconsistency
% variation
% variation
τ²
Tau-squared
Between-study var
Between-study var
PI
Prediction interval
Future studies
Future studies
Investigating Heterogeneity
When I² > 50%
High Heterogeneity
↓
Investigation Methods
Subgroup analysisPre-specified
Meta-regressionIf ≥10 studies
Sensitivity analysisExclude outliers
↓
Report unexplained heterogeneityLimitation
What if a meta-analysis of small positive trials is overturned by a single mega-trial?
REAL DATA
By the early 1990s, several small trials suggested intravenous magnesium reduced mortality after acute myocardial infarction. A meta-analysis (Teo et al., 1991) pooled these and found a significant benefit: OR 0.44, 95% CI 0.27-0.71. Then ISIS-4 (1995), a mega-trial with 58,050 patients, found no benefit at all. The small-study effects and heterogeneity had been ignored.
The Magnesium Controversy: 1991-1995
Your meta-analysis of small trials shows high heterogeneity (I² above 50%) but the pooled estimate is significant. How do you present this?
PATH A: Bury the Heterogeneity
Report the significant pooled estimate prominently; mention I² only in passing
↓
Clinicians adopt the treatment; a future large trial may contradict the meta-analysis, eroding trust in the method
OUTCOME: Premature guideline changes
PATH B: Investigate Transparently
Highlight heterogeneity; investigate sources (study size, quality); note that small-study effects may inflate the estimate
↓
Readers understand the uncertainty; recommendations call for a definitive large trial before changing practice
OUTCOME: Evidence-appropriate caution
THE REVELATION
Heterogeneity is a warning signal, not a footnote. Small-study effects can produce a falsely reassuring pooled estimate that a single large trial can overturn.
"I-squared is not just a number to report.
It is a question: Why do these studies disagree?
Investigate. Explain. Or acknowledge ignorance."
It is a question: Why do these studies disagree?
Investigate. Explain. Or acknowledge ignorance."
Have you not heard of the file drawer,
where negative studies go to die,
leaving only the positive survivors
to tell a distorted story?
where negative studies go to die,
leaving only the positive survivors
to tell a distorted story?
The Vioxx Disaster
MERCK, 2004
Vioxx (rofecoxib) was a blockbuster painkiller earning $2.5 billion/year.
Internal company documents showed Merck knew of cardiovascular risks but suppressed unfavorable data and published only favorable analyses.
A meta-analysis using all available data revealed a 2-fold increased risk of heart attack.
Vioxx was withdrawn. It had caused an estimated 88,000-140,000 excess heart attacks.
Internal company documents showed Merck knew of cardiovascular risks but suppressed unfavorable data and published only favorable analyses.
A meta-analysis using all available data revealed a 2-fold increased risk of heart attack.
Vioxx was withdrawn. It had caused an estimated 88,000-140,000 excess heart attacks.
Topol EJ. N Engl J Med. 2004;351:1707-1709
Detecting Publication Bias
Assessment Methods
Publication Bias Assessment
↓
Funnel plotVisual inspection
Egger's testStatistical asymmetry
Trim and fillImpute missing
↓
Requires ≥10 studiesLow power otherwise
Preventing Bias: The AllTrials Campaign
ALLTRIALS.NET
"All trials registered. All results reported."
• Search trial registries (ClinicalTrials.gov, WHO ICTRP)
• Contact companies for unpublished data
• Cite registration numbers in your review
• Report which registered trials are missing from your analysis
• Search trial registries (ClinicalTrials.gov, WHO ICTRP)
• Contact companies for unpublished data
• Cite registration numbers in your review
• Report which registered trials are missing from your analysis
"The file drawer is not empty.
It holds the studies that companies hid,
the results that journals rejected.
Your job is to open that drawer—or say you could not."
It holds the studies that companies hid,
the results that journals rejected.
Your job is to open that drawer—or say you could not."
The forest plot is the face of your meta-analysis.
It shows the reader everything:
each study, each weight, each confidence interval,
and the final pooled estimate.
It shows the reader everything:
each study, each weight, each confidence interval,
and the final pooled estimate.
Reading the Forest Plot
Elements of the Forest Plot
Forest Plot
↓
Study namesLeft column
SquaresPoint estimates
Lines95% CI
DiamondPooled estimate
↓
Square size = study weightLarger = more precise
Forest Plot Checklist
What to Include
□ Study identifiers (author, year)
□ Sample size per arm
□ Effect estimate with 95% CI
□ Weight (% contribution)
□ Line of no effect (RR=1 or MD=0)
□ Pooled estimate with 95% CI
□ Heterogeneity statistics (I², τ², Q)
□ Test for overall effect (Z, p-value)
The Vioxx Numbers That Changed Everything
2004 | THE APPROVe TRIAL AND MARKET WITHDRAWAL
For years, the forest plot for Vioxx (rofecoxib) cardiovascular safety showed a reassuring pattern. Point estimates from earlier trials clustered around the line of no effect. The diamond suggested the drug was safe.
Then came the APPROVe trial. When its data was added to the forest plot, the picture changed dramatically. APPROVe's large square pulled the pooled diamond definitively toward harm. The visual was unmistakable.
That forest plot ended Vioxx. Merck withdrew the drug voluntarily. The subsequent litigation cost the company $4.85 billion in settlements. Thousands of patients had suffered heart attacks while the earlier, smaller trials showed ambiguous results.
Then came the APPROVe trial. When its data was added to the forest plot, the picture changed dramatically. APPROVe's large square pulled the pooled diamond definitively toward harm. The visual was unmistakable.
That forest plot ended Vioxx. Merck withdrew the drug voluntarily. The subsequent litigation cost the company $4.85 billion in settlements. Thousands of patients had suffered heart attacks while the earlier, smaller trials showed ambiguous results.
THE LESSON
One well-conducted, adequately powered trial can shift the entire pooled estimate. Forest plots tell the story of how evidence accumulates—and sometimes, how it reverses course.
The Misleading Forest Plot
You are designing a forest plot for your meta-analysis. The axis scale and study ordering can change the visual impression. How do you proceed?
PATH A: Design for Impact
Use a compressed axis scale to make effect sizes look larger; order studies to build a visual narrative
↓
Readers form exaggerated impressions of effect magnitude; the plot becomes an advocacy tool rather than a data display
OUTCOME: Visual distortion of evidence
PATH B: Design for Clarity
Use standard axis scaling; order studies by year or alphabetically; include all standard elements (weights, CIs, I²)
↓
Readers can make their own judgments; the plot serves as a transparent data visualization
OUTCOME: Honest visual communication
"The forest plot hides nothing.
Every study visible. Every weight transparent.
Let the reader see what you saw—
and judge for themselves."
Every study visible. Every weight transparent.
Let the reader see what you saw—
and judge for themselves."
A pooled estimate is not enough.
You must also tell the reader:
How confident should they be in this result?
You must also tell the reader:
How confident should they be in this result?
GRADE Certainty Assessment
Rating the Evidence
Start: RCTs = High, Obs = Low
↓
Reasons to Downgrade?
Risk of bias-1 or -2
Inconsistency-1 or -2
Indirectness-1 or -2
Imprecision-1 or -2
Pub. bias-1 or -2
GRADE Certainty Levels
⊕⊕⊕⊕
HIGH
Very confident
Very confident
⊕⊕⊕◯
MODERATE
Likely close
Likely close
⊕⊕◯◯
LOW
May differ
May differ
⊕◯◯◯
VERY LOW
Uncertain
Uncertain
What happens when a GRADE assessment of "low certainty" collides with a public health emergency?
REAL DATA
The 2023 Cochrane review of physical interventions to reduce respiratory virus spread (Jefferson et al.) found that the evidence for masks in community settings was low certainty per GRADE, with wide confidence intervals. The review was widely reported as proving "masks don't work," though the authors stated the evidence was insufficient to draw firm conclusions in either direction.
The Cochrane Mask Review: 2023
Your systematic review on a politically sensitive topic receives a GRADE rating of "low certainty." How do you communicate this?
PATH A: Soften the Rating
Downplay the GRADE assessment to avoid political controversy; emphasize the point estimate over the certainty level
↓
The review loses methodological credibility; GRADE becomes seen as optional rather than rigorous
OUTCOME: Compromised methodology
PATH B: Report Faithfully
Report the GRADE rating honestly; explain clearly what "low certainty" means (and does not mean); distinguish absence of evidence from evidence of absence
↓
The public may misinterpret, but the scientific record is accurate; future research directions become clear
OUTCOME: Methodological integrity
THE REVELATION
"Low certainty" does not mean "no effect." GRADE ratings must be reported faithfully, with clear explanation of what they mean, especially for politically charged topics.
"The effect size is the what.
GRADE certainty is the how sure.
Report both—or the reader cannot judge
how much to trust your conclusion."
GRADE certainty is the how sure.
Report both—or the reader cannot judge
how much to trust your conclusion."
The Discussion is where you interpret.
Not to spin. Not to overstate.
But to explain what your findings mean—
and what they do not mean.
Not to spin. Not to overstate.
But to explain what your findings mean—
and what they do not mean.
Discussion Structure
1
Summary of Findings
Restate main results with certainty rating
2
Comparison with Existing Literature
How do your findings relate to prior reviews?
3
Strengths and Limitations
Both of the review AND of the included studies
4
Implications for Practice
What should clinicians/policymakers do?
5
Implications for Research
What studies are still needed?
Common Mistakes in Discussion
What NOT to Do
✗ Overstating conclusions beyond the data
✗ Ignoring limitations
✗ Treating statistical significance as clinical importance
✗ Failing to address heterogeneity
✗ Making causal claims from observational data
What if the most-cited methodology paper ever published warns that most research findings are false?
REAL DATA
John Ioannidis's 2005 paper in PLoS Medicine, "Why Most Published Research Findings Are False," has been cited over 10,000 times. Using mathematical modeling, he argued that the probability a research finding is true depends on study power, bias, and the number of tested relationships. For many research designs, the post-study probability of a true finding can be below 50%.
The Ioannidis Wake-Up Call: 2005
Your meta-analysis has a statistically significant result, but the included studies are small, heterogeneous, and many have high risk of bias. How do you write the discussion?
PATH A: Oversell the Finding
Lead with the significant pooled estimate; minimize limitations; make strong practice recommendations
↓
The finding enters guidelines prematurely; when replication fails, the entire meta-analysis method is blamed
OUTCOME: Eroded trust in evidence synthesis
PATH B: Calibrate the Interpretation
Discuss the result in context of study quality, heterogeneity, and certainty; match recommendation strength to evidence strength
↓
Readers understand the degree of confidence warranted; future research priorities become clear
OUTCOME: Proportionate, trustworthy conclusions
THE REVELATION
The Discussion must calibrate enthusiasm to evidence quality. Strong claims from weak evidence damage the credibility of the entire field.
"The Discussion is not for advocacy.
It is for honest interpretation.
Say what the evidence shows.
Admit what it does not show."
It is for honest interpretation.
Say what the evidence shows.
Admit what it does not show."
Have you not heard of the Wakefield paper,
where conflicts of interest were hidden,
where data was fabricated,
and millions of children went unvaccinated?
where conflicts of interest were hidden,
where data was fabricated,
and millions of children went unvaccinated?
The MMR-Autism Fraud
THE LANCET, 1998-2010
Andrew Wakefield published a study linking MMR vaccine to autism.
He did not disclose that he was paid £435,643 by lawyers seeking to sue vaccine manufacturers.
He did not disclose that he had filed a patent for a competing single-dose measles vaccine.
The study was eventually retracted. Wakefield was struck off. But the damage was done: vaccination rates plummeted, and measles outbreaks returned.
He did not disclose that he was paid £435,643 by lawyers seeking to sue vaccine manufacturers.
He did not disclose that he had filed a patent for a competing single-dose measles vaccine.
The study was eventually retracted. Wakefield was struck off. But the damage was done: vaccination rates plummeted, and measles outbreaks returned.
Deer B. BMJ. 2011;342:c5347
Transparency Checklist
What to Declare
□ Protocol registration number
□ Funding sources (all)
□ Role of funder in the review
□ Conflicts of interest (all authors)
□ Data availability statement
□ Deviations from protocol (with reasons)
□ Author contributions
The AllTrials Initiative
2013-PRESENT | A GRASSROOTS TRANSPARENCY MOVEMENT
In 2013, Ben Goldacre and colleagues at the Cochrane Collaboration launched AllTrials after discovering a disturbing truth: approximately half of all clinical trials were never published. The missing trials were disproportionately those with negative or inconvenient results.
The campaign gathered over 90,000 individual signatories and 700+ organizations. It demanded that all past and future trials be registered, with full methods and results reported.
The impact was transformative. The EU now requires trial registration and results reporting. The FDA strengthened its own requirements. Journals began demanding prospective registration. What started as advocacy became global policy.
The campaign gathered over 90,000 individual signatories and 700+ organizations. It demanded that all past and future trials be registered, with full methods and results reported.
The impact was transformative. The EU now requires trial registration and results reporting. The FDA strengthened its own requirements. Journals began demanding prospective registration. What started as advocacy became global policy.
THE LESSON
Demanding data access works. A grassroots movement changed international regulations—proving that transparency advocates can reshape the evidence ecosystem.
The Hydroxychloroquine Pre-print Cascade: 2020
It is early 2020. Your team has preliminary meta-analysis results on a COVID-19 treatment. The Gautret preprint (non-randomized, 42 patients) has already gone viral. The Surgisphere scandal will soon show fabricated data in major journals. Do you rush to preprint or wait?
PATH A: Preprint for Speed
Post immediately to medRxiv to influence policy; skip peer review for urgency
↓
If included studies have fabricated data or flawed methods, your meta-analysis amplifies the errors; retraction damages credibility
OUTCOME: Accelerated misinformation
PATH B: Verify, Then Publish
Rigorously assess study quality; contact authors for raw data; undergo rapid peer review before posting
↓
Slower to publish, but the analysis is robust; conclusions survive when flawed studies are retracted
OUTCOME: Durable, trustworthy evidence
"Transparency is not optional.
Declare your funding. Declare your conflicts.
The reader has a right to know
who paid for this work—and why."
Declare your funding. Declare your conflicts.
The reader has a right to know
who paid for this work—and why."
Before You Submit
The Final Checklist
PRISMA 2020 Final Check
Have You...
□ Completed all 27 PRISMA checklist items?
□ Included the PRISMA flow diagram?
□ Provided full search strategy in appendix?
□ Listed excluded studies with reasons?
□ Reported risk of bias for each study?
□ Provided forest plot(s)?
□ Assessed publication bias (if ≥10 studies)?
□ Graded certainty of evidence (GRADE)?
□ Declared all conflicts of interest?
□ Cited protocol registration?
Supplementary Materials
WHAT TO INCLUDE
• Full search strategies for all databases
• List of excluded studies with reasons
• Data extraction forms (blank and completed)
• Risk of bias details for each study
• Additional forest plots (subgroups, sensitivity)
• Funnel plot and statistical tests
• GRADE evidence profile tables
• List of excluded studies with reasons
• Data extraction forms (blank and completed)
• Risk of bias details for each study
• Additional forest plots (subgroups, sensitivity)
• Funnel plot and statistical tests
• GRADE evidence profile tables
How long can a fraudulent paper survive peer review, editorial scrutiny, and public challenge?
REAL DATA
Andrew Wakefield's 1998 Lancet paper linking MMR vaccine to autism took 12 years to be fully retracted (2010). During that time, journalist Brian Deer uncovered financial conflicts, ethical violations, and data manipulation. Multiple large studies (including a Danish cohort of over 650,000 children) found no association, yet the original paper's influence persisted.
The Wakefield Retraction: 1998-2010
During peer review, a reviewer raises serious concerns about a study included in your meta-analysis, citing data inconsistencies. How do you respond?
PATH A: Deflect the Concern
Dismiss the reviewer's concern as overly cautious; keep the study included without further investigation
↓
If the study is later retracted, your meta-analysis is contaminated; the conclusion may need to be withdrawn
OUTCOME: Contaminated evidence synthesis
PATH B: Investigate Thoroughly
Contact the study authors for raw data; run sensitivity analysis excluding the questioned study; disclose the concern transparently
↓
Your meta-analysis is robust to the inclusion or exclusion of the suspect study; the sensitivity analysis is documented
OUTCOME: Resilient, self-correcting review
THE REVELATION
Peer review is your last defense before publication. Engage with reviewer concerns as opportunities to strengthen your work, not obstacles to overcome.
"You have gathered the evidence.
You have weighed it fairly.
You have written it transparently.
Now submit your work—
and let truth be found, and found again."
You have weighed it fairly.
You have written it transparently.
Now submit your work—
and let truth be found, and found again."
References
Key Sources
- Page MJ et al. BMJ. 2021;372:n71. [PRISMA 2020]
- Jefferson T et al. Cochrane 2014;4:CD008965. [Tamiflu]
- Turner EH et al. N Engl J Med. 2008;358:252-260. [Antidepressants]
- Boutron I et al. JAMA. 2010;303:2058-2064. [Spin]
- Topol EJ. N Engl J Med. 2004;351:1707-1709. [Vioxx]
- Deer B. BMJ. 2011;342:c5347. [Wakefield]
- Sterne JAC et al. BMJ. 2019;366:l4898. [RoB 2]
- Higgins JPT et al. Cochrane Handbook. 2023.
- Schünemann HJ et al. GRADE Handbook. 2013.
- Ioannidis JPA. PLoS Med. 2005;2:e124. [Why most research is false]
What percentage of antidepressant trials appeared positive in published literature vs. FDA data?
Published 51%, FDA 94%
Published 94%, FDA 51%
Both about 75%
Published 80%, FDA 60%
What is the purpose of registering a protocol before conducting a systematic review?
To get funding
To claim priority
To prevent outcome switching and data-driven decisions
To make the review publishable
When should you NOT pool studies in a meta-analysis?
When there are fewer than 10 studies
When studies are too clinically or methodologically heterogeneous
When the effect is not statistically significant
When studies are from different countries
✔
Course Complete
"You now know the covenant of evidence:
Register before you search.
Search comprehensively. Select transparently.
Extract carefully. Assess bias honestly.
Pool wisely—or not at all.
Write so that truth may be found,
and found again, by those who follow."
Register before you search.
Search comprehensively. Select transparently.
Extract carefully. Assess bias honestly.
Pool wisely—or not at all.
Write so that truth may be found,
and found again, by those who follow."
The methods are nothing without the tools to execute them.
Which software will carry your analysis
from protocol to forest plot?
Which software will carry your analysis
from protocol to forest plot?
Software Decision Tree
Choosing Your Tools
Meta-Analysis Software
↓
Your Context?
Cochrane Review
RevManFree, official
Academic/Flexible
R metaforFree, powerful
Institution License
Stata metaComprehensive
Point-and-Click
CMAUser-friendly
The Essential Toolkit
RevMan
Cochrane official
Free download
Free download
R + metafor
Most flexible
Reproducible code
Reproducible code
GRADEpro
Certainty tables
SoF tables
SoF tables
Rayyan
Screening tool
AI-assisted
AI-assisted
REPRODUCIBILITY
Code-based tools (R, Stata) create reproducible analyses.
Share your code so others can verify your work.
R metafor Example
BASIC META-ANALYSIS IN R
library(metafor)
# Calculate effect sizes
dat <- escalc(measure="RR", ai=events_tx, bi=noevents_tx,
ci=events_ctrl, di=noevents_ctrl, data=mydata)
# Random effects model
res <- rma(yi, vi, data=dat, method="REML")
# Forest plot
forest(res, slab=paste(author, year))
# Calculate effect sizes
dat <- escalc(measure="RR", ai=events_tx, bi=noevents_tx,
ci=events_ctrl, di=noevents_ctrl, data=mydata)
# Random effects model
res <- rma(yi, vi, data=dat, method="REML")
# Forest plot
forest(res, slab=paste(author, year))
How do you coordinate writing when a systematic review has dozens of authors?
REAL DATA
The SPRINT trial (2015) listed over 100 authors from dozens of institutions. The writing group included a steering committee, site investigators, and statisticians. Coordinating contributions, managing version control, and determining authorship credit required formal structures. ICMJE criteria define authorship as requiring substantial contribution, drafting or revision, final approval, and accountability.
Team Writing Challenges: Large Collaborative Reviews
Your systematic review team has 12 members across 4 countries. How do you manage the writing process?
PATH A: Informal Coordination
Pass drafts via email; resolve authorship at the end; no writing plan or version control
↓
Duplication of effort; authorship disputes at submission; inconsistent voice and formatting; lost contributions
OUTCOME: Delays and conflict
PATH B: Structured Process
Assign section leads upfront; use shared platforms with version control; agree ICMJE criteria and authorship order at the start
↓
Clear accountability; consistent output; transparent contributions; authorship decided before results are known
OUTCOME: Efficient, fair collaboration
THE REVELATION
Agree on authorship criteria and writing responsibilities before the work begins. Disputes are hardest to resolve after the results are known.
"The tool does not make the analysis.
The analyst does.
But choose your tool wisely—
and share your code so truth can be verified."
The analyst does.
But choose your tool wisely—
and share your code so truth can be verified."
Not everyone writes meta-analyses.
But every clinician, every policymaker, every patient
must know how to read them.
But every clinician, every policymaker, every patient
must know how to read them.
The HRT Reversal
WOMEN'S HEALTH INITIATIVE, 2002
For decades, observational studies suggested hormone replacement therapy (HRT)
protected women from heart disease.
Meta-analyses of these studies showed a 35-50% reduction in cardiovascular risk.
Then the WHI randomized trial revealed the truth: HRT increased heart attack risk by 29%.
The observational meta-analyses had pooled confounded data— healthier women chose HRT, not the reverse.
Meta-analyses of these studies showed a 35-50% reduction in cardiovascular risk.
Then the WHI randomized trial revealed the truth: HRT increased heart attack risk by 29%.
The observational meta-analyses had pooled confounded data— healthier women chose HRT, not the reverse.
Rossouw JE et al. JAMA. 2002;288:321-333
How to Read a Forest Plot
Consumer's Guide
Reading the Forest Plot
↓
Line of no effectRR=1 or MD=0
Diamond positionLeft=benefit, Right=harm
Diamond widthNarrow=precise
↓
Does diamond cross the line?
NoStatistically significant
YesNot significant
Red Flags When Reading
Warning Signs in Published Meta-Analyses
⚠ No protocol registration cited
⚠ Single database searched
⚠ No risk of bias assessment
⚠ High I² but no investigation
⚠ Asymmetric funnel plot ignored
⚠ Industry funding, no sensitivity analysis
⚠ Conclusions exceed the evidence
What GRADE Ratings Mean
FOR CLINICIANS AND PATIENTS
HIGH: We are very confident. Future research unlikely to change.
MODERATE: Probably close to truth. Future research may change estimate.
LOW: Uncertain. Future research likely to change substantially.
VERY LOW: Very uncertain. Any estimate is speculative.
MODERATE: Probably close to truth. Future research may change estimate.
LOW: Uncertain. Future research likely to change substantially.
VERY LOW: Very uncertain. Any estimate is speculative.
"Read the forest, not just the diamond.
Look for the protocol. Check the bias.
Ask: Who funded this? What did they hide?
The informed reader is the guardian of truth."
Look for the protocol. Check the bias.
Ask: Who funded this? What did they hide?
The informed reader is the guardian of truth."
What if you must compare treatments
that were never tested head-to-head?
This is the realm of network meta-analysis.
that were never tested head-to-head?
This is the realm of network meta-analysis.
When to Use Network MA
NMA Decision Tree
Multiple Treatments
↓
Direct comparisons available?
All pairs directly compared
Pairwise MAStandard approach
Some indirect only
Network MABorrow strength
↓
Check transitivity assumptionSimilar populations across comparisons
The Network Geometry
VISUALIZING THE EVIDENCE
Nodes = Treatments (size = sample)
Edges = Direct comparisons (width = studies)
Dashed = Indirect evidence only
League Tables
READING NMA RESULTS
League tables show all pairwise comparisons from the network.
• Each cell: effect estimate + 95% CI
• Row vs. Column: Treatment A vs. Treatment B
• Green = Favors row treatment
• Red = Favors column treatment
• Rankings (SUCRA/P-score) help identify best options
• Each cell: effect estimate + 95% CI
• Row vs. Column: Treatment A vs. Treatment B
• Green = Favors row treatment
• Red = Favors column treatment
• Rankings (SUCRA/P-score) help identify best options
CRITICAL ASSUMPTION
Transitivity: If A vs. B and B vs. C are compared in similar patients,
we can estimate A vs. C indirectly.
What happens to meta-analyses when a prolific author's entire body of work is retracted?
REAL DATA
Joachim Boldt, a German anesthesiologist, had over 220 papers retracted for data fabrication (discovered 2010-2011). His studies on colloid solutions had been included in multiple systematic reviews and meta-analyses. When the retractions came, every meta-analysis containing his work had to be re-evaluated. Some conclusions changed substantially when his fabricated data was removed.
The Boldt Retraction Cascade: 2010-2011
You discover that a study included in your published meta-analysis has been retracted for data fabrication. What do you do?
PATH A: Hope No One Notices
Ignore the retraction; the meta-analysis is already published and the retracted study was small
↓
Others cite your meta-analysis; the fabricated data propagates through secondary citations; patient care decisions are based on contaminated evidence
OUTCOME: Cascading harm from inaction
PATH B: Self-Correct Publicly
Publish a correction or updated analysis excluding the retracted study; notify the journal; state whether conclusions change
↓
The scientific record is corrected; readers see the updated analysis; your reputation for integrity is enhanced
OUTCOME: Scientific self-correction
THE REVELATION
The integrity of evidence synthesis depends on ongoing vigilance. When included studies are retracted, the ethical duty is to update and correct, not to remain silent.
"When treatments have never met,
the network builds a bridge of evidence.
But the bridge rests on transitivity—
verify that the populations are comparable."
the network builds a bridge of evidence.
But the bridge rests on transitivity—
verify that the populations are comparable."
Have you not seen the AI that predicted cancer
with 99% accuracy in the training set—
and failed catastrophically
when deployed in the real world?
with 99% accuracy in the training set—
and failed catastrophically
when deployed in the real world?
The Sepsis Algorithm Failure
EPIC SEPSIS MODEL, 2021
Epic's sepsis prediction algorithm was deployed in hundreds of hospitals.
Internal validation showed excellent performance.
But an independent study at Michigan Medicine found the model missed 67% of sepsis cases and generated excessive false alarms.
The algorithm had been validated on the same population it was trained on— a recipe for overfitting and failure.
Internal validation showed excellent performance.
But an independent study at Michigan Medicine found the model missed 67% of sepsis cases and generated excessive false alarms.
The algorithm had been validated on the same population it was trained on— a recipe for overfitting and failure.
Wong A et al. JAMA Intern Med. 2021;181:1065-1070
AI Validation Decision Tree
Levels of Evidence for AI/ML
AI Prediction Model
↓
Validation Level?
Internal onlySame data split
HIGH RISK
TemporalDifferent time
MODERATE
ExternalDifferent site
BETTER
Impact RCTPatient outcomes
BEST
PROBAST & TRIPOD
PROBAST
Prediction model
Risk of Bias
Risk of Bias
TRIPOD
Reporting
guideline
guideline
TRIPOD-AI
AI-specific
extension
extension
CALIBRATION VS. DISCRIMINATION
AUC/c-statistic: Can the model rank patients? (discrimination)
Calibration: Are predicted probabilities accurate?
A model can have good AUC but poor calibration—and harm patients.
Calibration: Are predicted probabilities accurate?
A model can have good AUC but poor calibration—and harm patients.
"The algorithm learned from the data,
and the data was biased.
It validated on itself,
and called its reflection truth.
External validation is not optional—it is survival."
and the data was biased.
It validated on itself,
and called its reflection truth.
External validation is not optional—it is survival."
The meta-analysis speaks in numbers.
But the patient hears in fears and hopes.
How will you bridge the gap?
But the patient hears in fears and hopes.
How will you bridge the gap?
Translating Numbers to Meaning
Communication Decision Tree
Meta-Analysis Result
↓
Effect Size Type?
Relative (RR, OR)
Convert to NNTMore intuitive
Absolute (RD)
Use directly"X fewer per 1000"
Continuous (MD)
ContextualizeMinimal important diff
Scripts for Patients
EXPLAINING A POSITIVE RESULT
"The research pooled 15 studies with 8,000 patients.
It found that this treatment reduces the risk of [outcome] by about 30%.
In practical terms: if we treat 100 people like you, about 5 fewer will have [outcome] compared to no treatment.
We're moderately confident in this—future research might change it slightly.
What questions do you have about this?"
It found that this treatment reduces the risk of [outcome] by about 30%.
In practical terms: if we treat 100 people like you, about 5 fewer will have [outcome] compared to no treatment.
We're moderately confident in this—future research might change it slightly.
What questions do you have about this?"
Questions Patients Should Ask
Empowering Patients
1 "How many studies and patients were included?"
2 "How confident are the researchers in this result?"
3 "What are the benefits AND harms?"
4 "Were people like me included in these studies?"
5 "Who funded this research?"
6 "What does this mean for my specific situation?"
Can a spreadsheet error in an academic paper directly shape the economic policy of entire nations?
REAL DATA
Reinhart and Rogoff's 2010 paper claimed that countries with public debt exceeding 90% of GDP experienced dramatically lower growth. This finding was widely cited to justify austerity policies across Europe. In 2013, Herndon, Ash, and Pollin discovered a spreadsheet error: several countries had been accidentally excluded from the calculations. After correction, the sharp 90% threshold disappeared.
The Reinhart-Rogoff Policy Impact: 2010-2013
Your meta-analysis results have clear policy implications. Policymakers are eager for a simple message. How do you write the policy brief?
PATH A: Oversimplify for Impact
Provide a clean threshold or headline number; omit caveats and uncertainty ranges for maximum policy influence
↓
Policy is adopted based on simplified findings; when nuances emerge or errors are found, both the research and resulting policy are discredited
OUTCOME: Policy built on a fragile foundation
PATH B: Communicate with Integrity
Present the evidence with appropriate uncertainty; distinguish strong from suggestive findings; provide actionable summaries that preserve nuance
↓
Policymakers understand what the evidence supports and where uncertainty remains; decisions are made with appropriate caution
OUTCOME: Durable, evidence-informed policy
THE REVELATION
Policy briefs must communicate uncertainty honestly. Oversimplified findings may gain influence quickly but collapse when scrutinized, damaging trust in research-policy relationships.
"The meta-analysis speaks in numbers.
The patient hears in fears and hopes.
Your job is to be the translator—
faithful to the evidence, compassionate to the person."
The patient hears in fears and hopes.
Your job is to be the translator—
faithful to the evidence, compassionate to the person."
A systematic review captures evidence at a moment in time.
But science does not stop.
How do we keep the evidence alive?
But science does not stop.
How do we keep the evidence alive?
Living Systematic Reviews
COVID-19 PANDEMIC, 2020-2023
During the pandemic, evidence emerged faster than traditional reviews could synthesize.
Living systematic reviews were continuously updated as new trials reported— sometimes within days of publication.
The COVID-NMA consortium produced living reviews on treatments, vaccines, and diagnostics, updating recommendations in real-time as the evidence evolved.
Hydroxychloroquine went from "promising" to "ineffective" within months.
Living systematic reviews were continuously updated as new trials reported— sometimes within days of publication.
The COVID-NMA consortium produced living reviews on treatments, vaccines, and diagnostics, updating recommendations in real-time as the evidence evolved.
Hydroxychloroquine went from "promising" to "ineffective" within months.
Defined by Cochrane: continual updates at ≤monthly intervals
When to Use Living Reviews
Living Review Decision Tree
Review Type Decision
↓
Is Evidence Rapidly Evolving?
Yes + High Priority
Living ReviewContinuous updates
No / Stable
Standard ReviewUpdate every 2-5 years
↓
Resource intensiveRequires ongoing funding
The Future of Evidence Synthesis
Automation
ML-assisted
screening
screening
IPD-MA
Individual patient
data pooling
data pooling
Real-World
EHR-based
evidence
evidence
Adaptive
Platform trials
+ MA
+ MA
"The covenant of evidence is not static.
It grows with each new study, each new question.
Keep your reviews alive.
Keep your methods transparent.
Keep truth at the center of all you do."
It grows with each new study, each new question.
Keep your reviews alive.
Keep your methods transparent.
Keep truth at the center of all you do."
1 / 4