DTA > The Fraud
Have you not heard the tale of the woman
who promised to change the world with a drop of blood,
who raised billions on a test that never worked?
who promised to change the world with a drop of blood,
who raised billions on a test that never worked?
Palo Alto, 2003
STANFORD UNIVERSITY
A nineteen-year-old dropped out with a vision: hundreds of blood tests from a single drop.
Investors believed. Walgreens believed. The Pentagon believed.
They gave her $9 billion.
But the tests gave wrong results. Patients were told they had HIV when they didn't. Patients were told their blood was normal when they were dying.
Investors believed. Walgreens believed. The Pentagon believed.
They gave her $9 billion.
But the tests gave wrong results. Patients were told they had HIV when they didn't. Patients were told their blood was normal when they were dying.
Carreyrou J. Bad Blood. 2018
The Decision Tree of Deception
What Theranos Did vs. What Should Happen
New Diagnostic Test
↓
SHOULD DO
Validate Against Gold Standard
↓
Publish TP/FP/FN/TN
↓
FDA Approval
THERANOS DID
Skip Validation
↓
Hide Failures
↓
Harm Patients
"And the test lied,
and the lie was dressed in certainty,
and no one asked for the 2×2 table."
and the lie was dressed in certainty,
and no one asked for the 2×2 table."
This is why we study Diagnostic Test Accuracy.
When a test speaks,
there are only four possible truths.
Two are blessings. Two are curses.
there are only four possible truths.
Two are blessings. Two are curses.
What happens when a systematic review trusts every study equally?
REAL DATA
Sensitivity analyses in DTA systematic reviews consistently demonstrate that excluding high risk-of-bias studies changes pooled estimates. In mammography screening, case-control designs with unblinded interpretation tend to inflate sensitivity. The general principle is well-documented: QUADAS-2 quality assessment can shift pooled sensitivity by 10-15 percentage points when biased studies are removed.
The QUADAS-2 Mammography Audit
A review team pools 15 mammography DTA studies. Five have high risk of bias due to case-control design and unblinded interpretation.
PATH A: Pool All Studies
Include all 15 studies regardless of quality
↓
Biased 2x2 tables inflate TP counts, producing a pooled sensitivity of 87%
OUTCOME: Overconfidence in screening accuracy
PATH B: Apply Quality Assessment
Exclude high risk-of-bias studies using QUADAS-2
↓
Remaining 10 low-RoB studies yield sensitivity of approximately 75%
OUTCOME: Honest numbers guide honest decisions
THE REVELATION
The four outcomes (TP, FP, FN, TN) are only trustworthy if the study that produced them is trustworthy. A biased study contaminates the entire 2x2 table.
The Tree of Outcomes
Every Test Result Has a Reality Behind It
Patient Tested
↓
What is the TRUTH?
Has Disease
D+
↓
TPTest +
FNTest -
No Disease
D-
↓
FPTest +
TNTest -
The Sacred 2×2 Table
HIV Rapid Test Example (Real Data)
| HIV+ | HIV- | Total | |
|---|---|---|---|
| Test + | 98 | 3 | 101 |
| Test - | 2 | 895 | 897 |
| Total | 100 | 898 | 998 |
FROM THIS TABLE COMES ALL TRUTH
Sensitivity = 98/100 = 98%
Specificity = 895/898 = 99.7%
Specificity = 895/898 = 99.7%
"Two outcomes save. Two outcomes harm.
TP, TN: the test spoke true.
FP, FN: the test lied.
Know them by name, for they determine fate."
TP, TN: the test spoke true.
FP, FN: the test lied.
Know them by name, for they determine fate."
Have you not heard of the blood that was tested,
found clean,
and given to thousands—
while death swam within it?
found clean,
and given to thousands—
while death swam within it?
The Blood Supply Crisis, 1985
UNITED STATES
When HIV testing began, doctors celebrated: they could now screen the blood supply.
But the test had a window period—weeks after infection when the virus was present but undetectable.
Blood was tested. Blood was "negative." Blood was transfused.
8,000-12,000 Americans were infected through transfusions before better tests closed the window.
But the test had a window period—weeks after infection when the virus was present but undetectable.
Blood was tested. Blood was "negative." Blood was transfused.
8,000-12,000 Americans were infected through transfusions before better tests closed the window.
CDC. MMWR. 1987;36(49):833-840
The Window Period Decision Tree
Why False Negatives Are Deadly
Person Recently Infected
↓
Time Since Infection?
< 2 weeks
Test NEGATIVEVirus present!
↓
Blood DonatedOthers infected
> 4 weeks
Test POSITIVECorrectly detected
↓
Blood DiscardedSupply safe
Sensitivity Changes Over Time
0%
Day 1-7
Eclipse period
Eclipse period
~50%
Day 14
Seroconversion
Seroconversion
~95%
Day 21
Most detected
Most detected
99.9%
Day 45+
Window closed
Window closed
THE LESSON
Sensitivity is not fixed. It depends on when you test.
A "99% sensitive" test may be 0% sensitive in early infection.
"And the test said 'clean,'
for the virus had not yet shown its face.
And the blood was shared,
and the infection spread to the innocent."
for the virus had not yet shown its face.
And the blood was shared,
and the infection spread to the innocent."
Have you not heard of the pill given to mothers
to protect their pregnancies,
that planted cancer in their daughters
twenty years before it bloomed?
to protect their pregnancies,
that planted cancer in their daughters
twenty years before it bloomed?
The DES Tragedy, 1938-1971
UNITED STATES & EUROPE
Diethylstilbestrol (DES) was given to millions of pregnant women to prevent miscarriage.
No proper clinical trial was ever conducted. Doctors assumed it worked because it seemed reasonable.
Decades later, their daughters developed a rare cancer: clear cell adenocarcinoma of the vagina. A cancer so rare it was a diagnostic signal in itself.
5-10 million women were exposed. The harm crossed generations.
No proper clinical trial was ever conducted. Doctors assumed it worked because it seemed reasonable.
Decades later, their daughters developed a rare cancer: clear cell adenocarcinoma of the vagina. A cancer so rare it was a diagnostic signal in itself.
5-10 million women were exposed. The harm crossed generations.
Herbst AL et al. N Engl J Med. 1971;284:878-881
The Validation Decision Tree
What Should Have Happened
New Medical Intervention
↓
Was It Properly Tested?
YES
Randomized Trial
↓
Long-term Follow-up
↓
Know True EffectsBenefits AND harms
NO (DES)
Assumption Only
↓
Widespread Use
↓
Hidden HarmDiscovered too late
The Diagnostic Signal
WHEN RARITY BECOMES EVIDENCE
Clear cell adenocarcinoma of the vagina was so rare in young women that
7 cases in one hospital triggered an investigation.
The cluster itself was the diagnostic test:
Sensitivity to DES exposure: nearly 100%
If you have this cancer at this age, you were almost certainly exposed.
The cluster itself was the diagnostic test:
Sensitivity to DES exposure: nearly 100%
If you have this cancer at this age, you were almost certainly exposed.
1:1000
Risk of clear cell
cancer in DES daughters
cancer in DES daughters
5-10M
Women exposed
worldwide
worldwide
"And the mothers took the pill in hope,
and the daughters grew in shadow,
and twenty years later the cancer bloomed—
a diagnosis that indicted a generation of medicine."
and the daughters grew in shadow,
and twenty years later the cancer bloomed—
a diagnosis that indicted a generation of medicine."
A test has two virtues and two vices.
Sensitivity: Can it find the sick?
Specificity: Can it spare the healthy?
Sensitivity: Can it find the sick?
Specificity: Can it spare the healthy?
Can you trust a sensitivity number from a laboratory when the test is used in the real world?
REAL DATA
The BinaxNOW COVID-19 rapid antigen test reported sensitivity of approximately 84-97% in symptomatic individuals in manufacturer studies. However, real-world evaluations found sensitivity as low as 35-64% in asymptomatic individuals, depending on viral load and timing. The Cochrane review of rapid antigen tests (Dinnes 2022) confirmed average sensitivity of 73% in symptomatic and only 55% in asymptomatic populations across over 100 study evaluations.
The COVID Rapid Test Paradox: 2020-2021
A university plans to screen asymptomatic students weekly before allowing campus access. They read the manufacturer's claim of high sensitivity.
PATH A: Trust Lab Sensitivity
Rely on manufacturer's high sensitivity figure
↓
Asymptomatic carriers with low viral loads test negative and attend classes, spreading the virus
OUTCOME: False sense of safety; campus outbreaks
PATH B: Demand Real-World Data
Seek studies in the actual target population (asymptomatic students)
↓
Discover sensitivity is roughly 55% in asymptomatic people; add serial testing and other safeguards
OUTCOME: Layered safety catches more cases
THE REVELATION
Sensitivity is not a fixed property of a test. It changes with the population, the disease stage, and the setting. Always ask: sensitivity in whom?
Sensitivity: The Hunter
THE FORMULA
Sensitivity = TP / (TP + FN)
"Of all the sick, how many did we catch?"
Worked Example: COVID PCR Test
Given: 200 infected patients tested
TP = 196 (correctly positive), FN = 4 (missed)
Sensitivity = 196 / (196 + 4) = 196/200 = 98%
Interpretation: Test catches 98 of every 100 infected people
Specificity: The Guardian
THE FORMULA
Specificity = TN / (TN + FP)
"Of all the healthy, how many did we spare?"
Worked Example: Same COVID PCR Test
Given: 1000 uninfected people tested
TN = 999 (correctly negative), FP = 1 (false alarm)
Specificity = 999 / (999 + 1) = 999/1000 = 99.9%
Interpretation: Test correctly clears 999 of every 1000 healthy people
The Memory Rules
When to Use Which Test
What Do You Need?
RULE OUT disease
Use HIGH SENSITIVITY
↓
SnNoutSensitive Negative = OUT
RULE IN disease
Use HIGH SPECIFICITY
↓
SpPinSpecific Positive = IN
"Sensitivity catches the sick.
Specificity spares the well.
But no test masters both perfectly—
this is the burden we bear."
Specificity spares the well.
But no test masters both perfectly—
this is the burden we bear."
Have you not seen the physician
who saw 99% accurate
and believed a positive result meant 99% certainty?
This is the deadliest error in medicine.
who saw 99% accurate
and believed a positive result meant 99% certainty?
This is the deadliest error in medicine.
The Base Rate Fallacy
THE PUZZLE
A disease affects 1 in 1000 people.
A test is 99% sensitive and 99% specific.
A patient tests positive.
What is the probability they have the disease?
Most doctors say ~99%. The real answer is about 9%.
A test is 99% sensitive and 99% specific.
A patient tests positive.
What is the probability they have the disease?
Most doctors say ~99%. The real answer is about 9%.
The Math Revealed
Testing 100,000 People (Prevalence 1/1000)
Step 1: 100 have disease, 99,900 healthy
Step 2: Of 100 sick: 99 test positive (TP), 1 negative (FN)
Step 3: Of 99,900 healthy: 999 test positive (FP), 98,901 negative (TN)
Step 4: Total positives = 99 + 999 = 1,098
PPV = TP / All Positives = 99 / 1,098 = 9%
91% of positive results are FALSE POSITIVES!
Interactive Base Rate Calculator
See How Prevalence Changes PPV
Prevalence:
1%
Sensitivity:
99%
Specificity:
99%
9%
Positive Predictive Value (PPV)
91% of positives are false alarms
The Decision Tree of Prevalence
Same Test, Different Settings
Test: 99% Sens, 99% Spec
↓
Where Is Testing Done?
General Pop
0.1%
0.1%
PPV = 9%91% false +
High-Risk
10%
10%
PPV = 92%8% false +
Confirmatory
50%
50%
PPV = 99%1% false +
"And the doctor said '99% accurate,'
and the patient heard '99% certain,'
and both were deceived—
for they forgot to ask: How rare is this disease?"
and the patient heard '99% certain,'
and both were deceived—
for they forgot to ask: How rare is this disease?"
Have you not heard of the machine
that could find TB in two hours,
that was called revolutionary—
but missed the drug-resistant strains?
that could find TB in two hours,
that was called revolutionary—
but missed the drug-resistant strains?
The GeneXpert Story, South Africa
CAPE TOWN, 2010
For a century, TB diagnosis required growing bacteria for weeks.
Then came GeneXpert: results in 2 hours.
South Africa deployed it nationwide. The WHO endorsed it.
But in patients with low bacterial loads—often HIV co-infected— sensitivity dropped to 67%. One in three cases missed.
And for detecting rifampicin resistance, it missed 5% of resistant cases. Those patients received the wrong treatment. The resistant TB spread.
South Africa deployed it nationwide. The WHO endorsed it.
But in patients with low bacterial loads—often HIV co-infected— sensitivity dropped to 67%. One in three cases missed.
And for detecting rifampicin resistance, it missed 5% of resistant cases. Those patients received the wrong treatment. The resistant TB spread.
Steingart KR et al. Cochrane Database Syst Rev. 2014;1:CD009593
TB Diagnosis Decision Tree
When GeneXpert Is Not Enough
Suspected TB Patient
↓
GeneXpert Test
↓
Positive
↓
Rifampicin?
SensitiveStandard Tx
ResistantMDR-TB Tx
Negative
↓
HIV+ or High Suspicion?
YesCulture needed
NoLikely negative
Sensitivity by Patient Type
98%
Smear-positive
(high bacterial load)
(high bacterial load)
67%
Smear-negative
(low bacterial load)
(low bacterial load)
61%
HIV co-infected
(immune suppressed)
(immune suppressed)
THE LESSON
A test's sensitivity in clinical trials may not match its sensitivity in your patients.
Know your population.
"And the machine said 'negative,'
and the doctor believed the machine,
and the patient went home with TB in their lungs,
coughing resistance into the world."
and the doctor believed the machine,
and the patient went home with TB in their lungs,
coughing resistance into the world."
Have you not heard of the test for men
that found cancers that would never kill,
and led to treatments that destroyed lives?
that found cancers that would never kill,
and led to treatments that destroyed lives?
The PSA Screening Tragedy
UNITED STATES, 1990s-2010s
PSA (Prostate-Specific Antigen) could detect prostate cancer early.
Doctors screened millions of men. Cancers were found. Prostates were removed.
But many of these "cancers" would never have caused symptoms. The surgery caused impotence and incontinence in men who would have died of old age, not cancer.
Doctors screened millions of men. Cancers were found. Prostates were removed.
But many of these "cancers" would never have caused symptoms. The surgery caused impotence and incontinence in men who would have died of old age, not cancer.
Moyer VA. Ann Intern Med. 2012;157:120-134
The PSA Screening Dilemma: 2012
A 60-year-old man asks his doctor about PSA screening. PSA at the 4.0 ng/mL cutoff has sensitivity of approximately 21% for high-grade cancer but detects many indolent cancers.
PATH A: Screen All Men
Routine PSA screening for all men over 50
↓
Per 1,000 screened over 13 years: 1-2 deaths prevented, but 100+ false alarms and 30-40 men left impotent or incontinent from treating indolent cancers
OUTCOME: Net harm exceeds benefit at population level
PATH B: Shared Decision-Making
Discuss harms vs. benefits; individualize with risk factors, life expectancy, and patient values
↓
High-risk men can choose screening; low-risk men can decline; active surveillance replaces immediate surgery for low-grade findings
OUTCOME: Fewer unnecessary treatments; patient autonomy preserved
THE REVELATION
A test with high detection rates can cause more harm than good when it finds conditions that do not need finding. Overdiagnosis is the hidden cost of high sensitivity in indolent disease.
The Numbers of Harm
1
Life saved from
prostate cancer
per 1000 screened
prostate cancer
per 1000 screened
30-40
Men made impotent
or incontinent
per 1000 screened
or incontinent
per 1000 screened
100+
False positives
(biopsies, anxiety)
per 1000 screened
(biopsies, anxiety)
per 1000 screened
THE REVERSAL
In 2012, the US Preventive Services Task Force recommended against
routine PSA screening. The test was finding too much that didn't need finding.
Patient Decision Aid: PSA Screening
If 1,000 Men Age 55-69 Are Screened for 13 Years
Deaths from prostate cancer prevented
1-2 men
Men who will have false positive requiring biopsy
100-120 men
Men diagnosed with cancer that would never harm them
20-50 men
Men left impotent or incontinent from treatment
30-40 men
Is this trade-off acceptable to you?
"And the test found the shadow,
and the surgeon cut,
and the man lived—impotent, incontinent—
from a cancer that would never have woken."
and the surgeon cut,
and the man lived—impotent, incontinent—
from a cancer that would never have woken."
Have you not heard of the man with chest pain
whose first troponin was normal,
who was sent home—
and died before morning?
whose first troponin was normal,
who was sent home—
and died before morning?
The Troponin Timing Problem
EMERGENCY DEPARTMENTS WORLDWIDE
Troponin is the gold standard for heart attack diagnosis.
But it takes 3-6 hours to rise after myocardial injury.
A patient arrives one hour after chest pain begins. Troponin is tested: normal. "You're fine. Go home."
The heart was dying. The protein hadn't leaked yet.
Studies show 2-5% of MI patients sent home from ED die within 30 days.
A patient arrives one hour after chest pain begins. Troponin is tested: normal. "You're fine. Go home."
The heart was dying. The protein hadn't leaked yet.
Studies show 2-5% of MI patients sent home from ED die within 30 days.
Pope JH et al. N Engl J Med. 2000;342:1163-1170
Serial Testing Decision Tree
The Two-Troponin Protocol
Chest Pain Patient
↓
First Troponin
↓
Elevated
↓
Treat as MI
Normal
↓
When Did Pain Start?
<6 hrs
Wait 3 hrsRepeat troponin
>6 hrs
Low riskConsider d/c
High-Sensitivity Troponin
~70%
Conventional troponin
sensitivity at 0 hrs
sensitivity at 0 hrs
~95%
hs-Troponin
sensitivity at 0 hrs
sensitivity at 0 hrs
99%
hs-Troponin
at 3 hrs serial
at 3 hrs serial
THE TRADE-OFF
High-sensitivity troponin catches more heart attacks early.
But it also has more false positives—elevated in kidney disease,
heart failure, sepsis, and marathon runners.
"And the test said 'normal,'
for the heart had just begun to die.
And the patient was reassured,
and went home to finish dying."
for the heart had just begun to die.
And the patient was reassured,
and went home to finish dying."
Sensitivity describes the test.
Specificity describes the test.
But the patient asks:
"I tested positive. What are MY chances?"
Specificity describes the test.
But the patient asks:
"I tested positive. What are MY chances?"
What if the published sensitivity of a test is higher than the truth, and the likelihood ratios you calculate are therefore wrong?
REAL DATA
Rapid strep tests (RADT) showed pooled sensitivity of approximately 86% in published studies included in Cochrane reviews. However, FDA 510(k) regulatory submissions, which include unpublished manufacturer data, revealed sensitivity estimates of only 70-75%. Published studies with higher sensitivity were more likely to be submitted for publication—a classic case of publication bias inflating the apparent accuracy.
The Rapid Strep Test Publication Gap
A clinician calculates LR+ from published data (sensitivity 86%, specificity 95%) to decide whether to treat a child's sore throat. But the true sensitivity may be only 70%.
PATH A: Trust Published Meta-Analysis
Use LR+ from published data (86/5 = 17.2)
↓
Overestimated LR+ leads to overconfidence in a negative result; children with strep are sent home without antibiotics
OUTCOME: Missed strep leads to rheumatic fever risk
PATH B: Seek Regulatory Data
Use LR+ from FDA submissions (70/5 = 14), and note the LR- is worse (0.32 vs 0.15)
↓
Recognize a negative RADT cannot confidently exclude strep; back up with throat culture when clinical suspicion is high
OUTCOME: Appropriate caution protects children
THE REVELATION
Likelihood ratios are only as honest as the sensitivity and specificity that produce them. Publication bias inflates accuracy, making LR+ too optimistic and LR- too reassuring. Always ask: are unpublished studies missing?
Likelihood Ratios
POSITIVE LIKELIHOOD RATIO
LR+ = Sensitivity / (1 - Specificity)
How much more likely is a + result in sick vs healthy?
NEGATIVE LIKELIHOOD RATIO
LR- = (1 - Sensitivity) / Specificity
How much more likely is a - result in sick vs healthy?
The Fagan Nomogram
From Pre-Test to Post-Test Probability
Pre-Test
Probability
Probability
99%
50%
20%
5%
1%
Likelihood
Ratio
Ratio
100
10
1
0.1
0.01
Post-Test
Probability
Probability
99%
80%
50%
20%
1%
Draw a line from pre-test through LR to find post-test probability
Interpreting Likelihood Ratios
How Powerful Is This Test?
LR+ Value?
LR+ > 10Strong rule-in
5-10Moderate
2-5Weak
1-2Useless
LR- Value?
< 0.1Strong rule-out
0.1-0.2Moderate
0.2-0.5Weak
0.5-1Useless
"Sensitivity tells of the sick.
Specificity tells of the well.
But the likelihood ratio answers:
What does this result mean for THIS patient?"
Specificity tells of the well.
But the likelihood ratio answers:
What does this result mean for THIS patient?"
Have you not seen the child with fever in the village,
the rapid test that said negative,
and the Plasmodium that kept multiplying?
the rapid test that said negative,
and the Plasmodium that kept multiplying?
The Malaria RDT Problem
SUB-SAHARAN AFRICA
Malaria kills 600,000 people yearly, mostly children under 5.
Rapid Diagnostic Tests were meant to guide treatment in remote areas without microscopes or laboratories.
But when parasitemia is low—the RDT misses cases. And when P. falciparum deletes the HRP2 gene— the RDT sees nothing at all.
Rapid Diagnostic Tests were meant to guide treatment in remote areas without microscopes or laboratories.
But when parasitemia is low—the RDT misses cases. And when P. falciparum deletes the HRP2 gene— the RDT sees nothing at all.
WHO. Malaria RDT Performance. 2022
The Clinical Decision Tree
Child with Fever in Malaria-Endemic Area
Febrile Child
↓
Perform RDT
↓
RDT Positive
↓
Treat for Malaria
RDT Negative
↓
Clinical Suspicion?
High
Treat Anywayor Microscopy
Low
Look forOther Cause
Sensitivity Varies by Parasitemia
95%
High parasitemia
(>200/μL)
(>200/μL)
75%
Low parasitemia
(100-200/μL)
(100-200/μL)
50%
Very low
(<100/μL)
(<100/μL)
THE CLINICAL LESSON
A negative RDT does not rule out malaria in endemic areas.
Clinical judgment must override the test when suspicion is high.
"And the test said 'negative,'
and the child was sent home,
and the parasites multiplied in the dark,
and by morning the child could not wake."
and the child was sent home,
and the parasites multiplied in the dark,
and by morning the child could not wake."
In the year of pestilence,
the world needed a test that was fast.
But fast is not the same as accurate.
the world needed a test that was fast.
But fast is not the same as accurate.
When a new generation of test arrives with higher sensitivity, does that automatically make it better?
REAL DATA
High-sensitivity troponin (hs-cTn) assays increased sensitivity for acute myocardial infarction from approximately 70% (conventional troponin at presentation) to over 95%. But specificity dropped from approximately 95% to around 80% because hs-cTn detects myocardial injury from many non-MI causes (heart failure, sepsis, renal disease, pulmonary embolism). The net clinical effect required HSROC modeling across multiple studies to understand the tradeoff.
The Troponin Generation Shift: 2010s
An emergency department adopts hs-troponin. More patients now test positive, but many do not have acute MI.
PATH A: Adopt Based on Sensitivity Alone
Celebrate that MI detection jumped from 70% to over 95%
↓
More false positives lead to unnecessary catheterizations, hospital admissions, and patient anxiety for non-cardiac troponin elevations
OUTCOME: Overdiagnosis and wasted resources
PATH B: Model the Tradeoff
Use serial measurements (0h/1h or 0h/3h protocols) and clinical context to maintain specificity
↓
Rapid rule-out algorithms safely discharge low-risk patients; sensitivity remains high while managing the false positive rate
OUTCOME: Faster, safer triage of chest pain
THE REVELATION
Sensitivity and specificity trade off against each other. A new test generation that raises sensitivity will often lower specificity. The HSROC curve is the tool that reveals whether the net tradeoff helps or harms patients.
The Cochrane Verdict
COVID-19 Rapid Antigen Tests (Dinnes 2022 Cochrane Review)
| Population | Sensitivity | Missed |
|---|---|---|
| Symptomatic | 73% | 27% |
| Asymptomatic | 55% | 45% |
| First 7 days | 80% | 20% |
Dinnes J et al. Cochrane Database Syst Rev. 2022;7:CD013705
The False Security Decision Tree
Thanksgiving 2020: What Happened
Family Member Tests Negative
↓
Truly Negative?
55% if asymptomatic
True NegativeSafe to gather
45% if asymptomatic
FALSE NegativeInfectious!
↓
Gathers with FamilyGrandparents infected
"And the test said 'negative,'
and the family embraced,
and by winter's end,
the grandfather was buried."
and the family embraced,
and by winter's end,
the grandfather was buried."
Have you not heard of the screening
that found cancers that would never kill,
and led to treatments that caused more harm than the disease?
that found cancers that would never kill,
and led to treatments that caused more harm than the disease?
Can you trust a DTA meta-analysis done in a spreadsheet?
REAL DATA
DTA meta-analysis requires the bivariate model or HSROC—both need maximum likelihood estimation of correlated sensitivity and specificity on the logit scale. Research has documented that manual Excel calculations frequently introduce errors: a landmark study by Reinhart & Rogoff (2010, economics) demonstrated how a simple spreadsheet error led to global policy changes. In DTA, manually applying logit transformations and pooling sensitivity/specificity separately in Excel ignores the correlation between them, and can produce pooled estimates that differ meaningfully from validated bivariate models in software (R mada/reitsma, Stata metandi, SAS NLMIXED).
The QUADAS Excel Error
A research team needs pooled sensitivity and specificity for a DTA systematic review. They have 12 studies. One team member builds an Excel model; another uses R's mada package.
PATH A: Use the Spreadsheet
Pool sensitivity and specificity separately in Excel using simple averages or fixed-effect formulas
↓
Ignores the correlation between sensitivity and specificity; logit transformation errors compound; pooled sensitivity off by approximately 12 percentage points
OUTCOME: Wrong numbers published; clinical guidelines misled
PATH B: Use Validated Software
Use R (mada/reitsma), Stata (metandi), or SAS (NLMIXED) with the bivariate model
↓
Proper bivariate GLMM accounts for the sensitivity-specificity tradeoff, produces valid confidence regions, and handles between-study heterogeneity
OUTCOME: Reproducible, auditable, correct results
THE REVELATION
DTA meta-analysis is not simple pooling. The bivariate nature of the data (paired sensitivity and specificity) requires specialized statistical software. A spreadsheet error is not just an inconvenience—it can change clinical practice.
The Overdiagnosis Problem
3-4
Lives saved
per 10,000 screened
per 10,000 screened
50-130
Overdiagnosed
(treated unnecessarily)
(treated unnecessarily)
~500
False alarms
(anxiety, biopsies)
(anxiety, biopsies)
THE QUESTION
To save 3-4 lives, an estimated 50-130 women receive surgery, radiation, or chemotherapy
for cancers that would never have harmed them.
Is this trade-off worth it?
Is this trade-off worth it?
Patient Decision Aid: Mammography
If 10,000 Women Age 50-69 Are Screened for 10 Years
Deaths from breast cancer prevented
3-4 women
Women called back for false alarms
~500 women
Unnecessary biopsies
~200 women
Women treated for cancer that would never harm them
~15 women
Is screening right for you?
The Screening Cascade Decision Tree
10,000 Women Screened Over 10 Years
10,000 Women
↓
~1,000 RecalledAbnormal
↓
~500 False
Alarm
Alarm
~500 Biopsy
~50 cancer
~50 cancer
~9,000 Cleared
Of ~50 Cancers Found
~35 Would Kill3-4 saved
~15 Would Never KillOverdiagnosed
"And the test found the shadow,
and called it cancer,
and the woman was cut and burned—
for a shadow that would never have darkened her days."
and called it cancer,
and the woman was cut and burned—
for a shadow that would never have darkened her days."
Have you not heard of the scan
that finds the plaques in the brain,
but cannot tell you
if the mind will fade?
that finds the plaques in the brain,
but cannot tell you
if the mind will fade?
The Amyloid Paradox
ALZHEIMER'S RESEARCH, 2010s-2020s
PET scans can now detect amyloid plaques—the hallmark of Alzheimer's.
But 30% of cognitively normal elderly have amyloid plaques. They may never develop dementia.
And 10-20% of people with dementia have no amyloid.
The test finds the plaques. But plaques are not the disease. We are testing for a surrogate, not the outcome.
But 30% of cognitively normal elderly have amyloid plaques. They may never develop dementia.
And 10-20% of people with dementia have no amyloid.
The test finds the plaques. But plaques are not the disease. We are testing for a surrogate, not the outcome.
Jack CR et al. Lancet Neurol. 2018;17:760-773
Surrogate vs. Outcome Decision Tree
What Are We Really Testing For?
Diagnostic Test
↓
What Does It Detect?
Outcome itself
Direct Diagnosise.g., Biopsy for cancer
↓
High clinical value
Surrogate marker
Indirect Signale.g., Amyloid for dementia
↓
Validated link?
YesUse cautiously
NoLimited value
"And the scan found the plaques,
and the doctor named it Alzheimer's,
and the patient lived in terror—
of a forgetting that might never come."
and the doctor named it Alzheimer's,
and the patient lived in terror—
of a forgetting that might never come."
Not all studies are created equal.
Some are biased.
Some are poorly designed.
Some should not be trusted.
How do we separate the wheat from the chaff?
Some are biased.
Some are poorly designed.
Some should not be trusted.
How do we separate the wheat from the chaff?
What if most DTA studies do not even report enough information to judge their quality?
REAL DATA
Before the STARD initiative was published in 2003, a systematic assessment found that fewer than half of DTA studies reported whether index test interpretation was blinded, and reference standard descriptions were frequently inadequate. After STARD, reporting improved: multiple meta-epidemiological assessments found adherence to STARD items rose substantially, though many studies still fell short on key items such as flow diagrams and indeterminate result handling.
The STARD Revolution: 2003
A team completes a DTA study of a new point-of-care test. They are eager to publish quickly. They have the 2x2 data but have not documented blinding, patient flow, or indeterminate results.
PATH A: Publish Quickly
Submit without a STARD flow diagram or complete reporting of methods
↓
Readers cannot assess blinding, patient spectrum, or verification. QUADAS-2 assessment rates every domain as "unclear." The study may be excluded from future systematic reviews or, worse, included with inflated weight.
OUTCOME: Waste of research; uninterpretable results
PATH B: Follow STARD Guidelines
Complete the STARD checklist, create a patient flow diagram, report indeterminate results, and describe blinding
↓
Reviewers can fully assess quality. QUADAS-2 domains are answerable. The study contributes meaningfully to systematic reviews and clinical guidelines.
OUTCOME: Trustworthy evidence that advances care
THE REVELATION
You cannot assess quality if the study does not report its methods. STARD ensures that DTA studies are complete enough to be judged by QUADAS-2. Incomplete reporting is not neutral—it hides bias.
QUADAS-2: The Quality Checklist
Four Domains of Risk of Bias
1
Patient Selection
Was a consecutive or random sample enrolled? Was a case-control design avoided?
2
Index Test
Was the test interpreted without knowledge of the reference standard? Was the threshold pre-specified?
3
Reference Standard
Is the reference standard likely to correctly classify the condition? Was it interpreted blind?
4
Flow and Timing
Was there appropriate interval between tests? Did all patients receive the same reference standard?
QUADAS-2 Decision Tree
Should You Trust This Study?
DTA Study
↓
Check All 4 Domains
All Low Risk
High QualityTrust results
Some Unclear
ModerateUse with caution
Any High Risk
Low QualityResults may be biased
Common Biases in DTA Studies
!
Verification Bias
Only positive tests get the reference standard → inflates sensitivity
!
Spectrum Bias
Study population differs from clinical reality → results don't generalize
!
Incorporation Bias
Index test is part of reference standard → artificially high accuracy
!
Review Bias
Index test interpreted knowing reference result → inflates both metrics
"Before you trust the numbers,
ask: How were they gathered?
A biased study speaks with confidence—
but its confidence is a lie."
ask: How were they gathered?
A biased study speaks with confidence—
but its confidence is a lie."
One study may deceive.
One study may flatter.
But when you gather all the evidence—
the truth becomes harder to hide.
One study may flatter.
But when you gather all the evidence—
the truth becomes harder to hide.
What happens when different studies use different thresholds for the same test, and you try to pool them?
REAL DATA
D-dimer testing for pulmonary embolism (PE) traditionally used a fixed cutoff of 500 µg/L. The ADJUST-PE trial (Righini et al., JAMA 2014) showed that an age-adjusted cutoff (age × 10 µg/L for patients over 50) increased the proportion of elderly patients with negative D-dimer results from ~6% to ~30%, with a 3-month VTE risk of only 0.3% in the age-adjusted negative group. A DTA meta-analysis of D-dimer studies must use the bivariate model because different thresholds create a sensitivity-specificity tradeoff visible on the SROC curve.
The D-dimer Threshold Dilemma: ADJUST-PE 2014
An elderly patient (age 75) presents to the ED with possible PE. D-dimer is 620 µg/L. Using the fixed cutoff, this is positive. Using the age-adjusted cutoff (750 µg/L), this is negative.
PATH A: Use Fixed Cutoff (500 µg/L)
Apply one threshold to all patients regardless of age
↓
Elderly patients almost always exceed 500 µg/L. Specificity drops below 10% in those over 80. Nearly every elderly patient gets a CT pulmonary angiogram—with contrast dye, radiation, and incidental findings.
OUTCOME: D-dimer becomes useless in the elderly
PATH B: Use Bivariate Model with Threshold Covariate
Apply the age-adjusted cutoff; model threshold variation in the meta-analysis
↓
The SROC curve shows that age-adjusted thresholds move along the curve, trading a small amount of sensitivity for a large gain in specificity. 30% more elderly patients safely avoid CT imaging.
OUTCOME: Fewer unnecessary scans; no missed PEs
THE REVELATION
Threshold variation is the reason DTA meta-analysis needs the bivariate model. Different studies use different cutoffs, creating a tradeoff between sensitivity and specificity. The SROC curve is the map of that tradeoff.
Why DTA Meta-Analysis Is Different
THE PROBLEM
Sensitivity and specificity are correlated.
When one goes up, the other tends to go down.
You cannot pool them separately like treatment effects. You need the bivariate model.
You cannot pool them separately like treatment effects. You need the bivariate model.
The SROC Curve
Summary Receiver Operating Characteristic
Sensitivity
1 - Specificity (False Positive Rate)
Individual studies
Summary estimate
Reading the SROC
What Does the Curve Tell You?
SROC Curve Position
↓
Top-Left Corner
Excellent TestHigh sens + spec
Near Diagonal
Useless TestNo better than chance
Points Scattered
High HeterogeneityInvestigate sources
"One study may deceive.
Many studies, weighed together,
trace the path of truth—
the SROC curve that reveals what the test can truly do."
Many studies, weighed together,
trace the path of truth—
the SROC curve that reveals what the test can truly do."
But what if the studies disagree?
One says sensitivity is 95%.
Another says 60%.
Which truth do you believe?
One says sensitivity is 95%.
Another says 60%.
Which truth do you believe?
What if a test works well in the general population but fails in the patients who need it most?
REAL DATA
HRP2-based malaria rapid diagnostic tests (RDTs) achieve sensitivity of approximately 95% in the general population in endemic areas. However, in pregnant women, sensitivity can drop to as low as 56-76% due to placental sequestration of parasites—the parasites hide in the placenta, keeping peripheral blood parasitemia low and below the RDT detection threshold. A Cochrane review of malaria RDTs found substantial heterogeneity (I² often exceeding 80%) driven by population subgroups including pregnancy, children under 5, and HIV co-infection.
The Malaria RDT in Pregnancy
A meta-analysis pools 25 malaria RDT studies and reports pooled sensitivity of 93%. A clinician in an antenatal clinic uses this to reassure a pregnant woman with a negative RDT.
PATH A: Trust the Overall Pooled Estimate
Apply the 93% sensitivity from the general-population meta-analysis
↓
In pregnant women, the true sensitivity may be as low as 56-76%. A substantial proportion of infected pregnant women are falsely reassured. Untreated malaria in pregnancy causes severe maternal anemia, low birth weight, and stillbirth.
OUTCOME: Preventable maternal and neonatal deaths
PATH B: Investigate Heterogeneity by Subgroup
Conduct subgroup meta-analysis for pregnant women; explore I² and sources of variation
↓
Discover that pregnancy is a major source of heterogeneity. Recommend microscopy confirmation for all pregnant women with negative RDTs in endemic areas.
OUTCOME: Targeted protocols save mothers and babies
THE REVELATION
Heterogeneity is not just statistical noise. It often signals that the test performs differently in different populations. Ignoring I² and pooling everything together can be fatal for vulnerable subgroups.
Sources of Heterogeneity
Why Studies Disagree
Same Test, Different Results?
ThresholdDifferent cutoffs
PopulationSeverity, age
SettingPrimary vs specialist
QualityBias, blinding
Measuring Disagreement: I²
I² < 25%
Low
Studies agree
Studies agree
I² 25-75%
Moderate
Some variation
Some variation
I² > 75%
High
Major disagreement
Major disagreement
THE WARNING
When I² > 75%, the pooled estimate may be meaningless.
Explain the disagreement before averaging.
"When the studies disagree,
do not silence the dissent.
Ask: Why do they see differently?
The disagreement itself teaches."
do not silence the dissent.
Ask: Why do they see differently?
The disagreement itself teaches."
Your DTA Toolkit
The essential measures and when to use them
When an AI claims to diagnose better than doctors, should you trust the overall AUC?
REAL DATA
Deep learning models for skin cancer detection have reported AUC values as high as 0.91-0.94 in development datasets. However, external validation revealed alarming disparities: Daneshjou et al. (2022, Nature Medicine) found that commercial AI dermatology tools performed at near-chance levels on darker skin (Fitzpatrick types V-VI), with AUC as low as 0.50-0.57 — essentially random. Training datasets were heavily biased toward lighter skin tones, meaning the 2x2 table was never properly filled for all populations.
The AI Dermatology Promise: 2020s
A hospital considers deploying an AI skin cancer screening tool in a dermatology clinic serving a diverse urban population. The manufacturer reports AUC of 0.94.
PATH A: Deploy Based on Overall AUC
Trust the headline AUC of 0.94 and deploy to all patients
↓
Melanomas on darker skin are missed at higher rates. The overall sensitivity figure conceals a dangerous gap. Patients with the highest mortality from late diagnosis are the ones the AI fails most.
OUTCOME: Health disparity amplified by technology
PATH B: Demand Fairness-Stratified Evaluation
Require sensitivity and specificity broken down by skin tone (Fitzpatrick scale), age, and lesion location
↓
Discover the performance gap. Require retraining on diverse datasets or restrict use to validated populations. Pair AI with dermatologist oversight for underrepresented groups.
OUTCOME: Equitable deployment; no one left behind
THE REVELATION
A single AUC number can hide dangerous disparities. Emerging AI-based diagnostic tools must be evaluated with the same rigor as any diagnostic test: stratified by population, validated externally, and held to STARD and QUADAS-2 standards.
The Checklist
✓
Was there a valid reference standard?
Gold standard applied to ALL patients?
✓
Were interpreters blinded?
Test readers unaware of diagnosis?
✓
Was the spectrum appropriate?
Patients similar to your population?
✓
Was the threshold pre-specified?
Or chosen to maximize results?
When Results Don't Match Suspicion
The Clinical Override Decision Tree
Test Negative, High Suspicion
↓
What Is the LR-?
LR- < 0.1
Strong rule-outAccept negative
LR- 0.1-0.5
Repeat testOr different test
LR- > 0.5
Trust judgmentTest is weak
Sequential Testing Decision Tree
When One Test Isn't Enough
Initial Screening Test
↓
Positive
↓
Confirmatory TestHigh specificity
↓
PositiveDiagnose
NegativeFalse alarm
Negative
↓
Likely negativeIf high sens screen
"Armed with sensitivity, specificity, likelihood,
armed with the SROC and the measure of agreement,
you can see through the lie of the test—
and judge its truth for yourself."
armed with the SROC and the measure of agreement,
you can see through the lie of the test—
and judge its truth for yourself."
Have you not heard of the patient
who received the wrong blood,
not because the test was wrong,
but because no one performed it?
who received the wrong blood,
not because the test was wrong,
but because no one performed it?
The Test That Wasn't Done
HOSPITALS WORLDWIDE
ABO blood typing is nearly 100% accurate when performed.
Yet transfusion reactions still kill—not from test failure, but from human failure:
• Wrong blood drawn from wrong patient
• Labels switched in the lab
• Bedside check skipped in emergency
In the UK, 1 in 13,000 transfusions goes to the wrong patient. The test worked. The system failed.
Yet transfusion reactions still kill—not from test failure, but from human failure:
• Wrong blood drawn from wrong patient
• Labels switched in the lab
• Bedside check skipped in emergency
In the UK, 1 in 13,000 transfusions goes to the wrong patient. The test worked. The system failed.
Bolton-Maggs PHB. Transfus Med. 2016;26:303-311
Test vs. System Decision Tree
Where Can Things Go Wrong?
Diagnostic Process
↓
Error Source?
Test itself
Analytical ErrorSens/Spec issue
↓
Better test needed
Pre-analytical
Wrong sampleID error
↓
System fix needed
Post-analytical
Wrong actionReporting error
↓
Process fix needed
"The perfect test means nothing
if the wrong blood is drawn,
the wrong label is applied,
the wrong bag is hung."
if the wrong blood is drawn,
the wrong label is applied,
the wrong bag is hung."
DTA studies measure test accuracy. They do not measure system accuracy.
References
Key Sources
- Carreyrou J. Bad Blood. Knopf, 2018. [Theranos]
- CDC. MMWR. 1987;36(49):833-840. [HIV blood supply]
- Herbst AL et al. N Engl J Med. 1971;284:878-881. [DES]
- Moyer VA. Ann Intern Med. 2012;157:120-134. [PSA]
- Pope JH et al. N Engl J Med. 2000;342:1163-1170. [Troponin]
- Steingart KR et al. Cochrane 2014;1:CD009593. [GeneXpert]
- Dinnes J et al. Cochrane 2022;7:CD013705. [COVID RAT]
- UK Panel. Lancet. 2012;380:1778-1786. [Mammography]
- Jack CR et al. Lancet Neurol. 2018;17:760-773. [Amyloid]
- WHO. Malaria RDT Performance. 2022.
- Reitsma JB et al. J Clin Epidemiol. 2005;58:982-990. [Bivariate]
- Whiting PF et al. Ann Intern Med. 2011;155:529-536. [QUADAS-2]
- Bolton-Maggs PHB. Transfus Med. 2016;26:303-311.
A test is 99% sensitive and 99% specific. Disease prevalence is 1/1000. A patient tests positive. What is the probability they have the disease?
99%
90%
About 9%
50%
What does "SnNout" mean?
A highly Sensitive test, when Negative, rules OUT disease
A highly Specific test, when Negative, rules OUT disease
Sensitivity should be used for screening
Specificity should be above 90%
Why did the blood supply become contaminated with HIV despite testing?
The tests had low specificity
Tests had a window period with zero sensitivity in early infection
The tests were not performed correctly
The tests were too expensive
Which QUADAS-2 domain assesses whether the test was interpreted without knowing the diagnosis?
Patient Selection
Index Test
Reference Standard
Flow and Timing
✔
Course Complete
"Now you know the four outcomes,
the two virtues of a test,
the fallacy of the base rate,
the art of pooling evidence,
and the biases that hide the truth.
When the next test lies to you—
you will know."
the two virtues of a test,
the fallacy of the base rate,
the art of pooling evidence,
and the biases that hide the truth.
When the next test lies to you—
you will know."
1 / 4