Inference

Significance & truth

12 min

Learning goals

•You understand why a significant result is not the same as a true result.
•You can derive PPV and FDR from prior probability, α, and power.
•You can explain how power and p-hacking influence the credibility of published findings.

Positive predictive value (PPV)

85.0 %

Share of real effects among all significant findings

False discovery rate (FDR)

15.0 %

Share of false positives among all significant findings

True positives

410

significant & H₁ true

False positives

significant, but H₀ true

False negatives

n.s., but H₁ true

True negatives

427

n.s. & H₀ true

All 1000 studies

Each dot is one study. The colour shows its true category.

True positive — significant, H₁ trueFalse positive — significant, H₀ trueFalse negative — n.s., H₁ trueTrue negative — n.s., H₀ true

Only significant findings (483)

These are the studies that typically get published in the literature.

Of these,410 are truly positive73 are falsely positive→ PPV = 85.0 %

Under your assumptions, 85.0 % of the significant findings are actually real (PPV) – the remaining 15.0 % are false positives (FDR).

Formula (Ioannidis 2005): PPV = (p · power + u · p · (1 − power)) / (p · power + u · p · (1 − power) + (1 − p) · α + u · (1 − p) · (1 − α)). p = P(H₁), u = bias / p-hacking share.