Definition
Multiple testing occurs when many hypotheses or candidate specifications are evaluated and attention is focused on selected results. Even if every null hypothesis is true, the chance of at least one small p-value rises with the number and dependence of tests. The tested family and correction target must be defined before interpreting the result.
How it is calculated
For m independent tests, if each true null has rejection probability exactly alpha, the family-wise false-rejection probability is 1 minus (1 minus alpha) to the power m; otherwise the expression is an upper bound.
| Symbol | Meaning | Unit |
|---|---|---|
| P | probability | probability |
| alpha | per-test level | probability |
| m | number of tests | count |
-
Find the probability that each independent true-null test avoids false rejection.
-
Raise that probability to the number of tests for the probability that all avoid false rejection.
-
Take the complement to obtain the family-wise false-rejection probability.
Interpretation
If you keep opening doors, one may look unusually bright by chance. Searching over signals, universes and horizons creates a testing family even when no single test was labeled a hypothesis test. Dependence means the simple formula may not apply, but the selection issue remains.
Assumptions and conventions
-
The displayed expression assumes independent tests and true nulls.
-
The family of tested hypotheses is identified, including tried variants where relevant.
Probabilities are dimensionless; m is a count.
Limitations and common misreadings
-
The number of tests is not always the number of independent tests.
-
A correction must match the inferential target; no single adjustment answers every question.
References
- Yoav Benjamini, Yosef Hochberg. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B: Statistical Methodology. Sections 1–2. doi:10.1111/j.2517-6161.1995.tb02031.x. Source

