Sign up
ContentsHideShow

Quant glossary · Research & validation

Multiple testing

Multiple testing occurs when many hypotheses or candidate specifications are evaluated and attention is focused on selected results.

Definition

Multiple testing occurs when many hypotheses or candidate specifications are evaluated and attention is focused on selected results. Even if every null hypothesis is true, the chance of at least one small p-value rises with the number and dependence of tests. The tested family and correction target must be defined before interpreting the result.

How it is calculated

P(at least one false rejection)=1−(1−α)mP(\text{at least one false rejection})=1-(1-\alpha)^m

For m independent tests, if each true null has rejection probability exactly alpha, the family-wise false-rejection probability is 1 minus (1 minus alpha) to the power m; otherwise the expression is an upper bound.

Symbol Meaning Unit
P probability probability
alpha per-test level probability
m number of tests count
  1. Find the probability that each independent true-null test avoids false rejection.

  2. Raise that probability to the number of tests for the probability that all avoid false rejection.

  3. Take the complement to obtain the family-wise false-rejection probability.

Interpretation

If you keep opening doors, one may look unusually bright by chance. Searching over signals, universes and horizons creates a testing family even when no single test was labeled a hypothesis test. Dependence means the simple formula may not apply, but the selection issue remains.

Assumptions and conventions

  • The displayed expression assumes independent tests and true nulls.

  • The family of tested hypotheses is identified, including tried variants where relevant.

Probabilities are dimensionless; m is a count.

Limitations and common misreadings

  • The number of tests is not always the number of independent tests.

  • A correction must match the inferential target; no single adjustment answers every question.

References

  1. Yoav Benjamini, Yosef Hochberg. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B: Statistical Methodology. Sections 1–2. doi:10.1111/j.2517-6161.1995.tb02031.x. Source

Multiple testing FAQ

What is Multiple testing?

Multiple testing occurs when many hypotheses or candidate specifications are evaluated and attention is focused on selected results.

How is Multiple testing calculated?

For m independent tests, if each true null has rejection probability exactly alpha, the family-wise false-rejection probability is 1 minus (1 minus alpha) to the power m; otherwise the expression is an upper bound.

What does Multiple testing not tell you or assume?

The number of tests is not always the number of independent tests.

Browse all glossary termsResearch and validation termsQuant research library

Definitions are educational and descriptive, not investment advice. Worked examples use labelled sample data.