Sign up
ContentsHideShow

Quant glossary · Research & validation

Out-of-sample validation

Out-of-sample validation evaluates a model or rule on observations that were not used to select its specification or parameters.

Definition

Out-of-sample validation evaluates a model or rule on observations that were not used to select its specification or parameters. The split, time ordering, retraining policy and evaluation metric define what generalization question is being asked. Reusing the validation set for repeated choices turns it into another development sample.

How it is calculated

R^test=1ntest∑i∈Itestℓ(f^train(Xi),Yi)\widehat{R}_{test}=\frac{1}{n_{test}}\sum_{i\in\mathcal{I}_{test}}\ell(\hat f_{train}(X_i),Y_i)

Test risk is the average loss on test indices of predictions from a model fit using training data only.

Symbol Meaning Unit
R estimated test risk loss units
n test sample size count
ell loss function loss units
f fitted prediction rule prediction mapping
X features feature units
Y outcome outcome units
I_test test-index set unitless

Interpretation

Validation is a rehearsal on data the model has not seen while being chosen. It gives evidence about a defined transfer from training conditions to test conditions. Changing the split or repeatedly tuning on the holdout changes the evidence.

Assumptions and conventions

  • Test observations are not used, directly or indirectly, for model selection.

  • The split respects temporal, group or cross-sectional dependence relevant to deployment.

The validation metric retains its loss or return units; n is a count.

Limitations and common misreadings

  • Out-of-sample does not mean future-proof.

  • A random split can leak future information in time-series data.

References

  1. Trevor Hastie, Robert Tibshirani, Jerome Friedman. (2009). The Elements of Statistical Learning. Springer Series in Statistics. Chapter 7, Model Assessment and Selection. doi:10.1007/978-0-387-84858-7. Source

Out-of-sample validation FAQ

What is Out-of-sample validation?

Out-of-sample validation evaluates a model or rule on observations that were not used to select its specification or parameters.

How is Out-of-sample validation calculated?

Test risk is the average loss on test indices of predictions from a model fit using training data only.

What does Out-of-sample validation not tell you or assume?

Out-of-sample does not mean future-proof.

Browse all glossary termsResearch and validation termsQuant research library

Definitions are educational and descriptive, not investment advice. Worked examples use labelled sample data.