Definition
Out-of-sample validation evaluates a model or rule on observations that were not used to select its specification or parameters. The split, time ordering, retraining policy and evaluation metric define what generalization question is being asked. Reusing the validation set for repeated choices turns it into another development sample.
How it is calculated
Test risk is the average loss on test indices of predictions from a model fit using training data only.
| Symbol | Meaning | Unit |
|---|---|---|
| R | estimated test risk | loss units |
| n | test sample size | count |
| ell | loss function | loss units |
| f | fitted prediction rule | prediction mapping |
| X | features | feature units |
| Y | outcome | outcome units |
| I_test | test-index set | unitless |
Interpretation
Validation is a rehearsal on data the model has not seen while being chosen. It gives evidence about a defined transfer from training conditions to test conditions. Changing the split or repeatedly tuning on the holdout changes the evidence.
Assumptions and conventions
-
Test observations are not used, directly or indirectly, for model selection.
-
The split respects temporal, group or cross-sectional dependence relevant to deployment.
The validation metric retains its loss or return units; n is a count.
Limitations and common misreadings
-
Out-of-sample does not mean future-proof.
-
A random split can leak future information in time-series data.
References
- Trevor Hastie, Robert Tibshirani, Jerome Friedman. (2009). The Elements of Statistical Learning. Springer Series in Statistics. Chapter 7, Model Assessment and Selection. doi:10.1007/978-0-387-84858-7. Source

