Training, validation and test data: what each set tells you

Foundation lesson · Study time: about 20 minutes. Prerequisite: averages and the idea of a predictive model. Aim: assign each dataset a distinct job and identify one source of leakage.

Give each partition one job

Training data is used to fit model parameters. Validation data helps compare development choices, such as model complexity or a decision threshold. A final test set estimates performance after those choices are fixed. Repeatedly selecting models using the test score makes that set part of development.

An original engineering example

Imagine 12 laboratory pumps, each contributing 80 labelled vibration recordings: 960 recordings in total. The intended use is prediction on pumps absent from development. One illustrative plan assigns 8 whole pumps to training, 2 to validation, and 2 to testing. That gives 640, 160, and 160 recordings. No pump appears in more than one partition.

This allocation is a teaching example, not a universally suitable split. With only two test pumps, an apparently strong score may be sensitive to which pumps were selected. Record the pump identifiers and operating conditions, compare against a simple baseline, and report this limitation.

Keep information in its proper place

A random split of individual recordings can place measurements from the same pump in several partitions. For the stated new-pump goal, that can give an overly optimistic result. If the intended use is future measurements on existing pumps, a chronological design may be more relevant; choose the split to match deployment.

Fit a scaler, imputer, or feature selector using training data only, then apply the fitted transformation to validation data. In cross-validation, refit these operations within each training fold. Freeze the procedure before the final test. Do not use future measurements to create features for earlier predictions.

Check your understanding

A colleague chooses a threshold by trying 20 values on the final test set. Is that still a held-out final evaluation? No. The threshold now depends on the test outcomes. Use development validation for that choice and reserve an untouched evaluation set. What would you change if the project instead predicted the next month on the same pumps? Write down a time-based split and explain it.

Editorial note

Written independently for IIMEEE with AI assistance. Technical statements and the worked example were checked against the references listed below. No external prose, code, diagrams, or tables have been reproduced. These introductory notes use simplified assumptions; they are not instructions for operating machinery or live electrical equipment.