Evaluation metrics · part 12Splitting data without leaking
Every number in parts 1 to 11 assumes the test emails were never seen while the model was built.
A metric only predicts how a model will do on new data if it was measured on data the model never learned from, directly or indirectly. That depends entirely on how the data is split. The machine below sorts 20 blocks onto three stations: a training bench, a validation pan and a sealed test vault. Each block is one sender's 5 emails, 100 in all. Pick a scheme and watch which blocks go where.
Drag to turn. Pick a scheme.
Three sets, three jobs
Pick Hold-out. The data is split once, and each part has one job:
- Training set: the model learns from these emails.
- Validation set: you make choices here. That covers which model, which settings (the hyperparameters, such as a tree's depth or a learning rate), and which threshold (part 7).
- Test set: you measure the final, fixed model here, once. It stays sealed until every choice is made.
60 / 20 / 20 or 70 / 15 / 15 are common. Split stratified so every part keeps the same share of spam. The weakness is plain in the machine: one split, so every result depends on which few senders happened to land on the pan and in the vault. With small data, that is mostly luck.
k-fold cross-validation
Pick k-fold. The test set stays sealed in the vault. The other 16 blocks are cut into k = 4 folds, and the machine runs 4 times. Each time, one fold sits on the validation pan and the other three are used for training. Every block is validated exactly once.
CV score = average of the k fold scores report it ± their spread
The cross-validation average is for making choices. Once they are made, the final model is retrained on all 16 senders and the vault opens once, exactly as in hold-out. The test score is the one you report. This uses all the non-test data for both training and validation, and the spread across folds shows how stable the score is. 5 or 10 folds are usual. With classes of very different sizes, use stratified k-fold so each fold keeps the class mix.
Leave one group out
Samples often come in groups: several emails from one sender, several scans from one patient, many frames from one video. Emails from the same sender look alike, so if one sits in training and its twin in validation, the model is tested on something it has nearly seen.
Pick Leave one group out. Each run validates on one whole sender and trains on all the others, so the score answers "how does the model do on a sender it has never seen?" In medicine this is leave-one-patient-out (LOPO). With many groups, group k-fold does the same with fewer runs: whole groups are dealt into k folds, and no group is ever split across training and validation.
Nested cross-validation
Suppose you try several settings, run cross-validation for each, keep the best, and report its score. That score is too optimistic. Each score is a little noisy, and picking the highest of several noisy numbers picks up lucky noise. The same data both chose the setting and graded it.
Nested cross-validation gives the two jobs separate data with two loops. Pick Nested:
- Outer loop: 4 blocks go into the vault as the outer test fold.
- Inner loop: on the remaining 16 blocks, a full 4-fold cross-validation tries every setting and picks the best. The vault stays shut.
- The chosen setting is retrained on all 16, and the vault opens once to score it.
- Repeat for all 5 outer folds and average the 5 vault scores.
The average estimates how the whole procedure, tuning included, will do on new senders. It does not hand you one final model: once you trust the procedure, run the inner search once on all the data to get it.
Leaks
A leak is any route by which information about the test data reaches the model or the choices made about it. Leaks rarely cause errors. They make results look better, which is why they survive. The figure shows the most common one. Each sender's 5 emails are near-copies, plotted here on two features. Pick a split and see which validation emails still have a twin on the training side.
Split emails
Show validation fold
- spam
- not spam
- in the validation fold
- same sender, in training
- Random 5-fold: validation emails with a twin in training
- By sender: validation emails with a twin in training
With a random split, most validation emails have near-copies from the same sender in training, so the validation score partly measures how well the model recognises emails it has nearly seen. Split by sender and the twins are gone: the score then measures what you care about, emails from senders the model has never met. How much the score falls depends on the model. Models that memorise their training data fall the most. The common leaks:
- Near-duplicates across splits: the same sender, patient, product or forwarded message on both sides. Split by group.
- Tuning on the test set: choosing the threshold, model or features by test score. Every look leaks a little. Use validation data, or nested CV.
- Preprocessing before splitting: scaling, feature selection, oversampling or imputing on all the data at once. Fit every step inside each training fold, which a pipeline does for you.
- Time: training on next month to predict last month. For data that drifts, split by date: train on the past, test on the future.
- Target leakage: a feature that is only known after the label, such as "user moved to spam folder". It scores well in testing and is missing in production.
In code
splits.py
import numpy as np
from sklearn.model_selection import (train_test_split, StratifiedKFold,
GroupKFold, LeaveOneGroupOut)
idx = np.arange(len(y))
# hold-out 60 / 20 / 20, keeping the spam share in every part
rest, test = train_test_split(idx, test_size=0.2, stratify=y, random_state=0)
train, val = train_test_split(rest, test_size=0.25, stratify=y[rest], random_state=0)
# k-fold: every sample is validated exactly once
for train, val in StratifiedKFold(5, shuffle=True, random_state=0).split(X, y):
...
# by sender: no sender ever appears on both sides
for train, val in GroupKFold(5).split(X, y, groups=sender):
assert set(sender[train]).isdisjoint(sender[val])
# leave one sender out
for train, val in LeaveOneGroupOut().split(X, y, groups=sender):
...
nested.py
for outer_train, test in GroupKFold(5).split(X, y, groups=sender): # outer loop
Xo, yo, go = X[outer_train], y[outer_train], sender[outer_train]
for train, val in GroupKFold(4).split(Xo, yo, groups=go): # inner loop
... # try each setting: fit on train, score on val
# keep the best setting, refit on all of outer_train, score once on test
# report the average of the outer test scores
Next: Part 13: regression metrics, for models that predict a number rather than a label.