Splitting Without Leaks

Evaluation metrics · part 12Splitting data without leaking

Every number in parts 1 to 11 assumes the test emails were never seen while the model was built.

A metric only predicts how a model will do on new data if it was measured on data the model never learned from, directly or indirectly. That depends entirely on how the data is split. The machine below sorts 20 blocks onto three stations: a training bench, a validation pan and a sealed test vault. Each block is one sender's 5 emails, 100 in all. Pick a scheme and watch which blocks go where.

Drag to turn. Pick a scheme.

01

Three sets, three jobs

Pick Hold-out. The data is split once, and each part has one job:

60 / 20 / 20 or 70 / 15 / 15 are common. Split stratified so every part keeps the same share of spam. The weakness is plain in the machine: one split, so every result depends on which few senders happened to land on the pan and in the vault. With small data, that is mostly luck.

02

k-fold cross-validation

Pick k-fold. The test set stays sealed in the vault. The other 16 blocks are cut into k = 4 folds, and the machine runs 4 times. Each time, one fold sits on the validation pan and the other three are used for training. Every block is validated exactly once.

CV score = average of the k fold scores    report it ± their spread

The cross-validation average is for making choices. Once they are made, the final model is retrained on all 16 senders and the vault opens once, exactly as in hold-out. The test score is the one you report. This uses all the non-test data for both training and validation, and the spread across folds shows how stable the score is. 5 or 10 folds are usual. With classes of very different sizes, use stratified k-fold so each fold keeps the class mix.

03

Samples often come in groups: several emails from one sender, several scans from one patient, many frames from one video. Emails from the same sender look alike, so if one sits in training and its twin in validation, the model is tested on something it has nearly seen.

Pick Leave one group out. Each run validates on one whole sender and trains on all the others, so the score answers "how does the model do on a sender it has never seen?" In medicine this is leave-one-patient-out (LOPO). With many groups, group k-fold does the same with fewer runs: whole groups are dealt into k folds, and no group is ever split across training and validation.

04

Nested cross-validation

Suppose you try several settings, run cross-validation for each, keep the best, and report its score. That score is too optimistic. Each score is a little noisy, and picking the highest of several noisy numbers picks up lucky noise. The same data both chose the setting and graded it.

Nested cross-validation gives the two jobs separate data with two loops. Pick Nested:

  1. Outer loop: 4 blocks go into the vault as the outer test fold.
  2. Inner loop: on the remaining 16 blocks, a full 4-fold cross-validation tries every setting and picks the best. The vault stays shut.
  3. The chosen setting is retrained on all 16, and the vault opens once to score it.
  4. Repeat for all 5 outer folds and average the 5 vault scores.

The average estimates how the whole procedure, tuning included, will do on new senders. It does not hand you one final model: once you trust the procedure, run the inner search once on all the data to get it.

05

Leaks

A leak is any route by which information about the test data reaches the model or the choices made about it. Leaks rarely cause errors. They make results look better, which is why they survive. The figure shows the most common one. Each sender's 5 emails are near-copies, plotted here on two features. Pick a split and see which validation emails still have a twin on the training side.

Split emails

Show validation fold

Validation emails and their twins in training
  • spam
  • not spam
  • in the validation fold
  • same sender, in training
Random 5-fold: validation emails with a twin in training
By sender: validation emails with a twin in training

With a random split, most validation emails have near-copies from the same sender in training, so the validation score partly measures how well the model recognises emails it has nearly seen. Split by sender and the twins are gone: the score then measures what you care about, emails from senders the model has never met. How much the score falls depends on the model. Models that memorise their training data fall the most. The common leaks:

06

In code

splits.py

import numpy as np
from sklearn.model_selection import (train_test_split, StratifiedKFold,
                                     GroupKFold, LeaveOneGroupOut)

idx = np.arange(len(y))

# hold-out 60 / 20 / 20, keeping the spam share in every part
rest, test = train_test_split(idx, test_size=0.2, stratify=y, random_state=0)
train, val = train_test_split(rest, test_size=0.25, stratify=y[rest], random_state=0)

# k-fold: every sample is validated exactly once
for train, val in StratifiedKFold(5, shuffle=True, random_state=0).split(X, y):
    ...

# by sender: no sender ever appears on both sides
for train, val in GroupKFold(5).split(X, y, groups=sender):
    assert set(sender[train]).isdisjoint(sender[val])

# leave one sender out
for train, val in LeaveOneGroupOut().split(X, y, groups=sender):
    ...

nested.py

for outer_train, test in GroupKFold(5).split(X, y, groups=sender):    # outer loop
    Xo, yo, go = X[outer_train], y[outer_train], sender[outer_train]
    for train, val in GroupKFold(4).split(Xo, yo, groups=go):         # inner loop
        ...   # try each setting: fit on train, score on val
    # keep the best setting, refit on all of outer_train, score once on test
# report the average of the outer test scores

Next: Part 13: regression metrics, for models that predict a number rather than a label.