← Back to the research

Contamination taxonomy · v1.1

Five kinds of contamination. Only one is a leak.

Organised by the mitigation each type defeats rather than by how the leak occurred — which is why a held-out set only fixes the first. Types 1–4 happen at training time and are properties of the benchmark. Type 5 happens during the evaluation run, so it is a property of that run and not of the benchmark at all.

TRAINING TIME · PASSIVE · THE DATA REACHES THE MODEL THE WEB everything scraped PRETRAINING CORPUS what the model actually read MODEL BENCHMARK SOURCE DATA what the benchmark was built from tested on built from 1 2 3 cutoff AFTER the phenomenon 4 pattern, not items EVALUATION TIME · ACTIVE · THE MODEL REACHES THE DATA SANDBOX isolation, no egress MODEL UNDER TEST THE ANSWER KEY git history · web · database 5 breaks out, goes and gets it

  1. 1 Direct

    The items are in the corpus. Someone scraped the repo.

    Missed by: nothing. The only one a held-out set fixes.

  2. 2 Derivative

    The benchmark never leaked — the material it was built from did.

    Missed by: held-out private sets.

  3. 3 Temporal

    The cutoff sits after the phenomenon. You believe you are measuring prediction. You are measuring recall.

    Missed by: overlap checks.

  4. 4 Distributional

    The items are novel; the pattern is not — so heavily represented that the model never needs the reasoning you think you are testing.

    Missed by: every overlap-based method.

  5. 5 Acquired

    The model gets the answer key during the evaluation itself. 5a environment · 5b retrieval · 5c breaks isolation

    Missed by: everything above. At 5c, by isolation itself. A property of the run, not of the benchmark.

Full taxonomy ↗ Schema, validator & audit log ↗ 10.5281/zenodo.22182948 ↗ Read the argument →

← Back to the research