Data Contamination in ML Benchmarks

If your benchmark questions appeared in the training data, the model isn’t reasoning, it’s remembering. Contamination has quietly broken many published comparisons.

What contamination is

Test set leakage, when training data includes examples from a benchmark you later evaluate on. The model "scores well" because it memorised the answers, not because it generalised. As benchmarks proliferate and training corpora scale to the entire internet, contamination is the rule, not the exception.

The simple version. A benchmark publishes a test set. The test set ends up on a webpage. Web crawls used for training include the page. The model trains on the test answers. Evaluation reports inflated numbers.

The structural version. Even when test sets aren't directly published, derivative copies exist, student notes, blog posts, GitHub repos, paper appendices. Crawlers pick up derivatives. By the time a model trains, the benchmark has been "leaked" through ten different paths the original authors never controlled.

The implication. A model that scores 95% on a contaminated benchmark might genuinely deserve 70% on the underlying skill. The 25% gap is memorisation. Decisions made on contaminated benchmarks (which model to ship, which research direction is promising) are partially based on memorisation, not capability.

The cost. Real progress is harder to measure than the leaderboards suggest. The pace of improvement looks faster than it is; the differences between models look bigger than they are. Sober capability measurement requires fighting contamination.

How it happens

Benchmark posted on GitHub, scraped into Common Crawl, included in training. Or: paper published, derivative datasets created with the same examples, those scraped. Or: question-and-answer sites where benchmark questions are posted by curious users. The pipelines are accidental, not malicious; they're also nearly impossible to prevent at training-data-collection time.

The Common Crawl pipeline. Most large training datasets include Common Crawl or its derivatives. Common Crawl includes most of the public web. Most benchmark test sets, in some form, exist on the public web. The contamination is essentially structural for any model trained on web data.

The derivative dataset pipeline. Researchers create datasets that include benchmark questions for various reasons (translations, augmentations, study aids). The derivative datasets get included in training corpora. Contamination spreads through these channels even when the original benchmark wasn't directly included.

The Q&A site pipeline. Stack Exchange, Quora, Reddit. Users post benchmark questions seeking help. Answers (correct and incorrect) get indexed. Models train on the discussions. The original benchmark's "secret" answers leak through community discussion.

The conference paper pipeline. Papers include benchmark examples in figures, tables, and appendices. Papers are scraped (especially arXiv). Models train on the paper text. Even examples not on the public web pre-publication leak through paper text post-publication.

Detection

Several techniques. Membership inference, does the model's loss on benchmark examples differ from loss on similar held-out examples? Question variation, paraphrase the benchmark; does performance drop? Released-after-training, use benchmarks released after the training cutoff. Each method has limitations; use multiple together.

The membership inference test. Compute model loss on benchmark examples vs syntactically similar non-benchmark examples. Trained-on examples have suspiciously low loss; held-out examples have higher loss. The difference is the contamination signal.

The paraphrase test. Take benchmark questions; rephrase them with the same semantics but different surface form. Run the model on both. If accuracy drops 10+ points on paraphrases, the model was memorising surface patterns, not solving the underlying problem.

The fresh-benchmark test. Use benchmarks released after the model's training cutoff. The model can't have trained on them. Performance on fresh benchmarks is the cleanest capability measurement.

The cross-checking. No single method is perfect; combine. A model that scores well on the original, drops on paraphrases, has higher loss on the original than similar examples, that combination is strong evidence of contamination.

Avoiding it

For evaluators: use private holdouts that haven't appeared online. Re-curate benchmarks regularly. For model trainers: filter training data for known benchmark strings. For consumers: trust benchmarks released after the training cutoff most. Treat older benchmarks as suggestive, not authoritative.

The private-holdout strategy. The benchmark has two parts: a public dev set everyone can train on, and a private test set held by the evaluation organisation. Models submit predictions; evaluators compute scores. The private set never leaks; contamination is structurally prevented. Examples: HELM, MLPerf inference.

The re-curation strategy. Replace contaminated benchmarks with fresh ones every 6-12 months. The new benchmarks haven't yet leaked. Models can't have memorised them. Examples: ARC-AGI updates, fresh code benchmarks (LiveCodeBench).

The training-data filtering strategy. Before training, filter for known benchmark substrings. Imperfect (paraphrases survive) but catches direct contamination. Most frontier labs do this; teams using publicly-available data don't have the filter and inherit contamination.

The consumer-of-models strategy. Discount benchmark scores. A model claiming 95% on benchmark X probably deserves 70-85% credit for actual capability. Use multiple benchmarks (especially fresh ones); use task-specific evals on YOUR data; the leaderboards are signal but not ground truth.

Common antipatterns

Trusting a single benchmark. Single benchmarks are most contaminated. Use a basket of benchmarks (including fresh ones) for any consequential decision.

Treating benchmark gaps as definitive. A 10-point gap between model A and B might be capability or might be differential contamination. Validate on YOUR tasks before drawing conclusions.

Skipping the paraphrase check. Paraphrase robustness is a cheap, useful signal. Adding it to evals takes hours; surfaces contamination effectively.

Reporting benchmark scores without disclosing training cutoff. The cutoff is essential context. Without it, scores are uncomparable to other models.

What to do this week

Three moves. (1) For any benchmark you cite, check its release date vs your model's training cutoff. Older-than-cutoff benchmarks deserve discount. (2) Build a paraphrase test for your top eval task. The paraphrase robustness will surprise you. (3) If you publish benchmark results, include training cutoff and contamination methodology. The transparency makes your results more useful (and better-respected by readers who know).