Benchmarks — Can You Trust "#1 in AI Performance"?

"This model is number one on the benchmarks." Pick a model on that one sentence and you miss two things: the score may be the result of having seen the exam in advance, and the exam may have nothing to do with your work. In 2026 a benchmark score is less a report card than a marketing line the vendor chose.

---

Background / Why This Term Now

In 2026, trust in benchmarks has visibly wobbled. The signature event was the retirement of SWE-bench Verified. OpenAI announced it would no longer report this benchmark, which had been used almost as the standard for coding-agent performance, citing "signs that models had already learned the answer patches." According to OpenAI, an audit of 138 hard tasks found that 59.4% had flaws in the test design itself, and major frontier models reproduced the answer patches for some tasks verbatim (OpenAI, CodeSOTA, 2026).

In August of the same year, MLCommons, the industry standards body, published an official document titled "Seven questions for deciding whether to trust a benchmark" (MLCommons, Aug 2026). The people who make benchmarks issued a guide saying "doubt benchmarks."

Korea is no different. As the government selected national flagship AI models, critics pointed out that questions translated from English could not properly measure Korean-language ability, and the Ministry of Science and ICT and NIA began building independent benchmarks of 10,000 questions each in three areas: Korean language, instruction following, and safety (Money Today, Apr 2026).

This article lays out what the word "benchmark" actually measures, why the number-one ranking is shaky, and how to read scores so that they are still useful.

Key Data & Current State

Item	Figure	Source / Date
Test-design flaws among 138 hard SWE-bench Verified tasks	59.4%	OpenAI audit, 2026
SWE-bench Verified data leakage rate	10.6%	buildmvpfast summary, 2026
SWE-bench accuracy on public vs. private data	70% → 15–18%	MLCommons, Aug 2026
GSM8K accuracy drop when contaminated items are removed	About 13 points	Digital Applied summary, 2026
Effect of contamination on rankings (rank correlation, standard vs. contamination-controlled leaderboard)	0.997	Xiao & Cheng, arXiv, Jul 2026
LM Arena data share: Google / OpenAI vs. 83 open-weight models combined	19.2% / 20.4% vs. 29.7%	The Leaderboard Illusion, Apr 2025
Number of Llama 4 variants Meta tested privately before release	27 (1 released publicly)	Same paper
Top score on Humanity's Last Exam	59.1%	Artificial Analysis, as of Sep 2026
Major model scores at ARC-AGI-3 launch	Under 1% (humans: 100%)	ARC-AGI-3 analysis, Mar 2026

Interpretation: Two opposing facts hold at once. Contamination inflates absolute scores heavily (13 points, 70%→18%), but because every model is contaminated to a similar degree, the rankings themselves barely move (correlation 0.997). In other words, you cannot trust the number "80 points," but you can trust the order "A is above B" to a fair degree. That said, there are separate factors that shake rankings: private testing and selective disclosure.

In-Depth Analysis

1) What Exactly Is a Benchmark?

A benchmark is a fixed set of problems plus a scoring rule. MMLU is multiple-choice questions across 57 subjects, GSM8K is grade-school math word problems, SWE-bench is tasks that fix real GitHub issues so the tests pass, and HLE (Humanity's Last Exam) is the hardest questions written by experts in each field. Give the models the same problems, count the correct answers, and you get a score.

The key word is "fixed." Because the problems are fixed and mostly public, comparison is possible, but for exactly that reason the problems and answers float around the internet and get mixed into training data. This is called contamination. It is like an exam whose paper has leaked.

Separately, there are vote-based leaderboards like LM Arena. People look at two models' answers and vote for the better one, and points accumulate like a chess rating. With no fixed problems there is less contamination, but there are other problems instead.

2) Why "Number One" Is Shaky — Three Cracks

First, contamination. Benchmarks published before 2024 are effectively all contaminated to some degree. Remove the contaminated items from GSM8K and scores drop 13 points; SWE-bench accuracy falls from 70% on public problems to 15–18% on private ones. OpenAI abandoned SWE-bench Verified because models were not "solving" the answers but "remembering" them.

Second, selective disclosure. In April 2025, a paper titled "The Leaderboard Illusion" from Cohere Labs, AI2, Princeton, Stanford, and others pointed out the practice of some large vendors privately testing dozens of variants on LM Arena before release and publishing only the single best score. Meta tested 27 Llama 4 variants and released one. It is rolling the dice many times and reporting only the best roll.

Third, measurement conditions. Even with identical model weights, scores move by 10–20 points depending on prompt format, number of attempts, and thinking budget. A model that scored 59.1% at a "Max Effort" setting may be far lower at default settings, but the leaderboard shows the number from the highest setting.

3) So Are Benchmarks Useless? No, You Just Read Them Differently

There is an interesting counterargument. In July 2026, researchers at Stanford and City University of Macau compared 47 public models and 74 deliberately contaminated models across four benchmarks, and the rank correlation between the contamination-controlled leaderboard and the uncontrolled one was 0.997. Of 188 model-benchmark combinations, only 3 involved a specific model being unusually contaminated. If everyone has seen the exam to a similar degree, the rankings do not change much.

So the practical conclusion is this. Treat absolute scores ("MMLU 90%") as marketing copy, and use only the relative order measured under the same conditions and the size of the gaps. And the factors that shake the order, namely private testing and differences in measurement conditions, must be checked separately, outside the leaderboard.

4) The Benchmark Side's Response — Living Exams

The answer to contamination comes in two branches. One is to keep swapping in new problems. LiveBench and LiveCodeBench add new problems regularly, and SWE-rebench automatically collects new GitHub issues. SWE-bench Pro, which OpenAI recommended as an alternative, also reduces contamination with a private split.

The other is to set problems that cannot be memorized in the first place. ARC-AGI-3, launched on March 24, 2026, consists of interactive tasks with never-before-seen rules, and major models scored under 1% at launch; HLE consists of expert-written questions and the top score is still 59.1%. When scores rise on benchmarks like these, the case that "ability really improved" is relatively strong.

In Korea, the limits of translated benchmarks were the problem. An exam translated from English questions cannot measure spelling, spacing, dialect, or cultural context, so the K-AI leaderboard uses five metrics including KMMLU-Pro (questions from 14 national professional licensing exams) and CLIcK (Korean cultural and linguistic understanding), and the government is adding sets of 10,000 questions per area by the end of this year. In the January national flagship AI selection, the overall winner was LG and the Korean-language winner was SKT, with first place split by category (Money Today, Jan 2026), which shows well why "number one" cannot be summarized in a single line.

In Practice — 6 Principles for Reading Benchmarks Properly

1. Look at the gap, not the absolute score. The information is not "90 points" but whether it is "3 points ahead of second place" or "15 points ahead." A 3-point gap flips just by changing measurement conditions.

2. Check the benchmark's age. Read scores on pre-2024 public benchmarks (MMLU, HumanEval, GSM8K) on the assumption of contamination. If there is a successor with the same name (MMLU-Pro, LiveCodeBench, SWE-bench Pro), look at that one.

3. Read the measurement conditions alongside. Exclude from comparison any score that does not state the number of attempts (pass@1 or pass@5), thinking budget, and whether tools were used.

4. For vote-based leaderboards, look at data share too. Use LM Arena rankings as a reference, but remember that the structure allows top-tier large vendors to test private variants.

5. Give trust weight only to benchmarks that resemble your work. There is no reason to look at multiple-choice knowledge scores when choosing a coding agent. The first of the MLCommons seven questions is "Does this benchmark measure the decision I am about to make?"

6. Make the final call with 20 of your own problems. Collecting 20 inputs from your actual work and running them identically through candidate models is more accurate than any leaderboard. A few dollars is enough to cover the cost.