When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek Šuppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, Yiyang Nan, Jyoutir Raj, Yu Fan, Shubham Singh, Subramanyam Sahoo, Eliya Habba, Usman Gohar, Siddhesh Pawar, Robert Scholz, Arjun Subramonian, Jingwei Ni, Mykel Kochenderfer, Sanmi Koyejo, Mrinmaya Sachan, Stella Biderman, Zeerak Talat, Avijit Ghosh, Irene Solaiman · ICML 2026 · 2026

In one sentence

A study of 60 text-based LLM benchmarks that defines saturation as the loss of statistically reliable separation among top models, measures it with an uncertainty-aware saturation index from leaderboard scores, and finds age and test set scale predict it while private test sets and open-ended formats do not.

Abstract

Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly"saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches.

Questions this paper answers

has anyone measured how many language model benchmarks no longer tell the best models apart?
is there a quantitative operationalization of benchmark saturation across LLM leaderboards?
where do I start reading about whether LLM benchmarks still separate frontier models?
I need a citable source on benchmark saturation for an evaluation report, what should I cite?
"When AI Benchmarks Plateau" defines benchmark saturation as the loss of reliable discriminative power among top-performing models. Saturation is operationalized as an uncertainty-aware index computed from leaderboard scores, with no reliance on human baselines.
Holds for: The index uses the top k=5 leaderboard scores and an effective test set size n^0.5; it is designed for accuracy-like metrics averaged over a fixed test set, and other metrics such as Elo, pass@k or judge-based scores need their own variance estimates.
Of 60 widely used text-based LLM benchmarks, 29 show high or very high saturation (saturation index at least 0.7), of which 14 are very high (at least 0.9).
Holds for: Text-only benchmarks with usable leaderboard data, from developer reports (Jan 2022-Nov 2025) and cited papers; static snapshots.
how many of the popular language model benchmarks have stopped separating the top models?
what share of widely used text LLM benchmarks exhibit high saturation index values?
how do I tell whether the benchmarks in my evaluation suite still have discriminative power?
are the standard benchmarks I report on still worth reporting?
Of 60 widely used text-based LLM benchmarks, 29 show high or very high saturation (saturation index at least 0.7), of which 14 are very high (at least 0.9).
Holds for: Text-only benchmarks with usable leaderboard data, from developer reports (Jan 2022-Nov 2025) and cited papers; static snapshots.
The share of saturated benchmarks rises from 42.9% among benchmarks released within the past 24 months to 54.5% among those older than 60 months. Mean saturation indices across the age bins are 0.51, 0.52 and 0.60.
Holds for: N=60 benchmarks aged 1 to 114 months; the trend is directionally consistent but modest and not statistically significant at conventional thresholds.
do benchmarks get worse at separating models the longer they have been out?
is time since benchmark release a predictor of saturation index?
how do I use a benchmark's release date to judge whether it is still useful?
should I drop a benchmark from my evaluation suite once it is a few years old?
The share of saturated benchmarks rises from 42.9% among benchmarks released within the past 24 months to 54.5% among those older than 60 months. Mean saturation indices across the age bins are 0.51, 0.52 and 0.60.
Holds for: N=60 benchmarks aged 1 to 114 months; the trend is directionally consistent but modest and not statistically significant at conventional thresholds.
A Bayesian regression predicting the saturation index of 60 benchmarks attains R-squared of 0.884 +/- 0.012, with benchmark age and test set size the most consistent predictors. Predictors also included adoption proxies, accessibility, output format, templating, language coverage, curation strategy and documented quality issues.
Holds for: Fitted on 60 benchmarks with time-invariant annotations; accessibility, output format and templating show no reliable associations once confounders are included, and the fit is descriptive rather than causal.
does hiding the test answers keep a benchmark from going stale?
do private held-out test splits slow saturation relative to fully public benchmarks?
if I want my new benchmark to last, is withholding the test set enough?
should I pay for a benchmark with a hidden test set instead of using a public one?
Public (N=56) and private (N=4) benchmarks show similar saturation distributions, with no statistically meaningful difference in saturation index. Private held-out test sets therefore do not protect against saturation, rejecting the hypothesis that public benchmarks saturate faster.
Holds for: Only 4 private benchmarks in the sample, so the comparison has low power; based on annotated public availability of benchmark data and labels rather than on measured contamination.
does asking for written answers instead of multiple choice make a benchmark last longer?
does open-ended generation versus closed-ended answer format affect saturation rates?
when designing an eval, should I use free-form responses to keep it discriminative for longer?
is it worth rebuilding my multiple-choice eval as open-ended generation?
Closed-ended benchmarks (N=28) and open-ended generation benchmarks (N=31) show no meaningful difference in saturation, and the comparison is age-balanced (p=0.40).
Holds for: 60 text-only LLM benchmarks; output format annotated as MCQ versus free-form generation, so open-ended scoring protocols are not separated.
do benchmarks covering many languages stay useful longer than English-only ones?
is the lower observed saturation of multilingual benchmarks intrinsic or confounded by release recency?
can I extend an evaluation's shelf life by adding more languages to it?
should I invest in translating my benchmark into more languages to keep it hard?
Multilingual benchmarks (N=16) show lower raw saturation rates than English-only benchmarks (N=44). The advantage is explained by recency rather than intrinsic resistance, as the multilingual benchmarks are substantially younger on average (32.9 vs. 48.9 months).
Holds for: Observational comparison across 60 benchmarks with age as a confounder; no causal test isolating language coverage from maturity.
what actually makes some evaluations keep separating the best models for years?
which design factors, such as expert curation or templated item generation, are associated with slower saturation?
how should I build a benchmark that stays discriminative as models improve?
is it worth paying domain experts to write my eval items instead of crowdsourcing them?
Expert-curated benchmarks show lower saturation than crowdsourced ones at comparable ages, and several such benchmarks including ARC-AGI and BIG-Bench Hard remain unsaturated despite prolonged exposure.
Holds for: Curation categories differ significantly in age (p=0.0017), and fully synthetic benchmarks are too recent for causal interpretation; the finding is associational, not a controlled comparison.
Templated benchmarks (N=14) do not differ significantly from non-templated ones (N=46) in saturation behaviour (p=0.10), indicating that surface-level template diversity alone does not delay saturation.
Holds for: Templating annotated as whether prompts use fixed patterns versus natural variation, on 60 text benchmarks; age-balanced comparison group.
do the most cited and most reported benchmarks wear out faster?
after adjusting for age, are citation counts or appearance in developer technical reports associated with saturation?
can I use a benchmark's popularity to predict whether it is still discriminative?
should I avoid the most widely reported benchmarks on the assumption that heavy use burns them out?
After controlling for benchmark age, citation counts are not significantly associated with saturation (rho=0.22, p=0.12), nor are citation growth rates (rho=0.13, p=0.37) or frequency of appearance in developer technical reports (rho=0.05, p=0.73).
Holds for: Adoption measured only through citation counts, citation growth and inclusion in industry model release reports for 60 benchmarks; raw uncontrolled correlations do show higher saturation at higher citation counts.
The share of saturated benchmarks rises from 42.9% among benchmarks released within the past 24 months to 54.5% among those older than 60 months. Mean saturation indices across the age bins are 0.51, 0.52 and 0.60.
Holds for: N=60 benchmarks aged 1 to 114 months; the trend is directionally consistent but modest and not statistically significant at conventional thresholds.
does having more test questions help tell the best models apart?
how does test set size relate to saturation index and measurement resolution among top models?
how many examples should my benchmark contain if I want it to distinguish frontier models?
is it worth expanding my 500-item eval to a few thousand items?
Larger test sets are associated with lower saturation indices across the 60 benchmarks, and the relationship persists in the joint Bayesian regression, indicating that measurement resolution limits discriminative power.
Holds for: Test set sizes span a few dozen to several hundred thousand samples; the index's uncertainty term uses n^0.5 rather than raw n, so the size dependence is deliberately down-weighted.
A Bayesian regression predicting the saturation index of 60 benchmarks attains R-squared of 0.884 +/- 0.012, with benchmark age and test set size the most consistent predictors. Predictors also included adoption proxies, accessibility, output format, templating, language coverage, curation strategy and documented quality issues.
Holds for: Fitted on 60 benchmarks with time-invariant annotations; accessibility, output format and templating show no reliable associations once confounders are included, and the fit is descriptive rather than causal.
how do you put a number on benchmark staleness, and does the number move if you change the settings?
how sensitive is a leaderboard saturation index to the number of top models k and the weighting parameter alpha?
if I compute a saturation score for my own benchmarks, which parameter choices do I need to worry about?
can I trust a saturation ranking of benchmarks enough to act on it?
Saturation-index rankings are stable to parameter choices, with Spearman correlations of 0.92 for k=3 vs k=5, 0.88 for alpha=0.5 vs alpha=0 and 0.92 for alpha=0.5 vs alpha=1. Only 18.3-48.3% of benchmarks stay in the same one of the five saturation bins.
Holds for: Sensitivity checked over k in {3,5} and alpha in {0,0.5,1} on the 60-benchmark set; most bin changes are between neighbouring bins, so absolute values shift more than ordering.
"When AI Benchmarks Plateau" defines benchmark saturation as the loss of reliable discriminative power among top-performing models. Saturation is operationalized as an uncertainty-aware index computed from leaderboard scores, with no reliance on human baselines.
Holds for: The index uses the top k=5 leaderboard scores and an effective test set size n^0.5; it is designed for accuracy-like metrics averaged over a fixed test set, and other metrics such as Elo, pass@k or judge-based scores need their own variance estimates.
can a benchmark be exhausted even though the best scores are nowhere near 100%?
does a high saturation index imply the task is solved, or can score clustering occur well below ceiling?
how do I tell whether flat leaderboard scores mean my benchmark is used up or that models have stopped improving?
top models on my benchmark all sit around 79% and within a point of each other, is the benchmark still informative?
Saturation can occur far below the score ceiling: LiveBench reaches a saturation index of 0.99 with top models clustered in a 1.09-point range at roughly 79% performance. Such clustering reflects model-level stagnation rather than task completion.
Holds for: Single-benchmark case study from a static leaderboard snapshot of n=1000 with SE_delta=0.1028; saturation as defined additionally requires nearness to the empirical ceiling.
"When AI Benchmarks Plateau" defines benchmark saturation as the loss of reliable discriminative power among top-performing models. Saturation is operationalized as an uncertainty-aware index computed from leaderboard scores, with no reliance on human baselines.
Holds for: The index uses the top k=5 leaderboard scores and an effective test set size n^0.5; it is designed for accuracy-like metrics averaged over a fixed test set, and other metrics such as Elo, pass@k or judge-based scores need their own variance estimates.
what should the people who build evaluations do once all the best models score the same?
what lifecycle practices are recommended for benchmarks that have lost discriminative power among frontier models?
how do I decide when to refresh, stratify or retire a benchmark I maintain?
my leaderboard's top entries are within noise of each other, should I retire it or refresh the test set?
"When AI Benchmarks Plateau" argues benchmarks should be managed as ageing measurement instruments rather than fixed targets. Its recommendations are larger or stratified test sets, periodic or adversarial refreshes, uncertainty-aware leaderboard reporting, and explicit revision or retirement criteria.
Holds for: Recommendations are derived from associational analysis of 60 text-only LLM benchmarks and are not themselves experimentally validated; as of the 2026 publication.
which properties of a benchmark best explain whether it still separates the top models?
in a joint Bayesian regression over benchmark metadata, which covariates explain variance in the saturation index?
can I predict from a benchmark's metadata alone whether it is saturated before running any models?
if I only know a benchmark's age, size and format, can I judge whether it is worth running?
A Bayesian regression predicting the saturation index of 60 benchmarks attains R-squared of 0.884 +/- 0.012, with benchmark age and test set size the most consistent predictors. Predictors also included adoption proxies, accessibility, output format, templating, language coverage, curation strategy and documented quality issues.
Holds for: Fitted on 60 benchmarks with time-invariant annotations; accessibility, output format and templating show no reliable associations once confounders are included, and the fit is descriptive rather than causal.
The interaction model separating saturated from non-saturated benchmarks reaches a posterior AUROC tightly concentrated near a median of about 0.98.
Holds for: In-sample discrimination on the same 60 annotated benchmarks used for fitting, with labels from the paper's own index.

Claims and scope

Common misreadings

Terminology in this paper

benchmark saturation
The loss of reliable discriminative power among top-performing models on a benchmark: top scores are statistically indistinguishable and also approach the benchmark's empirically observed ceiling.
stagnation
Statistical indistinguishability among a benchmark's top models without performance being near the empirical ceiling, which may be overcome by future architectural, training or evaluation advances.
saturation index
A continuous score in [0,1] equal to exp(-R_norm^2), where R_norm is the top-1-to-top-k score gap divided by the standard error of that difference; higher values mean stronger evidence of saturation.
normalized score range (R_norm)
The spread between the best and k-th best leaderboard score divided by the standard error of their difference, interpretable as a signal-to-noise ratio for model separability.
effective test set size
The nominal test set size n raised to the power alpha (default 0.5), used in the standard-error term so that very large benchmarks do not dominate uncertainty estimates.
model-level saturation
Tight clustering of top models at a low absolute performance level, indicating that a benchmark no longer separates contemporary systems rather than that the task is solved.

How to cite

@article{akhtar2026ai,
  title={When {AI} Benchmarks Plateau: A Systematic Study of Benchmark Saturation},
  author={Mubashara Akhtar and Anka Reuel and Prajna Soni and Sanchit Ahuja and Pawan Sasanka Ammanamanchi and Ruchit Rawal and Vilém Zouhar and Srishti Yadav and Chenxi Whitehouse and Dayeon Ki and Jennifer Mickel and Leshem Choshen and Marek Šuppa and Jan Batzner and Jenny Chim and Jeba Sania and Yanan Long and Hossein A. Rahmani and Christina Knight and Yiyang Nan and Jyoutir Raj and Yu Fan and Shubham Singh and Subramanyam Sahoo and Eliya Habba and Usman Gohar and Siddhesh Pawar and Robert Scholz and Arjun Subramonian and Jingwei Ni and Mykel Kochenderfer and Sanmi Koyejo and Mrinmaya Sachan and Stella Biderman and Zeerak Talat and Avijit Ghosh and Irene Solaiman},
  journal={International Conference on Machine Learning (ICML) 2026},
  year={2026}
}

References

See the full reference list in the paper.