Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

a shared JSON schema and crowdsourced database for AI evaluation results

Jan Batzner, Sree Harsha Nelaturu, Damian Stachura, Anastassia Kornilova, Jon Crall, Tommaso Cerruti, Yanan Long, Yifan Mai, Sanchit Ahuja, Asaf Yehudai, Marek Suppa, John P. Lalor, Oluwagbemike Olowe, Jatin Ganhotra, Brian H. Hu, Eliya Habba, Andrew M. Bean, Chang Liu, Sander Land, Steven Dillmann, Aniketh Garikaparthi, Elron Bandel, Saki Imai, James Edgell, Wm. Matthew Kennedy, Jenny Chim, Patrick Meusling, Asteria Kaeberlein, Venkata Ramachandra Karthik Chundi, Manasi Patwardhan, Martin Ku, Austin Meek, Leon Knauer, Brian Wingenroth, Srishti Yadav, Usman Gohar, Felix Friedrich, Michelle Lin, Jennifer Mickel, Arman Cohan, Stella Biderman, Irene Solaiman, Zeerak Talat, Anka Reuel, Mubashara Akhtar, Gjergji Kasneci, Avijit Ghosh, Leshem Choshen · arXiv · 2026

In one sentence

Every Eval Ever is a community-governed JSON schema plus converters and a Hugging Face datastore that records not just a benchmark score but who ran it, on which model, under what generation settings, and what the metric means — spanning 22,235 models, 2,273 benchmarks and 31 evaluation formats.

Abstract

AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First, results are saved in incompatible formats, scattered across leaderboards, papers, blog posts, evaluation harness logs, and custom repositories. Second, results are created by different evaluation frameworks, which produce divergent scores for nominally identical evaluations and record metadata inconsistently, hindering comparison, cross-community evaluation science, cost reduction, and reuse. We introduce Every Eval Ever, the first shared schema and community-crowdsourced repository for AI evaluation results. The schema standardizes how evaluations are represented in a unified, single JSON document. It is source-agnostic by design, ingesting results from evaluation harnesses and papers alike, and optionally stores per-instance outputs for fine-grained analysis. We contribute: (i) a community-governed metadata schema with a companion instance-level schema, the first standardization effort of its kind; (ii) automatic converters from popular formats, evaluation harnesses, and leaderboards to the unified schema; and (iii) a crowdsourced community database hosted on Hugging Face, currently spanning to date 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats.

Questions this paper answers

is there an agreed way to write down the details of a benchmark run so other people can reuse the score?
what schema unifies evaluation result records across harnesses, leaderboards and reporting formats?
how do I publish my benchmark results in a format other people can aggregate with theirs?
should I log my eval runs into a shared schema instead of my own CSV columns?
Every Eval Ever is a shared schema and crowdsourced repository for AI evaluation results, filling the documentation gap left by dataset and model standards. Croissant, Datasheets for Datasets and Model Cards cover data and models but not the run-time context that determines whether two scores can be compared.
Holds for: Positioned as the first standardization effort of its kind for evaluation results as of the 2026 publication; concurrent efforts collecting instance-level or Inspect-specific outputs exist and are being aggregated into it.
The Every Eval Ever schema keeps required fields minimal and assigns each run a UUID rather than a canonical fingerprint. Partially specified, repeated and conflicting evaluation records are therefore all admitted and stay visible for later deduplication.
Holds for: Design decision of schema version 0.2.2 with companion instance_level_eval_0.2.2; the cost is that deduplication shifts to the analysis layer, where reference implementations of equivalence criteria are planned rather than shipped.
what should I read first about why the same benchmark score means different things in different tools?
which work covers cross-framework comparability and metadata provenance for LLM evaluation results?
where do I start reading if I need to compare scores reported by different evaluation pipelines?
I keep getting different numbers than a published leaderboard, is there a paper that explains what to check?
Every Eval Ever offers a starting point for readers looking for work on why nominally identical benchmark scores are not comparable across evaluation frameworks. It also documents what metadata has to be recorded before such scores can be compared.
Holds for: Coverage is strongest for text-based, single-model evaluations; multi-modal evaluations, human preference judgments such as Chatbot Arena Elo, and multi-agent settings are only partially supported as of publication.
Every Eval Ever is a shared schema and crowdsourced repository for AI evaluation results, filling the documentation gap left by dataset and model standards. Croissant, Datasheets for Datasets and Model Cards cover data and models but not the run-time context that determines whether two scores can be compared.
Holds for: Positioned as the first standardization effort of its kind for evaluation results as of the 2026 publication; concurrent efforts collecting instance-level or Inspect-specific outputs exist and are being aggregated into it.
how many models and tests are covered by the big crowdsourced collection of AI evaluation results?
what is the scale of the Every Eval Ever datastore in models, benchmarks and reporting formats?
where can I find already-collected benchmark scores for thousands of models instead of running them?
is the collection large enough that my model or my benchmark is likely already in it?
The Every Eval Ever datastore holds more than 200K aggregated evaluation results covering 22,235 models, 2,273 unique benchmarks and 31 distinct evaluation reporting formats, drawn from over a hundred community data contributions.
Holds for: Counts as of the May 4th 2026 snapshot; coverage is biased toward the sources contributors ingested, and 71.23% of runs come from one organization, alphaXiv.
when people publish benchmark scores, which run details do they usually leave out?
what are the field fill rates for inference platform, decoding temperature and parameter count across evaluation reporting formats?
which metadata do I need to chase down myself before I can trust a reported score?
can I tell from a published leaderboard entry what serving stack and sampling settings produced the number?
The inference platform is marked unknown or omitted entirely in 98% of evaluation rows in the Every Eval Ever datastore. Weighting each of the 31 formats equally, the field is reported in only 27% of rows on average.
Holds for: 98% is a micro-average over evaluation runs, 27% a macro-average over the 31 harnesses and formats in the datastore; measures the ingested sources, not the schema.
Across the 31 formats in the Every Eval Ever datastore, model name is filled 100% of the time but model parameter count and model license are each filled only 3%. Temperature and max tokens are filled 23% of the time.
Holds for: Macro-average fill rates over the 31 evaluation harnesses and formats ingested as of the paper's snapshot; measures what upstream sources report, and the schema records absence rather than defaulting missing values.
why do two papers report different perplexity numbers on the same WikiText text and the same model?
how much does token-level versus word-level normalization shift WikiText perplexity for Llama-2-7B?
how do I make my perplexity number comparable to one reported by a different codebase?
my perplexity is higher than the published one for the same model, is the normalization denominator the reason?
The same summed cross-entropy on WikiText yields a token-level perplexity of 5.4687 and a word-level perplexity of 8.7939 for Llama-2-7B, a gap of 3.3252 that comes only from the normalization denominator.
Holds for: Two implementations compared: a GPTQ-style script reporting token-normalized perplexity, and vLLM plus lm-eval-harness reporting word_perplexity; shown for OPT-6.7B (gap 1.4301) and Llama-2-7B on WikiText.
if I rerun a published evaluation myself, how close do the per-example results come out?
what per-instance agreement do local reruns of official HELM records achieve across comparable scenarios?
how do I check whether my local rerun of a public benchmark actually matches the official record?
can I trust the official numbers for a model, or should I rerun the benchmark myself?
Reproducing official HELM records locally and comparing per-instance scores gives 91% to 100% agreement for Pythia-6.9B and Vicuna-7B v1.3 across 13 comparable benchmarks, but drops to 78.8% for Falcon-7B on SyntheticReasoning-Natural.
Holds for: 3 models on 14 single-turn HELM benchmarks, both sides converted to the Every Eval Ever schema; agreement counts aligned (instance, core metric) pairs with identical scores up to numerical tolerance.
Instance-level comparison of official and reproduced HELM runs traces disagreements to concrete causes. Official Pythia completions on SyntheticReasoning-Natural are empty and score zero while local ones are non-empty, and Entity-Matching selects different Abt–Buy examples despite the same HELM recipe.
Holds for: 3 models on 14 single-turn HELM benchmarks; the schema surfaces mismatched example sets, empty completions and stochastic disagreement, but when serving details are missing it cannot always determine the exact cause.
when two runs of the same benchmark disagree, what is usually going wrong?
what concrete failure modes explain instance-level disagreement between official and reproduced HELM runs?
how do I debug why my rerun of a public benchmark gives different scores from the released ones?
my rerun disagrees with the official record on a handful of items, is that a real difference or a pipeline bug?
Instance-level comparison of official and reproduced HELM runs traces disagreements to concrete causes. Official Pythia completions on SyntheticReasoning-Natural are empty and score zero while local ones are non-empty, and Entity-Matching selects different Abt–Buy examples despite the same HELM recipe.
Holds for: 3 models on 14 single-turn HELM benchmarks; the schema surfaces mismatched example sets, empty completions and stochastic disagreement, but when serving details are missing it cannot always determine the exact cause.
do two coding agents that solve the same fraction of tasks cost the same money and time?
how do agentic scaffolds with an identical backbone differ in cost and wall-clock latency at matched accuracy on CocoaBench?
how do I choose between two coding agent harnesses when their accuracy is the same?
is accuracy enough for me to pick a coding agent, or do I need the dollars-per-task numbers too?
On CocoaBench, the Codex and OpenClaw scaffolds with a GPT-5.4 backbone both reach 45.1% accuracy, but Codex averages $0.7 and 377.8 s per task against OpenClaw's $1.0 and 502.1 s.
Holds for: Aggregate CocoaBench records for 6 scaffold–backbone pairs, re-represented in Every Eval Ever; source-reported accuracy, time and cost rather than reruns.
On CORE-Bench Hard from HAL, Claude Code beats CORE-Agent with a Claude Opus 4.5 backbone, 77.8% versus 42.2%. With Claude Opus 4.1 the ranking flips, 42.2% for Claude Code against 51.1% for CORE-Agent.
Holds for: Representative HAL records for 6 scaffold–backbone pairs on CORE-Bench Hard re-represented in Every Eval Ever; source-reported accuracies and costs rather than reruns.
does the best agent wrapper change depending on which underlying model you plug into it?
do scaffold rankings on CORE-Bench Hard invert across Claude Opus backbone versions?
how do I pick an agent scaffold when I plan to swap the backbone model later?
if I upgrade my backbone model, will the agent framework I already chose still be the better one?
On CORE-Bench Hard from HAL, Claude Code beats CORE-Agent with a Claude Opus 4.5 backbone, 77.8% versus 42.2%. With Claude Opus 4.1 the ranking flips, 42.2% for Claude Code against 51.1% for CORE-Agent.
Holds for: Representative HAL records for 6 scaffold–backbone pairs on CORE-Bench Hard re-represented in Every Eval Ever; source-reported accuracies and costs rather than reruns.
can you tell which benchmark has harder individual questions rather than just lower average scores?
what does a 1PL IRT fit to instance-level records show about item difficulty and its spread across GPQA Diamond, JudgeBench and Wordle Arena?
how do I compare benchmarks at the item level instead of comparing their headline accuracies?
do I have enough per-item results across models to fit an item-difficulty model on my benchmark?
A 1PL Item Response Theory model fit to instance-level Every Eval Ever records shows Wordle Arena items are harder on average and more variable in difficulty than GPQA Diamond or JudgeBench items. The response matrices cover 198 items from 69 models, 63 items from 46 models, and 350 items from 55 models.
Holds for: Unidimensional 1PL model fit with py-irt 0.7.1 via variational inference, using the is_correct field of each instance-level record; the difficulty comparison is descriptive, not a tested prediction about future saturation.
how much would it cost in API bills to rerun every collected model-and-benchmark evaluation from scratch?
what is the estimated inference spend to reproduce roughly 230,000 model-benchmark evaluation pairs, with and without an LLM judge?
how do I justify reusing published evaluation results instead of budgeting to rerun them?
is it cheaper for me to reuse aggregated published eval results than to rerun the benchmarks myself?
Re-running the roughly 230,000 model–benchmark evaluation pairs collected in Every Eval Ever is estimated to cost about $221K with a mid-tier model plus an LLM judge. A higher-end model raises that to about $368K, and a no-judge lower bound puts it at about $4.1K.
Holds for: Assumes 1,000 examples per benchmark, 100 input and 20 output tokens per example, 60% LLM-as-judge token overhead and list API prices; excludes agentic evaluations, reasoning models, repeated runs and human labeling.
are reported AI benchmark results spread across many systems, or piled onto a few popular ones?
how concentrated is evaluation activity across models and benchmarks in aggregated evaluation records?
how do I find out whether the model or benchmark I care about has any reported results at all?
if my model is not from a big lab, should I expect to find published evaluations of it?
Evaluation activity in the Every Eval Ever datastore follows a long tail: the top 25 models and top 25 benchmarks each account for barely 25% of all results. Excluding human baselines, 5 companies supply 23 of the 24 most frequently evaluated systems.
Holds for: Descriptive statistics over the ingested corpus as of the paper's snapshot; coverage is biased by which leaderboards and papers contributors converted, so the concentration describes reported evaluations rather than all evaluations run.
can a shared collection of benchmark results accept a run that is missing some of its settings?
how does a minimal-required-field schema with per-run UUIDs handle partial, duplicate and conflicting evaluation records?
how do I contribute an old evaluation run when I no longer know the temperature or the serving platform?
my logs are incomplete, will my results be rejected from a shared evaluation repository?
The Every Eval Ever schema keeps required fields minimal and assigns each run a UUID rather than a canonical fingerprint. Partially specified, repeated and conflicting evaluation records are therefore all admitted and stay visible for later deduplication.
Holds for: Design decision of schema version 0.2.2 with companion instance_level_eval_0.2.2; the cost is that deduplication shifts to the analysis layer, where reference implementations of equivalence criteria are planned rather than shipped.
The Every Eval Ever schema was built from structured feedback from about 40 researchers and unstructured feedback from about 110. A field was admitted only if some existing framework or published result already reports it and a majority of contributors judged it necessary to interpret the score.
Holds for: Contributors included benchmark creators, evaluation framework developers, governance experts, leaderboard operators and industry practitioners; disagreements were resolved by consensus among core maintainers, who retain final authority on contested changes.
who decided which run details a shared benchmark-reporting format should ask for?
what inclusion criteria and community feedback process determined the fields of the Every Eval Ever schema?
how do I decide which metadata fields to require when designing an evaluation reporting format?
can I trust that a shared evaluation schema asks for fields my group actually needs, and can I get one added?
The Every Eval Ever schema was built from structured feedback from about 40 researchers and unstructured feedback from about 110. A field was admitted only if some existing framework or published result already reports it and a majority of contributors judged it necessary to interpret the score.
Holds for: Contributors included benchmark creators, evaluation framework developers, governance experts, leaderboard operators and industry practitioners; disagreements were resolved by consensus among core maintainers, who retain final authority on contested changes.

Claims and scope

Common misreadings

Terminology in this paper

aggregate evaluation record
A single JSON document describing one evaluation run of one model, holding source provenance, model identity and access mode, evaluation framework, generation configuration, and one or more metric results.
instance-level companion file
An optional JSONL sidecar, one object per benchmark sample, storing prompts, model outputs, references, per-sample scores, token usage and latency, linked to an aggregate record by evaluation_id.
evaluator_relationship
A metadata field recording whether an evaluation was run by the model developer (first_party), an independent party (third_party), or the metadata contributor themselves (self).
source_type
A metadata field distinguishing results scraped from a leaderboard or paper (documentation) from results produced by a local evaluation run (evaluation_run).
interaction_type
The interaction regime of an instance-level record, one of single_turn, multi_turn, or agentic, which determines whether the record uses an output object or a messages array with tool calls.
metric_config
A per-result object capturing metric semantics — whether lower values are better, the score type (continuous, binary, ordinal), the score range, and level names for ordinal metrics — so that a bare number is not misread.
macro-average fill rate
The share of records in which a metadata field is populated, averaged with each source format weighted equally rather than by its number of records.

How to cite

@article{batzner2026every,
  author       = {Jan Batzner and
                  Sree Harsha Nelaturu and
                  Damian Stachura and
                  Anastassia Kornilova and
                  Jon Crall and
                  Tommaso Cerruti and
                  Yanan Long and
                  Yifan Mai and
                  Sanchit Ahuja and
                  Asaf Yehudai and
                  Marek Suppa and
                  John P. Lalor and
                  Oluwagbemike Olowe and
                  Jatin Ganhotra and
                  Brian H. Hu and
                  Eliya Habba and
                  Andrew M. Bean and
                  Chang Liu and
                  Sander Land and
                  Steven Dillmann and
                  Aniketh Garikaparthi and
                  Elron Bandel and
                  Saki Imai and
                  James Edgell and
                  Wm. Matthew Kennedy and
                  Jenny Chim and
                  Patrick Meusling and
                  Asteria Kaeberlein and
                  Venkata Ramachandra Karthik Chundi and
                  Manasi Patwardhan and
                  Martin Ku and
                  Austin Meek and
                  Leon Knauer and
                  Brian Wingenroth and
                  Srishti Yadav and
                  Usman Gohar and
                  Felix Friedrich and
                  Michelle Lin and
                  Jennifer Mickel and
                  Arman Cohan and
                  Stella Biderman and
                  Irene Solaiman and
                  Zeerak Talat and
                  Anka Reuel and
                  Mubashara Akhtar and
                  Gjergji Kasneci and
                  Avijit Ghosh and
                  Leshem Choshen},
  title        = {Every Eval Ever: {A} Unifying Schema and Community Repository for
                  {AI} Evaluation Results},
  journal      = {CoRR},
  volume       = {abs/2606.14516},
  year         = {2026},
  url          = {https://doi.org/10.48550/arXiv.2606.14516},
  doi          = {10.48550/ARXIV.2606.14516},
  eprinttype   = {arXiv},
  eprint       = {2606.14516},
  timestamp    = {Sat, 18 Jul 2026 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2606-14516.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.