Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting

a unified, machine-populated record that joins benchmark metadata, evaluation run data and model metadata so a reported score can be traced, compared and read for what it omits

Avijit Ghosh, Anka Reuel, Jenny Chim, Wm. Matthew Kennedy, Srishti Yadav, Jennifer Mickel, Yanan Long, Andrew Tran, Anastassia Kornilova, Damian Stachura, Kevin Klyman, Felix Friedrich, Jeba Sania, Jan Batzner, Anoop Mishra, Eliya Habba, Yixiong Hao, Nathan Heath, Shalaleh Rismani, Usman Gohar, Andrea Loehr, David Manheim, Ruchira Dhar, Sree Harsha Nelaturu, Aarush Sinha, Leshem Choshen, Drishti Sharma, Ishan Khire, Amit Saha, Subramanyam Sahoo, Michael Hardy, Michael Alexander Riegler, Kabir Manghnani, Michelle Lin, Yanan Jiang, Yilin Huang, Asaf Yehudai, Jessica Ji, Aris Hofmann, Mubashara Akhtar, Max Lamparth, Nuno Moniz, Yacine Jernite, Stella Biderman, Zeerak Talat, Sanmi Koyejo, Mykel Kochenderfer, Irene Solaiman · 2026

In one sentence

Evaluation Cards is a deployed reporting layer that joins benchmark metadata, evaluation run data and model metadata into one record, resolves every score to a family-composite-benchmark-split-metric path, and computes four interpretive signals (reproducibility, completeness, provenance, comparability) over it.

Abstract

AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present \EvalCards{}, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies \EvalCards{} across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.

Questions this paper answers

when a company publishes a benchmark score for a language model, do they usually say enough for someone else to get the same number?
how prevalent are missing decoding-configuration fields such as temperature and max_tokens across publicly reported LLM benchmark results?
how do I find out whether a published model score lists the generation settings I would need to re-run the evaluation?
can I trust a leaderboard score enough to reproduce it on my own infrastructure, or will the sampling settings be missing?
Across 50,461 (model, benchmark, metric-path) triples in the Evaluation Cards corpus, 48,698 (96.5%) are missing at least one field of the minimal reproducibility sub-schema. Within that sub-schema, max_tokens is absent from 95.6% of triples and temperature from 93.9%.
Holds for: Public LLM evaluation reporting ingested as of June 4, 2026 from EEE converters (HELM, lm-eval-harness, Inspect AI), leaderboard scrapes and community contributions; the sub-schema checks only temperature and max_tokens, not seeds, hardware or determinism.
do the companies that build AI models write down less about how they tested them than outside testers do?
does first-party evaluation reporting populate fewer reproducibility fields than third-party reporting on the same model-benchmark pairs?
how do I tell whether a developer-reported score or an independent evaluator's score comes with more usable configuration detail?
if I need documented evaluation settings, should I take the number from the model developer or from an independent evaluator?
On 180 (model, benchmark) pairs reported by both first- and third-party evaluators, first-party rows populate 0.0% of the base reproducibility fields on average against 16.6% for third-party rows. The reproducibility gap is therefore widest in developer self-reporting.
Holds for: 180 paired first/third-party cases in the ingested corpus; base fields are temperature and max_tokens only; evaluator relationship is self-declared in the EEE evaluator_relationship field.
how much of the basic information about a benchmark is actually written down where people report scores from it?
what is the per-benchmark field population rate against an operationalized evaluation documentation schema, and which fields saturate at 100% or 0%?
how do I check which parts of a benchmark's documentation, like preregistration or lifecycle status, I can expect to find at all?
if I want benchmark metadata beyond the score itself, which fields will I realistically get and which will I have to chase down?
Median per-benchmark documentation completeness against the 28-field operationalized Evaluation Cards schema is 10.7% across 635 benchmarks. Per-field population runs from 100.0% for score and metric score_type down to 0.0% for preregistration_url and lifecycle_status.
Holds for: 635 benchmarks with warehouse completeness rows, of which 211 carry matched Auto-BenchmarkCards records; completeness measures artifact-side documentation adequacy only, and partial fields score as the fraction of sub-items populated.
is the same AI model usually tested on the same benchmark by more than one organization, and do their numbers match?
what fraction of (model, benchmark) pairs have multi-organization coverage, and how many metric groups exceed a cross-party score-divergence threshold?
how do I check whether a reported model score has been independently corroborated by another organization?
should I treat a single reported benchmark number as corroborated, or look for a second organization's run of the same evaluation?
Of 49,865 (model, benchmark) pairs in the Evaluation Cards corpus, 98.2% are reported by only one party, and among the 181 multi-organization metric groups 94 (51.9%) exceed the 5% cross-party score-divergence threshold.
Holds for: Ingested public reporting from 30 organizations as of June 4, 2026; the 5% threshold is applied uniformly on each metric's native scale and ignores sampling variance.
which types of AI tests are mostly scored by the companies that built the models, rather than by outsiders?
how does the rate of first-party-only reporting vary across benchmark categories such as agentic, general and safety?
how do I find out whether agentic or safety scores for a model have any independent reporting behind them?
for agentic capability claims, can I expect an independent evaluation to compare against, or only the developer's own numbers?
First-party-only reporting is most prevalent for agentic benchmarks (15.1%) and general benchmarks (12.5%), and least prevalent for safety benchmarks (0.8%). Independent reporting is thus scarcest in the categories where comparability problems would matter most.
Holds for: Benchmark categories assigned by LLM-assisted labelling from benchmark name only into an 18-category taxonomy, with human review and manual corrections; the ingested corpus overrepresents English-language benchmarks and frontier-scale models.
do different websites report different benchmark scores for GPT-5, and by how much?
how do reported GPT-5 results vary across reporting organizations on MATH-500, and how many lack decoding configuration fields?
how do I reconcile conflicting MATH-500 numbers reported for GPT-5 by different evaluation organizations?
which GPT-5 math benchmark number should I cite when several organizations report different ones?
In the GPT-5 Evaluation Cards profile, 202 of 213 documented results (95%) lack temperature or max_tokens, and 13% of results are first-party. MATH-500 is reported by 3 organizations with scores from 84.7% (LLM Stats) to 98.9% (Artificial Analysis).
Holds for: One model view rendered from the Evaluation Cards interface on June 4, 2026, covering 118 benchmarks and 19 reporting organizations; a single frontier model rather than a corpus-wide statistic.
do scores reported for the MMLU-Pro test disagree depending on who ran it?
what cross-party score divergence and missing-field rates appear when MMLU-Pro results are aggregated across reporting organizations?
how do I check whether an MMLU-Pro number for a model like Llama 3.2 is consistent across the sources that report it?
if I am comparing models on MMLU-Pro, can I mix numbers from different leaderboards?
Aggregating MMLU-Pro across 8 reporting organizations, 4,975 of 5,079 reported results (98%) lack a minimal reproducibility field. 6 model entries diverge across parties beyond the threshold, including Llama 3.2 at 20.9% from Hugging Face versus 61.8% from Arcadia Impact.
Holds for: One benchmark view rendered on June 4, 2026 over 401 models and 8 organizations; MMLU-Pro's own Auto-BenchmarkCards record populates 26 of 28 fields, so it is a well-documented benchmark rather than a typical one.
where do the rules for what an AI evaluation report should contain actually come from?
what corpus of recommendation items and stakeholder input was coded to derive the Evaluation Cards five-part reporting structure?
how do I justify the set of fields I ask evaluators to report, rather than inventing a checklist?
is the evaluation reporting structure grounded in prior recommendations, or is it one team's opinion about what to record?
The Evaluation Cards reporting framework is a five-part structure covering design, before execution, execution, lifecycle, and reporting and publication. It was derived from 730 recommendation items coded from 52 papers out of 748 screened candidates, plus 12 stakeholder interviews.
Holds for: Preregistered review of AI evaluation practice papers published 2020-2025; two independent coders reached Cohen's kappa in [0.865, 0.895] and Krippendorff's alpha in [0.916, 0.964]; interviewees recruited through author networks, mostly North America-based.
can model and test names collected from many different websites be matched up reliably to one canonical name?
what accuracy does the Evaluation Cards entity resolver achieve when canonicalizing model, benchmark and metric strings?
how do I merge evaluation results from several leaderboards when each one spells the model and benchmark names differently?
if I build on this aggregated evaluation corpus, how much name-matching error should I expect on benchmarks versus models?
The Evaluation Cards entity resolver reaches 98.3% accuracy on models, 77.4% on benchmarks and 86.7% on metrics, over 200 randomly sampled entities per type. The resolver maps observed model, benchmark and metric strings to canonical identifiers.
Holds for: In-domain EEE corpus data with manual labelling of each prediction, and matching rules were curated on the full dataset, so figures are not held-out; unresolved strings are retained rather than dropped.
why is saying "model X scores 0.99 on MATH" not enough to know what was actually measured?
how does a five-level rollout hierarchy of family, composite, benchmark, split and metric resolve an aggregate benchmark claim to its underlying metric?
how do I trace a headline benchmark number back to the specific split and metric it came from?
if I record evaluation results in my own database, is a flat model-benchmark-score row going to be enough?
Evaluation Cards replaces the flat (model, benchmark, score) triple with a five-level rollout hierarchy of family, composite, benchmark, split and metric. A claim such as "GPT-5 achieves 0.994 on MATH" resolves to MATH-family / artificial_analysis / MATH-500 / advanced-math / accuracy.
Holds for: LLM benchmarks ingested into Evaluation Cards; the hierarchy lets integrity signals attach to a (model, metric-path) pair rather than to a benchmark label, and depends on the canonicalization layer resolving name variants correctly.
can the same record of how a model was tested be shown both to a researcher and to someone without a technical background?
how do the Evaluation Cards reader modes surface or compress the same underlying record for research versus policy audiences?
how do I present missing evaluation configuration fields to a non-technical reader without dumbing down the underlying record?
do I need two separate documents for researchers and policy readers, or can one evaluation record serve both?
Evaluation Cards renders identical records through two reader modes, differing only in which fields are surfaced or compressed. Research mode lists the specific missing configuration fields, while the default summary mode states the same signal as "How this model was prompted during testing is not documented."
Holds for: Two modes derived from 12 practitioner interviews across technical, developer and policy roles; systematic usability evaluation is planned post-deployment rather than completed.
does a low documentation completeness number mean an AI model or its developer is bad?
does Evaluation Cards convert documentation completeness into letter grades, pass/fail thresholds or developer rankings?
how do I report gaps in evaluation documentation without turning them into a scorecard that penalizes developers?
if my model's card shows low completeness, is that a quality judgement I have to answer for?
Evaluation Cards assigns no letter grades, pass/fail thresholds or completeness rankings, surfacing omitted fields to readers instead of penalizing developers for them.
Holds for: All records in the interface; signal outputs are flags, missing-field lists and a completeness score in [0,1].
is there a project that pulls benchmark descriptions, evaluation run details and model metadata into one place?
which work composes benchmark metadata schemas, evaluation run records and model catalogs into a single reader-facing evaluation record?
where do I start reading if I want to build infrastructure for AI evaluation reporting rather than another documentation standard?
do I have to fill in yet another documentation template, or is there an integration layer that assembles evaluation records from what already exists?
Evaluation Cards composes three previously separate efforts into a single reader-facing record: Auto-BenchmarkCards for benchmark metadata, EEE for evaluation run data, and community model catalogs. It is an integration layer rather than another documentation standard for evaluators to fill in by hand.
Holds for: As of the June 2026 preprint, and by the comparison in Table 1 against Datasheets, Data Cards, Model Cards, BenchmarkCards, Audit Cards, Eval Factsheets, SPHERE, STREAM, HELM, Inspect, Open LLM Leaderboard, EEE and BetterBench; covers LLM evaluation only.
Evaluation Cards is a deployed monitoring instrument for the state of public AI evaluation reporting, applied to 5,816 models, 635 benchmarks and 101,955 reported results from 30 organizations. Its code is open and the interface is live and hosted.
Holds for: LLM evaluation reporting as of June 4, 2026; the corpus is what EEE, leaderboard scrapes and community contributions supply rather than a census of public reporting.
is anyone actually keeping track of how well AI evaluation results are being reported across the whole field?
what deployed instrument monitors reporting completeness and cross-party divergence across thousands of models, benchmarks and reported results?
how do I get an at-scale picture of evaluation reporting gaps without auditing each leaderboard by hand?
is there a live, open tool I can point at to show how public AI evaluation reporting currently stands?
Evaluation Cards is a deployed monitoring instrument for the state of public AI evaluation reporting, applied to 5,816 models, 635 benchmarks and 101,955 reported results from 30 organizations. Its code is open and the interface is live and hosted.
Holds for: LLM evaluation reporting as of June 4, 2026; the corpus is what EEE, leaderboard scrapes and community contributions supply rather than a census of public reporting.

Claims and scope

Common misreadings

Terminology in this paper

rollout hierarchy
A five-level structure — family, composite, benchmark, split, metric — through which every reported evaluation score resolves to an explicit traceable path instead of a flat (model, benchmark, score) triple.
minimal reproducibility sub-schema
The small set of fields required to re-run a reported evaluation: temperature and max_tokens, extended with harness, eval_plan and eval_limits for agentic evaluations.
reporting completeness
The share of a 28-field operationalized reporting schema populated for a benchmark, where fields with sub-items score as the fraction of sub-items present.
provenance signal
An annotation on a reported score stating whether the evaluator was first-party, third-party or collaborative, whether any other party reported the same score, and which risk categories the benchmark carries.
comparability flag
A warning raised when reported scores for the same model, benchmark and metric differ by more than 5% of the metric's range, either across setup variants or across reporting parties.
reader modes
Two renderings of one identical evaluation record — a research mode foregrounding methodology and configuration, and a summary mode foregrounding plain-language accountability — differing only in which fields are surfaced, compressed or reframed.

How to cite

@misc{ghosh2026evaluationcardsinterpretivelayer,
      title={Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting}, 
      author={Avijit Ghosh and Anka Reuel and Jenny Chim and Wm. Matthew Kennedy and Srishti Yadav and Jennifer Mickel and Yanan Long and Andrew Tran and Anastassia Kornilova and Damian Stachura and Kevin Klyman and Felix Friedrich and Jeba Sania and Jan Batzner and Anoop Mishra and Eliya Habba and Yixiong Hao and Nathan Heath and Shalaleh Rismani and Usman Gohar and Andrea Loehr and David Manheim and Ruchira Dhar and Sree Harsha Nelaturu and Aarush Sinha and Leshem Choshen and Drishti Sharma and Ishan Khire and Amit Saha and Subramanyam Sahoo and Michael Hardy and Michael Alexander Riegler and Kabir Manghnani and Michelle Lin and Yanan Jiang and Yilin Huang and Asaf Yehudai and Jessica Ji and Aris Hofmann and Mubashara Akhtar and Max Lamparth and Nuno Moniz and Yacine Jernite and Stella Biderman and Zeerak Talat and Sanmi Koyejo and Mykel Kochenderfer and Irene Solaiman},
      year={2026},
      eprint={2606.09809},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.09809}, 
}

References

See the full reference list in the paper.