CUBE: A Standard for Unifying Agent Benchmarks

Common Unified Benchmark Environments — a wrap-once protocol so any agent benchmark runs on any evaluation or RL platform

Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko, Aman Jaiswal, Kusha Sareen, Shailesh Nanisetty, Joan Cabezas, Manuel Del Verme, Omar G. Younis, Simone Baratta, Matteo Avalle, Imene Kerboua, Xing Han Lù, Elron Bandel, Michal Shmueli-Scheuer, Asaf Yehudai, Leshem Choshen, Jonathan Lebensold, Sean Hughes, Massimo Caccia, Alexandre Drouin, Siva Reddy, Tao Yu, Yu Su, Graham Neubig, Dawn Song · arXiv · 2026

In one sentence

CUBE is a proposed protocol standard fusing MCP tool calls with Gym-style reset/step/evaluate semantics, splitting task, benchmark, package and registry concerns into separate API layers so an agent benchmark is wrapped once and usable by any compliant evaluation or RL-training platform.

Abstract

The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating an"integration tax"that limits comprehensive evaluation. We propose CUBE (Common Unified Benchmark Environments), a universal protocol standard built on MCP and Gym that allows benchmarks to be wrapped once and used everywhere. By separating task, benchmark, package, and registry concerns into distinct API layers, CUBE enables any compliant platform to access any compliant benchmark for evaluation, RL training, or data generation without custom integration. We call on the community to contribute to the development of this standard before platform-specific implementations deepen fragmentation as benchmark production accelerates through 2026.

Questions this paper answers

is there a common interface that lets an AI agent benchmark run on any evaluation platform without custom glue code?
what interface contract does CUBE specify for agentic benchmarks across task, benchmark, package and registry layers?
how do I make my agent benchmark runnable by many harnesses without writing a wrapper for each one?
should I package my agent benchmark against a shared standard instead of maintaining per-platform adapters?
CUBE is a position paper naming the "integration tax" as the bottleneck that limits which labs can evaluate agents broadly. The tax is the per-benchmark custom wrapper work that N agents against M benchmarks demands, and CUBE's proposed fix is a single benchmark-side interface contract.
Holds for: A position paper from March 2026 arguing for a standard; it reports no agent accuracy results and no measured reduction in engineering effort. Concurrent with AgentBeats' Agentified Agent Assessment paradigm, which shares the motivation but differs in scope and design.
CUBE separates agent-benchmark integration into 4 distinct API layers: task-level interaction, benchmark-level discovery and spawning, package-level installation and parallelization, and a registry-level metadata catalog. A benchmark author implements Python classes once and CUBE generates the RPC layer automatically.
Holds for: Describes the proposed contract as of the v1 draft; the reference implementation is labelled alpha and interfaces may change. The layer split is argued on design grounds, not validated by a user study or an integration-effort measurement.
CUBE's task-level API keeps MCP's tools/list, tools/call, resources/list and resources/read unchanged, and adds cube/evaluate, cube/reset, cube/step, cube/close and cube/privileged_info. The result is a superset from which both plain MCP and plain Gym can be recovered, plus a non-blocking async-Gym pattern.
Holds for: MCP is chosen for actions because its non-blocking tools/call and tool-signature discovery suit benchmarks such as ARE and GAIA-2 that require agents to coordinate long-running operations; the traditional blocking Gym step function cannot express those patterns.
which write-up argues that setting up agent benchmarks is too much work and proposes a shared standard?
what position paper frames the integration cost of agentic evaluation environments and its relation to existing agent platforms?
where do I start reading about why integrating agentic benchmarks costs so much engineering effort?
I want the background argument before committing my team to an agent evaluation standard — what should I read?
CUBE is a position paper naming the "integration tax" as the bottleneck that limits which labs can evaluate agents broadly. The tax is the per-benchmark custom wrapper work that N agents against M benchmarks demands, and CUBE's proposed fix is a single benchmark-side interface contract.
Holds for: A position paper from March 2026 arguing for a standard; it reports no agent accuracy results and no measured reduction in engineering effort. Concurrent with AgentBeats' Agentified Agent Assessment paradigm, which shares the motivation but differs in scope and design.
CUBE positions existing agent platforms as complementary rather than rivals, and targets the benchmark packaging and infrastructure lifecycle layer that none of them specifies. Each compared platform evolved from its own niche: NeMo Gym from RL training, AgentBeats from competition infrastructure, OpenEnv from the HuggingFace ecosystem, Harbor from SWE evaluation and HAL from academic benchmarking.
Holds for: Platform features and documentation available in March 2026. The paper states an AgentBeats judge agent could consume a CUBE benchmark through a thin connector, but reports no such connector as built.
how many benchmarks for AI agents exist today, and is that number still climbing?
what is the current count and projected growth of agentic benchmarks?
is the agent benchmark landscape stable enough to pick a few, or will it keep expanding on me?
Over 300 agentic benchmarks exist, many of them innovative but largely unknown because they are too difficult to set up. CUBE's authors forecast that number to double by the end of 2026.
Holds for: A community count and an authors' projection as of March 2026, not a systematic survey with an enumerated list; the doubling is attributed to coding-agent and RL post-training activity.
why bolt a tool-calling protocol onto the usual reset-and-step loop instead of picking one of them?
why does CUBE extend MCP with Gym-style reset, step and evaluate methods rather than using Gym alone?
how do I let an agent make tool calls that do not block while still driving an episode step by step?
if my agent already speaks MCP, do I have to rewrite it to use a Gym-style benchmark API?
CUBE's task-level API keeps MCP's tools/list, tools/call, resources/list and resources/read unchanged, and adds cube/evaluate, cube/reset, cube/step, cube/close and cube/privileged_info. The result is a superset from which both plain MCP and plain Gym can be recovered, plus a non-blocking async-Gym pattern.
Holds for: MCP is chosen for actions because its non-blocking tools/call and tool-signature discovery suit benchmarks such as ARE and GAIA-2 that require agents to coordinate long-running operations; the traditional blocking Gym step function cannot express those patterns.
how do you run benchmarks that need one shared web server or virtual machine behind all their tasks?
how is benchmark-level shared infrastructure lifecycle handled for environments like WebArena's micro-internet or OSWorld's desktop VM?
how do I spin up a persistent VM or server once and hand each task a handle to it?
can I run WebArena or OSWorld through a standard benchmark interface, or does shared infrastructure break it?
CUBE's benchmark-level layer handles benchmarks whose infrastructure is shared across tasks, such as WebArena's persistent GitLab, e-commerce and forum micro-internet or OSWorld's full desktop OS. Those resources are initialized once in Benchmark.setup() and each spawned task receives a RuntimeContext.
Holds for: OpenEnv requires the user to provision shared infrastructure externally via environment variables, and Harbor's per-task container model has no cross-task shared-infrastructure lifecycle, which is reflected in its adapter catalogue excluding WebArena and OSWorld.
Four popular agent benchmarks disagree on nearly every infrastructure axis: SWE-bench uses per-task Docker containers, WebArena and OSWorld need benchmark-level VMs, and GAIA ships static files with no tools at all. OSWorld's scaling bottleneck is 20 GB+ of RAM per agent plus heavy disk I/O for state resets.
Holds for: A 4-benchmark comparison (WebArena, SWE-bench, OSWorld, GAIA) chosen to illustrate heterogeneity in hosting format, action space, integration effort and scaling bottleneck; not an exhaustive benchmark survey.
what actually differs between agent benchmarks that makes each one so much work to set up?
how do SWE-bench, WebArena, OSWorld and GAIA differ on containerization, VM requirements and hardware footprint?
how do I budget RAM, disk and container setup for running several different agent benchmarks?
how much machine do I need per agent to run a desktop-OS benchmark like OSWorld?
Four popular agent benchmarks disagree on nearly every infrastructure axis: SWE-bench uses per-task Docker containers, WebArena and OSWorld need benchmark-level VMs, and GAIA ships static files with no tools at all. OSWorld's scaling bottleneck is 20 GB+ of RAM per agent plus heavy disk I/O for state resets.
Holds for: A 4-benchmark comparison (WebArena, SWE-bench, OSWorld, GAIA) chosen to illustrate heterogeneity in hosting format, action space, integration effort and scaling bottleneck; not an exhaustive benchmark survey.
can a grading harness see the answer key or the scoring code to explain why an agent failed?
how does CUBE expose privileged environment state and ground truth to LLM judges and to policy distillation?
how do I get ground-truth answers and evaluator source code out of a benchmark to diagnose agent failures?
I want my judge to explain agent failures more reliably — can a benchmark hand me the grading code?
CUBE standardizes an optional info field carrying privileged information at both task and step level, such as the evaluation function's source code, ground-truth answers, or summaries of internal environment state. Its stated purposes are better LLM-judge failure diagnosis and privileged policy distillation during training.
Holds for: The field is curated by benchmark designers and optional, so its content varies by benchmark; the paper motivates it from documented LLM-judge limitations but reports no measurement of judge accuracy with versus without it.
how can you tell an agent benchmark still works without paying for a language model run?
how does CUBE support deterministic end-to-end episode testing of a benchmark package in CI?
how do I add a benchmark to continuous integration and assert a full episode still scores correctly?
can I regression-test my agent environment on every commit without LLM API calls?
Every CUBE package must expose get_debug_task_configs() and make_debug_agent(task_id), a scripted agent guaranteed to solve a debug task. Any consumer can then run a full episode end-to-end and assert the reward reaches 1.0 without calling a live language model.
Holds for: Makes benchmarks testable in standard CI pipelines; it checks that the harness plumbing and reward path work, not that the benchmark's task difficulty or evaluation logic is well designed.
does a catalogue of agent benchmarks store the benchmarks themselves, or just point at where they live?
what metadata fields does the CUBE Registry record, and does it host benchmark code or data?
how do I find an agent benchmark whose runtime, hardware needs and licence fit what I can run?
before I list my benchmark in a registry, what do I have to declare about licences and copyrighted content?
The CUBE Registry indexes metadata only and hosts no benchmark code or data, pointing at PyPI via a package field. The recorded metadata covers runtime type (docker, apptainer, vm, docker-root, docker-in-docker, live), hardware needs, task count, separate package and benchmark licenses, and a content notice for cloned or copyrighted material.
Holds for: Registration triggers a GitHub compliance-verification job; the design intent is that authors retain control of distribution and that researchers can filter out benchmarks incompatible with their infrastructure before installing anything.
how many environments does each of the agent evaluation platforms cover, and how do they relate to each other?
how does CUBE's wrapped-benchmark coverage compare with NeMo Gym, AgentBeats, OpenEnv, Harbor and HAL?
how do I choose between the available agent evaluation platforms when I need broad benchmark coverage?
if another platform already wraps far more environments, why would I adopt a benchmark packaging standard as well?
CUBE had 9 wrapped CUBEs at the time of writing, an explicitly early-stage catalogue. The compared platforms list 40+ environments in NeMo Gym, 250+ benchmarks across 17 domains in AgentBeats, 30+ in OpenEnv and 46+ adapters in Harbor.
Holds for: Counts as documented in March 2026, which move quickly. CUBE is the only entry in the comparison whose stated coverage includes shared-infrastructure and VM-based benchmarks.
CUBE positions existing agent platforms as complementary rather than rivals, and targets the benchmark packaging and infrastructure lifecycle layer that none of them specifies. Each compared platform evolved from its own niche: NeMo Gym from RL training, AgentBeats from competition infrastructure, OpenEnv from the HuggingFace ecosystem, Harbor from SWE evaluation and HAL from academic benchmarking.
Holds for: Platform features and documentation available in March 2026. The paper states an AgentBeats judge agent could consume a CUBE benchmark through a thin connector, but reports no such connector as built.
do the people proposing a new agent benchmark standard think their design is the correct one?
what design objections to CUBE's layered RPC and MCP-plus-Gym API do its own authors concede?
I would rather have explicit async primitives and fewer abstraction layers — has that objection to CUBE been addressed?
CUBE's authors state explicitly that their design is not claimed to be optimal, only that some standard is necessary. Objections they list a critic might prefer include explicit async primitives, fewer abstraction layers, pure Gym without MCP, and message-passing over RPC.
Holds for: A draft standard put forward for community feedback in March 2026; the listed alternative designs are not evaluated against CUBE's own.
how does a new benchmark standard get off the ground when platforms and benchmark authors each wait for the other to move first?
what adoption strategy does CUBE propose to break the two-sided network effect between platforms and benchmark authors?
how do I get a proposed evaluation standard adopted when neither side will move first?
which platforms are committing to CUBE connectors, and by when should I expect to see them?
CUBE's adoption plan addresses a two-sided deadlock in which platforms wait for compliant benchmarks while benchmark authors wait for platform demand. The plan recruits early platform supporters to build reference connectors for NVIDIA NeMo Gym and OpenEnv, and targets critical mass by the end of 2026.
Holds for: A plan and a target date stated in March 2026, not an achieved outcome; the initial corpus spans web navigation, software engineering and desktop environments, the reference implementation is alpha and the connectors are proposed deliverables.

Claims and scope

Common misreadings

Terminology in this paper

Integration Tax
The redundant engineering cost of writing a separate driver or wrapper for every agent-benchmark pair, so that evaluating one agent on five benchmarks means five unique wrappers, and switching evaluation frameworks means starting over.
CUBE
Common Unified Benchmark Environments: a protocol standard defining task-, benchmark-, package- and registry-level interfaces that an agent benchmark implements once to become usable by any compliant evaluation or RL-training platform.
a CUBE
A single benchmark wrapped to comply with the CUBE standard, distributed as a Python package and indexed in the CUBE registry.
async-Gym API
An interaction pattern obtained by fusing MCP's non-blocking tools/call with Gym-style reward and termination semantics, letting an agent issue multiple concurrent long-running tool calls instead of waiting inside a blocking step function.
privileged information
Optional context exposed to the evaluation harness but not the agent — evaluation function source code, ground-truth answers, or summaries of internal environment state — used for judge-based failure diagnosis and privileged policy distillation.
debug agent
A scripted, non-LLM agent shipped with a benchmark package that is guaranteed to solve a designated debug task, so a full episode can be run in CI and asserted to reach a reward of 1.0.
compliance badge
A registry-visible marker earned by a benchmark that passes a stress suite verifying idempotent resets, isolation between concurrent task instances, and resource usage within declared bounds.

How to cite

@misc{lacoste2026cube,
  title={CUBE: A Standard for Unifying Agent Benchmarks},
  author={Alexandre Lacoste and Nicolas Gontier and Oleh Shliazhko and Aman Jaiswal and Kusha Sareen and Shailesh Nanisetty and Joan Cabezas and Manuel Del Verme and Omar G. Younis and Simone Baratta and Matteo Avalle and Imene Kerboua and Xing Han Lù and Elron Bandel and Michal Shmueli-Scheuer and Asaf Yehudai and Leshem Choshen and Jonathan Lebensold and Sean Hughes and Massimo Caccia and Alexandre Drouin and Siva Reddy and Tao Yu and Yu Su and Graham Neubig and Dawn Song},
  year={2026},
  url       = {https://arxiv.org/abs/2603.15798},
  doi       = {10.48550/ARXIV.2603.15798},
  publisher = {arXiv}
}

References

See the full reference list in the paper.