TextArena

a suite of text-based games for benchmarking and training language-model agents through competitive play

Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, Cheston Tan · arXiv · 2025

In one sentence

TextArena is an open-source, Gym-style collection of 57+ competitive text games — single-, two- and multi-player — that evaluates and trains LLM agents on social skills like negotiation, theory of mind and deception through relative, TrueSkill-rated online play against other models and humans.

Abstract

TextArena is an open-source collection of competitive text-based games for training and evaluation of agentic behavior in Large Language Models (LLMs). It spans 57+ unique environments (including single-player, two-player, and multi-player setups) and allows for easy evaluation of model capabilities via an online-play system (against humans and other submitted models) with real-time TrueSkill scores. Traditional benchmarks rarely assess dynamic social skills such as negotiation, theory of mind, and deception, creating a gap that TextArena addresses. Designed with research, community and extensibility in mind, TextArena emphasizes ease of adding new games, adapting the framework, testing models, playing against the models, and training models. Detailed documentation of environments, games, leaderboard, and examples are available on https://github.com/LeonGuertler/TextArena and https://www.textarena.ai/.

Questions this paper answers

where can I read about testing language models by making them play games against each other instead of answering quiz questions?
which work motivates competitive-play relative rankings over saturating static LLM benchmarks?
how do I evaluate a language model's negotiation and deception ability when quiz-style benchmarks give me nothing to measure?
my model tops the usual multiple-choice benchmarks, so what should I read to start evaluating it in interactive games?
TextArena argues that relative, competitive-play rankings are a more sustainable evaluation paradigm than absolute benchmarks. Saturating scores on MMLU, HumanEval and ARC-AGI leave no headroom, while a ranking survives as long as models differ in capability.
Holds for: An argument positioned against Chatbot Arena's human-preference voting, as of the April 2025 report; the two paradigms' validity is not empirically compared.
TextArena targets dynamic social capabilities that static question-answering benchmarks such as MMLU and HumanEval do not probe. The environments are built to elicit negotiation, persuasion, deception and theory of mind in interactive multi-agent play.
Holds for: Text-based games only, in English as released; what the environments are designed to elicit, not a validated measurement of these constructs.
TextArena's initial release ships 16 single-player, 47 two-player and 11 multi-player text-based environments. That is more in every category than the 8 prior game-based LLM benchmarks compared against, including SPIN-Bench (21/3/2) and GameBench (0/3/6).
Holds for: Counts as of the paper's initial release; a footnote states the collection had grown to 74 games by publication, and some listed games are marked as not yet fully implemented.
how many text games can a language model be tested on in TextArena, and how many players do they take?
what is the environment count of TextArena by single-player, two-player and multi-player category relative to other game-based LLM suites?
how do I find a text-game suite with enough single-player, head-to-head and group games to cover a whole evaluation?
if I want breadth of games rather than a handful, is TextArena bigger than the other LLM game benchmarks?
TextArena's initial release ships 16 single-player, 47 two-player and 11 multi-player text-based environments. That is more in every category than the 8 prior game-based LLM benchmarks compared against, including SPIN-Bench (21/3/2) and GameBench (0/3/6).
Holds for: Counts as of the paper's initial release; a footnote states the collection had grown to 74 games by publication, and some listed games are marked as not yet fully implemented.
TextArena is the only framework among the 9 compared game-based LLM benchmarks that supports all four of a Gym-compatible API, online evaluation, model-vs-model play and model-vs-human play.
Holds for: Clembench, LMRL-Gym, GameBench, Game-theoretic LLM, LAMEN, GTBench, GameArena and SPIN-Bench as characterised by the authors, on those 4 capability dimensions only.
how does a text-game leaderboard turn win-loss records between language models into a skill number?
why is TrueSkill rather than Elo used to rate LLMs from competitive text-game match outcomes?
how do I get a stable skill rating for my model from a limited number of games against other models?
should I rate my models with TrueSkill or Elo if I want a reliable ranking from few matches?
TextArena rates models with TrueSkill (initialised at mu=25, sigma=25/3) rather than Elo, and reports that in the authors' experiments TrueSkill converged faster to a reliable skill estimate than Elo.
Holds for: Online matches with varying player counts and team play; the convergence comparison is reported without numbers or a described protocol.
TextArena's online system had evaluated 283 models at the time of writing, including community submissions and 64 official models hosted by the platform for free play.
Holds for: Counts as of the paper's writing on a live, continuously updating leaderboard; submissions are unrestricted, so entries can include repeated model variants.
can a language model's game rating be compared with how well actual people play the same games?
how does TextArena produce a human baseline rating on the same scale as model ratings for model-vs-human play?
how do I check whether my model actually beats people at negotiation or bluffing games rather than just other models?
can I put my model up against human opponents and see a single human rating to compare against?
Human players in TextArena are pooled into a single leaderboard entry called "Humanity", giving frontier models a directly comparable opponent rating rather than a static human score.
Holds for: Human ratings are aggregated collectively rather than per player, so the entry reflects the mixed skill of whoever chose to play online.
TextArena is the only framework among the 9 compared game-based LLM benchmarks that supports all four of a Gym-compatible API, online evaluation, model-vs-model play and model-vs-human play.
Holds for: Clembench, LMRL-Gym, GameBench, Game-theoretic LLM, LAMEN, GTBench, GameArena and SPIN-Bench as characterised by the authors, on those 4 capability dimensions only.
can a text-game score be split up to show whether a model is good at bluffing versus reading other players?
how are per-skill aptitudes such as theory of mind and persuasion derived from environment-level ratings and skill tags?
how do I find out which social abilities my model is weak at rather than just its overall rank?
can I get a skill breakdown for my model instead of one leaderboard position?
TextArena estimates a model's aptitude in each of 10 soft skills as a weighted average of its ratings on the environments tagged with that skill. Each environment carries up to 5 skill tags, among them Theory of Mind, Bluffing, Persuasion and Uncertainty Estimation.
Holds for: Skill tags and weights are assigned by the authors, not measured; per-skill scores in the published radar figure are each normalised separately for presentation, so cross-skill magnitudes are not comparable.
if a model loses a text game, does that mean it plays badly or that it did not follow the rules?
do TextArena's preliminary rankings conflate strategic play with rule and output-format compliance?
how do I tell whether my model's poor game results come from weak strategy or from failing to obey the game format?
should I trust an early text-game leaderboard position as a measure of my model's reasoning?
TextArena's preliminary model rankings conflate playing skill with rule and format comprehension, and some reasoning models were observed revealing their own cards or hidden roles during play.
Holds for: Preliminary rankings over a subset of models and games in a 5-page work-in-progress report; how much ranking variance the confound explains is not quantified.
can games between language models generate the training data needed to make them better at multi-step reasoning?
can competitive self-play in text games serve as a reinforcement-learning signal with an adaptive difficulty curriculum?
how do I generate multi-turn interaction data for RL training without writing a new reward function for every task?
is self-play in text games worth using as an RL data source for my agent?
TextArena positions competitive self-play in text games as a near-infinite reinforcement-learning data source with a difficulty curriculum that adapts as agents improve, aimed at multi-turn agentic reasoning rather than single-turn answer quality.
Holds for: A stated design motivation and future direction as of the April 2025 report; the paper trains no models and reports no RL results.
TextArena keeps its interface close to OpenAI Gym with stackable wrappers, so a full two-model match across several games runs in about 10 lines of Python via ta.make, get_observation and step.
Holds for: The offline API shown in the paper's example script; online play uses ta.make_online with model name, description and email, and the code setup may change over time.
how much code does it take to set up a match between two language models in a text game?
does TextArena expose an OpenAI Gym-style API with stackable wrappers for model-vs-model episodes?
how do I run a head-to-head match between two models across several games without building the harness myself?
will TextArena drop into my existing Gym-based evaluation code?
TextArena keeps its interface close to OpenAI Gym with stackable wrappers, so a full two-model match across several games runs in about 10 lines of Python via ta.make, get_observation and step.
Holds for: The offline API shown in the paper's example script; online play uses ta.make_online with model name, description and email, and the code setup may change over time.
TextArena is the only framework among the 9 compared game-based LLM benchmarks that supports all four of a Gym-compatible API, online evaluation, model-vs-model play and model-vs-human play.
Holds for: Clembench, LMRL-Gym, GameBench, Game-theoretic LLM, LAMEN, GTBench, GameArena and SPIN-Bench as characterised by the authors, on those 4 capability dimensions only.
how many language models have already been scored on a competitive text-game leaderboard?
what is the size of the evaluated model pool on TextArena's online leaderboard, including community submissions?
if I submit my model, how many other models will it already have ratings to be compared against?
TextArena's online system had evaluated 283 models at the time of writing, including community submissions and 64 official models hosted by the platform for free play.
Holds for: Counts as of the paper's writing on a live, continuously updating leaderboard; submissions are unrestricted, so entries can include repeated model variants.
which abilities does a quiz-style benchmark like MMLU miss that showing up in games with other players would reveal?
why target dynamic multi-agent social capabilities instead of building harder static question-answering benchmarks?
how do I measure negotiation, persuasion and theory of mind in a model when the standard benchmarks only test knowledge?
is it worth moving to interactive game evaluation for my model, or should I just find a harder QA benchmark?
TextArena targets dynamic social capabilities that static question-answering benchmarks such as MMLU and HumanEval do not probe. The environments are built to elicit negotiation, persuasion, deception and theory of mind in interactive multi-agent play.
Holds for: Text-based games only, in English as released; what the environments are designed to elicit, not a validated measurement of these constructs.
TextArena argues that relative, competitive-play rankings are a more sustainable evaluation paradigm than absolute benchmarks. Saturating scores on MMLU, HumanEval and ARC-AGI leave no headroom, while a ranking survives as long as models differ in capability.
Holds for: An argument positioned against Chatbot Arena's human-preference voting, as of the April 2025 report; the two paradigms' validity is not empirically compared.

Claims and scope

Common misreadings

Terminology in this paper

Humanity
A single pooled leaderboard entry in TextArena representing all human players collectively, so that models can be rated against human play as if against one opponent.
Soft-skill profiling
Estimating a model's aptitude in each of 10 named skill categories by taking a weighted average of its ratings across environments tagged with that skill.
Online play
A mode in which a submitted model is matched over the network against other submitted models, platform-hosted models, or human players, with TrueSkill ratings updated after every match.
Relative evaluation
Scoring models by outcomes against each other rather than against a fixed answer key, so that no maximum attainable score exists as long as models differ in capability.

How to cite

@article{guertler2025textarena,
  title={TextArena},
  author={Guertler, Leon and Cheng, Bobby and Yu, Simon and Liu, Bo and Choshen, Leshem and Tan, Cheston},
  journal={arXiv preprint arXiv:2504.11442},
  year={2025}
}

References

See the full reference list in the paper.