The Mighty ToRR: A Benchmark for Table Reasoning and Robustness

Shir Ashury-Tahan; Yifan Mai; Rajmohan C; Ariel Gera; Yotam Perlitz; Asaf Yehudai; Elron Bandel; Leshem Choshen; Eyal Shnarch; Percy Liang; Michal Shmueli-Scheuer

doi:10.48550/arXiv.2502.19412

The Mighty ToRR: A Benchmark for Table Reasoning and Robustness

a benchmark for Table Reasoning and Robustness: table tasks scored across many equivalent table formats

Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer · SurgLLM@ICML · 2025

In one sentence

ToRR evaluates 14 LLMs on 10 table reasoning datasets under 35 semantically equivalent prompt configurations — 7 serializations crossed with 4 structural perturbations — and shows that table performance tops out around 0.50 while every model's score swings widely across formats that carry identical information.

Abstract

Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to adopt. To address this gap, we create ToRR, a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks. The benchmark includes 10 datasets that cover different types of table reasoning capabilities across varied domains. ToRR goes beyond model performance rankings, and is designed to reflect whether models can handle tabular data consistently and robustly, across a variety of common table representation formats. We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR. Our results reveal a striking pattern of brittle model behavior, where even strong models are unable to perform robustly on tabular data tasks. Although no specific table format leads to consistently better performance, we show that testing over multiple formats is crucial for reliably estimating model capabilities. Moreover, we show that the reliability boost from testing multiple prompts can be equivalent to adding more test examples. Overall, our findings show that table understanding and reasoning tasks remain a significant challenge.

Questions this paper answers

how good are today's chatbots at answering questions about spreadsheets and tables?
what overall accuracy do frontier LLMs reach on tabular reasoning tasks, and how far apart are models within one family?
how do I find out whether any available LLM is accurate enough to run question answering over my data tables?
if I need table question answering, does it matter much which top model I pick?: On ToRR, the best-performing LLMs (claude-3-5-sonnet, gpt-4o and deepseek-v3) reach an overall performance score of only 0.50. The weakest evaluated model, llama-3-1-8b-instruct, reaches 0.29.
Holds for: 14 models across 7 families, 10 table datasets, 100 sampled examples per dataset, 5-shot greedy decoding; scores averaged over 35 prompt configurations.; Performance differences between models within the same family on ToRR average 0.07, and paired Cohen's d shows most model comparisons on ToRR have small, often non-significant practical differences.
Holds for: 14 models on 10 table datasets over 35 prompt configurations; aggregated p-values still find all pairwise comparisons significant.
does writing a table as CSV instead of HTML change how well a language model answers questions about it?
how sensitive is LLM table reasoning accuracy to the serialization format of the input table?
how do I tell whether my table formatting choice is costing me accuracy on a table QA pipeline?
should I worry that switching my tables from markdown to JSON will change my model's answers?: Every model evaluated on ToRR is unrobust: across the 35 semantically equivalent prompt configurations of one example, the minimum and maximum scores give entirely different estimates of performance. Robustness scores run from 0.49 for mixtral-8x7b-instruct to 0.70 for claude-3-5-sonnet.
Holds for: Robustness is 1 minus the mean per-example score range over 7 serializations x 4 structural perturbations plus 7 unperturbed variants; tables embedded directly in the prompt, no tool use.; No table serialization format consistently wins on ToRR: aggregated over all models and datasets no serializer outperforms the others, and per-model serializer preferences shift overall performance by at most 0.06.
Holds for: 7 serializations — HTML, CSV, JSON, Markdown, Indexed Row Major, DataFrame and Concatenation — measured by example-level win rate over 14 models and 10 datasets; individual model-dataset pairs still vary much more.; On ToRR the gap between a model's best and worst serializer on a single dataset averages about 0.05. It reaches 0.22 for llama-3-1-8b-instruct on TableBench FC, and 0.23 for llama-3-1-405b-instruct on the same dataset.
Holds for: Largest max-minus-min score difference across the 7 serializations, computed per model-dataset pair over 14 models and 10 datasets.
what is the best way to write a table into a prompt so a language model understands it?
is there a serialization format for tabular input that dominates across models and table reasoning datasets?
which table format should I use when I feed tables into an LLM prompt?
can I just pick one table format for all my models, or do I have to test each one?: No table serialization format consistently wins on ToRR: aggregated over all models and datasets no serializer outperforms the others, and per-model serializer preferences shift overall performance by at most 0.06.
Holds for: 7 serializations — HTML, CSV, JSON, Markdown, Indexed Row Major, DataFrame and Concatenation — measured by example-level win rate over 14 models and 10 datasets; individual model-dataset pairs still vary much more.; On ToRR the gap between a model's best and worst serializer on a single dataset averages about 0.05. It reaches 0.22 for llama-3-1-8b-instruct on TableBench FC, and 0.23 for llama-3-1-405b-instruct on the same dataset.
Holds for: Largest max-minus-min score difference across the 7 serializations, computed per model-dataset pair over 14 models and 10 datasets.
if I shuffle the rows of a table or flip it sideways, do language models get more answers wrong?
what effect do structural table perturbations such as row and column swapping or transposition have on LLM task scores?
do I need to normalize row order and orientation in my tables before sending them to an LLM?
is it worth cleaning up transposed tables and blank rows in my data before running a model on them?: Structural table perturbations in ToRR have no consistent direction of effect, changing model scores by an average of 0.03 relative to the unperturbed baseline. The 4 perturbations are row swapping, column swapping, transposition and adding empty rows.
Holds for: 4 perturbations applied on top of each of 7 serializations, 14 models, 10 datasets; per-example absolute impact is larger, and more pronounced for smaller models and for Table QA and fact-checking datasets than for Table-to-Text.
can you trust a leaderboard that tested every model with the same single prompt?
how many prompt configurations are needed before model rankings become stable in Kendall's W agreement?
how many prompt variants should I evaluate over to get a model ranking that does not flip?
I ranked a few models with one prompt template, should I redo it with more?: Model rankings derived from a single table prompt configuration agree poorly with one another on ToRR, so a benchmark that fixes one serialization format yields an unreliable ranking of models.
Holds for: Kendall's W over 30 sampled prompt-configuration sets, 14 models, 100 examples per dataset; table serializations and structural perturbations only, not instruction wording.; Increasing the number of table prompt configurations from 1 to 10 raises Kendall's W ranking agreement on ToRR by more than 0.35 on average. The largest gains fall between roughly 2 and 8 configurations.
Holds for: Averaged over 30 sampled configuration sets and over ToRR's datasets; per-dataset gains differ sharply, FinQA rising from 0.35 to 0.93 with 11 prompts while NumericNLG rises only from 0.29 to 0.54.
is it better to test a model on more examples or on more versions of the same prompt?
can additional prompt configurations substitute for test-set size in achieving reliable model rankings?
I have a limited evaluation budget for table tasks, how should I split it between examples and prompt variants?
should I spend my annotation budget labelling more test examples or writing more prompt formats?: Adding prompt configurations can substitute for test examples: on ToRR, 50 examples evaluated with 2 prompt configurations give about the same model-ranking reliability as 100 examples with 1 configuration.
Holds for: Kendall's W over 30 randomly sampled example-and-configuration sets, averaged per dataset, on table reasoning tasks at these small sample sizes.; Increasing the number of table prompt configurations from 1 to 10 raises Kendall's W ranking agreement on ToRR by more than 0.35 on average. The largest gains fall between roughly 2 and 8 configurations.
Holds for: Averaged over 30 sampled configuration sets and over ToRR's datasets; per-dataset gains differ sharply, FinQA rising from 0.35 to 0.93 with 11 prompts while NumericNLG rises only from 0.29 to 0.54.
is there a benchmark that checks both whether models can read tables and whether their answers survive reformatting?
which table reasoning benchmark treats robustness across semantically equivalent prompts as a reported metric rather than a fixed prompt leaderboard?
where should I start reading about how table formatting affects language model evaluation?
which table benchmark should I cite if I want to argue single-prompt leaderboards are unreliable?: ToRR is a table reasoning benchmark that measures robustness as a first-class quantity, pairing 10 datasets over 6 tabular tasks with 35 semantically equivalent prompt configurations per example. Scores therefore reflect consistency across table formats rather than one chosen format.
Holds for: Covers tables that fit directly in the prompt as text; excludes tool-using, agentic, retrieval, multi-table and image-based table settings, and hierarchical tables. Compared against TableBench, DataBench, TQA-Bench, InfiAgent-DABench and TableVQA-Bench.; ToRR is a worked example for benchmark designers that evaluating over many semantically equivalent prompt variants, rather than one canonical prompt, is what makes a leaderboard's model ranking reproducible.
Holds for: Shown for table serialization formats and structural table perturbations in English table tasks as of 2025; other prompt dimensions are argued by analogy to prior work, not measured.
on table question answering tests, can you actually tell two models apart or do their scores overlap?
what fraction of model pairs are separated by non-overlapping confidence intervals on aggregated versus individual table reasoning datasets?
which table dataset should I evaluate on if I need to show a statistically distinguishable difference between two models?
is one table dataset enough to prove my model beats a competitor, or do I need to aggregate several?: Aggregated ToRR separates 79% of model pairs with non-overlapping confidence intervals, while individual datasets range from 38% for TableBench FC to over 71% for WikiTQ.
Holds for: Bootstrapping 1K seeds over samples of 100 examples per dataset with the 14 evaluated models; separability depends on which models are compared.
are open models catching up with the big commercial ones at reading tables?
do open-weight LLMs match proprietary models on tabular reasoning performance and robustness, and does parameter count track accuracy?
how do I decide between a self-hosted open model and a commercial API for reasoning over tables?
can I swap a paid API for an open-weight model on my table extraction workload without losing accuracy?: Closed proprietary models outperform open-weight models across most ToRR datasets, though the open qwen2-72b-instruct beats llama-3-1-405b-instruct on both performance and robustness.
Holds for: 14 models released through late 2024 and early 2025, served via Together AI and vendor APIs; no reasoning-mode or tool-using configurations.; Performance differences between models within the same family on ToRR average 0.07, and paired Cohen's d shows most model comparisons on ToRR have small, often non-significant practical differences.
Holds for: 14 models on 10 table datasets over 35 prompt configurations; aggregated p-values still find all pairwise comparisons significant.
what kinds of table questions and data sources go into a table reasoning test set?
which tabular tasks, datasets and prompt configurations make up the ToRR benchmark suite?
what table skills would I be covering if I evaluated on a multi-task table reasoning benchmark?
does a table benchmark I might adopt cover the task types my product actually does, like fact checking and table QA?: ToRR is a table reasoning benchmark that measures robustness as a first-class quantity, pairing 10 datasets over 6 tabular tasks with 35 semantically equivalent prompt configurations per example. Scores therefore reflect consistency across table formats rather than one chosen format.
Holds for: Covers tables that fit directly in the prompt as text; excludes tool-using, agentic, retrieval, multi-table and image-based table settings, and hierarchical tables. Compared against TableBench, DataBench, TQA-Bench, InfiAgent-DABench and TableVQA-Bench.
how do you put a number on whether a model gives consistent answers when a table is rewritten?
how is a per-example robustness score computed across semantically equivalent table prompt configurations?
how do I measure the spread between a model's best-case and worst-case score across equivalent prompts?
how should I report my model's consistency across table formats rather than just its average accuracy?: Every model evaluated on ToRR is unrobust: across the 35 semantically equivalent prompt configurations of one example, the minimum and maximum scores give entirely different estimates of performance. Robustness scores run from 0.49 for mixtral-8x7b-instruct to 0.70 for claude-3-5-sonnet.
Holds for: Robustness is 1 minus the mean per-example score range over 7 serializations x 4 structural perturbations plus 7 unperturbed variants; tables embedded directly in the prompt, no tool use.

Claims and scope

On ToRR, the best-performing LLMs (claude-3-5-sonnet, gpt-4o and deepseek-v3) reach an overall performance score of only 0.50. The weakest evaluated model, llama-3-1-8b-instruct, reaches 0.29. (Table 2)
Scope: 14 models across 7 families, 10 table datasets, 100 sampled examples per dataset, 5-shot greedy decoding; scores averaged over 35 prompt configurations.
Performance differences between models within the same family on ToRR average 0.07, and paired Cohen's d shows most model comparisons on ToRR have small, often non-significant practical differences. (Section 3.2 and Section 3.1)
Scope: 14 models on 10 table datasets over 35 prompt configurations; aggregated p-values still find all pairwise comparisons significant.
Every model evaluated on ToRR is unrobust: across the 35 semantically equivalent prompt configurations of one example, the minimum and maximum scores give entirely different estimates of performance. Robustness scores run from 0.49 for mixtral-8x7b-instruct to 0.70 for claude-3-5-sonnet. (Table 2 and Figure 3)
Scope: Robustness is 1 minus the mean per-example score range over 7 serializations x 4 structural perturbations plus 7 unperturbed variants; tables embedded directly in the prompt, no tool use.
No table serialization format consistently wins on ToRR: aggregated over all models and datasets no serializer outperforms the others, and per-model serializer preferences shift overall performance by at most 0.06. (Section 3.3 and Figure 9)
Scope: 7 serializations — HTML, CSV, JSON, Markdown, Indexed Row Major, DataFrame and Concatenation — measured by example-level win rate over 14 models and 10 datasets; individual model-dataset pairs still vary much more.
On ToRR the gap between a model's best and worst serializer on a single dataset averages about 0.05. It reaches 0.22 for llama-3-1-8b-instruct on TableBench FC, and 0.23 for llama-3-1-405b-instruct on the same dataset. (Table 7)
Scope: Largest max-minus-min score difference across the 7 serializations, computed per model-dataset pair over 14 models and 10 datasets.
Structural table perturbations in ToRR have no consistent direction of effect, changing model scores by an average of 0.03 relative to the unperturbed baseline. The 4 perturbations are row swapping, column swapping, transposition and adding empty rows. (Figure 12 and Figure 15)
Scope: 4 perturbations applied on top of each of 7 serializations, 14 models, 10 datasets; per-example absolute impact is larger, and more pronounced for smaller models and for Table QA and fact-checking datasets than for Table-to-Text.
Model rankings derived from a single table prompt configuration agree poorly with one another on ToRR, so a benchmark that fixes one serialization format yields an unreliable ranking of models. (Figure 4)
Scope: Kendall's W over 30 sampled prompt-configuration sets, 14 models, 100 examples per dataset; table serializations and structural perturbations only, not instruction wording.
Increasing the number of table prompt configurations from 1 to 10 raises Kendall's W ranking agreement on ToRR by more than 0.35 on average. The largest gains fall between roughly 2 and 8 configurations. (Figure 4)
Scope: Averaged over 30 sampled configuration sets and over ToRR's datasets; per-dataset gains differ sharply, FinQA rising from 0.35 to 0.93 with 11 prompts while NumericNLG rises only from 0.29 to 0.54.
Adding prompt configurations can substitute for test examples: on ToRR, 50 examples evaluated with 2 prompt configurations give about the same model-ranking reliability as 100 examples with 1 configuration. (Figure 5)
Scope: Kendall's W over 30 randomly sampled example-and-configuration sets, averaged per dataset, on table reasoning tasks at these small sample sizes.
Aggregated ToRR separates 79% of model pairs with non-overlapping confidence intervals, while individual datasets range from 38% for TableBench FC to over 71% for WikiTQ. (Section 4 and Figure 19)
Scope: Bootstrapping 1K seeds over samples of 100 examples per dataset with the 14 evaluated models; separability depends on which models are compared.
Closed proprietary models outperform open-weight models across most ToRR datasets, though the open qwen2-72b-instruct beats llama-3-1-405b-instruct on both performance and robustness. (Figure 8 and Table 2)
Scope: 14 models released through late 2024 and early 2025, served via Together AI and vendor APIs; no reasoning-mode or tool-using configurations.
ToRR is a table reasoning benchmark that measures robustness as a first-class quantity, pairing 10 datasets over 6 tabular tasks with 35 semantically equivalent prompt configurations per example. Scores therefore reflect consistency across table formats rather than one chosen format. (Table 8)
Scope: Covers tables that fit directly in the prompt as text; excludes tool-using, agentic, retrieval, multi-table and image-based table settings, and hierarchical tables. Compared against TableBench, DataBench, TQA-Bench, InfiAgent-DABench and TableVQA-Bench.
ToRR is a worked example for benchmark designers that evaluating over many semantically equivalent prompt variants, rather than one canonical prompt, is what makes a leaderboard's model ranking reproducible.
Scope: Shown for table serialization formats and structural table perturbations in English table tasks as of 2025; other prompt dimensions are argued by analogy to prior work, not measured.

Common misreadings

ToRR's finding that no serializer wins on average does not mean table format is irrelevant in practice: individual model-dataset pairs vary by about 0.05 between their best and worst serializer and by up to 0.23 in the worst case, so a deployment still needs its format chosen by case-by-case tuning.
The low robustness scores in ToRR are not evidence that models prefer particular table formats. The variation is largely idiosyncratic per example, which the paper reads as general prompt sensitivity rather than a format preference.
ToRR's low absolute scores are not simply a small-model story: the strongest closed models reach only 0.50 overall, and the gap between the strongest and weakest large models is narrow.
The claim that prompt configurations can substitute for test examples is demonstrated for ranking reliability at small sample sizes — 50 examples with 2 configurations matching 100 examples with 1 — and is not a claim that prompt variation replaces data collection in general.
ToRR does not measure agentic or tool-using table capabilities: datasets requiring SQL, retrieval, code execution, multi-table joins or table images were deliberately excluded, so its scores are not an upper bound on what a table-using pipeline can do.

Terminology in this paper

Prompt configuration: A pairing of one table serialization format with at most one structural table perturbation; ToRR uses 35 of them per example (7 serializations x 4 perturbations, plus the 7 unperturbed serializations).
Serialization: A method for converting a table into a string that can be embedded in a prompt, such as HTML, CSV, JSON, Markdown, Indexed Row Major, DataFrame or Concatenation.
Structural perturbation: A change to how a table's content is laid out without changing its content or cell relations: row swapping, column swapping, transposition, or inserting empty rows.
Robustness score (R): One minus the average per-example range (maximum minus minimum) of a model's score across all prompt configurations of that example, so 1.0 means identical scores on every table format.
Mean Absolute Impact: The mean of the absolute score change caused by a perturbation relative to its unperturbed baseline, used so that positive and negative effects do not cancel out in an average.
Separability with Confidence: The percentage of model pairs whose bootstrapped score confidence intervals do not overlap, used as a measure of how well a benchmark distinguishes models.

How to cite

@inproceedings{ashurytahan2025mightytorrbenchmarktable,
      title={The Mighty ToRR: A Benchmark for Table Reasoning and Robustness}, 
      author={Shir Ashury-Tahan and Yifan Mai and Rajmohan C and Ariel Gera and Yotam Perlitz and Asaf Yehudai and Elron Bandel and Leshem Choshen and Eyal Shnarch and Percy Liang and Michal Shmueli-Scheuer},
      year={2025},
      eprint={2502.19412},
      booktitle={SurgLLM@ICML},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.19412}, 
}

References

See the full reference list in the paper.