ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models

an LLM-based pipeline that turns a benchmark's wrong answers into a hierarchical taxonomy of why the model failed

Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen · arXiv · 2026

In one sentence

ErrorMap analyses each wrong prediction in a benchmark run with an analyst LLM and recursively groups the resulting error labels into a layered taxonomy, and applying it to 83 models on 35 datasets yields ErrorAtlas, a static taxonomy of 17 high-level LLM error categories.

Abstract

Large Language Models (LLM) benchmarks tell us when models fail, but not why they fail. A wrong answer on a reasoning dataset may stem from formatting issues, calculation errors, or dataset noise rather than weak reasoning. Without disentangling such causes, benchmarks remain incomplete and cannot reliably guide model improvement. We introduce ErrorMap, the first method to chart the sources of LLM failure. It extracts a model's unique"failure signature", clarifies what benchmarks measure, and broadens error identification to reduce blind spots. This helps developers debug models, aligns benchmark goals with outcomes, and supports informed model selection. ErrorMap works on any model or dataset with the same logic. Applying our method to 35 datasets and 83 models we generate ErrorAtlas, a taxonomy of model errors, revealing recurring failure patterns. ErrorAtlas highlights error types that are currently underexplored in LLM research, such as omissions of required details in the output and question misinterpretation. By shifting focus from where models succeed to why they fail, ErrorMap and ErrorAtlas enable advanced evaluation - one that exposes hidden weaknesses and directs progress. Unlike success, typically measured by task-level metrics, our approach introduces a deeper evaluation layer that can be applied globally across models and tasks, offering richer insights into model behavior and limitations. We make the taxonomy and code publicly available with plans to periodically update ErrorAtlas as new benchmarks and models emerge.

Questions this paper answers

why do language models get benchmark questions wrong, not just how often?
is there an automated pipeline that produces per-instance error diagnoses of LLM benchmark failures rather than aggregate accuracy?
how do I find out what kinds of mistakes my model is making on a benchmark instead of just its score?
my model scores 62% on a benchmark and I need to know why it loses the other 38% -- what can tell me that?
ErrorMap addresses the gap between benchmark scores, which say when a model fails, and diagnosis, which says why. It analyses the model's actual output alongside the input rather than characterising difficulty by properties of the question alone.
Holds for: Positioning as of the 2026 arXiv release; earlier diagnostic work the paper surveys either targets a single subdomain or model, or infers difficulty from inputs, and the authors report finding no prior general LLM error taxonomy.
ErrorAtlas is a taxonomy of 17 high-level LLM error categories, led by Missing Required Element at 15.56% of errors, Specification Misinterpretation at 11.5%, and Logical Reasoning Error at 9.09% of errors.
Holds for: Sampled wrong predictions from 83 models on 35 datasets spanning HELM Capabilities, MedHELM, ToRR, BFCL-v4 and code benchmarks, with gpt-oss-120b as analyst and clustering LLM.
what should I read first about diagnosing why language models fail rather than just scoring them?
which work introduced a reusable taxonomy of LLM error types for diagnostic evaluation?
where do I start if I want to build error analysis into my model evaluation pipeline?
ErrorMap addresses the gap between benchmark scores, which say when a model fails, and diagnosis, which says why. It analyses the model's actual output alongside the input rather than characterising difficulty by properties of the question alone.
Holds for: Positioning as of the 2026 arXiv release; earlier diagnostic work the paper surveys either targets a single subdomain or model, or infers difficulty from inputs, and the authors report finding no prior general LLM error taxonomy.
ErrorAtlas is released as a static, publicly available taxonomy so error analyses of new models can be compared across time and papers. Applying it needs only per-instance analysis plus category assignment, skipping taxonomy generation.
Holds for: Reuse is intended for tasks whose outputs contain interpretable content or chain-of-thought; classification-style tasks with no explanation cannot be analysed, and the authors plan periodic ErrorAtlas updates rather than freezing it.
what kinds of mistakes do large language models make most often?
which error categories dominate the failure distribution across LLM benchmarks and models?
how do I know which failure types to prioritise fixing in my language model?
if I only have time to fix one class of model mistake, which one is most common?
ErrorAtlas is a taxonomy of 17 high-level LLM error categories, led by Missing Required Element at 15.56% of errors, Specification Misinterpretation at 11.5%, and Logical Reasoning Error at 9.09% of errors.
Holds for: Sampled wrong predictions from 83 models on 35 datasets spanning HELM Capabilities, MedHELM, ToRR, BFCL-v4 and code benchmarks, with gpt-oss-120b as analyst and clustering LLM.
Omissions of required detail are the single most prevalent LLM error category in ErrorAtlas, at 15.56% of errors, appearing in 31 of 35 datasets and for 82 of 83 models. Question or task misinterpretation is second at 11.5%, and both types receive little attention in LLM research.
Holds for: Prevalence measured on the sampled errors of the 35 datasets studied; the judgement that these types are under-discussed comes from the authors' related-work survey of existing error taxonomies, not from a bibliometric count.
are there common ways language models fail that hardly anyone studies?
which prevalent LLM failure modes are underrepresented in the research literature relative to their frequency?
how do I check whether my model is leaving out required parts of an answer or misreading the question?
should I be worried about incomplete answers and misread instructions rather than reasoning ability?
Omissions of required detail are the single most prevalent LLM error category in ErrorAtlas, at 15.56% of errors, appearing in 31 of 35 datasets and for 82 of 83 models. Question or task misinterpretation is second at 11.5%, and both types receive little attention in LLM research.
Holds for: Prevalence measured on the sampled errors of the 35 datasets studied; the judgement that these types are under-discussed comes from the authors' related-work survey of existing error taxonomies, not from a bibliometric count.
when a model gets a hard reasoning question wrong, is bad reasoning actually the cause?
what share of errors on MMLU-Pro, GPQA and Omni-MATH have low reasoning orientation?
how do I tell whether my model's losses on a reasoning benchmark come from arithmetic and formatting slips instead of reasoning?
can I trust a reasoning benchmark score as a measure of my model's reasoning?
About 44% of model errors on MMLU-Pro, GPQA and Omni-MATH have a weak reasoning orientation. Those failures are instead technical, such as computation errors, missing required elements or counting errors, even though the 3 benchmarks are positioned as reasoning tests.
Holds for: Aggregated over the sampled failures of the models evaluated on these 3 datasets, with categories assigned by the ErrorAtlas classifier; failure is decided by each benchmark's primary metric and a threshold.
can you trust error categories that a language model assigned automatically?
how were the coverage and label accuracy of the ErrorAtlas category assignments validated?
how do I check that an automatic error categorisation is not just putting everything in a catch-all bucket?
before I report an automated error breakdown, how do I know the labels are right?
Automatically mapping the analysed wrong predictions back into ErrorAtlas gives a coverage score of 95.2%, with only 1 instance falling into "other" and 48 into "hard to analyze".
Holds for: Wan et al. (2024) protocol applied to the same errors used to build the taxonomy; 295 errors counted as uncovered, including rare or uninformative categories outside ErrorAtlas.
A meta-judge choosing between an error's assigned ErrorAtlas category and a random alternative agrees with the assigned label 92% of the time. The same meta-judge accepts the per-instance error analyses 91.1% of the time.
Holds for: Qwen2.5-72B-Instruct as meta-judge over the 3 experiments (ErrorAtlas construction, Gemini comparison, MMLU-Pro taxonomy), in a forced binary choice against one random negative.
if you rebuild an automatically generated list of error types with a different model or a smaller sample, do you get the same list?
how stable is LLM-induced error taxonomy construction under prompt rephrasing, example changes and sampling rate?
how big a sample of wrong predictions do I need before an induced error taxonomy stops changing?
can I use a cheaper judge model and a 5% sample and still get the same error categories?
Rebuilding ErrorAtlas with a rephrased prompt, different examples and a 15% sample reproduces the original categories with 88% precision and 88% recall. A weaker configuration using qwen-30b-thinking on a 5% sample reaches 1.00 precision but 0.52 recall.
Holds for: Manual comparison of high-level category descriptions and subcategories against the original gpt-oss-120b 10%-sample taxonomy; the lower-bound configuration still recovers 8 of the top 10 categories, with losses in the long tail.
does an automatic breakdown of model mistakes match what human annotators found?
how does an LLM-generated error distribution on MMLU-Pro compare with the benchmark paper's manual error annotation?
how do I validate automated error labelling against a hand-annotated error study?
can automated error analysis save me from hand-annotating my model's wrong answers?
Run on MMLU-Pro, ErrorMap produces 5 error categories whose GPT-4o distribution tracks the MMLU-Pro paper's manual annotation. ErrorMap reports 44% logical reasoning errors against 39% manual reasoning errors, and 5% prompt misinterpretation against 4% question understanding errors.
Holds for: Single dataset (MMLU-Pro) and single model (GPT-4o); ErrorMap splits the manual "lack of specific knowledge" category (35%) across two of its own categories and has no "other" bucket, which is 10% of the manual labels, so the label sets do not map one-to-one.
what kinds of mistakes did the newer, stronger version of a model stop making?
can error-category analysis localise the accuracy gap between Gemini 1.5 Flash and Gemini 1.5 Pro?
how do I work out which specific weaknesses a model upgrade actually fixed?
is it worth paying for the larger model version, and which failures does it fix?
Gemini 1.5 Pro outscores Gemini 1.5 Flash by a mean 4.8% on the HELM Capabilities benchmark. ErrorMap localises that gap: the Pro version makes significantly fewer computation errors and fewer incomplete-reasoning errors.
Holds for: One model pair on HELM Capabilities, run with the fixed ErrorAtlas categories using only ErrorMap stages 1 and 2.b over all predictions of the 2 Gemini models.
Models evaluated on HELM Capabilities show distinct failure signatures: Gemini 2.0 Flash Lite has the highest rate of incomplete-content errors and the fewest formatting errors. Claude 3.5 Haiku skews toward logical reasoning errors and Mixtral 8x22B Instruct v0.1 toward computation errors.
Holds for: Models compared only within HELM Capabilities so the instances are shared; differences between best- and worst-performing models per category are usually significant under binomial tests, except prompt misinterpretation (p=.075).
do two language models with similar scores get things wrong in different ways?
are per-model error-category distributions on HELM Capabilities distinguishable across frontier LLMs?
how do I compare candidate models by their failure profile rather than a single accuracy number?
two models score the same on my benchmark -- how do I pick the one whose mistakes I can live with?
Models evaluated on HELM Capabilities show distinct failure signatures: Gemini 2.0 Flash Lite has the highest rate of incomplete-content errors and the fewest formatting errors. Claude 3.5 Haiku skews toward logical reasoning errors and Mixtral 8x22B Instruct v0.1 toward computation errors.
Holds for: Models compared only within HELM Capabilities so the instances are shared; differences between best- and worst-performing models per category are usually significant under binomial tests, except prompt misinterpretation (p=.075).
do models make different kinds of mistakes in health questions than in maths questions?
how do LLM error-category distributions shift across MMLU-Pro subject domains?
how do I tell whether my model's errors in one subject area are factual rather than computational?
should I expect my model's failure pattern to change when I move it from maths to medical questions?
Error distributions on MMLU-Pro shift by subject domain: mathematics and physics have near-identical profiles, while the health domain shows a disproportionately high share of factual errors, exceeding even history.
Holds for: MMLU-Pro only, with errors grouped by the dataset's own domain labels; reported as an observed pattern in the error distribution figure rather than a per-domain significance test.
how much compute and time does it take to analyse why a model got thousands of questions wrong?
what is the inference cost of building an LLM error taxonomy over many model-dataset pairs at a 10% failure sample?
how do I run error analysis over a whole benchmark without sending every wrong prediction through a judge model?
can I afford to run LLM-based error analysis on my evaluation runs?
Building ErrorAtlas took roughly 3 hours of largely parallel inference by sampling approximately 10% of each model-dataset pair's failures. Applying ErrorMap to a single dataset or model pair is cheaper still, since cost scales with the number of wrong predictions analysed.
Holds for: gpt-oss-120b as the analyst LLM, with most inference calls run in parallel; the MMLU-Pro taxonomy and the Gemini 1.5 comparison each required fewer inferences than the ErrorAtlas build.
ErrorAtlas is released as a static, publicly available taxonomy so error analyses of new models can be compared across time and papers. Applying it needs only per-instance analysis plus category assignment, skipping taxonomy generation.
Holds for: Reuse is intended for tasks whose outputs contain interpretable content or chain-of-thought; classification-style tasks with no explanation cannot be analysed, and the authors plan periodic ErrorAtlas updates rather than freezing it.
can I reuse an existing list of model error types on my own model's wrong answers instead of building one?
does applying a static LLM error taxonomy skip the taxonomy-induction stage?
how do I label my model's failures with categories that are comparable to other papers' error analyses?
should I generate my own error categories or apply a published taxonomy to my model?
ErrorAtlas is released as a static, publicly available taxonomy so error analyses of new models can be compared across time and papers. Applying it needs only per-instance analysis plus category assignment, skipping taxonomy generation.
Holds for: Reuse is intended for tasks whose outputs contain interpretable content or chain-of-thought; classification-style tasks with no explanation cannot be analysed, and the authors plan periodic ErrorAtlas updates rather than freezing it.

Claims and scope

Common misreadings

Terminology in this paper

ErrorAtlas
A static, reusable taxonomy of 17 high-level LLM error categories built by running ErrorMap over sampled wrong predictions from 83 models on 35 datasets, intended for replicable cross-model and cross-benchmark comparison.
Failure signature
The distribution of a single model's errors over taxonomy categories, used to distinguish models that share the same benchmark score but fail in different ways.
Informative Correct Predictions (ICPs)
Correct answers produced by other models on the same benchmark instance, given to the analyst LLM as additional reference material when diagnosing a wrong prediction; up to 3 are included per prompt.
Taxonomy coverage
The fraction of analysed wrong predictions that an automatic classifier maps into a real taxonomy category rather than into 'other', 'hard to analyze', or rare uninformative categories.

How to cite

@article{ashury2026errormap,
  title={ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models},
  author={Ashury-Tahan, Shir and Mai, Yifan and Bandel, Elron and Shmueli-Scheuer, Michal and Choshen, Leshem},
  journal={arXiv preprint arXiv:2601.15812},
  year={2026}
}

References

See the full reference list in the paper.