# Holmes: A Benchmark to Assess the Linguistic Competence of Language Models

a probing benchmark that measures what language models internally encode about linguistic phenomena

Authors: Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, Iryna Gurevych
Venue: Trans. Assoc. Comput. Linguistics (2024)

## What this paper shows
Holmes measures the linguistic competence of language models by training linear probes on the frozen last-layer representations of 208 datasets covering morphology, syntax, semantics, reasoning and discourse, isolating linguistic knowledge from instruction following.

## Claims, with scope
- Holmes is a probing benchmark that assesses the English linguistic competence of language models with 208 datasets covering 66 phenomena. Its morphology, syntax, semantics, reasoning and discourse datasets consolidate resources found in a survey of 274 probing studies.
  Scope: English only, and last-layer internal representations only. Classifier-based (linear) probing, so generation and instruction following are not measured; 18 of the 208 datasets rest on licensed resources.
  Evidence: Section 4.1, Section 3.1
- A meta-study of 274 probing papers finds the literature collectively covers 289 tasks and 161 language models, yet individual studies stay narrow. Part-of-speech tagging, the most probed task, was evaluated on only 23% of the models, and the top-10 most mentioned models account for 80% of all model mentions.
  Scope: 28,063 papers from 2015 to August 2023 at ACL-family venues plus selected other venues, filtered by occurrences of 'probing'/'probe' and then manually reviewed; recent large models such as Pythia, UL2 and Llama-2 are almost absent from the surveyed work.
  Evidence: Section 3.2 (iv), Figure 5, Figure 6
- Among 274 surveyed probing studies, 74% use classifier-based probing and 20% use mask-based probing, while roughly 3% rely on attention patterns or other approaches.
  Scope: Categorisation of studies published 2015 to August 2023 at major NLP venues; each study assigned a single dominant probing method.
  Evidence: Section 3.2 (iii), Figure 4
- Probing results on Holmes vary little across seeds, with an average standard deviation of 0.02 over 5 random seeds, against the 0.07 deviation reported for prompt paraphrasing in prompting-based evaluation. Average compression is 1.9 and average selectivity 0.31.
  Scope: Averages over 208 probing datasets and the evaluated models, with fixed probe hyperparameters (20 epochs, batch size 64, learning rate 0.0005); selectivity computed only for base-sized models of 10M-200M parameters.
  Evidence: Section 5 (i), Figure 8
- Across 59 evaluated language models, linguistic competence is markedly stronger for formal phenomena (morphology and syntax) than for functional ones (semantics, reasoning and discourse), which score lower on the probing task metric.
  Scope: 59 models spanning sparse, static, encoder, decoder and encoder-decoder types, probed on last-layer representations of 208 English datasets.
  Evidence: Section 5 (ii), Figure 9
- The five phenomenon types in Holmes correlate at 68.4±7.5 Kendall-tau with the overall model ranking but only 54.7±13.9 with each other. Discourse is the most distinct type, at 44.4±14.7 average correlation with the others.
  Scope: Kendall-tau rank correlations over the models jointly evaluated within Holmes; semantics and reasoning correlate 73.9 and 75.6 with the overall ranking but only 58.4 with each other.
  Evidence: Section 5 (ii), Figure 10 (left)
- Encoder language models reach a mean winning rate of 52% on Holmes against 21% for decoder models of comparable size. Decoder models do not match encoder stability on frequent part-of-speech tokens even at 700 times the parameter count.
  Scope: 5 encoder and 6 decoder models up to 220M parameters for the mean-winning-rate comparison; token stability covers BERT, RoBERTa, GPT2, Pythia-12B and Llama-2-70B on the top-20 most common tokens of the pos, xpos and upos datasets.
  Evidence: Section 5 (iii), Figure 11
- Linguistic competence on Holmes scales with parameter count within the Pythia and T5 families. The jump is pronounced beyond 0.5B parameters for Pythia and 1.0B for T5, and is concentrated in morphology and syntax.
  Scope: 8 T5 sizes and 5 Pythia sizes only; T5 (encoder-decoder) reaches a mean winning rate of 40-70% versus 20-60% for Pythia (decoder-only), so architecture and scale are not separated in this comparison.
  Evidence: Section 5 (iv), Figure 12
- Instruction tuning raises mean winning rate on Holmes by an average of +10% for morphology, +5% for syntax and +4% for reasoning. It lowers it by -3% for semantics and -1% for discourse, for an overall average of +2%, with per-model effects from +41% to -13%.
  Scope: 11 instruction-tuned models each compared against its own pre-trained base (Llama-2-Chat, FLAN-T5, Dolly-v2, Tülu-2, Orca-2, Vicuna-v1.5, FLAN-UL2, Mixtral-Instruct) at 7B to 70B parameters.
  Evidence: Table 2
- Holmes probing rankings correlate only moderately with the prompting-based OpenLLM leaderboard, at 54.7 to 58.0 Kendall-tau for syntax, semantics and discourse, rising to 65.3 for morphology and 77.5 for reasoning.
  Scope: Models evaluated by both Holmes and OpenLLM (Beeching et al., 2023); correlation patterns hold across MMLU, TruthfulQA and GSM8K, with TruthfulQA the weakest correlate.
  Evidence: Section 5 (vi), Figure 10 (right)
- On the BLiMP datasets evaluated by both Holmes and HELM, probing-based and prompting-based model rankings barely agree, at a rank correlation of tau=0.05. Most HELM prompting results fall below the random baseline.
  Scope: 40 open decoder models and 22 BLiMP datasets covering quantifier, island effects, irregular forms and binding phenomena, using HELM's own evaluation code with its multiple-choice-joined prompting adaptation.
  Evidence: Figure 15
- FlashHolmes reproduces Holmes model rankings at a rank resolution of about 1.5 while requiring roughly 3% of the computation. It trains probes on 1/32 of the training instances and drops the 18 licensed datasets, versus about 6 GPU days of encoding for a 70B model on full Holmes.
  Scope: Rank resolution is the 95% CI of rank difference against full Holmes, where 1/2 of the training data gives about 0.9 and 1/512 about 2.6; English datasets, last-layer representations.
  Evidence: Section 6, Figure 13
- Holmes argues that probing-based evaluation retains validity under dataset contamination, because instruction tuning aligns a model's textual responses rather than explicitly aligning the internal representations that the probes read.
  Scope: An argument rather than a measurement; training data is unknown for several evaluated models, including Llama-2, Mixtral and Wizard.
  Evidence: Ethical Considerations and Limitations, Dataset Contamination

## Common misreadings
- Holmes does not test whether a language model can answer grammar questions; it trains linear probes on frozen last-layer representations, so a high Holmes score says nothing directly about a model's generated output.
- The finding that encoders beat decoders holds for models up to 220M parameters in the mean-winning-rate comparison, because no large encoder-only models were available; it is not a measured comparison of a 70B encoder against a 70B decoder.
- Instruction tuning is not uniformly harmful or uniformly helpful for linguistic competence: it averages +10% mean winning rate on morphology but -3% on semantics, and individual models range from -13% to +41% depending on phenomenon type.
- A moderate Holmes-OpenLLM rank correlation does not mean probing and prompting measure the same thing; on jointly evaluated BLiMP datasets the two rankings agree at only tau=0.05.
- FlashHolmes is not merely a random subsample of Holmes: besides training on 1/32 of the instances it also excludes the 18 datasets built from licensed resources.
- Holmes covers English only, so its rankings should not be read as claims about multilingual linguistic competence.

## Terminology
- linguistic competence: Following Chomsky (1965), a language model's unconscious internal understanding of linguistic phenomena, assessed via its internal representations rather than via its textual responses.
- linguistic performance: A language model's use of language in textual responses to instructions, which is what prompting-based benchmarks measure and which conflates linguistic knowledge with abilities such as instruction following.
- formal vs. functional phenomena: Formal phenomena are morphology and syntax (grammatical rules and statistical patterns); functional phenomena are semantics, reasoning and discourse (practical abilities like interpreting sentiment or detecting speculation).
- selectivity: The macro-F1 of a probe trained on the true labels minus the macro-F1 of the same probe trained on randomly assigned control-task labels; higher values mean the probe found phenomenon-relevant structure rather than memorising.
- compression: The ratio of a uniform encoding of instances and labels to their minimum description length under the probe; higher values mean the linguistic phenomenon is more cleanly encoded in the representation.
- discriminability: The Kendall-tau alignment between the model ranking induced by a single probing dataset and the overall benchmark ranking; low values mean no single dataset dominates the aggregate ranking.
- rank resolution: The 95% confidence interval of the difference in a model's rank between a subsampled benchmark and the full benchmark; a resolution of 1 means a model keeps its rank or swaps with a neighbour.
- FlashHolmes: The streamlined variant of the Holmes benchmark that trains probes on 1/32 of the training instances and excludes 18 licensed datasets.

## Links
- arXiv: https://arxiv.org/abs/2404.18923
- PDF: https://arxiv.org/pdf/2404.18923
- HTML: https://arxiv.org/html/2404.18923
- Hugging Face: https://huggingface.co/papers/2404.18923
- alphaXiv: https://www.alphaxiv.org/abs/2404.18923
- DOI: https://doi.org/10.1162/TACL\_A\_00718
- Semantic Scholar: https://www.semanticscholar.org/paper/274625741
- Publisher: https://doi.org/10.1162/tacl\_a\_00718
- project page: https://holmes-benchmark.github.io

## How to cite
@article{DBLP:journals/corr/abs-2404-18923,author       = {Andreas Waldis and
                  Yotam Perlitz and
                  Leshem Choshen and
                  Yufang Hou and
                  Iryna Gurevych},
  title        = {Holmes: {A} Benchmark to Assess the Linguistic Competence of Language Models},
  journal      = {Trans. Assoc. Comput. Linguistics},
  volume       = {12},
  year         = {2024},
  url          = {https://doi.org/10.1162/tacl\_a\_00718},
  doi          = {10.1162/TACL\_A\_00718},
  eprinttype    = {arXiv},
  eprint       = {2404.18923},
  timestamp    = {Tue, 14 Oct 2025 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/tacl/WaldisWPCCHG24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org},
  pages        = {1616--1647}
}