Bigger is not always better: The importance of human-scale language modeling for psycholinguistics

Ethan Gotlieb Wilcox; Michael Y Hu; Aaron Mueller; Alex Warstadt; Leshem Choshen; Chengxu Zhuang; Adina Williams; Ryan Cotterell; Tal Linzen

doi:10.1016/j.jml.2025.104650

Bigger is not always better: The importance of human-scale language modeling for psycholinguistics

a language model pretrained on roughly the amount of language a child hears, 100 million words or less

Ethan Gotlieb Wilcox, Michael Y Hu, Aaron Mueller, Alex Warstadt, Leshem Choshen, Chengxu Zhuang, Adina Williams, Ryan Cotterell, Tal Linzen · Journal of Memory and Language · 2025

In one sentence

Bigger is not always better: scaling language models to trillions of words weakens their value as cognitive models, and the first BabyLM Challenge shows that models trained on 100 million words or fewer reach near-human BLiMP performance -- though not by cognitively plausible means.

Abstract

When trained to place high probability on a training corpus, neural network language models can learn a surprising amount about language. Recent work has demonstrated that large performance improvements can arise from simply increasing, i.e., scaling, the size of the corpora they are trained on and the number of parameters in those models. Accordingly, many contemporary systems are trained on trillions of words. While largely beneficial to performance on language applications, scaling has several downsides for both computational psycholinguistics and natural language processing research. We discuss the scientific challenges presented by the scaling paradigm, as well as the benefits that would result from language models that can learn from human-scale data. In the second half of this paper, we report on findings from a recent effort to bring about human-scale language model pretraining: the first iteration of the BabyLM Challenge, a shared task organized by the authors that invited participants to train a language model on 100 million words or less. The challenge produced several concrete best practices for practitioners interested in small-scale language modeling. For cognitive scientists, the challenge demonstrated that robust linguistic generalizations can be learned by models trained on a human-scale dataset, though this is not yet achieved through cognitively plausible mechanisms. Furthermore, it established a population of “BabyLMs” that are all effective at data-efficient language learning. Studying such models can help us identify hypotheses for the computational mechanisms that underlie human language acquisition. • Psycholinguistics benefits from computational models trained at human data scale. • We report on the BabyLM Challenge, an effort to train models at human scale. • BabyLM models achieve close to human-level performance on some tasks. • High language modeling performance is attainable with academic computational resources. • We identify actionable insights for human-scale language modeling.

Questions this paper answers

why would researchers deliberately train a language model on only as much text as a child hears?
how does training on trillions of tokens compromise the use of language models as psycholinguistic models of acquisition and as surprisal estimators?
how do I justify a small-corpus pretraining setup to reviewers who expect web-scale data?
if I care about cognitive modelling rather than benchmark scores, should I be worried that my model saw far more text than a person ever could?: Scaling training corpora to trillions of words undermines two scientific uses of language models in psycholinguistics, Wilcox et al. argue. Those uses are assessing poverty-of-the-stimulus claims and estimating word probabilities for testing surprisal theory.
Holds for: An argument about neural language models used as cognitive models of language learning and processing, not about their engineering utility, which the paper grants is improved by scaling.; The BabyLM Challenge showed that robust linguistic generalizations are learnable from human-scale data, but not through cognitively plausible mechanisms. Winning systems relied on transformer optimization tricks, hundreds of training epochs, and large-scale data augmentation.
Holds for: The first (2023) iteration's submissions; only 7 teams ran the optional age-of-acquisition evaluation, and no submission was evaluated against incremental reading-time data.
what was the competition where teams trained language models on a child-sized amount of text, and how many took part?
what were the data budget and participation figures for the first BabyLM shared task at CoNLL 2023?
where do I find a set of openly released English language models pretrained on 100 million words or fewer?
is there an existing shared task and model pool I can build on instead of designing my own small-data pretraining benchmark?: The BabyLM Challenge, a CoNLL 2023 shared task, asked participants to pretrain language models on 100 million words of English or fewer. That budget is roughly the linguistic input available to a child in the United States by age 12.
Holds for: English only, text and transcribed speech only, with no limit on compute or number of training epochs; the 3 tracks were Strict (100M words), Strict-Small (10M words) and Loose (100M words plus non-linguistic data).; The first BabyLM Challenge received 31 papers and 162 submitted models, establishing a population of openly released models that are all effective data-efficient learners of English.
Holds for: CoNLL 2023 iteration; participants could enter multiple tracks, so unique-participant counts per track are lower than the model count.
how well can a language model do on English grammar tests if it only ever reads 100 million words?
what BLiMP accuracy do 100M-word and 10M-word pretrained models reach relative to a Llama 2 70B skyline?
how do I find out what grammatical accuracy is achievable at my training budget before I commit compute?
if I can only afford a small pretraining corpus, how much grammatical ability am I giving up compared with a huge model?: The best BabyLM submissions reached 0.85-0.86 on BLiMP while trained on 100 million words, above the Llama 2 70B skyline score of 0.84. The top model was a few percentage points shy of human performance on BLiMP.
Holds for: BLiMP minimal-pair accuracy on English syntax; Llama 2 was evaluated on GLUE and SuperGLUE with in-context learning rather than fine-tuning.; Cutting the BabyLM training budget from 100 million to 10 million words cost the best submissions about 5 points of BLiMP, 0.85 down to 0.80. The corresponding GLUE drop was 4 points, 0.78 down to 0.74.
Holds for: Top-3 submissions in the Strict and Strict-Small tracks; both corpora used the same source composition, with Strict-Small sampled at 10% from each source.
does feeding a model easy text before hard text actually help when there is little data?
did curriculum learning yield measurable gains over the BabyLM baselines, and which intervention classes did?
which pretraining interventions should I spend my effort on when my corpus is only 10 to 100 million words?
should I bother building a difficulty-ordered curriculum for my small-data pretraining run?: Curriculum learning was the most popular strategy among BabyLM submissions yet produced only marginal gains over the baselines, while data preprocessing and architectural modifications were the most effective strategies.
Holds for: Meta-analysis by hand-coding each submission into 9 approach categories, counting at most one model per participant per track; curricula tested were those participants chose, sorted mostly by simplicity metrics.
which encoder architecture is a good default for pretraining on a small English corpus?
do ELC-BERT's layer-wise skip connections beat LTG-BERT when both are pretrained on the 100M-word corpus?
which architecture should I start from for a data-efficient masked language model?
is it worth adopting ELC-BERT over LTG-BERT for my 100-million-word pretraining run?: LTG-BERT is the architecture Wilcox et al. recommend as a starting point for small-scale language modeling. ELC-BERT's added layer-wise skip connections gave no advantage over it when both were trained for 20 epochs on the 100-million-word Strict corpus.
Holds for: Reproduction runs averaged over 3 seeds on 4 NVIDIA RTX8000 GPUs, with smaller batch size and shorter sequence length than the original LTG-BERT paper; the 2 models tie on BLiMP at 0.83.
how many passes over a small text corpus are worth doing before the returns dry up?
how much of the winning BabyLM result is attributable to hundreds of epochs rather than to architecture, and where do BLiMP and GLUE gains saturate?
how do I set the number of training epochs for a 100-million-word pretraining corpus?
can I get most of the benefit in 20 epochs, or do I need to budget for hundreds of passes over my data?: Training ELC-BERT and LTG-BERT for 20 epochs instead of the 450 or more used in the winning submission cost about 2 points on BLiMP and GLUE. It also cost about 10 points on the BLiMP Supplement, leaving the 20-epoch models ahead of McGill-BERT on GLUE but behind it on BLiMP.
Holds for: Reproductions on the 100-million-word Strict corpus, 3 seeds, with batch size and sequence length differing from the original submissions.; BLiMP gains from additional training epochs on the BabyLM corpora diminish roughly exponentially, so most benefit of repeated exposure arrives within the first 20 epochs. Strict-Small GLUE performance declines after 50 epochs.
Holds for: LTG-BERT on the Strict and Strict-Small corpora, losses and scores averaged over 3 seeds; training loss correlates with BLiMP at -0.99 (Strict) and -0.95 (Strict-Small), but with GLUE at only 0.61 for Strict-Small.
can a model pick up hard grammar rules like which questions you cannot ask, from a child-sized amount of text?
do models pretrained on 100M words perform worse on island constraints, filler-gap dependencies and subject-aux inversion than on other BLiMP paradigms?
how do I check whether phenomena at the centre of nativist arguments are learnable at human-scale data volumes?
can I cite small-corpus model results when arguing about whether syntax has to be innate?: BabyLM submissions performed about the same on BLiMP subtasks central to poverty-of-the-stimulus debates as on the remaining BLiMP subtasks. The POS-relevant subtasks covered island constraints, filler-gap dependencies and subject-aux inversion.
Holds for: Post-hoc partition of BLiMP subtasks by the paper's authors, averaged across submissions within each track, with 95% CIs across model scores; English syntax only.; The best BabyLM submissions reached 0.85-0.86 on BLiMP while trained on 100 million words, above the Llama 2 70B skyline score of 0.84. The top model was a few percentage points shy of human performance on BLiMP.
Holds for: BLiMP minimal-pair accuracy on English syntax; Llama 2 was evaluated on GLUE and SuperGLUE with in-context learning rather than fine-tuning.
did giving language models pictures or speech alongside text make them learn more from less?
did BabyLM Loose-track submissions with non-linguistic supervision outperform text-only Strict-Small models in aggregate?
how do I decide whether to add images, audio or code to a small pretraining corpus?
should I add multimodal data to squeeze more out of my 10 million words of text?: Loose-track BabyLM models, permitted extra non-linguistic data such as audio, code or images, tended to score lower in the aggregate than Strict-Small models trained on 10 million words of text alone.
Holds for: First BabyLM iteration only, with 2023-era architectures not designed for multimodal pretraining; the extra modality data was not counted against the 100M-word budget.
do models trained on small corpora learn real grammar rules or just shallow word cues?
what do MSGS scores indicate about linguistic versus surface generalization preference in the top BabyLM models?
how do I test whether a small pretrained model prefers linguistic over surface features?
can I trust that my small-data model's grammar score reflects genuine generalization rather than shortcut features?: All top BabyLM models except Strict-Small McGill-BERT scored positively on MSGS, indicating a preference for linguistic over surface generalizations comparable to Llama 2 (0.26) and RoBERTa-base (0.24).
Holds for: MSGS Matthews correlation after fine-tuning on ambiguous training sets; MSGS has not been run with human subjects, so no human reference point exists, and Loose-track McGill-BERT scored -0.02.
what should I read first about training language models on the amount of language a child actually hears?
which paper is the entry point for developmentally plausible, data-efficient pretraining and its bearing on cognitive modelling?
where do I start a literature review on human-scale language model pretraining?
which single paper should I hand a student who wants to work on child-scale language model training?: Scaling training corpora to trillions of words undermines two scientific uses of language models in psycholinguistics, Wilcox et al. argue. Those uses are assessing poverty-of-the-stimulus claims and estimating word probabilities for testing surprisal theory.
Holds for: An argument about neural language models used as cognitive models of language learning and processing, not about their engineering utility, which the paper grants is improved by scaling.; The BabyLM Challenge, a CoNLL 2023 shared task, asked participants to pretrain language models on 100 million words of English or fewer. That budget is roughly the linguistic input available to a child in the United States by age 12.
Holds for: English only, text and transcribed speech only, with no limit on compute or number of training epochs; the 3 tracks were Strict (100M words), Strict-Small (10M words) and Loose (100M words plus non-linguistic data).; The BabyLM Challenge showed that robust linguistic generalizations are learnable from human-scale data, but not through cognitively plausible mechanisms. Winning systems relied on transformer optimization tricks, hundreds of training epochs, and large-scale data augmentation.
Holds for: The first (2023) iteration's submissions; only 7 teams ran the optional age-of-acquisition evaluation, and no submission was evaluated against incremental reading-time data.
do the models that won the child-scale training competition learn anything like the way children do?
can BabyLM winning systems be treated as cognitively plausible accounts of child language acquisition?
how do I tell whether a data-efficient pretrained model is usable as a model of human acquisition?
can I present a small-corpus language model as a simulation of how a child learns language?: The BabyLM Challenge showed that robust linguistic generalizations are learnable from human-scale data, but not through cognitively plausible mechanisms. Winning systems relied on transformer optimization tricks, hundreds of training epochs, and large-scale data augmentation.
Holds for: The first (2023) iteration's submissions; only 7 teams ran the optional age-of-acquisition evaluation, and no submission was evaluated against incremental reading-time data.
how much grammar and language understanding do you lose going from 100 million training words down to 10 million?
what is the BLiMP and GLUE penalty for the Strict-Small 10M budget relative to the 100M Strict budget?
how do I estimate the cost of shrinking my pretraining corpus by a factor of 10?
if I can only collect 10 million words instead of 100 million, how much performance should I expect to sacrifice?: Cutting the BabyLM training budget from 100 million to 10 million words cost the best submissions about 5 points of BLiMP, 0.85 down to 0.80. The corresponding GLUE drop was 4 points, 0.78 down to 0.74.
Holds for: Top-3 submissions in the Strict and Strict-Small tracks; both corpora used the same source composition, with Strict-Small sampled at 10% from each source.

Claims and scope

The BabyLM Challenge, a CoNLL 2023 shared task, asked participants to pretrain language models on 100 million words of English or fewer. That budget is roughly the linguistic input available to a child in the United States by age 12.
Scope: English only, text and transcribed speech only, with no limit on compute or number of training epochs; the 3 tracks were Strict (100M words), Strict-Small (10M words) and Loose (100M words plus non-linguistic data).
Scaling training corpora to trillions of words undermines two scientific uses of language models in psycholinguistics, Wilcox et al. argue. Those uses are assessing poverty-of-the-stimulus claims and estimating word probabilities for testing surprisal theory.
Scope: An argument about neural language models used as cognitive models of language learning and processing, not about their engineering utility, which the paper grants is improved by scaling.
The best BabyLM submissions reached 0.85-0.86 on BLiMP while trained on 100 million words, above the Llama 2 70B skyline score of 0.84. The top model was a few percentage points shy of human performance on BLiMP. (Table 1; Figure 3)
Scope: BLiMP minimal-pair accuracy on English syntax; Llama 2 was evaluated on GLUE and SuperGLUE with in-context learning rather than fine-tuning.
Cutting the BabyLM training budget from 100 million to 10 million words cost the best submissions about 5 points of BLiMP, 0.85 down to 0.80. The corresponding GLUE drop was 4 points, 0.78 down to 0.74. (Table 1)
Scope: Top-3 submissions in the Strict and Strict-Small tracks; both corpora used the same source composition, with Strict-Small sampled at 10% from each source.
Loose-track BabyLM models, permitted extra non-linguistic data such as audio, code or images, tended to score lower in the aggregate than Strict-Small models trained on 10 million words of text alone. (Table 1; Figure 3)
Scope: First BabyLM iteration only, with 2023-era architectures not designed for multimodal pretraining; the extra modality data was not counted against the 100M-word budget.
BabyLM submissions performed about the same on BLiMP subtasks central to poverty-of-the-stimulus debates as on the remaining BLiMP subtasks. The POS-relevant subtasks covered island constraints, filler-gap dependencies and subject-aux inversion. (Figure 4)
Scope: Post-hoc partition of BLiMP subtasks by the paper's authors, averaged across submissions within each track, with 95% CIs across model scores; English syntax only.
All top BabyLM models except Strict-Small McGill-BERT scored positively on MSGS, indicating a preference for linguistic over surface generalizations comparable to Llama 2 (0.26) and RoBERTa-base (0.24). (Table 1)
Scope: MSGS Matthews correlation after fine-tuning on ambiguous training sets; MSGS has not been run with human subjects, so no human reference point exists, and Loose-track McGill-BERT scored -0.02.
Curriculum learning was the most popular strategy among BabyLM submissions yet produced only marginal gains over the baselines, while data preprocessing and architectural modifications were the most effective strategies. (Figure 5; Figure 6)
Scope: Meta-analysis by hand-coding each submission into 9 approach categories, counting at most one model per participant per track; curricula tested were those participants chose, sorted mostly by simplicity metrics.
LTG-BERT is the architecture Wilcox et al. recommend as a starting point for small-scale language modeling. ELC-BERT's added layer-wise skip connections gave no advantage over it when both were trained for 20 epochs on the 100-million-word Strict corpus. (Table A.2; Appendix A)
Scope: Reproduction runs averaged over 3 seeds on 4 NVIDIA RTX8000 GPUs, with smaller batch size and shorter sequence length than the original LTG-BERT paper; the 2 models tie on BLiMP at 0.83.
Training ELC-BERT and LTG-BERT for 20 epochs instead of the 450 or more used in the winning submission cost about 2 points on BLiMP and GLUE. It also cost about 10 points on the BLiMP Supplement, leaving the 20-epoch models ahead of McGill-BERT on GLUE but behind it on BLiMP. (Table A.2)
Scope: Reproductions on the 100-million-word Strict corpus, 3 seeds, with batch size and sequence length differing from the original submissions.
BLiMP gains from additional training epochs on the BabyLM corpora diminish roughly exponentially, so most benefit of repeated exposure arrives within the first 20 epochs. Strict-Small GLUE performance declines after 50 epochs. (Figure A.8; Figure A.9)
Scope: LTG-BERT on the Strict and Strict-Small corpora, losses and scores averaged over 3 seeds; training loss correlates with BLiMP at -0.99 (Strict) and -0.95 (Strict-Small), but with GLUE at only 0.61 for Strict-Small.
The BabyLM Challenge showed that robust linguistic generalizations are learnable from human-scale data, but not through cognitively plausible mechanisms. Winning systems relied on transformer optimization tricks, hundreds of training epochs, and large-scale data augmentation.
Scope: The first (2023) iteration's submissions; only 7 teams ran the optional age-of-acquisition evaluation, and no submission was evaluated against incremental reading-time data.
The first BabyLM Challenge received 31 papers and 162 submitted models, establishing a population of openly released models that are all effective data-efficient learners of English. (Section 'Submitted systems and results'; Figure 2)
Scope: CoNLL 2023 iteration; participants could enter multiple tracks, so unique-participant counts per track are lower than the model count.

Common misreadings

A BabyLM model scoring near Llama 2 on BLiMP and GLUE is not equivalent to a large language model: BabyLMs generate repetitive and sometimes nonsensical text and are poor at following instructions or learning from in-context examples.
The BabyLM Challenge did not establish that models learn language the way children do -- the winning systems used hundreds of epochs over the same corpus, low-level transformer optimization tricks, and data augmentation, none of which is cognitively plausible.
The negative result for curriculum learning bears on model training regimes; it is not evidence that child-directed speech is useless for children, though the paper reads it as supporting skepticism that child-directed speech is necessary for effective learning.
All BabyLM Challenge results are for English, so whether the same architectures and data efficiencies hold for typologically different languages is untested.
ELC-BERT winning both Strict and Strict-Small tracks does not mean its layer-wise skip connections were what made it win: under a controlled 20-epoch budget it performs comparably to the plain LTG-BERT backbone.
The claim is not that scaling harms language technology performance -- the paper grants that scaling largely benefits applications, and locates the harm in scientific uses such as poverty-of-the-stimulus arguments and reading-time prediction.

Terminology in this paper

BabyLM Corpus: The 100-million-word English pretraining corpus released for the BabyLM Challenge, in which about 56% is transcribed or scripted speech and about 40% comes from sources intended or suitable for children, with the rest from adult writing such as Wikipedia and Project Gutenberg.
Strict, Strict-Small and Loose tracks: The three BabyLM Challenge entry conditions: 100 million English training tokens (Strict), 10 million (Strict-Small), and 100 million tokens plus optional non-linguistic data such as audio, code or images (Loose).
skyline model: A large, fully-scaled reference model included for comparison rather than as a competitor, such as the 70-billion-parameter Llama 2 evaluated alongside 100-million-word models.
POS-relevant BLiMP subtasks: The BLiMP minimal-pair subtasks whose phenomena have figured in poverty-of-the-stimulus learnability debates: island constraints, filler-gap dependencies and subject-aux inversion.
MSGS: The Mixed Signals Generalization Set, a fine-tuning benchmark that trains a model on labels consistent with both a linguistic and a surface generalization and then tests which one the model adopted, scored as a Matthews correlation where 1 is systematic linguistic generalization and -1 systematic surface generalization.

How to cite

@article{wilcox2024bigger,
  title={Bigger is not always better: The importance of human-scale language modeling for psycholinguistics},
  author={Wilcox, Ethan Gotlieb and Hu, Michael Y and Mueller, Aaron and Warstadt, Alex and Choshen, Leshem and Zhuang, Chengxu and Williams, Adina and Cotterell, Ryan and Linzen, Tal},
  journal={Journal of Memory and Language},
  volume={144},
  pages={104650},
  year={2025},
  publisher={Elsevier}
}

References

See the full reference list in the paper.