Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

shared task on pretraining language models with as little text as a child hears

Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, Ryan Cotterell · CoNLL 2023 · 2023

In one sentence

The BabyLM Challenge is a shared task in which participants pretrain language models on a fixed 10M- or 100M-word developmentally plausible corpus, and in its 2023 first iteration the winning LTG-BERT-based submission outscored Llama 2 and RoBERTa-base in aggregate while most curriculum-learning attempts failed.

Abstract

Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements.

Questions this paper answers

Is there an open competition for training language models on the amount of language a child hears?
What shared task benchmarks sample-efficient pretraining on developmentally plausible corpora of 10M or 100M words?
Where can I find a fixed small-scale pretraining corpus and evaluation pipeline to test my own language model?
If I want to compare my small-data pretraining run against others, is there an existing leaderboard I can enter?
The BabyLM Challenge established a shared task and public leaderboard for pretraining language models on developmentally plausible budgets of 10M or 100M words. It supplied a fixed corpus of child-directed speech, dialogue and children's literature plus a common evaluation pipeline.
Holds for: First iteration, 2023, English only, 31 papers and 162 models across the Strict, Strict-Small and Loose tracks; earlier data-limited work existed (LTG-BERT, BabyBERTa, MiniPile) but not as a community shared task with a common corpus and leaderboard.
Can a language model trained on 100 million words of text score better than one trained on trillions?
Did any BabyLM Strict-track submission exceed the Llama 2 and RoBERTa-base skylines on the aggregate evaluation score?
How do I find out what the strongest 100M-word pretrained encoder achieved and what it was compared against?
Should I expect a model I pretrain on 100M words to be competitive with off-the-shelf large models on grammar and understanding benchmarks?
ELC-BERT, trained on 100M words in the BabyLM Strict track, reached an aggregate score of 0.74. That is above the Llama 2 skyline at 0.71, trained on 2T tokens, and above RoBERTa-base at 0.70 trained on its full corpus.
Holds for: (Super)GLUE was the one task where ELC-BERT did not beat the skylines, at 0.78 against Llama 2's 0.84; Llama 2 was evaluated on (Super)GLUE by in-context learning.
The two top BabyLM Strict-track systems, ELC-BERT at 0.74 aggregate and Boot-BERT at 0.70, both build on the LTG-BERT encoder architecture. The participants' own baselines indicate the backbone rather than their added modifications drove most of the gain.
Holds for: Encoder-only masked language models on 100M-word and 10M-word English BabyLM corpora, trained for hundreds to thousands of epochs; ELC-BERT used over 450 epochs in Strict and over 2000 in Strict-Small.
Does feeding a language model simple text before hard text help when there is very little training data?
Did curriculum learning yield consistent gains for BabyLM 2023 submissions across the evaluation suite?
How do I decide whether to order my pretraining data from easy to hard when data is capped at 100M words?
I only have 10M words to pretrain on, is curriculum learning worth my engineering time?
Curriculum learning was the most popular BabyLM 2023 approach, attempted by 13 of 31 teams (41.9%), and the majority of those attempts produced no consistent improvement across the BabyLM evaluation tasks.
Holds for: Curricula ranking by surprisal, lexical frequency, length, syntactic complexity, dataset difficulty, vocabulary size and objective difficulty, on the 10M-word and 100M-word English BabyLM corpora.
Which model design worked best for training on only 10 to 100 million words of text?
Was the LTG-BERT encoder backbone responsible for the top aggregate scores on developmentally plausible data budgets?
Which architecture should I start from if I am pretraining on a 100M-word corpus?
Do I get more from picking a stronger encoder backbone or from adding my own training modifications at small data scale?
The two top BabyLM Strict-track systems, ELC-BERT at 0.74 aggregate and Boot-BERT at 0.70, both build on the LTG-BERT encoder architecture. The participants' own baselines indicate the backbone rather than their added modifications drove most of the gain.
Holds for: Encoder-only masked language models on 100M-word and 10M-word English BabyLM corpora, trained for hundreds to thousands of epochs; ELC-BERT used over 450 epochs in Strict and over 2000 in Strict-Small.
ELC-BERT, trained on 100M words in the BabyLM Strict track, reached an aggregate score of 0.74. That is above the Llama 2 skyline at 0.71, trained on 2T tokens, and above RoBERTa-base at 0.70 trained on its full corpus.
Holds for: (Super)GLUE was the one task where ELC-BERT did not beat the skylines, at 0.78 against Llama 2's 0.84; Llama 2 was evaluated on (Super)GLUE by in-context learning.
How much better does a language model get when its training text goes from 10 million to 100 million words?
Did BabyLM Strict-track models trained on 100M words outperform Strict-Small models trained on 10M words on GLUE and aggregate scores?
How do I decide whether collecting 10x more pretraining text will actually raise my scores at this scale?
Is it worth spending effort to get 100M words of clean text if I already have 10M?
BabyLM Strict-track models trained on 100M words did not outperform Strict-Small models trained on 10M words by a large margin. Only 2 Strict-track models achieved higher GLUE scores than the best Strict-Small model.
Holds for: 162 submitted models across the three 2023 tracks, evaluated on the vocabulary-filtered BabyLM versions of BLiMP, BLiMP Supplement, (Super)GLUE and MSGS; the Strict corpus is 98.04M words and Strict-Small 9.96M.
Did giving language models images or other non-text input help when their text data was limited?
How did BabyLM Loose-track submissions with unlimited non-linguistic data compare in aggregate to 10M-word Strict-Small models?
Should I add visual or audio data to a small text corpus to improve my model's language scores?
I have image-caption data available on top of 100M words of text, will using it beat a text-only model trained on far less?
BabyLM Loose-track models, which could add unlimited non-linguistic data to a 100M-word text budget, tended to score lower in aggregate than Strict-Small models limited to 10M words of text.
Holds for: 20 Loose-track models from 8 participants in 2023; few multimodal submissions were received, and one text-and-audio system (WhisBERT) was reported undertrained.
How close to human accuracy on grammar tests can a model trained on a child-sized amount of text get?
What BLiMP minimal-pair accuracy did the best BabyLM 2023 submission reach relative to reported human performance?
If I pretrain on at most 100M words, how much grammatical acceptability performance am I giving up compared with humans?
The best BabyLM 2023 submission came within about 3% of reported human performance on BLiMP, despite training on at most 100M words.
Holds for: Zero-shot minimal-pair accuracy on the vocabulary-filtered BLiMP used in the challenge; human performance is the figure reported by Warstadt et al. (2020a).
When a sentence can be read two ways, do models trained on small amounts of text go by word patterns or by grammar?
What did MSGS Matthews correlations reveal about linguistic versus surface inductive bias in models trained on 10M-100M words?
How do I tell whether my small-data pretrained model generalizes on syntactic features rather than surface cues?
Do I need billions of words of pretraining before my model prefers linguistic generalizations over surface ones?
MSGS Matthews correlations for BabyLM 2023 systems were largely negative, showing that models trained on 10M-100M words prefer surface features over linguistic ones in ambiguous contexts. ELC-BERT was the exception, at -0.01 in Strict-Small and -0.10 in Strict against Llama 2's -0.24.
Holds for: Six ambiguous MSGS subtasks under finetuned evaluation; macro-average MCC for the top systems per track plus baselines and skylines. Llama 2 was fully finetuned on MSGS.
Can language models tell that a robin is a bird and not the other way round?
How did BabyLM 2023 submissions and the skylines score on the Hypernym suite of the BLiMP Supplement?
Should I trust a small pretrained model to handle lexical entailment between general and specific words?
On the Hypernym test suite of the BLiMP Supplement, every BabyLM 2023 system and both skylines scored near chance, between 0.45 and 0.50 accuracy.
Holds for: Zero-shot minimal-pair scoring on 860 semi-automatically templated lexical-entailment items phrased as logical statements.
What extra language tests did the BabyLM Challenge add beyond standard grammar and understanding benchmarks?
Which BLiMP Supplement suites, such as Turn-Taking and Hypernym, discriminate between BabyLM systems and which saturate at chance?
Which evaluation suites should I use if I want to separate strong from weak models pretrained on 10M-100M words?
If I run the BabyLM Supplement suites on my model, which of them will actually tell me something about it?
The Turn-Taking suite of the BLiMP Supplement separated BabyLM 2023 systems sharply, with ELC-BERT (Strict) reaching 0.92 against RoBERTa's 0.73 and Llama 2's 0.83, while some systems scored near chance.
Holds for: 280 templated dialogue minimal pairs on indexical pronoun choice across a speaker change, scored zero-shot; transcribed dialogue is a large share of the BabyLM corpus.
On the Hypernym test suite of the BLiMP Supplement, every BabyLM 2023 system and both skylines scored near chance, between 0.45 and 0.50 accuracy.
Holds for: Zero-shot minimal-pair scoring on 860 semi-automatically templated lexical-entailment items phrased as logical statements.
Can a language model predict the age at which children learn particular words?
Did any BabyLM 2023 Strict-Small submission beat the OPT-125M baseline on age-of-acquisition prediction measured by mean average deviation?
How do I check whether a small pretrained model's word learning lines up with children's acquisition order?
Is a BabyLM-style model a better fit than a plain 125M-parameter baseline if I want to model children's word acquisition?
On the optional age-of-acquisition prediction task, no BabyLM 2023 Strict-Small submission beat the OPT-125M baseline's mean average deviation of 2.03 months, with the best submissions at 2.05.
Holds for: 7 of 31 teams (22.6%) evaluated on AoA prediction, almost all in Strict-Small; MAD in months across cross-validation folds.
Which training tweaks actually helped models learn more from a small amount of text?
Which modifications, such as reduced context length, sentence-level examples or same-corpus distillation, consistently improved BabyLM scores?
What changes should I make to my pretraining recipe to squeeze more out of a 10M-100M word corpus?
I am pretraining on 10 million words, should I shorten sequences or distill from a teacher trained on the same data?
Reducing context length or using single sentences as training examples, and distilling a student from a teacher trained on the same corpus, were the BabyLM 2023 modifications that most consistently improved scores.
Holds for: Based on hand-coding 162 submitted models into 9 approach categories and on participants' own controlled comparisons, not on organizer-run ablations; English BabyLM corpora of 10M and 100M words.
Was the amount of computation limited in the child-scale language model competition, or only the amount of text?
Did the BabyLM 2023 rules constrain compute alongside the data budget, and how many training samples did the winning Strict submission consume?
How do I estimate the training cost of reproducing a winning 100M-word pretrained model?
If I train on only 100M words, can I expect the run to be cheap, or will I need as many training steps as BERT?
The BabyLM 2023 rules capped training data but not compute, and the winning Strict submission consumed roughly as many training samples as BERT despite a training set only about 3% as large.
Holds for: The 2023 iteration's rules, under which repeated epochs did not count against the word budget; the organizers flag compute efficiency as a target for future iterations.
The two top BabyLM Strict-track systems, ELC-BERT at 0.74 aggregate and Boot-BERT at 0.70, both build on the LTG-BERT encoder architecture. The participants' own baselines indicate the backbone rather than their added modifications drove most of the gain.
Holds for: Encoder-only masked language models on 100M-word and 10M-word English BabyLM corpora, trained for hundreds to thousands of epochs; ELC-BERT used over 450 epochs in Strict and over 2000 in Strict-Small.
If a model scores higher on grammar and language understanding tests, does it also match how hard humans find text to read?
Does BabyLM aggregate benchmark performance correlate with predicting human reading times as a psycholinguistic measure?
How do I choose a small pretrained model if what I need is a predictor of human reading difficulty?
Can I pick the top-scoring BabyLM model as my cognitive model of reading, or is benchmark rank the wrong signal?
A BabyLM 2023 submission awarded for outstanding evaluation found that models scoring better on the BabyLM benchmark tasks were not better at predicting human reading difficulty.
Holds for: Decoder-only GPT-style models trained by Steuer et al. (2023) on BabyLM data; a single submission's finding, not an organizer-run meta-analysis across all 162 models.

Claims and scope

Common misreadings

Terminology in this paper

Strict track
BabyLM Challenge track requiring models to train exclusively on the released 100M-word English corpus of child-directed speech, dialogue and children's literature.
Strict-Small track
BabyLM Challenge track requiring models to train exclusively on a 10M-word subsample of the released BabyLM corpus, roughly the linguistic input of a child's first two to five years.
Loose track
BabyLM Challenge track allowing unlimited non-linguistic data (audio, images, code, music) and expert annotations alongside a 100M-word text budget that covers all language data used for any model in the pipeline.
skyline
In the BabyLM Challenge, a reference model trained on its full unrestricted corpus — RoBERTa-base and Llama 2 70B — run through the same evaluation pipeline to bound what large-scale pretraining achieves.
BLiMP Supplement
Five minimal-pair test suites released for the BabyLM Challenge covering phenomena BLiMP omits: hypernymy, subject-auxiliary inversion, turn-taking, and easy and tricky question-answer congruence.
MSGS
Mixed Signals Generalization Set: a finetuning benchmark whose training labels are ambiguous between a syntactic and a surface generalization, scored by Matthews correlation with the syntactic generalization, so 1 means systematic linguistic bias and -1 systematic surface bias.
age-of-acquisition prediction
Task converting a language model's average word surprisals into predicted ages at which children acquire those words, scored by mean average deviation in months from measured child acquisition ages.

How to cite

@inproceedings{warstadt2023findings,title = "Findings of the {B}aby{LM} Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora",
    author = "Warstadt, Alex  and
      Mueller, Aaron  and
      Choshen, Leshem  and
      Wilcox, Ethan  and
      Zhuang, Chengxu  and
      Ciro, Juan  and
      Mosquera, Rafael  and
      Paranjabe, Bhargavi  and
      Williams, Adina  and
      Linzen, Tal  and
      Cotterell, Ryan",
    editor = "Warstadt, Alex  and
      Mueller, Aaron  and
      Choshen, Leshem  and
      Wilcox, Ethan  and
      Zhuang, Chengxu  and
      Ciro, Juan  and
      Mosquera, Rafael  and
      Paranjabe, Bhargavi  and
      Williams, Adina  and
      Linzen, Tal  and
      Cotterell, Ryan",
    booktitle = "Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.conll-babylm.1",
    doi = "10.18653/v1/2023.conll-babylm.1",
    pages = "1--34",
}

References

See the full reference list in the paper.