Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

shared task on pretraining language models with as little text as a child hears

Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, Ryan Cotterell · CoNLL 2023 · 2023

In one sentence

The BabyLM Challenge is a shared task in which participants pretrain language models on a fixed 10M- or 100M-word developmentally plausible corpus, and in its 2023 first iteration the winning LTG-BERT-based submission outscored Llama 2 and RoBERTa-base in aggregate while most curriculum-learning attempts failed.

Abstract

Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements.

Questions this paper answers

What should I read first about training language models on small, child-scale amounts of data?
Is there a shared task for sample-efficient language model pretraining?
Where did the BabyLM Challenge come from and what did it set up?
The BabyLM Challenge established a shared task and public leaderboard for pretraining language models on developmentally plausible budgets of 10M or 100M words. It supplied a fixed corpus of child-directed speech, dialogue and children's literature plus a common evaluation pipeline.
Holds for: First iteration, 2023, English only, 31 papers and 162 models across the Strict, Strict-Small and Loose tracks; earlier data-limited work existed (LTG-BERT, BabyBERTa, MiniPile) but not as a community shared task with a common corpus and leaderboard.
Can a language model trained on 100 million words beat one trained on trillions?
Did any BabyLM submission outperform Llama 2?
Which model won the BabyLM Strict track and what did it score?
ELC-BERT, trained on 100M words in the BabyLM Strict track, reached an aggregate score of 0.74. That is above the Llama 2 skyline at 0.71, trained on 2T tokens, and above RoBERTa-base at 0.70 trained on its full corpus.
Holds for: (Super)GLUE was the one task where ELC-BERT did not beat the skylines, at 0.78 against Llama 2's 0.84; Llama 2 was evaluated on (Super)GLUE by in-context learning.
The two top BabyLM Strict-track systems, ELC-BERT at 0.74 aggregate and Boot-BERT at 0.70, both build on the LTG-BERT encoder architecture. The participants' own baselines indicate the backbone rather than their added modifications drove most of the gain.
Holds for: Encoder-only masked language models on 100M-word and 10M-word English BabyLM corpora, trained for hundreds to thousands of epochs; ELC-BERT used over 450 epochs in Strict and over 2000 in Strict-Small.
Does curriculum learning help when pretraining data is limited?
How well did curriculum learning work in the BabyLM Challenge?
Is ordering training sentences from easy to hard worth trying for small-data language models?
Curriculum learning was the most popular BabyLM 2023 approach, attempted by 13 of 31 teams (41.9%), and the majority of those attempts produced no consistent improvement across the BabyLM evaluation tasks.
Holds for: Curricula ranking by surprisal, lexical frequency, length, syntactic complexity, dataset difficulty, vocabulary size and objective difficulty, on the 10M-word and 100M-word English BabyLM corpora.
Which architecture works best for pretraining on 10M-100M words?
Was LTG-BERT effective on developmentally plausible data budgets?
What backbone did the winning BabyLM models use?
The two top BabyLM Strict-track systems, ELC-BERT at 0.74 aggregate and Boot-BERT at 0.70, both build on the LTG-BERT encoder architecture. The participants' own baselines indicate the backbone rather than their added modifications drove most of the gain.
Holds for: Encoder-only masked language models on 100M-word and 10M-word English BabyLM corpora, trained for hundreds to thousands of epochs; ELC-BERT used over 450 epochs in Strict and over 2000 in Strict-Small.
ELC-BERT, trained on 100M words in the BabyLM Strict track, reached an aggregate score of 0.74. That is above the Llama 2 skyline at 0.71, trained on 2T tokens, and above RoBERTa-base at 0.70 trained on its full corpus.
Holds for: (Super)GLUE was the one task where ELC-BERT did not beat the skylines, at 0.78 against Llama 2's 0.84; Llama 2 was evaluated on (Super)GLUE by in-context learning.
How much does going from 10 million to 100 million words of pretraining data actually buy you?
Did BabyLM Strict-track models beat Strict-Small models by much?
Is a 10x increase in pretraining data worth it at this scale?
BabyLM Strict-track models trained on 100M words did not outperform Strict-Small models trained on 10M words by a large margin. Only 2 Strict-track models achieved higher GLUE scores than the best Strict-Small model.
Holds for: 162 submitted models across the three 2023 tracks, evaluated on the vocabulary-filtered BabyLM versions of BLiMP, BLiMP Supplement, (Super)GLUE and MSGS; the Strict corpus is 98.04M words and Strict-Small 9.96M.
Did adding images or audio help language models trained on small data in BabyLM?
How did multimodal submissions to the BabyLM Loose track perform?
Does multimodal input improve sample efficiency for language modelling?
BabyLM Loose-track models, which could add unlimited non-linguistic data to a 100M-word text budget, tended to score lower in aggregate than Strict-Small models limited to 10M words of text.
Holds for: 20 Loose-track models from 8 participants in 2023; few multimodal submissions were received, and one text-and-audio system (WhisBERT) was reported undertrained.
How close are small-data language models to human grammar performance?
What BLiMP accuracy did the best BabyLM model reach relative to humans?
Can models trained on child-scale data pass minimal-pair grammar tests?
The best BabyLM 2023 submission came within about 3% of reported human performance on BLiMP, despite training on at most 100M words.
Holds for: Zero-shot minimal-pair accuracy on the vocabulary-filtered BLiMP used in the challenge; human performance is the figure reported by Warstadt et al. (2020a).
Do language models trained on small corpora prefer syntactic or surface generalizations?
What did MSGS reveal about the inductive bias of BabyLM models?
Does linguistic inductive bias require billions of words of pretraining?
MSGS Matthews correlations for BabyLM 2023 systems were largely negative, showing that models trained on 10M-100M words prefer surface features over linguistic ones in ambiguous contexts. ELC-BERT was the exception, at -0.01 in Strict-Small and -0.10 in Strict against Llama 2's -0.24.
Holds for: Six ambiguous MSGS subtasks under finetuned evaluation; macro-average MCC for the top systems per track plus baselines and skylines. Llama 2 was fully finetuned on MSGS.
Do language models understand hypernym and lexical entailment relations?
How did BabyLM models do on the Hypernym test suite?
Which BLiMP Supplement task defeated every model including the skylines?
On the Hypernym test suite of the BLiMP Supplement, every BabyLM 2023 system and both skylines scored near chance, between 0.45 and 0.50 accuracy.
Holds for: Zero-shot minimal-pair scoring on 860 semi-automatically templated lexical-entailment items phrased as logical statements.
What new evaluation tasks did the BabyLM Challenge add beyond BLiMP and GLUE?
Which BLiMP Supplement suite best separates strong from weak small-data models?
Can language models track pronoun shifts across a change of speaker?
The Turn-Taking suite of the BLiMP Supplement separated BabyLM 2023 systems sharply, with ELC-BERT (Strict) reaching 0.92 against RoBERTa's 0.73 and Llama 2's 0.83, while some systems scored near chance.
Holds for: 280 templated dialogue minimal pairs on indexical pronoun choice across a speaker change, scored zero-shot; transcribed dialogue is a large share of the BabyLM corpus.
On the Hypernym test suite of the BLiMP Supplement, every BabyLM 2023 system and both skylines scored near chance, between 0.45 and 0.50 accuracy.
Holds for: Zero-shot minimal-pair scoring on 860 semi-automatically templated lexical-entailment items phrased as logical statements.
Can small language models predict the age at which children acquire words?
Did any BabyLM submission beat the baseline on age-of-acquisition prediction?
How well do BabyLM models align with children's word learning?
On the optional age-of-acquisition prediction task, no BabyLM 2023 Strict-Small submission beat the OPT-125M baseline's mean average deviation of 2.03 months, with the best submissions at 2.05.
Holds for: 7 of 31 teams (22.6%) evaluated on AoA prediction, almost all in Strict-Small; MAD in months across cross-validation folds.
What practical tricks improved sample efficiency in the BabyLM Challenge?
Does shortening context length or sentence-level batching help low-resource pretraining?
Did knowledge distillation help models trained on 10 million words?
Reducing context length or using single sentences as training examples, and distilling a student from a teacher trained on the same corpus, were the BabyLM 2023 modifications that most consistently improved scores.
Holds for: Based on hand-coding 162 submitted models into 9 approach categories and on participants' own controlled comparisons, not on organizer-run ablations; English BabyLM corpora of 10M and 100M words.
Did the BabyLM Challenge limit compute as well as data?
How many epochs did the winning BabyLM model train for?
Were data-limited pretraining winners actually cheap to train?
The BabyLM 2023 rules capped training data but not compute, and the winning Strict submission consumed roughly as many training samples as BERT despite a training set only about 3% as large.
Holds for: The 2023 iteration's rules, under which repeated epochs did not count against the word budget; the organizers flag compute efficiency as a target for future iterations.
The two top BabyLM Strict-track systems, ELC-BERT at 0.74 aggregate and Boot-BERT at 0.70, both build on the LTG-BERT encoder architecture. The participants' own baselines indicate the backbone rather than their added modifications drove most of the gain.
Holds for: Encoder-only masked language models on 100M-word and 10M-word English BabyLM corpora, trained for hundreds to thousands of epochs; ELC-BERT used over 450 epochs in Strict and over 2000 in Strict-Small.
Do better BLiMP and GLUE scores mean a model is a better cognitive model of reading?
Does BabyLM benchmark performance correlate with predicting human reading times?
Which BabyLM paper won the award for outstanding evaluation and what did it find?
A BabyLM 2023 submission awarded for outstanding evaluation found that models scoring better on the BabyLM benchmark tasks were not better at predicting human reading difficulty.
Holds for: Decoder-only GPT-style models trained by Steuer et al. (2023) on BabyLM data; a single submission's finding, not an organizer-run meta-analysis across all 162 models.

Claims and scope

Common misreadings

Terminology in this paper

Strict track
BabyLM Challenge track requiring models to train exclusively on the released 100M-word English corpus of child-directed speech, dialogue and children's literature.
Strict-Small track
BabyLM Challenge track requiring models to train exclusively on a 10M-word subsample of the released BabyLM corpus, roughly the linguistic input of a child's first two to five years.
Loose track
BabyLM Challenge track allowing unlimited non-linguistic data (audio, images, code, music) and expert annotations alongside a 100M-word text budget that covers all language data used for any model in the pipeline.
skyline
In the BabyLM Challenge, a reference model trained on its full unrestricted corpus — RoBERTa-base and Llama 2 70B — run through the same evaluation pipeline to bound what large-scale pretraining achieves.
BLiMP Supplement
Five minimal-pair test suites released for the BabyLM Challenge covering phenomena BLiMP omits: hypernymy, subject-auxiliary inversion, turn-taking, and easy and tricky question-answer congruence.
MSGS
Mixed Signals Generalization Set: a finetuning benchmark whose training labels are ambiguous between a syntactic and a surface generalization, scored by Matthews correlation with the syntactic generalization, so 1 means systematic linguistic bias and -1 systematic surface bias.
age-of-acquisition prediction
Task converting a language model's average word surprisals into predicted ages at which children acquire those words, scored by mean average deviation in months from measured child acquisition ages.

How to cite

@inproceedings{warstadt2023findings,title = "Findings of the {B}aby{LM} Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora",
    author = "Warstadt, Alex  and
      Mueller, Aaron  and
      Choshen, Leshem  and
      Wilcox, Ethan  and
      Zhuang, Chengxu  and
      Ciro, Juan  and
      Mosquera, Rafael  and
      Paranjabe, Bhargavi  and
      Williams, Adina  and
      Linzen, Tal  and
      Cotterell, Ryan",
    editor = "Warstadt, Alex  and
      Mueller, Aaron  and
      Choshen, Leshem  and
      Wilcox, Ethan  and
      Zhuang, Chengxu  and
      Ciro, Juan  and
      Mosquera, Rafael  and
      Paranjabe, Bhargavi  and
      Williams, Adina  and
      Linzen, Tal  and
      Cotterell, Ryan",
    booktitle = "Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.conll-babylm.1",
    doi = "10.18653/v1/2023.conll-babylm.1",
    pages = "1--34",
}

References

See the full reference list in the paper.