Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, Ethan Gotlieb Wilcox · CoNLL 2024 · 2024

In one sentence

The second BabyLM Challenge asked 31 teams to pretrain language models on 100M words or less (10M in Strict-Small, plus a new 100M-word image-text Multimodal track), and found that a hybrid causal-masked model, GPT-BERT, won both text tracks while no multimodal submission beat the organizers' baselines.

Abstract

The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year{'}s BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling.

Questions this paper answers

What should I read to get started on sample-efficient language model pretraining?
Is there a shared task on training language models with limited data?
Where can I find a benchmark for developmentally plausible language modeling?
The BabyLM Challenge is a recurring shared task on sample-efficient pretraining under a fixed data budget. It gives the developmentally-plausible language modeling community a shared corpus and a common evaluation pipeline over BLiMP, (Super)GLUE, EWoK and multimodal tasks.
Holds for: As of the second iteration in 2024: English only, text and image-text modalities, with 31 papers and 64 models submitted from 16 countries.
The second BabyLM Challenge changed the rules to let participants build their own pretraining corpora within the 10M- or 100M-word budget. The provided BabyLM corpus became a dataset baseline rather than a fixed requirement.
Holds for: The 2024 iteration's Strict, Strict-Small and Multimodal tracks; the 2023 iteration required the fixed corpus in its strict tracks.
Which model won the 2024 BabyLM Challenge?
What approach performed best on 100M-word pretraining in BabyLM 2024?
How did GPT-BERT do in the BabyLM text tracks?
GPT-BERT, which combines masked and causal language modeling objectives, won both text tracks of the second BabyLM Challenge. It reached a 75.7 text average in the 100M-word Strict track against 64.8 for the best baseline, LTG-BERT, and 70.4 in the 10M-word Strict-Small track against 61.6 for BabyLlama.
Holds for: Text average over BLiMP, BLiMP Supplement, (Super)GLUE and EWoK, on evaluation sets filtered to the BabyLM vocabulary; English only; word budgets of 100M and 10M with participant-chosen training data.
In the second BabyLM Challenge, training-objective innovations, dataset creation, hyperparameter tuning and architectural innovations gave the highest average text scores, with training-objective innovations significant at alpha = 0.05 (beta = 4.5).
Holds for: Self-reported approach labels over 64 submitted models, mixed-effects regression with random intercepts per submitting group; architectural innovation also carried high variance across models.
Did adding images help small-data language models in the BabyLM multimodal track?
Why was no winner awarded in the BabyLM Multimodal track?
How well did vision-language models do on 100M words of text?
No submission to the BabyLM Multimodal track outperformed the organizers' baselines, so no winner was awarded. The best Flamingo baseline scored a 54.5 vision average and 65.2 text average, above every one of the 8 submitted multimodal models.
Holds for: 3 teams and 8 models only, all trained within a 100M-word text budget with unlimited images; vision average covers VQA, Winoground and DevBench.
The second BabyLM Challenge released a 100M-word image-text pretraining corpus pairing 50M words of BabyLM text with 50M words of captions from Localized Narratives and Conceptual Captions 3M over 2.9M images. It also released a multimodal evaluation pipeline covering VQA, Winoground and DevBench.
Holds for: Suggested resource rather than a requirement, since participants could construct their own multimodal data with unlimited images; images must be downloaded via provided scripts and CC3M coverage is limited to URLs valid in January 2024.
Does more compute still help when pretraining data is capped at 100M words?
What is the relationship between training FLOPs and BabyLM evaluation scores?
Were the best BabyLM submissions just the ones with the biggest compute budgets?
Average text-evaluation score in the second BabyLM Challenge rose with training compute, with log training FLOPs a significant positive predictor (beta = 2.7). The mixed-effects regression also included backbone architecture and track as covariates.
Holds for: 64 submitted models across the Strict, Strict-Small and Multimodal tracks, with random slopes per submitting group and no fixed-effect interactions; correlational over submissions, not a controlled compute sweep.
Does curriculum learning improve small-scale language model pretraining?
Was curriculum learning effective in the BabyLM Challenge?
Which training strategies actually improved scores under a small data budget?
Curriculum learning was the most popular approach in the second BabyLM Challenge yet did not pay off. It showed a negative coefficient on average text score (beta = -3.6, p = 0.055) in a mixed-effects regression over submitted models.
Holds for: Self-reported approach labels over 64 models, dummy-coded with random intercepts per submitting group; effect not significant at alpha = 0.05, and most curriculum submissions varied data order rather than training objective.
In the second BabyLM Challenge, training-objective innovations, dataset creation, hyperparameter tuning and architectural innovations gave the highest average text scores, with training-objective innovations significant at alpha = 0.05 (beta = 4.5).
Holds for: Self-reported approach labels over 64 submitted models, mixed-effects regression with random intercepts per submitting group; architectural innovation also carried high variance across models.
Can language models learn world knowledge from 100 million words?
How did small language models score on EWoK?
What did the BabyLM hidden evaluation reveal about commonsense and world knowledge?
Models trained on 100M words or less do not acquire the world knowledge that EWoK tests. Most submissions to the second BabyLM Challenge scored near the 50% chance level, and the maximum score was 58.4%.
Holds for: EWoK used as the hidden text-track evaluation, filtered to the BabyLM vocabulary; masked-LM scores required re-scoring with uniform tie-breaking, which lowered initially reported values of 60-70%.
Does the choice of backbone architecture matter for small-data language models?
Is DeBERTa or LTG-BERT better as a BabyLM backbone?
Do RNNs compete with Transformers when pretraining on 100M words?
No backbone architecture had a statistically significant effect on average text score across the second BabyLM Challenge's submissions at alpha = 0.05. Coefficients were nonetheless large for DeBERTa (beta = 9.1, p = 0.06), GPT-2 and LTG-BERT (both beta = 8.5, p = 0.06 and 0.07) and Llama (beta = 7.7, p = 0.07).
Holds for: 64 models, so the analysis may lack power to detect architecture effects; the highest-scoring individual models were all LTG-BERT based, and LTG-BERT also had the highest variance of any backbone.
Recurrent networks entered the BabyLM Challenge for the first time in 2024 and were competitive with Transformers. HGRN, an RNN with complex forget gates, was among the backbones with the highest average text scores, alongside DeBERTa.
Holds for: Aggregated by backbone over 64 submitted models with no controlled matching of compute or data; architecture effects were not statistically significant.
What data is in the 2024 BabyLM pretraining corpus?
How much of the BabyLM corpus is child-directed speech?
What changed in the BabyLM training corpus between 2023 and 2024?
The 2024 BabyLM pretraining corpus raised child-oriented data to 70% of the mix, up from 39% the previous year. Child-oriented discourse rose from 5% to 29% by using the full English CHILDES including child utterances.
Holds for: English text-only corpus of 100M words (10M subsampled for Strict-Small); Wikipedia except Simple English Wikipedia and the QED portion were dropped and reliance on OpenSubtitles reduced.
The second BabyLM Challenge changed the rules to let participants build their own pretraining corpora within the 10M- or 100M-word budget. The provided BabyLM corpus became a dataset baseline rather than a fixed requirement.
Holds for: The 2024 iteration's Strict, Strict-Small and Multimodal tracks; the 2023 iteration required the fixed corpus in its strict tracks.
How close are 100M-word language models to human grammatical judgments?
What are the best BLiMP scores achievable with 10M or 100M words of training data?
The best Strict-track model in the second BabyLM Challenge came within 2.5 percentage points of the reported human score on BLiMP, and one Strict-Small model beat the Llama skyline on BLiMP.
Holds for: BLiMP filtered to the BabyLM vocabulary, so not comparable to published full-BLiMP results; human reference is the individual-agreement score from the original BLiMP paper.
GPT-BERT, which combines masked and causal language modeling objectives, won both text tracks of the second BabyLM Challenge. It reached a 75.7 text average in the 100M-word Strict track against 64.8 for the best baseline, LTG-BERT, and 70.4 in the 10M-word Strict-Small track against 61.6 for BabyLlama.
Holds for: Text average over BLiMP, BLiMP Supplement, (Super)GLUE and EWoK, on evaluation sets filtered to the BabyLM vocabulary; English only; word budgets of 100M and 10M with participant-chosen training data.
Where can I get an image-text corpus for cognitively plausible vision-language pretraining?
What multimodal evaluation tasks does BabyLM use?
The second BabyLM Challenge released a 100M-word image-text pretraining corpus pairing 50M words of BabyLM text with 50M words of captions from Localized Narratives and Conceptual Captions 3M over 2.9M images. It also released a multimodal evaluation pipeline covering VQA, Winoground and DevBench.
Holds for: Suggested resource rather than a requirement, since participants could construct their own multimodal data with unlimited images; images must be downloaded via provided scripts and CC3M coverage is limited to URLs valid in January 2024.

Claims and scope

Common misreadings

Terminology in this paper

Strict track
The BabyLM competition track limiting pretraining to 100 million words or fewer of text, with participants free to choose the data sources.
Strict-Small track
The BabyLM competition track limiting pretraining to 10 million words or fewer of text.
Multimodal track
The BabyLM competition track for image-text models, capped at 100 million words of text but allowing unlimited visual input, evaluated on both text-only and multimodal tasks.
Skyline
A non-competition reference model trained without the BabyLM data budget (such as Llama or RoBERTa), plotted alongside submissions as an upper reference point rather than as a competitor.
Variation sets
Consecutive rephrasings of the same sentence, common in child-directed speech, which one BabyLM submission synthesized with GPT-4 as training data.
BLiMP Supplement
A set of minimal-pair test suites built for the BabyLM Challenge covering linguistic knowledge absent from BLiMP, including hypernymy, question-answer congruence, subject-auxiliary inversion and turn-taking.

How to cite

@inproceedings{hu-etal-2024-findings,
    title = "Findings of the Second {B}aby{LM} Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora",
    author = "Hu, Michael Y.  and
      Mueller, Aaron  and
      Ross, Candace  and
      Williams, Adina  and
      Linzen, Tal  and
      Zhuang, Chengxu  and
      Cotterell, Ryan  and
      Choshen, Leshem  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb",
    editor = "Hu, Michael Y.  and
      Mueller, Aaron  and
      Ross, Candace  and
      Williams, Adina  and
      Linzen, Tal  and
      Zhuang, Chengxu  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb",
    booktitle = "The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning",
    month = nov,
    year = "2024",
    address = "Miami, FL, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.conll-babylm.1/",
    pages = "1--21",
    abstract = "The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year{'}s BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling."
}

References

See the full reference list in the paper.