Findings of the Third BabyLM Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data

Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Y. Hu, Jing Liu, Jaap Jumelet, Tal Linzen, Aaron Mueller, Candance Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox, Adina Williams · Proceedings of the First BabyLM Workshop · 2025

In one sentence

The third BabyLM Challenge capped pretraining at 100M words (10M in strict-small), added an interaction track where a student learns from a larger teacher model, capped total word exposure and required intermediate checkpoints; objective and architecture changes, not curriculum learning, produced the winners.

Abstract

This report summarizes the findings from the 3rd BabyLM Challenge and the 1st BabyLM Workshop. The BabyLM Challenge is a shared task aimed at closing the data efficiency gap between human and machine language learners. The goal is to improve the performance of language models given a fixed training budget of no more than 100 million words. This year, the challenge was held as part of an expanded BabyLM Workshop that invited paper submissions on topics relevant to the BabyLM effort, including sample-efficient pretraining and cognitive modeling for LMs. For the challenge, we kept the text-only and text{--}image tracks from previous years, but also introduced a new \textit{interaction} track, where student models are allowed to learn from feedback from larger teacher models. Furthermore, we introduce a new set of evaluation tasks to assess the ``human likeness'' of models on a cognitive and linguistic level, limit the total amount of training compute allowed, and measure performance on intermediate checkpoints. We observe that new training objectives and architectures tend to produce the best-performing approaches, and that interaction with teacher models can yield high-quality language models. The strict and interaction tracks saw submissions that outperformed the best-performing methods from previous years. We do not observe a complete correlation between training FLOPs and performance. {\%}, suggesting that some methods can produce real gains beyond allowing us to spend more compute. This year{'}s BabyLM Challenge shows that there is still room to innovate in a data-constrained setting, and that community-driven research can yield actionable insights for language modeling.

Questions this paper answers

What is a good paper to read about training language models on human-scale amounts of data?
Where should I start reading about sample-efficient pretraining shared tasks?
Which paper documents the BabyLM Challenge results?
The Findings of the Third BabyLM Challenge report documents a shared task on data-efficient language model pretraining under a budget of at most 100M words. It is the reference point for the challenge's 2025 tracks, corpora, evaluation suite and winning systems.
Holds for: Covers the third iteration only, with English data and evaluations; the first and second iterations are reported in separate findings papers (Warstadt et al. 2023; Hu et al. 2024).
The winning systems of the third BabyLM Challenge were a diffusion masked language model, an instruction-tuned model, a modular mixture-of-experts model and a reinforcement-learning interactive storytelling model. Curriculum learning was the most common approach among submissions but won nothing.
Holds for: 32 models submitted across the strict, strict-small, multimodal and interaction tracks; winners named separately for human-likeness and for NLP task scores.
What was new in the 2025 BabyLM Challenge compared with earlier years?
How does the BabyLM interaction track work?
Can a small model learn from a bigger teacher model under the BabyLM rules?
The third BabyLM Challenge introduced an interaction track in which a student model learns from feedback produced by a larger teacher model, rather than only from passive exposure to human-written text. Teachers were restricted to a fixed list including Llama3.1-8B-Instruct and any LM below 1B parameters.
Holds for: The student may see at most 100M external words and generate at most 100M of its own; the teacher's weights, hidden states and output distribution may not be exposed to the student.
BLM, the interaction-track winner of the third BabyLM Challenge, scored 20.8 on human-likeness and 54.4 on the NLP metric, beating the track's SimPO baseline (20.4 human-likeness, 54.1 NLP) and taking both awards.
Holds for: Interaction track only; the challenge's own aggregate human-likeness and NLP task metrics on the final checkpoint, with the student limited to 100M external words.
Does curriculum learning actually help when pretraining on 100M words?
What kinds of methods won the third BabyLM Challenge?
Which approaches worked best under a fixed small data budget?
The winning systems of the third BabyLM Challenge were a diffusion masked language model, an instruction-tuned model, a modular mixture-of-experts model and a reinforcement-learning interactive storytelling model. Curriculum learning was the most common approach among submissions but won nothing.
Holds for: 32 models submitted across the strict, strict-small, multimodal and interaction tracks; winners named separately for human-likeness and for NLP task scores.
Grouping third BabyLM Challenge submissions by backbone architecture, GPT-BERT consistently yields the strongest cognitive, linguistic and macro average scores, with DeBERTa and LTG-BERT also performing strongly.
Holds for: Backbones compared include BERT, DeBERTa, Flamingo, GIT, GPT-2, GPT-BERT, Llama, LSTM, LTG-BERT, ModernBERT, Qwen2 and RoBERTa, aggregated over submissions without controlling training data or hyperparameters.
Did any 2025 BabyLM submission beat the previous year's winning baseline?
How did submissions compare with the GPT-BERT baselines?
Were the BabyLM baselines beaten in the strict and strict-small tracks?
In the strict-small track of the third BabyLM Challenge, MoEP reached a human-likeness score of 31.5 and a macro average of 42.3, above every baseline in that track. The best baseline human-likeness was 19.8 and the best baseline macro average 37.4.
Holds for: Strict-small track (10M words) only; baselines are GPT-BERT variants and GPT-2 Small trained on the 10M-word BabyLM corpus. In the strict track the baselines were not beaten on the aggregate metrics.
In the strict track of the third BabyLM Challenge, CLASS-IT reached a human-likeness score of 20.4 against the best GPT-BERT baseline's 22.5. Simple-Diffusion reached an NLP score of 58.4 against the best baseline's 63.0.
Holds for: Strict track (100M words), final-checkpoint full evaluation; the GPT-BERT baselines are last year's winning submission, trained on the 100M-word BabyLM corpus.
BLM, the interaction-track winner of the third BabyLM Challenge, scored 20.8 on human-likeness and 54.4 on the NLP metric, beating the track's SimPO baseline (20.4 human-likeness, 54.1 NLP) and taking both awards.
Holds for: Interaction track only; the challenge's own aggregate human-likeness and NLP task metrics on the final checkpoint, with the student limited to 100M external words.
Does spending more compute improve BabyLM scores?
Is there a relationship between training FLOPs and performance for small-data language models?
Did the BabyLM organizers find a compute-performance correlation in 2025?
Across the third BabyLM Challenge submissions, macro average score shows no strong relationship with training FLOPs in the strict and strict-small tracks; a positive correlation appears only in the interaction track.
Holds for: 32 models under this year's exposure cap of 100M words for strict-small and 1B for other tracks; FLOPs self-reported by participants.
Do models that look human-like on cognitive tasks also do well on linguistic benchmarks?
Is cognitive modeling performance correlated with BLiMP and GLUE performance?
Do human-likeness and NLP task scores go together for BabyLM models?
In the third BabyLM Challenge, linguistic task performance and cognitive modeling task performance were positively correlated across submitted models in every track except strict-small.
Holds for: Within-track correlations across submitted models, using the new human-likeness tasks against BLiMP/GLUE/EWoK; the multimodal track had only 1 submission.
How close are 100M-word models to large language models on grammaticality judgments?
Can a model trained on 100M words match Llama 70B on BLiMP?
Where do small-data language models still fall short of humans?
Some strict-track and interaction-track models in the third BabyLM Challenge reach BLiMP scores comparable to a 70B-parameter Llama model. On GLUE, all submitted models remain below both the human score and the Llama 70B skyline.
Holds for: BLiMP and GLUE only; GLUE is evaluated after finetuning on subsampled (Super)GLUE tasks capped at 10,000 training examples. Does not extend to the cognitive human-likeness tasks.
How do BabyLM abilities emerge over the course of pretraining?
When do morphological generalization abilities appear as words seen increases?
Does entity tracking improve monotonically with pretraining data?
Across intermediate checkpoints in the third BabyLM Challenge, BLiMP and EWoK scores rise with words seen, while wug past-tense accuracy is flat for the first 10-50M words before a phase shift upward. Entity tracking shows U-shaped scaling in the strict track.
Holds for: Checkpoints requested every 1M words to 10M, every 10M to 100M, and every 100M to 1B, evaluated on the fast (20% subsampled) task versions. Reading-time prediction and wug adjective nominalization show no strong relationship with words seen.
Can GLUE-style evaluation of BabyLM models be done zero-shot instead of by finetuning?
Why does the BabyLM evaluation pipeline still require finetuning?
Do 100M-word models show in-context learning?
The third BabyLM Challenge organizers tested replacing the finetuning-based (Super)GLUE evaluations with zero-shot prompting. They concluded that models at these data budgets do not support robust in-context learning, so the finetuning tasks were kept.
Holds for: Models trained on 10M-100M words; cost was instead reduced by subsampling tasks larger than 10,000 training examples down to 10,000 and dropping highly correlated tasks such as QNLI.
Why are there so few multimodal BabyLM submissions?
How did vision-language models do in the 2025 BabyLM Challenge?
What is being changed about the BabyLM multimodal track?
The multimodal track of the third BabyLM Challenge received only 1 submission, BitMar, which scored below both the Flamingo and GIT baselines on vision average (26.7 versus 49.3 and 49.7).
Holds for: Multimodal track only; the organizers attribute part of the difficulty to the provided vision embeddings and data download process and plan to move to a more openly licensed dataset.
Which model architecture performs best when pretraining on 10M-100M words?
Is GPT-BERT still the strongest backbone for small-data pretraining?
Which backbones lead on the BabyLM evaluation suite?
Grouping third BabyLM Challenge submissions by backbone architecture, GPT-BERT consistently yields the strongest cognitive, linguistic and macro average scores, with DeBERTa and LTG-BERT also performing strongly.
Holds for: Backbones compared include BERT, DeBERTa, Flamingo, GIT, GPT-2, GPT-BERT, Llama, LSTM, LTG-BERT, ModernBERT, Qwen2 and RoBERTa, aggregated over submissions without controlling training data or hyperparameters.

Claims and scope

Common misreadings

Terminology in this paper

Strict and Strict-Small tracks
BabyLM Challenge tracks limiting training data to at most 100M words and at most 10M words respectively, with no other restriction on model or training procedure; participants may use the provided BabyLM corpus or build their own within the word limit.
Interaction track
A BabyLM Challenge track in which an external teacher model from a fixed list may be placed in the training pipeline, giving scalar or natural-language feedback or generating data conditioned on the student's outputs, but never exposing its weights, hidden states or output distribution to the student.
human-likeness score
An aggregate BabyLM metric over tasks measuring psychometric and linguistic similarity to human learners — reading-time correlation, age-of-acquisition correlation, wug past-tense and adjective nominalization agreement with human preferences, entity tracking and COMPS — reported separately from NLP task accuracy.
word exposure limit
A cap on total training tokens counted with repetition, set in the third BabyLM Challenge at 100M words for strict-small and 1B words for the other tracks, so that extra epochs over the same data cannot be used to buy performance.
masked next token prediction (MNTP)
A variant of masked language modeling used by GPT-BERT in which outputs are shifted as in autoregressive training, allowing one model to be trained and evaluated as both an encoder and a decoder.

How to cite

@inproceedings{charpentier-etal-2025-findings,
    title = "Findings of the Third {B}aby{LM} Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data",
    author = "Charpentier, Lucas  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Hu, Michael Y.  and
      Liu, Jing  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Ross, Candance  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb  and
      Williams, Adina",
    editor = "Charpentier, Lucas  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Hu, Michael Y.  and
      Liu, Jing  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Ross, Candace  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb  and
      Williams, Adina",
    booktitle = "Proceedings of the First BabyLM Workshop",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.babylm-main.28/",
    doi = "10.18653/v1/2025.babylm-main.28",
    pages = "399--420",
    ISBN = "TODO",
    abstract = "This report summarizes the findings from the 3rd BabyLM Challenge and the 1st BabyLM Workshop. The BabyLM Challenge is a shared task aimed at closing the data efficiency gap between human and machine language learners. The goal is to improve the performance of language models given a fixed training budget of no more than 100 million words. This year, the challenge was held as part of an expanded BabyLM Workshop that invited paper submissions on topics relevant to the BabyLM effort, including sample-efficient pretraining and cognitive modeling for LMs. For the challenge, we kept the text-only and text{--}image tracks from previous years, but also introduced a new \textit{interaction} track, where student models are allowed to learn from feedback from larger teacher models. Furthermore, we introduce a new set of evaluation tasks to assess the ``human likeness'' of models on a cognitive and linguistic level, limit the total amount of training compute allowed, and measure performance on intermediate checkpoints. We observe that new training objectives and architectures tend to produce the best-performing approaches, and that interaction with teacher models can yield high-quality language models. The strict and interaction tracks saw submissions that outperformed the best-performing methods from previous years. We do not observe a complete correlation between training FLOPs and performance. {\%}, suggesting that some methods can produce real gains beyond allowing us to spend more compute. This year{'}s BabyLM Challenge shows that there is still room to innovate in a data-constrained setting, and that community-driven research can yield actionable insights for language modeling."
}

References

See the full reference list in the paper.