# Findings of the Third BabyLM Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data

Authors: Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Y. Hu, Jing Liu, Jaap Jumelet, Tal Linzen, Aaron Mueller, Candance Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox, Adina Williams
Venue: Proceedings of the First BabyLM Workshop (2025)

## What this paper shows
The third BabyLM Challenge capped pretraining at 100M words (10M in strict-small), added an interaction track where a student learns from a larger teacher model, capped total word exposure and required intermediate checkpoints; objective and architecture changes, not curriculum learning, produced the winners.

## Claims, with scope
- The Findings of the Third BabyLM Challenge report documents a shared task on data-efficient language model pretraining under a budget of at most 100M words. It is the reference point for the challenge's 2025 tracks, corpora, evaluation suite and winning systems.
  Scope: Covers the third iteration only, with English data and evaluations; the first and second iterations are reported in separate findings papers (Warstadt et al. 2023; Hu et al. 2024).
- The third BabyLM Challenge introduced an interaction track in which a student model learns from feedback produced by a larger teacher model, rather than only from passive exposure to human-written text. Teachers were restricted to a fixed list including Llama3.1-8B-Instruct and any LM below 1B parameters.
  Scope: The student may see at most 100M external words and generate at most 100M of its own; the teacher's weights, hidden states and output distribution may not be exposed to the student.
- The winning systems of the third BabyLM Challenge were a diffusion masked language model, an instruction-tuned model, a modular mixture-of-experts model and a reinforcement-learning interactive storytelling model. Curriculum learning was the most common approach among submissions but won nothing.
  Scope: 32 models submitted across the strict, strict-small, multimodal and interaction tracks; winners named separately for human-likeness and for NLP task scores.
  Evidence: Figure 1 and Section 6.1
- BLM, the interaction-track winner of the third BabyLM Challenge, scored 20.8 on human-likeness and 54.4 on the NLP metric, beating the track's SimPO baseline (20.4 human-likeness, 54.1 NLP) and taking both awards.
  Scope: Interaction track only; the challenge's own aggregate human-likeness and NLP task metrics on the final checkpoint, with the student limited to 100M external words.
  Evidence: Table 3
- In the strict track of the third BabyLM Challenge, CLASS-IT reached a human-likeness score of 20.4 against the best GPT-BERT baseline's 22.5. Simple-Diffusion reached an NLP score of 58.4 against the best baseline's 63.0.
  Scope: Strict track (100M words), final-checkpoint full evaluation; the GPT-BERT baselines are last year's winning submission, trained on the 100M-word BabyLM corpus.
  Evidence: Table 3
- In the strict-small track of the third BabyLM Challenge, MoEP reached a human-likeness score of 31.5 and a macro average of 42.3, above every baseline in that track. The best baseline human-likeness was 19.8 and the best baseline macro average 37.4.
  Scope: Strict-small track (10M words) only; baselines are GPT-BERT variants and GPT-2 Small trained on the 10M-word BabyLM corpus. In the strict track the baselines were not beaten on the aggregate metrics.
  Evidence: Table 3
- Across the third BabyLM Challenge submissions, macro average score shows no strong relationship with training FLOPs in the strict and strict-small tracks; a positive correlation appears only in the interaction track.
  Scope: 32 models under this year's exposure cap of 100M words for strict-small and 1B for other tracks; FLOPs self-reported by participants.
  Evidence: Figure 6
- In the third BabyLM Challenge, linguistic task performance and cognitive modeling task performance were positively correlated across submitted models in every track except strict-small.
  Scope: Within-track correlations across submitted models, using the new human-likeness tasks against BLiMP/GLUE/EWoK; the multimodal track had only 1 submission.
  Evidence: Figure 3
- Some strict-track and interaction-track models in the third BabyLM Challenge reach BLiMP scores comparable to a 70B-parameter Llama model. On GLUE, all submitted models remain below both the human score and the Llama 70B skyline.
  Scope: BLiMP and GLUE only; GLUE is evaluated after finetuning on subsampled (Super)GLUE tasks capped at 10,000 training examples. Does not extend to the cognitive human-likeness tasks.
  Evidence: Figure 4
- Across intermediate checkpoints in the third BabyLM Challenge, BLiMP and EWoK scores rise with words seen, while wug past-tense accuracy is flat for the first 10-50M words before a phase shift upward. Entity tracking shows U-shaped scaling in the strict track.
  Scope: Checkpoints requested every 1M words to 10M, every 10M to 100M, and every 100M to 1B, evaluated on the fast (20% subsampled) task versions. Reading-time prediction and wug adjective nominalization show no strong relationship with words seen.
  Evidence: Figure 5 and Section 7
- Grouping third BabyLM Challenge submissions by backbone architecture, GPT-BERT consistently yields the strongest cognitive, linguistic and macro average scores, with DeBERTa and LTG-BERT also performing strongly.
  Scope: Backbones compared include BERT, DeBERTa, Flamingo, GIT, GPT-2, GPT-BERT, Llama, LSTM, LTG-BERT, ModernBERT, Qwen2 and RoBERTa, aggregated over submissions without controlling training data or hyperparameters.
  Evidence: Figure 7
- The third BabyLM Challenge organizers tested replacing the finetuning-based (Super)GLUE evaluations with zero-shot prompting. They concluded that models at these data budgets do not support robust in-context learning, so the finetuning tasks were kept.
  Scope: Models trained on 10M-100M words; cost was instead reduced by subsampling tasks larger than 10,000 training examples down to 10,000 and dropping highly correlated tasks such as QNLI.
  Evidence: Section 4.2
- The multimodal track of the third BabyLM Challenge received only 1 submission, BitMar, which scored below both the Flamingo and GIT baselines on vision average (26.7 versus 49.3 and 49.7).
  Scope: Multimodal track only; the organizers attribute part of the difficulty to the provided vision embeddings and data download process and plan to move to a more openly licensed dataset.
  Evidence: Table 2 and Table 3

## Common misreadings
- The third BabyLM Challenge did not show that submissions beat the baselines everywhere: in the strict track the GPT-BERT baselines still had the best human-likeness (22.5) and best NLP score (63.0), and in the multimodal track the sole submission scored below both baselines on vision.
- The finding that FLOPs and performance are weakly related is not evidence that compute is irrelevant: a positive correlation still appears in the interaction track, this year's exposure cap removed one of the main ways compute was previously converted into score, and hyperparameter tuning remains a compute-dependent advantage.
- Curriculum learning being the most popular approach in the third BabyLM Challenge is not evidence that it was the most effective one; the best-performing entries modified the training objective or architecture instead.
- Matching a 70B Llama on BLiMP does not mean BabyLM-scale models match large models generally: on GLUE all submitted models remain below both the Llama 70B skyline and human scores.
- The interaction track's teacher model is not distilled into the student in the usual sense — exposing the teacher's weights, hidden states or output distribution to the student was prohibited, and interaction had to run through text or scalar feedback.

## Terminology
- Strict and Strict-Small tracks: BabyLM Challenge tracks limiting training data to at most 100M words and at most 10M words respectively, with no other restriction on model or training procedure; participants may use the provided BabyLM corpus or build their own within the word limit.
- Interaction track: A BabyLM Challenge track in which an external teacher model from a fixed list may be placed in the training pipeline, giving scalar or natural-language feedback or generating data conditioned on the student's outputs, but never exposing its weights, hidden states or output distribution to the student.
- human-likeness score: An aggregate BabyLM metric over tasks measuring psychometric and linguistic similarity to human learners — reading-time correlation, age-of-acquisition correlation, wug past-tense and adjective nominalization agreement with human preferences, entity tracking and COMPS — reported separately from NLP task accuracy.
- word exposure limit: A cap on total training tokens counted with repetition, set in the third BabyLM Challenge at 100M words for strict-small and 1B words for the other tracks, so that extra epochs over the same data cannot be used to buy performance.
- masked next token prediction (MNTP): A variant of masked language modeling used by GPT-BERT in which outputs are shifted as in autoregressive training, allowing one model to be trained and evaluated as both an encoder and a decoder.

## Links
- DOI: https://doi.org/10.18653/v1/2025.babylm-main.28
- Semantic Scholar: https://www.semanticscholar.org/paper/282893571
- Publisher: https://aclanthology.org/2025.babylm-main.28/
- anthology: https://aclanthology.org/2025.babylm-main.28/
- evaluation pipeline: https://github.com/babylm/evaluation-pipeline-2025
- Leaderboard: https://huggingface.co/spaces/BabyLM-community/babylm-leaderboard-2025-alltasks

## How to cite
@inproceedings{charpentier-etal-2025-findings,
    title = "Findings of the Third {B}aby{LM} Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data",
    author = "Charpentier, Lucas  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Hu, Michael Y.  and
      Liu, Jing  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Ross, Candance  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb  and
      Williams, Adina",
    editor = "Charpentier, Lucas  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Hu, Michael Y.  and
      Liu, Jing  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Ross, Candace  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb  and
      Williams, Adina",
    booktitle = "Proceedings of the First BabyLM Workshop",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.babylm-main.28/",
    doi = "10.18653/v1/2025.babylm-main.28",
    pages = "399--420",
    ISBN = "TODO",
    abstract = "This report summarizes the findings from the 3rd BabyLM Challenge and the 1st BabyLM Workshop. The BabyLM Challenge is a shared task aimed at closing the data efficiency gap between human and machine language learners. The goal is to improve the performance of language models given a fixed training budget of no more than 100 million words. This year, the challenge was held as part of an expanded BabyLM Workshop that invited paper submissions on topics relevant to the BabyLM effort, including sample-efficient pretraining and cognitive modeling for LMs. For the challenge, we kept the text-only and text{--}image tracks from previous years, but also introduced a new \textit{interaction} track, where student models are allowed to learn from feedback from larger teacher models. Furthermore, we introduce a new set of evaluation tasks to assess the ``human likeness'' of models on a cognitive and linguistic level, limit the total amount of training compute allowed, and measure performance on intermediate checkpoints. We observe that new training objectives and architectures tend to produce the best-performing approaches, and that interaction with teacher models can yield high-quality language models. The strict and interaction tracks saw submissions that outperformed the best-performing methods from previous years. We do not observe a complete correlation between training FLOPs and performance. {\%}, suggesting that some methods can produce real gains beyond allowing us to spend more compute. This year{'}s BabyLM Challenge shows that there is still room to innovate in a data-constrained setting, and that community-driven research can yield actionable insights for language modeling."
}