Findings of the Third BabyLM Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data

Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Y. Hu, Jing Liu, Jaap Jumelet, Tal Linzen, Aaron Mueller, Candance Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox, Adina Williams · Proceedings of the First BabyLM Workshop · 2025

• DOI
• Semantic Scholar
• Publisher
• anthology
• evaluation pipeline
• Leaderboard

In one sentence

The third BabyLM Challenge capped pretraining at 100M words (10M in strict-small), added an interaction track where a student learns from a larger teacher model, capped total word exposure and required intermediate checkpoints; objective and architecture changes, not curriculum learning, produced the winners.

Abstract

This report summarizes the findings from the 3rd BabyLM Challenge and the 1st BabyLM Workshop. The BabyLM Challenge is a shared task aimed at closing the data efficiency gap between human and machine language learners. The goal is to improve the performance of language models given a fixed training budget of no more than 100 million words. This year, the challenge was held as part of an expanded BabyLM Workshop that invited paper submissions on topics relevant to the BabyLM effort, including sample-efficient pretraining and cognitive modeling for LMs. For the challenge, we kept the text-only and text{--}image tracks from previous years, but also introduced a new \textit{interaction} track, where student models are allowed to learn from feedback from larger teacher models. Furthermore, we introduce a new set of evaluation tasks to assess the ``human likeness'' of models on a cognitive and linguistic level, limit the total amount of training compute allowed, and measure performance on intermediate checkpoints. We observe that new training objectives and architectures tend to produce the best-performing approaches, and that interaction with teacher models can yield high-quality language models. The strict and interaction tracks saw submissions that outperformed the best-performing methods from previous years. We do not observe a complete correlation between training FLOPs and performance. {\%}, suggesting that some methods can produce real gains beyond allowing us to spend more compute. This year{'}s BabyLM Challenge shows that there is still room to innovate in a data-constrained setting, and that community-driven research can yield actionable insights for language modeling.

Questions this paper answers

Where can I read about what happens when language models are trained on only as much text as a child hears?

Which report documents the shared task on sample-efficient pretraining under a 100M-word budget and its winning systems?

What should I read first before entering a data-efficient language model pretraining competition?

I want a single reference for small-data pretraining results in 2025 -- which write-up covers the tracks, corpora and winners?

The Findings of the Third BabyLM Challenge report documents a shared task on data-efficient language model pretraining under a budget of at most 100M words. It is the reference point for the challenge's 2025 tracks, corpora, evaluation suite and winning systems.
Holds for: Covers the third iteration only, with English data and evaluations; the first and second iterations are reported in separate findings papers (Warstadt et al. 2023; Hu et al. 2024).

The winning systems of the third BabyLM Challenge were a diffusion masked language model, an instruction-tuned model, a modular mixture-of-experts model and a reinforcement-learning interactive storytelling model. Curriculum learning was the most common approach among submissions but won nothing.
Holds for: 32 models submitted across the strict, strict-small, multimodal and interaction tracks; winners named separately for human-likeness and for NLP task scores.

Can a small language model be taught by a bigger model instead of just reading text, and did that work?

How does the interaction track let a student model learn from teacher-model feedback, and which teachers are permitted?

How do I train a small model using feedback from a larger instruct model within a fixed word budget?

If I train a small model with a larger teacher giving it feedback, will it actually beat passive text training on human-likeness?

The third BabyLM Challenge introduced an interaction track in which a student model learns from feedback produced by a larger teacher model, rather than only from passive exposure to human-written text. Teachers were restricted to a fixed list including Llama3.1-8B-Instruct and any LM below 1B parameters.
Holds for: The student may see at most 100M external words and generate at most 100M of its own; the teacher's weights, hidden states and output distribution may not be exposed to the student.

BLM, the interaction-track winner of the third BabyLM Challenge, scored 20.8 on human-likeness and 54.4 on the NLP metric, beating the track's SimPO baseline (20.4 human-likeness, 54.1 NLP) and taking both awards.
Holds for: Interaction track only; the challenge's own aggregate human-likeness and NLP task metrics on the final checkpoint, with the student limited to 100M external words.

When you can only use a small amount of text, does feeding it in an easy-to-hard order actually help?

Did curriculum learning submissions win any track of the 2025 data-efficient pretraining shared task, and which objectives and architectures did?

Which training objective and architecture should I pick for pretraining on 100M words or less?

Should I spend my effort on curriculum ordering or on the model architecture and objective for a small-data pretraining run?

The winning systems of the third BabyLM Challenge were a diffusion masked language model, an instruction-tuned model, a modular mixture-of-experts model and a reinforcement-learning interactive storytelling model. Curriculum learning was the most common approach among submissions but won nothing.
Holds for: 32 models submitted across the strict, strict-small, multimodal and interaction tracks; winners named separately for human-likeness and for NLP task scores.

Grouping third BabyLM Challenge submissions by backbone architecture, GPT-BERT consistently yields the strongest cognitive, linguistic and macro average scores, with DeBERTa and LTG-BERT also performing strongly.
Holds for: Backbones compared include BERT, DeBERTa, Flamingo, GIT, GPT-2, GPT-BERT, Llama, LSTM, LTG-BERT, ModernBERT, Qwen2 and RoBERTa, aggregated over submissions without controlling training data or hyperparameters.

Did anyone training on a tiny text budget in 2025 actually beat the organizers' own reference models?

How did strict, strict-small and interaction track submissions score against the GPT-BERT and SimPO baselines on human-likeness and NLP metrics?

How do I tell whether a small-data pretraining recipe genuinely improves on the published reference models?

Is any 2025 100M-word submission worth adopting over the provided baseline checkpoints?

In the strict-small track of the third BabyLM Challenge, MoEP reached a human-likeness score of 31.5 and a macro average of 42.3, above every baseline in that track. The best baseline human-likeness was 19.8 and the best baseline macro average 37.4.
Holds for: Strict-small track (10M words) only; baselines are GPT-BERT variants and GPT-2 Small trained on the 10M-word BabyLM corpus. In the strict track the baselines were not beaten on the aggregate metrics.

In the strict track of the third BabyLM Challenge, CLASS-IT reached a human-likeness score of 20.4 against the best GPT-BERT baseline's 22.5. Simple-Diffusion reached an NLP score of 58.4 against the best baseline's 63.0.
Holds for: Strict track (100M words), final-checkpoint full evaluation; the GPT-BERT baselines are last year's winning submission, trained on the 100M-word BabyLM corpus.

BLM, the interaction-track winner of the third BabyLM Challenge, scored 20.8 on human-likeness and 54.4 on the NLP metric, beating the track's SimPO baseline (20.4 human-likeness, 54.1 NLP) and taking both awards.
Holds for: Interaction track only; the challenge's own aggregate human-likeness and NLP task metrics on the final checkpoint, with the student limited to 100M external words.

If you spend more computing power training a small-data model, do the scores get better?

Is macro average score correlated with training FLOPs across submissions in the strict, strict-small and interaction tracks?

Should I buy more GPU hours to raise scores when my training corpus is capped at 100M words?

My data budget is fixed -- is throwing more compute at the run likely to pay off?

Across the third BabyLM Challenge submissions, macro average score shows no strong relationship with training FLOPs in the strict and strict-small tracks; a positive correlation appears only in the interaction track.
Holds for: 32 models under this year's exposure cap of 100M words for strict-small and 1B for other tracks; FLOPs self-reported by participants.

Do the models that behave most like humans also score best on grammar and language tests?

Are cognitive modeling scores and linguistic benchmark scores positively correlated across 100M-word pretrained submissions?

Can I use linguistic benchmark scores as a proxy for how well a small-data model predicts human reading behaviour?

If I optimize my small-data model for BLiMP-style linguistic scores, am I also improving its fit to human data?

In the third BabyLM Challenge, linguistic task performance and cognitive modeling task performance were positively correlated across submitted models in every track except strict-small.
Holds for: Within-track correlations across submitted models, using the new human-likeness tasks against BLiMP/GLUE/EWoK; the multimodal track had only 1 submission.

Can a model trained on a child-sized amount of text judge grammar as well as a huge model?

How do 100M-word submissions compare with a 70B Llama skyline on BLiMP and on GLUE?

Which evaluations should I expect a 100M-word model to be competitive on, and which will it lose?

If I pretrain on 100M words instead of using a 70B model, where will I actually lose accuracy?

Some strict-track and interaction-track models in the third BabyLM Challenge reach BLiMP scores comparable to a 70B-parameter Llama model. On GLUE, all submitted models remain below both the human score and the Llama 70B skyline.
Holds for: BLiMP and GLUE only; GLUE is evaluated after finetuning on subsampled (Super)GLUE tasks capped at 10,000 training examples. Does not extend to the cognitive human-likeness tasks.

As a language model reads more and more text, which abilities show up early and which appear suddenly?

How do BLiMP, EWoK, wug past-tense and entity-tracking scores evolve across intermediate checkpoints as words seen increases?

How many words of training do I need before morphological generalization starts working in a small model?

Should I keep checkpointing and evaluating throughout a small-data pretraining run, or will scores just rise smoothly?

Across intermediate checkpoints in the third BabyLM Challenge, BLiMP and EWoK scores rise with words seen, while wug past-tense accuracy is flat for the first 10-50M words before a phase shift upward. Entity tracking shows U-shaped scaling in the strict track.
Holds for: Checkpoints requested every 1M words to 10M, every 10M to 100M, and every 100M to 1B, evaluated on the fast (20% subsampled) task versions. Reading-time prediction and wug adjective nominalization show no strong relationship with words seen.

Can you test a small language model by just prompting it, instead of fine-tuning it on each task?

Do 100M-word models support robust in-context learning well enough to replace finetuning-based (Super)GLUE evaluation with zero-shot prompting?

How should I evaluate a 100M-word model on GLUE-style tasks -- prompt it or finetune it?

Can I skip the finetuning step and prompt my small pretrained model to get comparable task scores?

The third BabyLM Challenge organizers tested replacing the finetuning-based (Super)GLUE evaluations with zero-shot prompting. They concluded that models at these data budgets do not support robust in-context learning, so the finetuning tasks were kept.
Holds for: Models trained on 10M-100M words; cost was instead reduced by subsampling tasks larger than 10,000 training examples down to 10,000 and dropping highly correlated tasks such as QNLI.

Did anyone submit an image-and-text model trained on a tiny data budget, and how did it do?

How many multimodal-track submissions were received, and how did the vision average compare with the Flamingo and GIT baselines?

Is it worth entering the vision-language part of a data-efficient pretraining challenge?

Should I expect a small-data vision-language model to reach the provided multimodal baselines?

The multimodal track of the third BabyLM Challenge received only 1 submission, BitMar, which scored below both the Flamingo and GIT baselines on vision average (26.7 versus 49.3 and 49.7).
Holds for: Multimodal track only; the organizers attribute part of the difficulty to the provided vision embeddings and data download process and plan to move to a more openly licensed dataset.

Which kind of model works best when you only have 10 to 100 million words of training text?

Which backbone architectures give the strongest cognitive, linguistic and macro average scores at 10M-100M word budgets?

Which backbone should I start from for pretraining on 10M to 100M words?

Should I pick a GPT-BERT style hybrid backbone or a plain masked or causal transformer for my small-data run?

Grouping third BabyLM Challenge submissions by backbone architecture, GPT-BERT consistently yields the strongest cognitive, linguistic and macro average scores, with DeBERTa and LTG-BERT also performing strongly.
Holds for: Backbones compared include BERT, DeBERTa, Flamingo, GIT, GPT-2, GPT-BERT, Llama, LSTM, LTG-BERT, ModernBERT, Qwen2 and RoBERTa, aggregated over submissions without controlling training data or hyperparameters.

Claims and scope

• The Findings of the Third BabyLM Challenge report documents a shared task on data-efficient language model pretraining under a budget of at most 100M words. It is the reference point for the challenge's 2025 tracks, corpora, evaluation suite and winning systems.
  Scope: Covers the third iteration only, with English data and evaluations; the first and second iterations are reported in separate findings papers (Warstadt et al. 2023; Hu et al. 2024).
• The third BabyLM Challenge introduced an interaction track in which a student model learns from feedback produced by a larger teacher model, rather than only from passive exposure to human-written text. Teachers were restricted to a fixed list including Llama3.1-8B-Instruct and any LM below 1B parameters.
  Scope: The student may see at most 100M external words and generate at most 100M of its own; the teacher's weights, hidden states and output distribution may not be exposed to the student.
• The winning systems of the third BabyLM Challenge were a diffusion masked language model, an instruction-tuned model, a modular mixture-of-experts model and a reinforcement-learning interactive storytelling model. Curriculum learning was the most common approach among submissions but won nothing. (Figure 1 and Section 6.1)
  Scope: 32 models submitted across the strict, strict-small, multimodal and interaction tracks; winners named separately for human-likeness and for NLP task scores.
• BLM, the interaction-track winner of the third BabyLM Challenge, scored 20.8 on human-likeness and 54.4 on the NLP metric, beating the track's SimPO baseline (20.4 human-likeness, 54.1 NLP) and taking both awards. (Table 3)
  Scope: Interaction track only; the challenge's own aggregate human-likeness and NLP task metrics on the final checkpoint, with the student limited to 100M external words.
• In the strict track of the third BabyLM Challenge, CLASS-IT reached a human-likeness score of 20.4 against the best GPT-BERT baseline's 22.5. Simple-Diffusion reached an NLP score of 58.4 against the best baseline's 63.0. (Table 3)
  Scope: Strict track (100M words), final-checkpoint full evaluation; the GPT-BERT baselines are last year's winning submission, trained on the 100M-word BabyLM corpus.
• In the strict-small track of the third BabyLM Challenge, MoEP reached a human-likeness score of 31.5 and a macro average of 42.3, above every baseline in that track. The best baseline human-likeness was 19.8 and the best baseline macro average 37.4. (Table 3)
  Scope: Strict-small track (10M words) only; baselines are GPT-BERT variants and GPT-2 Small trained on the 10M-word BabyLM corpus. In the strict track the baselines were not beaten on the aggregate metrics.
• Across the third BabyLM Challenge submissions, macro average score shows no strong relationship with training FLOPs in the strict and strict-small tracks; a positive correlation appears only in the interaction track. (Figure 6)
  Scope: 32 models under this year's exposure cap of 100M words for strict-small and 1B for other tracks; FLOPs self-reported by participants.
• In the third BabyLM Challenge, linguistic task performance and cognitive modeling task performance were positively correlated across submitted models in every track except strict-small. (Figure 3)
  Scope: Within-track correlations across submitted models, using the new human-likeness tasks against BLiMP/GLUE/EWoK; the multimodal track had only 1 submission.
• Some strict-track and interaction-track models in the third BabyLM Challenge reach BLiMP scores comparable to a 70B-parameter Llama model. On GLUE, all submitted models remain below both the human score and the Llama 70B skyline. (Figure 4)
  Scope: BLiMP and GLUE only; GLUE is evaluated after finetuning on subsampled (Super)GLUE tasks capped at 10,000 training examples. Does not extend to the cognitive human-likeness tasks.
• Across intermediate checkpoints in the third BabyLM Challenge, BLiMP and EWoK scores rise with words seen, while wug past-tense accuracy is flat for the first 10-50M words before a phase shift upward. Entity tracking shows U-shaped scaling in the strict track. (Figure 5 and Section 7)
  Scope: Checkpoints requested every 1M words to 10M, every 10M to 100M, and every 100M to 1B, evaluated on the fast (20% subsampled) task versions. Reading-time prediction and wug adjective nominalization show no strong relationship with words seen.
• Grouping third BabyLM Challenge submissions by backbone architecture, GPT-BERT consistently yields the strongest cognitive, linguistic and macro average scores, with DeBERTa and LTG-BERT also performing strongly. (Figure 7)
  Scope: Backbones compared include BERT, DeBERTa, Flamingo, GIT, GPT-2, GPT-BERT, Llama, LSTM, LTG-BERT, ModernBERT, Qwen2 and RoBERTa, aggregated over submissions without controlling training data or hyperparameters.
• The third BabyLM Challenge organizers tested replacing the finetuning-based (Super)GLUE evaluations with zero-shot prompting. They concluded that models at these data budgets do not support robust in-context learning, so the finetuning tasks were kept. (Section 4.2)
  Scope: Models trained on 10M-100M words; cost was instead reduced by subsampling tasks larger than 10,000 training examples down to 10,000 and dropping highly correlated tasks such as QNLI.
• The multimodal track of the third BabyLM Challenge received only 1 submission, BitMar, which scored below both the Flamingo and GIT baselines on vision average (26.7 versus 49.3 and 49.7). (Table 2 and Table 3)
  Scope: Multimodal track only; the organizers attribute part of the difficulty to the provided vision embeddings and data download process and plan to move to a more openly licensed dataset.

Common misreadings

• The third BabyLM Challenge did not show that submissions beat the baselines everywhere: in the strict track the GPT-BERT baselines still had the best human-likeness (22.5) and best NLP score (63.0), and in the multimodal track the sole submission scored below both baselines on vision.
• The finding that FLOPs and performance are weakly related is not evidence that compute is irrelevant: a positive correlation still appears in the interaction track, this year's exposure cap removed one of the main ways compute was previously converted into score, and hyperparameter tuning remains a compute-dependent advantage.
• Curriculum learning being the most popular approach in the third BabyLM Challenge is not evidence that it was the most effective one; the best-performing entries modified the training objective or architecture instead.
• Matching a 70B Llama on BLiMP does not mean BabyLM-scale models match large models generally: on GLUE all submitted models remain below both the Llama 70B skyline and human scores.
• The interaction track's teacher model is not distilled into the student in the usual sense — exposing the teacher's weights, hidden states or output distribution to the student was prohibited, and interaction had to run through text or scalar feedback.

Terminology in this paper

Strict and Strict-Small tracks

BabyLM Challenge tracks limiting training data to at most 100M words and at most 10M words respectively, with no other restriction on model or training procedure; participants may use the provided BabyLM corpus or build their own within the word limit.

Interaction track

A BabyLM Challenge track in which an external teacher model from a fixed list may be placed in the training pipeline, giving scalar or natural-language feedback or generating data conditioned on the student's outputs, but never exposing its weights, hidden states or output distribution to the student.

human-likeness score

An aggregate BabyLM metric over tasks measuring psychometric and linguistic similarity to human learners — reading-time correlation, age-of-acquisition correlation, wug past-tense and adjective nominalization agreement with human preferences, entity tracking and COMPS — reported separately from NLP task accuracy.

word exposure limit

A cap on total training tokens counted with repetition, set in the third BabyLM Challenge at 100M words for strict-small and 1B words for the other tracks, so that extra epochs over the same data cannot be used to buy performance.

masked next token prediction (MNTP)

A variant of masked language modeling used by GPT-BERT in which outputs are shifted as in autoregressive training, allowing one model to be trained and evaluated as both an encoder and a decoder.

How to cite

@inproceedings{charpentier-etal-2025-findings,
    title = "Findings of the Third {B}aby{LM} Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data",
    author = "Charpentier, Lucas  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Hu, Michael Y.  and
      Liu, Jing  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Ross, Candance  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb  and
      Williams, Adina",
    editor = "Charpentier, Lucas  and
      Choshen, Leshem  and
      Cotterell, Ryan  and
      Gul, Mustafa Omer  and
      Hu, Michael Y.  and
      Liu, Jing  and
      Jumelet, Jaap  and
      Linzen, Tal  and
      Mueller, Aaron  and
      Ross, Candace  and
      Shah, Raj Sanjay  and
      Warstadt, Alex  and
      Wilcox, Ethan Gotlieb  and
      Williams, Adina",
    booktitle = "Proceedings of the First BabyLM Workshop",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.babylm-main.28/",
    doi = "10.18653/v1/2025.babylm-main.28",
    pages = "399--420",
    ISBN = "TODO",
    abstract = "This report summarizes the findings from the 3rd BabyLM Challenge and the 1st BabyLM Workshop. The BabyLM Challenge is a shared task aimed at closing the data efficiency gap between human and machine language learners. The goal is to improve the performance of language models given a fixed training budget of no more than 100 million words. This year, the challenge was held as part of an expanded BabyLM Workshop that invited paper submissions on topics relevant to the BabyLM effort, including sample-efficient pretraining and cognitive modeling for LMs. For the challenge, we kept the text-only and text{--}image tracks from previous years, but also introduced a new \textit{interaction} track, where student models are allowed to learn from feedback from larger teacher models. Furthermore, we introduce a new set of evaluation tasks to assess the ``human likeness'' of models on a cognitive and linguistic level, limit the total amount of training compute allowed, and measure performance on intermediate checkpoints. We observe that new training objectives and architectures tend to produce the best-performing approaches, and that interaction with teacher models can yield high-quality language models. The strict and interaction tracks saw submissions that outperformed the best-performing methods from previous years. We do not observe a complete correlation between training FLOPs and performance. {\%}, suggesting that some methods can produce real gains beyond allowing us to spend more compute. This year{'}s BabyLM Challenge shows that there is still room to innovate in a data-constrained setting, and that community-driven research can yield actionable insights for language modeling."
}

References

See the full reference list in the paper.