BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop

a shared task where a small language model learns from a teacher model's feedback under a 100M-word exposure budget

Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, A. Warstadt, E. Wilcox, Adina Williams · arXiv · 2025

In one sentence

The 2025 BabyLM workshop keeps the 100M-word and 10M-word data-efficient pretraining tracks, adds an Interaction track where a student model learns from a teacher model without seeing its weights or output distribution, and caps competition entries at roughly 10 epochs of data exposure.

Abstract

BabyLM aims to dissolve the boundaries between cognitive modeling and language modeling. We call for both workshop papers and for researchers to join the 3rd BabyLM competition. As in previous years, we call for participants in the data-efficient pretraining challenge in the general track. This year, we also offer a new track: INTERACTION. This new track encourages interactive behavior, learning from a teacher, and adapting the teaching material to the student. We also call for papers outside the competition in any relevant areas. These include training efficiency, cognitively plausible research, weak model evaluation, and more.

Questions this paper answers

what is BabyLM and what question about learning language from limited input does it try to answer?
which venue brings together child language acquisition research and sample-efficient language model pretraining?
where do I submit work that sits between cognitive science and small-data language modeling?
should I send my paper to BabyLM if I do not want to enter a competition track?
BabyLM is a recurring shared task and workshop framed around a single question: how a computational system can learn language from limited input. It brings cognitive scientists studying child language acquisition together with researchers building sample-efficient language models.
Holds for: The 2025 call describes the third edition; suggested topics include data-efficient architectures, data curation, cognitively inspired modeling and evaluation, scaling-law comparisons, and multimodal modeling.
BabyLM, previously run as a competition on training language models from developmentally plausible amounts of data, became a workshop in 2025. The workshop accepts research papers at the intersection of cognitive science and language modeling with no requirement to enter any competition track.
Holds for: Describes the 2025 (third) edition, held at EMNLP 2025 in Suzhou; the competition tracks continue alongside the paper track.
what is new in the 2025 round of the small-data language model competition compared with the previous year?
which rule changes did the 3rd BabyLM Challenge introduce over the 2nd, in tracks, data budget and checkpointing?
I entered the earlier BabyLM challenge, what do I need to change to submit in 2025?
if I already have a BabyLM-style pretraining pipeline, is it still eligible under the 2025 rules?
The 2025 BabyLM competition adds an Interaction track in which a pretrained external model may act as a teacher. The submitted student model may be exposed to no more than 100M word tokens and may itself generate no more than 100M words during training.
Holds for: External models must come from a predetermined list on the BabyLM website; they may be finetuned or distilled without restriction, but their weights, hidden states and output distributions may not be revealed to the student.
BabyLM 2025 caps leaderboard-eligible models at a fixed amount of input counting repeated exposures. The budget is at most 100M words for the Strict-small track and at most 1B words for all other tracks, roughly 10 epochs over the standard BabyLM corpora.
Holds for: Competition entries only; participants may train longer and report it in their paper, and workshop papers are exempt. Budget measured in whitespace-separated input words, and in the Interaction track input words plus generated tokens.
BabyLM 2025 requires competition entrants to upload intermediate checkpoints to the HuggingFace Hub. The intervals increase over training: every 1M words up to 10M, every 10M words up to 100M, and every 100M words up to 1B for tracks other than Strict-small.
Holds for: Checkpoints feed the updated evaluation pipeline's measures of learning efficiency and acquisition trajectories; precise evaluation details were deferred to the pipeline release.
The BabyLM 2025 training corpora are unchanged from the 2nd BabyLM Challenge. They comprise a 100M-word Strict set, a 10M-word Strict-small set, and a 100M-word Multimodal set that pairs image captions with text and includes about 2.9M images.
Holds for: Word counts are approximate; the Strict corpus draws mainly on CHILDES (29M words), Project Gutenberg children's stories (26M), OpenSubtitles (20M) and Simple English Wikipedia (15M). Participants need not use the official corpus.
can a small language model be trained by a bigger model giving it feedback in a limited-data competition?
how does the BabyLM Interaction track constrain teacher-student setups and the student's own generated tokens?
how do I use a large pretrained teacher to train a 100M-word budget model without breaking the entry rules?
am I allowed to distil from an off-the-shelf pretrained model in my BabyLM 2025 entry?
The 2025 BabyLM competition adds an Interaction track in which a pretrained external model may act as a teacher. The submitted student model may be exposed to no more than 100M word tokens and may itself generate no more than 100M words during training.
Holds for: External models must come from a predetermined list on the BabyLM website; they may be finetuned or distilled without restriction, but their weights, hidden states and output distributions may not be revealed to the student.
BabyLM's data budget is a closed-system accounting rule. Any tokenizer, parser, augmenter or ancillary language model used in the pipeline has its own training text counted toward the 100M-word limit, so off-the-shelf tools trained on outside language are disallowed.
Holds for: Synthetic data is permitted so long as the generators' training data is inside the budget; the Interaction track's listed external models and its interactive environment are the stated exceptions.
how many times can a model see the same training text in the small-data pretraining competition, and why is there a cap?
what epoch or multiple-exposure budget applies to BabyLM 2025 leaderboard eligibility, and what motivated it?
how long can I keep training on the 100M-word corpus before my run stops being eligible?
if more compute keeps improving my scores, can I just train for more epochs and still submit?
BabyLM 2025 caps leaderboard-eligible models at a fixed amount of input counting repeated exposures. The budget is at most 100M words for the Strict-small track and at most 1B words for all other tracks, roughly 10 epochs over the standard BabyLM corpora.
Holds for: Competition entries only; participants may train longer and report it in their paper, and workshop papers are exempt. Budget measured in whitespace-separated input words, and in the Interaction track input words plus generated tokens.
The BabyLM organizers justify the 2025 epoch limit by noting that a conclusion of the 2024 challenge was that more compute correlates with higher performance. That correlation conflicts with both developmental plausibility and the goal of democratizing pretraining research.
Holds for: The organizers deliberately did not cap compute or FLOPs, judging FLOP accounting too technically demanding and BabyLM compute unlikely to exceed what is available to children.
how big is the training text used for the small-data language model challenge and what is in it?
what are the composition and sizes of the Strict, Strict-small and Multimodal BabyLM pretraining corpora?
where do I get the pretraining data for a developmentally plausible language model, and how much of it is there?
do I need to build my own corpus for BabyLM 2025 or can I reuse the earlier release?
The BabyLM 2025 training corpora are unchanged from the 2nd BabyLM Challenge. They comprise a 100M-word Strict set, a 10M-word Strict-small set, and a 100M-word Multimodal set that pairs image captions with text and includes about 2.9M images.
Holds for: Word counts are approximate; the Strict corpus draws mainly on CHILDES (29M words), Project Gutenberg children's stories (26M), OpenSubtitles (20M) and Simple English Wikipedia (15M). Participants need not use the official corpus.
do entrants to the small-data language model challenge have to save models partway through training?
what intermediate checkpoint schedule does BabyLM 2025 require entrants to publish for learning-dynamics analysis?
how often should I save and upload checkpoints during a 100M-word pretraining run to stay eligible?
how much extra storage and uploading am I committing to if I enter BabyLM 2025?
BabyLM 2025 requires competition entrants to upload intermediate checkpoints to the HuggingFace Hub. The intervals increase over training: every 1M words up to 10M, every 10M words up to 100M, and every 100M words up to 1B for tracks other than Strict-small.
Holds for: Checkpoints feed the updated evaluation pipeline's measures of learning efficiency and acquisition trajectories; precise evaluation details were deferred to the pipeline release.
is a model judged on how human-like it is, or only on how well it does language tasks, in the small-data challenge?
does the BabyLM 2025 evaluation pipeline include psychometric measures such as reading-time prediction alongside NLP benchmarks?
how do I get credit for a model that fits human reading behaviour rather than topping accuracy benchmarks?
my model is not the most accurate but fits human data well, is there a category I can win?
The 2025 BabyLM evaluation pipeline adds psychometric tasks such as reading-time prediction. Human-likeness is treated as an award category separate from NLP task accuracy, so a system can win on either metric.
Holds for: The 2025 pipeline was rewritten from scratch with HuggingFace and plain PyTorch entry points; hidden evaluations release no less than 2 weeks before the model submission deadline.
which starter models are provided for the 2025 small-data language model challenge?
what pretrained and Interaction-track baselines does BabyLM 2025 release, including the winning 2024 submission and the multimodal ones?
what should I compare my limited-data pretrained model against, and where do the teacher-feedback baselines come from?
is there a released baseline I can fork rather than training a BabyLM entry from scratch?
BabyLM 2025 releases GPT-BERT, the winning submission of the 2024 challenge, and GPT-2 Small as Strict and Strict-small baselines. The GIT and Flamingo multimodal baselines are re-released because no 2024 submission beat them.
Holds for: The 2025 competition's provided baselines; the Multimodal track was re-released despite limited participation in 2024.
BabyLM 2025 ships two Interaction-track baselines, one using PPO with a learned reward and one using natural-language corrections. The reward model is a deberta-v3-xsmall trained on child-caregiver conversations, and the correction baseline has Llama-3.1 Instruct 8B revise GPT-2 Small completions over 20 rounds.
Holds for: The correction baseline trains on teacher-corrected text with language modeling loss, then with SimPO at learning rate 0.00005, beta=2, gamma=1 and a 0.2 language-modeling regularizer.
if I use another model or an existing tagger to prepare training text, does that text count against the word limit?
how does BabyLM's closed-system data accounting treat tokenizers, parsers and auxiliary LMs used for augmentation?
can I augment my 100M-word corpus with synthetic text or off-the-shelf preprocessing tools and stay within budget?
is an external POS tagger or pretrained parser going to disqualify my BabyLM submission?
BabyLM's data budget is a closed-system accounting rule. Any tokenizer, parser, augmenter or ancillary language model used in the pipeline has its own training text counted toward the 100M-word limit, so off-the-shelf tools trained on outside language are disallowed.
Holds for: Synthetic data is permitted so long as the generators' training data is inside the budget; the Interaction track's listed external models and its interactive environment are the stated exceptions.
how strict is reviewing for papers describing entries to the small-data language modeling challenge?
what acceptance criteria apply to BabyLM competition-entry papers versus its archival research submissions?
what do I need to include in a BabyLM entry paper so it is not rejected?
is it worth writing up a BabyLM submission if my results are not competitive?
BabyLM applies lenient acceptance to competition submissions, planning to reject only papers with incorrect or unjustified claims, significant technical issues, insufficient methodological detail for replication, or minimal time investment.
Holds for: Leniency covers competition submissions; non-competition workshop papers are evaluated on merit and relevance under double-blind review, up to 8 pages, via ARR or direct OpenReview submission.
BabyLM, previously run as a competition on training language models from developmentally plausible amounts of data, became a workshop in 2025. The workshop accepts research papers at the intersection of cognitive science and language modeling with no requirement to enter any competition track.
Holds for: Describes the 2025 (third) edition, held at EMNLP 2025 in Suzhou; the competition tracks continue alongside the paper track.

Claims and scope

Common misreadings

Terminology in this paper

Strict track
A BabyLM competition track requiring the submitted language model to be trained on a corpus of 100M words or fewer, evaluated on language-only tasks; participants need not use the official BabyLM corpus.
Strict-small track
A BabyLM competition track requiring the submitted language model to be trained on a corpus of 10M words or fewer.
Submission model
In the BabyLM Interaction track, the participant's own entry into the competition, subject to the 100M-word exposure limit — as opposed to the external model used only inside its training pipeline.
External model
In the BabyLM Interaction track, a secondary pretrained model drawn from an organizer-approved list that is used in the training pipeline but not submitted; its weights, hidden states and output distribution may not be exposed to the submission model.
Communicative response (CR)
A caregiver reply that indicates a child's utterance was understood; predicted by a reward model in one BabyLM Interaction baseline, where an utterance followed by such a response receives reward 0 and one not followed by it receives reward 1.
Hidden evaluations
BabyLM evaluation tasks withheld from participants to control for overfitting to the public suite, released no less than 2 weeks before the model submission deadline.

References

See the full reference list in the paper.