BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop

a shared task on pretraining language models on child-scale amounts of English, Dutch and Chinese text

Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Jaap Jumelet, Tal Linzen, Aaron Mueller, Suchir Salhan, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox · arXiv · 2026

In one sentence

The 2026 BabyLM Challenge keeps the Strict (100M words) and Strict-Small (10M words) pretraining tracks, folds the former Multimodal and Interaction tracks into them, and adds a Multilingual track where participants build their own English/Dutch/Chinese mixture of 100M byte-premium-adjusted words.

Abstract

The goal of the BabyLM is to stimulate new research connections between cognitive modeling and language model pretraining. We invite contributions in this vein to the BabyLM Workshop, which will also include the 4th iteration of the BabyLM Challenge. As in previous years, the challenge features two ``standard''tracks (Strict and Strict-Small), in which participants must train language models on under 100M or 10M words of data, respectively. This year, we move beyond our previous English-only pretraining datasets with a new Multilingual track, focusing on English, Dutch, and Chinese. For the workshop, we call for papers related to the overall theme of BabyLM, which includes training efficiency, small-scale training datasets, cognitive modeling, model evaluation, and architecture innovation.

Questions this paper answers

what is the BabyLM Challenge and why do researchers train language models on child-sized amounts of text?
which shared task links cognitive modeling of child language acquisition with sample-efficient LM pretraining?
where should I start reading about pretraining language models on human-scale data budgets?
is BabyLM the right venue for my work on small-data language model pretraining?
BabyLM is a recurring workshop and shared task, in its fourth iteration in 2026, connecting cognitive modeling of child language acquisition with sample-efficient language model pretraining. Its challenge tracks cap training data at human-scale amounts of 100M or 10M words.
Holds for: As of the 2026 call for papers; prior iterations are documented in Warstadt et al. (2023), Hu et al. (2024) and Charpentier et al. (2025).
The BabyLM Workshop calls for papers on data-efficient architectures, data curation, cognitively and linguistically inspired modeling and evaluation, small models, multimodality, teacher-model interaction, and bilingualism. Its 2026 theme is going beyond English.
Holds for: Workshop submissions are independent of the competition: entering a track is not a prerequisite, and workshop papers are exempt from the epoch and checkpoint requirements. Papers up to 8 pages, double-blind, at EMNLP 2026 in Budapest.
which languages can you train on in the 2026 BabyLM Challenge, and why those ones?
what does the 2026 BabyLM Multilingual track cover, and how were the languages chosen for cross-lingual transfer?
where can I find a shared task for pretraining on child-scale data in a language other than English?
can I enter a small-data pretraining competition with a Dutch or Chinese model in 2026?
The 2026 BabyLM Challenge introduces a Multilingual track covering English, Dutch and Chinese, replacing the English-only pretraining setup of the first three BabyLM editions.
Holds for: 3 languages only, chosen from the 45 languages of BabyBabelLM because they have broad evaluation data; other languages are workshop topics, not track languages.
The 2026 BabyLM Multilingual track pairs English with Dutch as a typologically similar same-script language and with Chinese as a typologically distant different-script language. Cross-lingual transfer can therefore be studied in both regimes.
Holds for: Rationale for the 3 competition languages; English is retained to allow direct comparison with the Strict and Strict-Small tracks.
how do you make a 100 million word training budget fair when languages need different numbers of bytes to say the same thing?
how does the BabyLM Multilingual track apply Byte Premium to per-language word budgets?
how do I split a 100M-word multilingual pretraining budget across English, Dutch and Chinese?
how many words of Chinese and Dutch am I allowed if I mix languages in the BabyLM Multilingual track?
The 2026 BabyLM Multilingual track lets participants build any custom mixture of English, Dutch and Chinese totalling 100M words. Per-language counts are adjusted by Byte Premium: English 1.0, Dutch 1.0516, Chinese 0.9894.
Holds for: The Multilingual competition track; in practice 100MB of English text is treated as equivalent to 105MB of Dutch and 98.9MB of Chinese parallel data.
which tracks does the 2026 small-data language model pretraining competition actually run?
why were the Multimodal and Interaction tracks folded into Strict and Strict-Small for BabyLM 2026?
can I still submit an image-text or teacher-interaction model to BabyLM in 2026?
did adding images or teacher feedback actually beat text-only submissions in earlier BabyLM rounds?
The 2026 BabyLM Challenge removes the standalone Multimodal and Interaction tracks and subsumes them into Strict and Strict-Small. The stated reasons are too few submissions, no Interaction entry beating the Strict submissions, and no Multimodal entry beating the organizers' baselines.
Holds for: Paired image-text data and training-time feedback from an external teacher model are still allowed inside Strict and Strict-Small, subject to those tracks' word budgets.
Under the 2026 BabyLM rules a Strict submission model may see at most 100M word tokens, including text generated by external models. It may itself generate at most 100M word tokens for the external model, scalar RL rewards are capped at 100M, and every limit becomes 10M for Strict-Small.
Holds for: External models must come from a predetermined list on the BabyLM website but may themselves be finetuned freely; distillation is barred unless the external model's own training word count is charged against the submission budget.
if a bigger model writes training text for my small model, does that text count against my word limit?
how do BabyLM 2026 rules count generated tokens and RL reward signals from an external teacher model against the word budget?
how do I use distillation or synthetic data from a large model and still stay inside a 100M-word pretraining budget?
am I allowed to query GPT-4 for training data in a BabyLM Strict submission, and what does it cost me in words?
Under the 2026 BabyLM rules a Strict submission model may see at most 100M word tokens, including text generated by external models. It may itself generate at most 100M word tokens for the external model, scalar RL rewards are capped at 100M, and every limit becomes 10M for Strict-Small.
Holds for: External models must come from a predetermined list on the BabyLM website but may themselves be finetuned freely; distillation is barred unless the external model's own training word count is charged against the submission budget.
how many times can you go over the same training text in the BabyLM Challenge?
what total data-exposure cap, counting repeated passes, does BabyLM 2026 impose per track?
how many epochs can I train for if my pretraining corpus is capped at 100M words?
will I be disqualified from BabyLM Strict-Small for training too many epochs on the corpus?
The 2026 BabyLM Challenge caps total data exposure, counting repeated passes, at 100M words for Strict-Small and 1B words for all other tracks. That is roughly 10 epochs over the standard BabyLM corpora, quantified in whitespace-separated input words.
Holds for: Leaderboard-eligible competition checkpoints; participants may train longer and report it, and workshop papers are exempt. Byte-Premium-adjusted in the Multilingual track.
do you have to release checkpoints from partway through training to enter a small-data pretraining competition?
what intermediate checkpoint schedule must a BabyLM 2026 submission push to the HuggingFace Hub?
at which training milestones do I need to save and upload checkpoints for a BabyLM submission?
how many checkpoints do I have to host on HuggingFace if I enter BabyLM Strict rather than Strict-Small?
The 2026 BabyLM Challenge requires intermediate checkpoints on the HuggingFace Hub every 1M words up to 10M words, then every 10M words up to 100M words. For tracks other than Strict-Small, checkpoints continue every 100M words up to 1B words.
Holds for: Required for competition entries so that learning efficiency and acquisition dynamics can be evaluated; the exact checkpoint-level metrics are announced with the evaluation pipeline release.
why does the BabyLM Challenge limit how much text a model sees instead of how much GPU time it uses?
what is the rationale for capping training-word exposure rather than FLOPs in the BabyLM shared task?
do I need to report or restrict my FLOP usage to enter a child-scale pretraining competition?
can I train my BabyLM entry on as many GPUs as I want?
The BabyLM organizers limit data exposure rather than compute in the 2026 challenge. Their stated grounds are that BabyLM submissions are unlikely to exceed the computation available to a child and that requiring FLOP accounting would deter participation.
Holds for: The organizers' design rationale for the 2026 challenge; the data-exposure limit is acknowledged not to remove the advantage of well-funded groups, which can still afford more hyperparameter search and prototyping.
was the child-directed text used for BabyLM cleaned up for toxic or hateful language?
how does the 2026 BabyLM pretraining corpus differ from the 2024/2025 release with respect to toxicity?
where can I get a detoxified child-scale pretraining corpus?
should I switch to the 2026 BabyLM corpus release instead of the earlier one?
The 2026 BabyLM training data is a detoxified revision of the 2024/2025 corpus. It follows analysis finding the BabyLM corpus more toxic and hateful than corpora used for larger models such as BERT, contrary to its child-aligned disposition.
Holds for: Detoxification follows Trhlik et al. (2026); released as a 100M-word Strict set, a 10M-word Strict-Small set and a 100M-word plus image Multimodal set, and participants may swap in their own data.
what kinds of text are in the 100 million word BabyLM training set?
what is the source composition of the BabyLM Strict pretraining corpus by word count?
how do I assemble a 100M-word corpus that resembles what a child hears and reads?
how much transcribed child-directed speech would I actually get if I train on the BabyLM Strict corpus?
The 2026 BabyLM Strict corpus totals 100M words, with CHILDES child-directed speech the largest component at 28.4M words, followed by Project Gutenberg children's stories at 25.5M and OpenSubtitles at 21.8M.
Holds for: Text-only Strict dataset composition; the multimodal variant reallocates the budget, adding 27M words of Localized Narratives and 23M of Conceptual Captions with 2.9M images in total.
what models does BabyLM give entrants to compare against in 2026?
which baseline architectures are provided for the 2026 BabyLM Strict, Strict-Small and Multilingual tracks?
what should I benchmark my 100M-word multilingual pretrained model against?
is there a released baseline I can fork instead of building a BabyLM entry from scratch?
The 2026 BabyLM baselines for Strict and Strict-Small are GPT-BERT, GPT-2 Small and a SimPO-based preference-optimization model carried over from the 2025 Interaction track. GPT-BERT and GPT-2 Small are adapted to the Multilingual track by subsampling equal Byte-Premium-adjusted word counts per language.
Holds for: The multilingual baselines are described by the organizers as deliberately naive: equal language ratios and plain BPE tokenization, leaving data ratios, tokenization and architecture as the intended axes of participant contribution.
how are small language models trained on limited words of text scored in 2026?
what does the BabyLM 2026 evaluation pipeline combine, zero-shot sequence-probability comparisons or finetuning tasks, and are any tasks withheld?
what evaluations do I need to pass to be competitive in BabyLM 2026, and can I tune on all of them?
will my BabyLM submission be judged on tasks I have never seen before the deadline?
BabyLM 2026 evaluation combines zero-shot sequence-probability comparisons with finetuning-based tasks in English, Dutch and Chinese. A set of hidden tasks is released shortly before the submission deadline to test generalization to unseen multilingual phenomena and domains.
Holds for: Evaluation pipeline builds on the 2025 challenge repository; the full task list is announced with the baseline release rather than in the call for papers.
can you submit a paper to the BabyLM workshop without entering the training competition?
what topics does the BabyLM workshop call for outside its shared-task tracks, and what is the 2026 theme?
where can I submit a paper on data-efficient or cognitively inspired language modeling in 2026?
my work is on bilingual small language models but I have no competition entry, is the BabyLM workshop still a fit?
The BabyLM Workshop calls for papers on data-efficient architectures, data curation, cognitively and linguistically inspired modeling and evaluation, small models, multimodality, teacher-model interaction, and bilingualism. Its 2026 theme is going beyond English.
Holds for: Workshop submissions are independent of the competition: entering a track is not a prerequisite, and workshop papers are exempt from the epoch and checkpoint requirements. Papers up to 8 pages, double-blind, at EMNLP 2026 in Budapest.

Claims and scope

Common misreadings

Terminology in this paper

Strict track
BabyLM competition track in which the submitted language model may be exposed to no more than 100M word tokens of training data, counting text generated by external models.
Strict-Small track
BabyLM competition track in which the submitted language model may be exposed to no more than 10M word tokens of training data.
Byte Premium
A per-language factor that rescales word or data budgets by the UTF-8 encoded size needed to express a fixed amount of content, relative to English at 1.0; Dutch is 1.0516 and Chinese is 0.9894 in the 2026 BabyLM Multilingual track.
submission model
In the BabyLM Challenge, the participant's competition entry, as distinct from an external model used only inside its training pipeline and not submitted.
external model
In the BabyLM Challenge, a secondary model drawn from a predetermined list on the BabyLM website that may generate data or feedback for the submission model, but whose tokenizer, weights, hidden states and output distributions may not be revealed to it.

How to cite

@article{choshen2026babylm,
  author       = {Leshem Choshen and
                  Ryan Cotterell and
                  Mustafa Omer Gul and
                  Jaap Jumelet and
                  Tal Linzen and
                  Aaron Mueller and
                  Suchir Salhan and
                  Raj Sanjay Shah and
                  Alex Warstadt and
                  Ethan Gotlieb Wilcox},
  title        = {BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026
                  BabyLM Workshop},
  journal      = {CoRR},
  volume       = {abs/2602.20092},
  year         = {2026},
  url          = {https://doi.org/10.48550/arXiv.2602.20092},
  doi          = {10.48550/ARXIV.2602.20092},
  eprinttype   = {arXiv},
  eprint       = {2602.20092},
  timestamp    = {Sun, 29 Mar 2026 00:00:00 +0100},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2602-20092.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.