A Hitchhiker's Guide to Scaling Law Estimation

Leshem Choshen, Yang Zhang, Jacob Andreas · ICML 2025 · 2025

In one sentence

A meta-analysis of over 1,000 scaling laws fit to a released dataset of losses and evaluations from 485 published pretrained models, yielding concrete best practices for estimating a scaling law for a new model family.

Abstract

Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets. This provides an efficient way for practitioners and researchers alike to compare pretraining decisions involving optimizers, datasets, and model architectures. Despite the widespread use of scaling laws to model the dynamics of language model training, there has been little work on understanding how to best estimate and interpret them. We collect (and release) a large-scale dataset containing losses and downstream evaluations for 485 previously published pretrained models. We use these to estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families. We find that fitting scaling laws to intermediate checkpoints of training runs (and not just their final losses) substantially improves accuracy, and that -- all else equal -- estimates of performance are generally most accurate when derived from other models of similar sizes. However, because there is a significant degree of variability across model seeds, training multiple small models is sometimes more useful than training a single large one. Moreover, while different model families differ scaling behavior, they are often similar enough that a target model's behavior can be predicted from a single model with the same architecture, along with scaling parameter estimates derived from other model families.

Questions this paper answers

which paper should I read to learn how to predict how a language model will improve with more size and data?
is there a study of scaling law estimation methodology, rather than another proposed functional form for one model family?
where do I start if I need to fit a scaling law for my own pretraining runs and do not know how to choose the small models?
I am planning a pretraining budget and want guidance on scaling-law fitting practice, what should I read first?
A Hitchhiker's Guide to Scaling Law Estimation is a practical guide to choosing the preliminary models used to fit a language-model scaling law. It asks how many, how large and how fully trained they should be, rather than proposing a law for one family.
Holds for: Loss-based scaling laws in the Hoffmann et al. (2022) functional form, fit with square loss and replicated with Huber loss; as of publication in 2025.
A Hitchhiker's Guide to Scaling Law Estimation releases a public dataset of pretraining losses and downstream evaluations for 485 published pretrained models. The dataset covers more than 40 scaled model families and 1.9M recorded training steps.
Holds for: Language models whose largest family member exceeds 3B parameters and whose data was public or privately shared; some losses were manually extracted from published figures.
is there an open collection of training loss curves from many published language models?
is there a released corpus of pretraining losses and downstream evaluations spanning many scaled model families and training steps?
where can I get loss curves across model families so I can test scaling-law fitting procedures without pretraining anything myself?
can I reuse someone else's published loss curves instead of training models to study scaling behaviour?
A Hitchhiker's Guide to Scaling Law Estimation releases a public dataset of pretraining losses and downstream evaluations for 485 published pretrained models. The dataset covers more than 40 scaled model families and 1.9M recorded training steps.
Holds for: Language models whose largest family member exceeds 3B parameters and whose data was public or privately shared; some losses were manually extracted from published figures.
when predicting how a big model will do, is it better to use only each small model's final loss or its whole training curve?
do intermediate checkpoint losses improve scaling law fits relative to fitting final-loss points only?
I logged loss every few thousand steps for my small runs, should I feed all of those points into the scaling-law fit or just the last one?
is it worth keeping and using all my intermediate checkpoints when fitting a scaling law, or can I just use the end of training?
Fitting scaling laws to full training curves rather than to final losses alone substantially lowers prediction error, and relying only on the end of training produces significantly worse fits across model families.
Holds for: OPT, GPT-3 and Pythia families, using checkpoint subsets defined by the fraction of training tokens retained; assumes the learning-rate schedule's effect on intermediate losses is negligible.
Discarding checkpoints from the first 10B training tokens cuts scaling-law error from above 15% to 4-10% for OPT and Pythia, because the earliest phase of training contains loss spikes and non-monotonic behaviour.
Holds for: OPT and Pythia families; cutting fewer than 10B tokens gave noisier results and cutting more had negligible effect in preliminary experiments.
do the very first stages of training mess up predictions of how a model will improve at larger scale?
should early-training checkpoints be filtered out of a scaling law fit, and above what token count do fits stabilise?
how do I decide which early checkpoints to drop from my loss curves before fitting a scaling law?
my early loss curve has spikes and weird jumps, should I exclude that part before fitting a scaling law?
Discarding checkpoints from the first 10B training tokens cuts scaling-law error from above 15% to 4-10% for OPT and Pythia, because the earliest phase of training contains loss spikes and non-monotonic behaviour.
Holds for: OPT and Pythia families; cutting fewer than 10B tokens gave noisier results and cutting more had negligible effect in preliminary experiments.
how precise does a prediction of a large model's loss have to be before it is actually useful for making decisions?
what absolute relative error threshold does a scaling law need to discriminate between pretraining interventions given seed noise?
how do I know whether my scaling-law prediction is accurate enough to choose between two pretraining recipes?
my scaling law is off by several percent, is that good enough to trust for a pretraining decision?
A scaling law needs about 4% absolute relative error to be useful for comparing pretraining decisions. No widely adopted pretraining change surveyed was motivated by less than a 4% relative loss difference, and seed-to-seed variation alone reaches 3.5%.
Holds for: Based on a survey of published A/B tests on pretraining decisions, where reported effects run from 4% up to 50%; errors up to 20% still separate many modeling choices.
can I borrow numbers from someone else's published scaling study instead of training a whole ladder of small models?
can model-size scaling parameters be fixed from prior published fits so a single partially trained run suffices for a new family?
I can only afford one small training run, how do I still get a loss prediction for the larger model I want to train?
do I really need to train five or six models of different sizes, or can one run plus published parameters do?
With the model-size scaling parameters fixed to values from earlier published work, a single partially trained model in a new family can suffice. OLMo's 7B loss is predicted from 1B checkpoints with under 1% error.
Holds for: Transformer families with model-size parameters borrowed from Muennighoff et al. (2024); fails for the encoder-decoder T5-Pile, and errors reach 37%, 25% and 15% for OPT predictions to 175B from 8.7B, 13B and 30B.
Estimated scaling-law parameters E, A, alpha, B and beta differ dramatically across model families, so the rate at which extra data or parameters help depends on architectural details.
Holds for: The more than 40 scaled families in the released dataset, all fit with the Hoffmann et al. (2022) form; the differences are in fitted parameters, not the form.
do models with different architectures improve at different rates as you add data or parameters?
do fitted scaling coefficients such as the data and parameter exponents vary across model families, or can a law be transferred unchanged?
can I take a published scaling law from one model family and apply it to my own architecture without refitting?
my architecture differs from the one a published scaling law was fit on, should I refit it?
Estimated scaling-law parameters E, A, alpha, B and beta differ dramatically across model families, so the rate at which extra data or parameters help depends on architectural details.
Holds for: The more than 40 scaled families in the released dataset, all fit with the Hoffmann et al. (2022) form; the differences are in fitted parameters, not the form.
With the model-size scaling parameters fixed to values from earlier published work, a single partially trained model in a new family can suffice. OLMo's 7B loss is predicted from 1B checkpoints with under 1% error.
Holds for: Transformer families with model-size parameters borrowed from Muennighoff et al. (2024); fails for the encoder-decoder T5-Pile, and errors reach 37%, 25% and 15% for OPT predictions to 175B from 8.7B, 13B and 30B.
can I just train the model I actually care about part of the way and guess where its loss ends up?
how much of a target model's token budget must be completed before extrapolating its final pretraining loss is reliable?
instead of a ladder of small models, how far into the target run do I need to go to predict its final loss?
should I spend my compute on several small models or on partially training the big model I want?
Training the target model itself part-way is a viable substitute for training many small models, but reliable loss estimates require training it on roughly 30% of the full run.
Holds for: Predicting within a single parameter-count family, so only the token term is extrapolated; shown on OPT, GPT-2, OLMo, Pythia variants and T5-Pile.
how much worse is simply using the best small model's loss as a guess for a bigger model?
what absolute relative error do no-fit baselines that assume the target is no better than the largest preliminary model incur?
do I need to fit a curve at all, or can I just report the smallest-loss small model as an estimate for the large one?
is fitting a scaling law worth the effort compared with eyeballing my best small run?
Simple no-fitting baselines that assume the target model is no better than the best small model incur more than 15% error and mostly above 10%. Across all scaled families studied they average 18% absolute relative error.
Holds for: Two baselines, the best loss in the training set and the loss of the most-compute model, each given the full available family.
is it better to train more small models or a few bigger ones when trying to predict a large model's loss?
how many preliminary models does a reliable scaling law fit need, and does adding models help without increasing maximum size?
I have a fixed compute budget for preliminary runs, how many models should I train before fitting a scaling law?
I have trained 3 small models so far, should I train more before trusting the extrapolation?
Increasing the number of preliminary models lowers scaling-law error even when the added models are not larger, and 5 models is a reasonable minimum for reliable predictions.
Holds for: GPT-3, Gopher, OPT and Pythia families; the trend is not monotonic, since a single badly behaved model can dominate the fit.
Preliminary models closer in parameter count to the target give better fits, but the effect is neither strong nor monotonic. The 4 smallest models available already reach under 10% error for GPT-3, Gopher and OPT.
Holds for: Predicting the largest model in each family; Pythia's smallest models are not predictive, and extrapolating 34x up in Pythia is still reliable when other factors are accounted for.
how big do the practice models need to be compared with the large model whose loss I want to predict?
how much does the parameter-count gap between preliminary models and the target affect extrapolation error in a scaling law fit?
how do I pick the sizes of the small models I train so the extrapolation to my target size holds up?
can I get away with only very small preliminary runs, or does my largest one need to be near the target size?
Preliminary models closer in parameter count to the target give better fits, but the effect is neither strong nor monotonic. The 4 smallest models available already reach under 10% error for GPT-3, Gopher and OPT.
Holds for: Predicting the largest model in each family; Pythia's smallest models are not predictive, and extrapolating 34x up in Pythia is still reliable when other factors are accounted for.
Increasing the number of preliminary models lowers scaling-law error even when the added models are not larger, and 5 models is a reasonable minimum for reliable predictions.
Holds for: GPT-3, Gopher, OPT and Pythia families; the trend is not monotonic, since a single badly behaved model can dominate the fit.
if one of my practice runs behaved strangely, can I find and remove it by holding models out one at a time?
does leave-one-out cross-validation identify which preliminary models corrupt a scaling law fit?
how do I detect which preliminary model is poisoning my scaling-law fit before I extrapolate?
one of my small models looks like an outlier, should I trust cross-validation to tell me whether to drop it?
Cross-validation does not identify which preliminary models will corrupt a scaling-law fit. In 58% of cases, removing the model flagged as hard to predict produced the worst possible error on the actual target.
Holds for: Leave-one-out over parameter-count families within each scaled family, using the highest-token models as targets; tested on the paper's collected families only.
Increasing the number of preliminary models lowers scaling-law error even when the added models are not larger, and 5 models is a reasonable minimum for reliable predictions.
Holds for: GPT-3, Gopher, OPT and Pythia families; the trend is not monotonic, since a single badly behaved model can dominate the fit.
are the numbers fitted in a scaling formula actually independent, or do some of them move together?
how many effective degrees of freedom do the 5 fitted parameters of a Chinchilla-style scaling law have across families?
can I reduce the number of free parameters I fit in a scaling law by exploiting relationships between them?
should I fit all 5 scaling-law parameters for my family, or constrain some of them?
Scaling laws appear to have fewer degrees of freedom than their 5 fitted parameters suggest, with 3 principal components explaining 99.49% of the variance across fitted parameters. A is linearly related to alpha, and B to beta.
Holds for: Across the fitted families in the released dataset; exceptions are the encoder-decoder T5-Pile and 4 families trained with multiple passes over one training set, which show a different B-beta relationship.
can the same curve that predicts big models also predict a small one from the large ones?
can a scaling law extrapolate downward, fitting the smallest model's loss in a family from the largest models?
how do I estimate what a tiny version of my model would score without training it, given results from my larger runs?
I already trained large models, can I predict a small one's loss instead of running it?
Scaling laws also predict downward, fitting the loss of the smallest model in a family from the largest models. Good fits need at least 30-40% of training and enough models in the fitting set.
Holds for: OPT, Pythia variants and T5-Pile; the percentage of training used for the fitting models is not reversed.

Claims and scope

Common misreadings

Terminology in this paper

scaled model family
A set of language models differing only in parameter count and number of training tokens, sharing architecture and training distribution; different checkpoints of one run and different seeds count as different models.
absolute relative error (ARE)
The mean over target models of the absolute difference between the true loss and the scaling-law-predicted loss, divided by the true loss.
q-maximal token family
The subset of a scaled model family containing all models trained on at least a q fraction of the largest training run's tokens.
scale-up factor
The ratio between the parameter count of the target model and that of the largest model used to fit the scaling law.

How to cite

@inproceedings{choshen2024hitchhiker,
  title={A Hitchhiker's Guide to Scaling Law Estimation},
  author={Choshen, Leshem and Zhang, Yang and Andreas, Jacob},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=KugSHTH0c8}
}

References

See the full reference list in the paper.