A Latent Variable Framework for Scaling Laws in Large Language Models

Peiyao Cai, Chengyu Cui, Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Mikhail Yurochkin, Moulinath Banerjee, Yuekai Sun, Kean Ming Tan, Gongjun Xu · arXiv · 2025

In one sentence

A latent variable scaling law in which each LLM family gets a random ability vector and each benchmark a loading vector, so benchmark performance is modelled as a small set of interpretable skills that scale with model size and training tokens, with consistency, asymptotic normality and prediction intervals.

Abstract

We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated by the rapid emergence of numerous new LLM families with distinct architectures and training strategies, evaluated on an increasing number of benchmarks. This heterogeneity makes a single global scaling curve inadequate for capturing how performance varies across families and benchmarks. To address this, we propose a latent variable modeling framework in which each LLM family is associated with a latent variable that captures the common underlying features in that family. An LLM's performance on different benchmarks is then driven by its latent skills, which are jointly determined by the latent variable and the model's own observable features. We develop an estimation procedure for this latent variable model and establish its statistical properties. We also design efficient numerical algorithms that support estimation and various downstream tasks. Empirically, we evaluate the approach on 12 widely used benchmarks from the Open LLM Leaderboard (v1/v2).

Questions this paper answers

does making a language model bigger help the same skills that feeding it more text helps?
do benchmark-specific scaling coefficients on log parameters and log tokens differ across skill dimensions such as mathematics and common-sense reasoning?
how do I work out whether adding parameters or adding training tokens will improve the particular capability I care about?
my model needs better math more than better common-sense answers, so should I spend my next run on more data or a bigger network?
Mathematical skill (anchored on MATH) loads more on training tokens than on parameters, with coefficients 0.774 (SE 0.101) for log tokens versus 0.432 (SE 0.065) for log parameters. Common-sense reasoning (HellaSwag) reverses this, at 0.840 (SE 0.055) for log parameters versus 0.384 (SE 0.085) for log tokens.
Holds for: K=4 fit with covariates log(size), log(tokens) and their interaction, on combined Open LLM Leaderboard v1/v2 data; the interaction term is not consistently significant.
Compute-optimal allocation under a fixed FLOPs budget differs by skill: at 100.37E19 FLOPs the MATH-optimal configuration is 0.07B parameters with 2.39T tokens. The HellaSwag- and BBH-optimal configuration at the same budget is 1.12B parameters with 0.15T tokens.
Holds for: Allocations are constrained to the range of sizes and token counts observed in the training data (up to 180B parameters and 15T tokens), so budgets at the extremes saturate at those bounds; derived from the fitted K=4 model.
how many underlying abilities are enough to explain how language models score across a dozen benchmarks?
how is the latent dimension K selected in a factor model of LLM benchmark scores, and where does the AIC curve flatten?
how do I decide how many latent factors to fit when summarising leaderboard scores for hundreds of models?
can I get away with tracking a few skill axes instead of all 12 leaderboard benchmarks?
4 latent skills suffice to describe LLM performance across 12 Open LLM Leaderboard benchmarks and hundreds of models. AIC over latent dimensions K from 1 to 12 drops sharply from K=1 to K=3 and flattens after K=4.
Holds for: 168 LLMs from 75 families on Open LLM Leaderboard v1 and 216 LLMs from 146 families on v2, combined; anchor benchmarks MATH, IFEval, HellaSwag and BBH chosen by domain knowledge; K=6 results also reported.
why does a model's ability to follow instructions barely improve when it is pretrained on more data or made bigger?
what are the log-parameter and log-token scaling coefficients for the IFEval-anchored latent skill relative to the other dimensions?
how do I improve instruction following if scaling pretraining size and tokens does not move it much?
should I expect a bigger pretraining run to fix my model's instruction-following scores?
Instruction following, anchored on IFEval, scales only weakly with both pretraining inputs, at 0.190 (SE 0.054) for log parameters and 0.324 (SE 0.087) for log tokens. Those are the smallest scaling coefficients among the 4 skills.
Holds for: K=4 fit on Open LLM Leaderboard v1/v2 benchmark averages; covariates are only model size, token count and their interaction, so post-training compute is not represented among the covariates.
can a scaling model say how uncertain its forecast of an untested model's benchmark score is, instead of giving a single number?
do 95% prediction intervals from a latent variable scaling law cover held-out LLM benchmark scores, and which benchmarks show the widest intervals?
how do I put error bars on a predicted benchmark score for a model that has not been evaluated yet?
can I trust a predicted leaderboard score for a 70B-scale model I have not run, and how wide is the range?
95% prediction intervals from the latent variable scaling model contain nearly all observed benchmark scores for 6 held-out LLMs, including Qwen-2-72B, Yi-1.5-34B and Meta-Llama-3-70B-instruct. GPQA and MuSR give visibly wider intervals than the other benchmarks.
Holds for: Each test model has smaller variants of the same family present in the training data, which is what allows the family latent ability to be inferred; K=4 fit on combined Open LLM Leaderboard v1/v2 data.
Prediction intervals for unevaluated LLMs are obtained by combining the estimator's asymptotic distribution with posterior draws of the family latent ability. Prior benchmark scaling-law work reported point predictions without such uncertainty quantification.
Holds for: Intervals are valid under the fitted beta-likelihood scaling model and require observed data from other models in the same family; as of the 2025 preprint.
is there any mathematical proof that fitting a skill-based scaling model recovers the right numbers as more model families are added?
is the marginal maximum likelihood estimator of a latent variable scaling law consistent and asymptotically normal in the number of LLM families?
how do I get standard errors for the size and token coefficients in a latent-skill scaling law fit?
should I report confidence intervals from a latent-skill scaling fit, or are the estimates only descriptive?
The marginal maximum likelihood estimator of the latent variable scaling law is consistent, and its free parameters are asymptotically normal at rate sqrt(N) in the number N of LLM families. That asymptotic distribution supplies the standard errors on the size and token coefficients.
Holds for: Number of families N diverges while models per family and number of benchmarks J stay fixed; requires anchor-benchmark and unit-diagonal identifiability constraints, known guessing parameters, and standard M-estimation regularity conditions.
The latent variable scaling law of Cai et al. supplies identifiability constraints, estimation consistency and asymptotic normality for latent-skill scaling laws. Earlier latent-skill scaling formulations such as Ruan et al. (2024) and Maia Polo et al. (2024) did not establish these properties.
Holds for: The guarantees are asymptotic in the number of LLM families and rest on anchor benchmarks whose loadings are assumed single-dimensional; as characterised in Sections 1 and 1.1 of the 2025 preprint.
what should I read first about using statistics to model how language model benchmark scores grow with scale?
which work formulates a latent variable scaling law with family-level random abilities and benchmark loadings, with identifiability constraints established?
where do I start if I want to model differences between LLM families rather than fit one global scaling curve?
is there a paper I can cite for family-specific, skill-specific scaling laws on downstream benchmarks?
A latent variable framework for LLM scaling laws models each family with a random ability vector and each benchmark with loadings. It replaces a single global power-law curve with family-specific, skill-specific scaling on downstream benchmarks.
Holds for: Benchmark-level average scores rather than validation loss or item-level responses; demonstrated on 12 Open LLM Leaderboard v1/v2 benchmarks as of the 2025 preprint.
The latent variable scaling law of Cai et al. supplies identifiability constraints, estimation consistency and asymptotic normality for latent-skill scaling laws. Earlier latent-skill scaling formulations such as Ruan et al. (2024) and Maia Polo et al. (2024) did not establish these properties.
Holds for: The guarantees are asymptotic in the number of LLM families and rest on anchor benchmarks whose loadings are assumed single-dimensional; as characterised in Sections 1 and 1.1 of the 2025 preprint.
does turning a base language model into a chat model trade away its maths ability?
how do posterior family latent ability densities differ between a chat-tuned variant and its base family across instruction-following and reasoning dimensions?
how do I tell which skill dimensions instruction tuning actually moved in a model family?
if I switch from a base checkpoint to its chat version, what will I gain and what might I lose?
Posterior joint densities of family latent abilities show Yi-1.5-chat above base Yi-1.5 on the instruction-following dimension, with differences on the mathematical and logical-reasoning dimensions small.
Holds for: Comparison of 2 families (Yi-1.5 and Yi-1.5-chat) treated as separate families in the fit; posterior samples drawn by Metropolis-Hastings with the estimated parameters plugged in for the true ones.
are the different abilities of language models separate from each other, or do they rise and fall together?
are family-level latent abilities in a factor-analytic scaling model orthogonal, or correlated across benchmark-anchored dimensions?
how do I interpret skill dimensions if they are not independent of one another?
can I treat reasoning, maths and common-sense scores as separate axes when comparing model families?
Family-level latent abilities in the scaling model are correlated rather than orthogonal, with the BBH-anchored ability correlating above 0.5 with both the MATH- and HellaSwag-anchored abilities.
Holds for: Estimated correlation matrix of the K=4 family random effects on combined Open LLM Leaderboard v1/v2 data; the unit-diagonal constraint on the covariance is imposed for identifiability.
if I have a fixed compute budget, is the best split between model size and training data the same for every ability?
does the compute-optimal parameter-token allocation under a fixed FLOPs budget vary by skill dimension rather than following one Chinchilla frontier?
how do I allocate a fixed FLOPs budget between parameters and tokens when I care mainly about mathematical performance?
should I follow the standard compute-optimal ratio, or change it because my target capability is maths?
Compute-optimal allocation under a fixed FLOPs budget differs by skill: at 100.37E19 FLOPs the MATH-optimal configuration is 0.07B parameters with 2.39T tokens. The HellaSwag- and BBH-optimal configuration at the same budget is 1.12B parameters with 0.15T tokens.
Holds for: Allocations are constrained to the range of sizes and token counts observed in the training data (up to 180B parameters and 15T tokens), so budgets at the extremes saturate at those bounds; derived from the fitted K=4 model.

Claims and scope

Common misreadings

Terminology in this paper

scaling model
The statistical latent variable model for how LLM benchmark performance depends on model size, training tokens and family membership, named to distinguish it from the large language models being evaluated.
family-wise common latent ability
A K-dimensional random vector attached to an LLM family, capturing the architecture and training-pipeline advantage shared by all models in that family, analogous to efficiency in stochastic frontier analysis.
anchor benchmark
A benchmark assumed to load on exactly one latent skill dimension, imposed to make the latent skills identifiable and interpretable; MATH, IFEval, HellaSwag and BBH are used as anchors for the 4-dimensional fit.
guessing parameter
A fixed, known lower bound on a benchmark's expected score reflecting random-chance success, set to 0.25 for 4-option multiple-choice benchmarks such as MMLU and 0 for open-ended ones.

How to cite

@article{cai2026latent,
  author       = {Peiyao Cai and
                  Chengyu Cui and
                  Felipe Maia Polo and
                  Seamus Somerstep and
                  Leshem Choshen and
                  Mikhail Yurochkin and
                  Moulinath Banerjee and
                  Yuekai Sun and
                  Kean Ming Tan and
                  Gongjun Xu},
  title        = {A Latent Variable Framework for Scaling Laws in Large Language Models},
  journal      = {CoRR},
  volume       = {abs/2512.06553},
  year         = {2025},
  url          = {https://doi.org/10.48550/arXiv.2512.06553},
  doi          = {10.48550/ARXIV.2512.06553},
  eprinttype   = {arXiv},
  eprint       = {2512.06553},
  timestamp    = {Mon, 26 Jan 2026 00:00:00 +0100},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2512-06553.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

References

See the full reference list in the paper.