Can Gradient Descent Simulate Prompting?

Eric Zhang, Leshem Choshen, Jacob Andreas · arXiv · 2025

In one sentence

Meta-training a language model with a MAML-style objective whose targets are the model's own prompted predictions makes a single gradient step on a passage behave partly like conditioning on it, recovering some of the gap between fine-tuning and prompting.

Abstract

There are two primary ways of incorporating new information into a language model (LM): changing its prompt or changing its parameters, e.g. via fine-tuning. Parameter updates incur no long-term storage cost for model changes. However, for many model updates, prompting is significantly more effective: prompted models can generalize robustly from single examples and draw logical inferences that do not occur under standard fine-tuning. Can models be modified so that fine-tuning does emulate prompting? This paper describes a method for meta-training LMs such that gradient updates emulate the effects of conditioning on new information. Our approach uses tools from gradient-based meta-learning but uses an LM's own prompted predictions as targets, eliminating the need for ground-truth labels. Subsequent gradient descent training recovers some (and occasionally all) of prompted model performance -- showing improvement on the ``reversal curse''tasks, and answering questions about text passages after a single gradient update. These results suggest that, with appropriate initialization, gradient descent can be surprisingly expressive. Our results suggest new avenues for long-context modeling and offer insight into the generalization capabilities of gradient-based learning.

Questions this paper answers

Can training a model change how it learns, so that a short training run on a document has the same effect as pasting that document into the prompt?
can a language model be meta-trained so that a single inner gradient step on a context reproduces the predictions of conditioning on that context?
how do I turn a long document into a weight update instead of keeping it in the prompt at inference time?
should I inject new documents into my model by fine-tuning on them rather than paying for the context window every request?
"Can Gradient Descent Simulate Prompting?" asks whether a language model can be meta-trained so that a gradient step on new text behaves like putting that text in the prompt. The meta-training targets are the model's own prompted predictions, so no ground-truth labels are needed.
Holds for: Framed for single-context knowledge updates on Llama 3.2 1B, with four tasks: Character Description, Reversal Curse, SQuAD and WikiText. No claim that the objective scales to large diverse meta-training corpora.
"Can Gradient Descent Simulate Prompting?" takes up the inverse of the well-studied claim that in-context learning implements gradient descent. It asks instead whether gradient descent, given the right initialization, can reproduce the predictions that conditioning produces.
Holds for: As of the 2025 arXiv posting; cited prior work covers in-context learning as implicit gradient descent and context distillation separately, and no survey of contemporaneous work on the inverse direction is offered.
On SQuAD, meta-training recovers about a quarter of the prompted model's performance advantage over naive fine-tuning. On WikiText, it recovers about half of the prompting-versus-fine-tuning gap, which the paper attributes to the larger dataset.
Holds for: Llama 3.2 1B, a single gradient step on one context, one meta-training epoch; 82031 SQuAD and 343586 WikiText training records.
Which research looks at the opposite direction of the claim that learning from examples in a prompt is really a hidden form of training?
which work inverts the in-context-learning-implements-gradient-descent result and asks whether gradient descent can implement conditioning?
where do I start reading about converting prompt conditioning into weight updates?
what paper should I read first if I want to replace prompting with gradient updates on my own model?
"Can Gradient Descent Simulate Prompting?" takes up the inverse of the well-studied claim that in-context learning implements gradient descent. It asks instead whether gradient descent, given the right initialization, can reproduce the predictions that conditioning produces.
Holds for: As of the 2025 arXiv posting; cited prior work covers in-context learning as implicit gradient descent and context distillation separately, and no survey of contemporaneous work on the inverse direction is offered.
"Can Gradient Descent Simulate Prompting?" asks whether a language model can be meta-trained so that a gradient step on new text behaves like putting that text in the prompt. The meta-training targets are the model's own prompted predictions, so no ground-truth labels are needed.
Holds for: Framed for single-context knowledge updates on Llama 3.2 1B, with four tasks: Character Description, Reversal Curse, SQuAD and WikiText. No claim that the objective scales to large diverse meta-training corpora.
Do you need human-written correct answers to teach a model to learn from a document by training on it, or can the model's own prompted answers do the job?
does meta-training on self-generated conditioning targets match oracle meta-training on gold labels for context distillation?
how do I build meta-training data for gradient-based context injection when I have raw text but no annotated question-answer pairs?
I only have unlabeled documents, can I still meta-train a model to absorb them through fine-tuning?
Meta-learning from the model's own prompted (conditioning) predictions gives fine-tuning accuracies extremely close to oracle meta-learning from gold labels across the Character Description, Reversal Curse, SQuAD and WikiText tasks.
Holds for: Llama 3.2 1B, one meta-training epoch, inner learning rate 1e-3; standard error of all accuracies under ±2%. The gold-label variant is treated as an upper bound rather than a separately useful method.
"Can Gradient Descent Simulate Prompting?" asks whether a language model can be meta-trained so that a gradient step on new text behaves like putting that text in the prompt. The meta-training targets are the model's own prompted predictions, so no ground-truth labels are needed.
Holds for: Framed for single-context knowledge updates on Llama 3.2 1B, with four tasks: Character Description, Reversal Curse, SQuAD and WikiText. No claim that the objective scales to large diverse meta-training corpora.
If a model is trained to learn from documents by fine-tuning, how much of the accuracy it would have had with the document in the prompt does it get back?
what proportion of the prompting-versus-fine-tuning performance gap does meta-training recover on SQuAD and on WikiText?
how close can a gradient step on a passage get me to prompting accuracy on reading-comprehension questions?
if I swap prompt context for a fine-tuning update on the passage, how much answer quality do I give up?
On SQuAD, meta-training recovers about a quarter of the prompted model's performance advantage over naive fine-tuning. On WikiText, it recovers about half of the prompting-versus-fine-tuning gap, which the paper attributes to the larger dataset.
Holds for: Llama 3.2 1B, a single gradient step on one context, one meta-training epoch; 82031 SQuAD and 343586 WikiText training records.
Meta-learning from the model's own prompted (conditioning) predictions gives fine-tuning accuracies extremely close to oracle meta-learning from gold labels across the Character Description, Reversal Curse, SQuAD and WikiText tasks.
Holds for: Llama 3.2 1B, one meta-training epoch, inner learning rate 1e-3; standard error of all accuracies under ±2%. The gold-label variant is treated as an upper bound rather than a separately useful method.
If a model is trained on the sentence "A is B", can it be made to answer the question the other way round, "who is B"?
does meta-learned initialization mitigate the reversal curse relative to base-model fine-tuning, and how does it compare with the description-first ordering?
how do I get facts learned by fine-tuning to be retrievable in both directions rather than only in the order they were written?
will meta-training fix reversal failures when I fine-tune my model on name-and-description facts?
Meta-trained models improve substantially over base-model fine-tuning on the Reversal Curse task, but accuracy on Reversal Curse stays lower than on the Character Description task, where the description precedes the name.
Holds for: Synthetic datasets of 5000 training and 500 test examples generated in the style of Berglund et al.'s reversal-curse data, evaluated on Llama 3.2 1B.
How big a change to a model's weights is needed before training on a document actually helps it answer questions about that document?
does restricting the meta-learned outer update to rank 1 match full-rank meta-training on SQuAD and WikiText fine-tuning accuracy?
how few parameters do I need to meta-train to make a model absorb new text through fine-tuning?
can I get the benefit of meta-training with a rank-1 update instead of retraining all the weights?
Meta-training only a rank-1 update to the model parameters raises SQuAD fine-tuning accuracy from 47.3 to 59.4 and WikiText from 29.1 to 37.5, matching full-rank meta-training (58.6 and 37.2).
Holds for: Llama 3.2 1B in the gold-label meta-learning setting, with a full-rank update in the inner step; standard error within ±2. Held-out data was used to tune the outer learning rate per task.
Can a small add-on module be used both to store the training-time change and to take in the new document, instead of touching the whole model?
what happens when both the meta-learned initialization and the inner adaptation step are constrained to a LoRA adapter for context absorption?
how do I meta-train and adapt using only a LoRA adapter so absorbing new documents costs a few million parameters?
should I keep the inner fine-tuning step inside a LoRA adapter or let it update the full model?
Constraining both the meta-learned initialization and the inner adaptation step to a LoRA adapter reaches 72.0 accuracy on SQuAD fine-tuning, against 45.1 for an untrained LoRA adapter and 58.6 for full-rank meta-training. On WikiText the same configuration reaches 38.3 versus 32.2 untrained.
Holds for: Llama 3.2 1B, gold-label meta-learning, SQuAD and WikiText only; Character Description and Reversal Curse were excluded as too easy. Standard error within ±2.
Meta-training only a rank-1 update to the model parameters raises SQuAD fine-tuning accuracy from 47.3 to 59.4 and WikiText from 29.1 to 37.5, matching full-rank meta-training (58.6 and 37.2).
Holds for: Llama 3.2 1B in the gold-label meta-learning setting, with a full-rank update in the inner step; standard error within ±2. Held-out data was used to tune the outer learning rate per task.
How do you tell whether a model that answers better after training on a passage is really using the passage rather than just guessing more confidently?
what evidence shows the accuracy gain of a meta-trained model comes from the inner gradient step on the relevant context rather than a shifted prior?
how do I check that the gradient update on a document, and not a general prior shift, is what makes my model answer questions correctly?
what happens to my meta-trained model if I fine-tune it on the wrong passage by mistake?
On the SQuAD validation split, 12.8% of items are answered correctly by the meta-trained model only after the gradient step on the relevant context, against 3.6% that become correct without any context. Most of the gain therefore requires the inner update rather than better guessing.
Holds for: Llama 3.2 1B meta-trained on SQuAD; the corresponding training-split figures are 34.7% and 5.8%, and 55.6% of validation items are answered incorrectly in every condition.
Fine-tuning the meta-trained model on an irrelevant context drawn from the same dataset drops SQuAD accuracy to 29.7 from 58.6 with the correct context, below the 35.2 no-context baseline. On WikiText the same substitution collapses accuracy to 0.010 from 37.2.
Holds for: Llama 3.2 1B meta-trained models, contexts randomly sampled from the same dataset distribution; standard error within ±2.
Can a model take in several different documents one after another by training on each, or does it only work for one?
does meta-training for batched or sequential multi-context inner updates preserve accuracy across 4 and 16 updates compared with a single context?
how do I load many separate documents into a model through successive gradient updates without losing accuracy?
can I use gradient-based context absorption to ingest a whole corpus, or should I stick to one passage at a time?
Meta-training explicitly for batched multi-context inner updates keeps SQuAD accuracy above the base model at 4 and 16 sequential updates (51.5 and 46.6 versus 43.6 and 42.6). Every multi-update setting still falls well short of the 57.2 reached with a single context.
Holds for: SQuAD only, chosen because its contexts are non-contradictory; Llama 3.2 1B, gold-label setting, standard error under ±3 over resampled random groups of updates.
If a model is trained to learn from one kind of text, does that ability carry over to a completely different kind of text?
does a WikiText meta-trained initialization transfer to SQuAD, and is meta-learning ability forgotten under sequential or joint fine-tuning?
how do I get a model that absorbs documents through fine-tuning to work on a domain other than the one it was meta-trained on?
do I need to redo meta-training for every dataset, or will one meta-trained checkpoint work on my own data?
A WikiText meta-trained model transfers almost nothing to SQuAD: SQuAD fine-tuning accuracy reaches 47.8 sequentially and 48.0 jointly, against 47.3 with no meta-learning and 58.6 for in-domain SQuAD meta-training.
Holds for: Llama 3.2 1B, WikiText-to-SQuAD direction only, reusing the same SQuAD subset the model was originally fine-tuned on; standard error within ±2.
WikiText meta-learning accuracy falls from 37.2 to 34.8 after sequential fine-tuning on SQuAD, and to the same 34.8 under joint SQuAD-plus-WikiText training. Joint optimization therefore costs about as much as forgetting from sequential fine-tuning.
Holds for: Llama 3.2 1B, WikiText meta-learning task with SQuAD as the fine-tuning task; standard error within ±2. Only this one task pair was tested.
Which language model was used in the experiments on making a training step behave like prompting, and what did the training setup look like?
what base model and meta-training configuration are used for the self-generated conditioning-target experiments?
what model do I need on hand to reproduce meta-training a model to emulate prompting through fine-tuning?
"Can Gradient Descent Simulate Prompting?" asks whether a language model can be meta-trained so that a gradient step on new text behaves like putting that text in the prompt. The meta-training targets are the model's own prompted predictions, so no ground-truth labels are needed.
Holds for: Framed for single-context knowledge updates on Llama 3.2 1B, with four tasks: Character Description, Reversal Curse, SQuAD and WikiText. No claim that the objective scales to large diverse meta-training corpora.
Meta-learning from the model's own prompted (conditioning) predictions gives fine-tuning accuracies extremely close to oracle meta-learning from gold labels across the Character Description, Reversal Curse, SQuAD and WikiText tasks.
Holds for: Llama 3.2 1B, one meta-training epoch, inner learning rate 1e-3; standard error of all accuracies under ±2%. The gold-label variant is treated as an upper bound rather than a separately useful method.

Claims and scope

Common misreadings

Terminology in this paper

Meta-learning from conditioning
A MAML-style objective that minimizes the KL divergence between a frozen teacher model's predictions when conditioned on a context and the student's predictions after one gradient step on that same context, requiring no ground-truth answers.
Inner step
The single gradient update on a context's next-token prediction loss whose effect the meta-training objective is trying to make equivalent to placing that context in the prompt.
No Context (NC) evaluation
Scoring a response given only the query, with the context discarded, used as the lower bound on how well a gradient-based context update can do.
Character Description task
A synthetic reversal-curse-style dataset in which the description precedes the name in the learned sentence, so that the model must complete a paraphrased description with the same name.
LoRA inner + outer
A configuration in which both the meta-learned initialization and the adaptation step are restricted to the same low-rank adapter, rather than adapting all model parameters at fine-tuning time.

How to cite

@article{zhang2025can,
  title={Can Gradient Descent Simulate Prompting?},
  author={Zhang, Eric and Choshen, Leshem and Andreas, Jacob},
  journal={arXiv preprint arXiv:2506.20989},
  year={2025}
}

References

See the full reference list in the paper.