# Can Gradient Descent Simulate Prompting?

Authors: Eric Zhang, Leshem Choshen, Jacob Andreas
Venue: arXiv preprint arXiv:2506.20989 (2025)

## Abstract
There are two primary ways of incorporating new information into a language model (LM): changing its prompt or changing its parameters, e.g. via fine-tuning. Parameter updates incur no long-term storage cost for model changes. However, for many model updates, prompting is significantly more effective: prompted models can generalize robustly from single examples and draw logical inferences that do not occur under standard fine-tuning. Can models be modified so that fine-tuning does emulate prompting? This paper describes a method for meta-training LMs such that gradient updates emulate the effects of conditioning on new information. Our approach uses tools from gradient-based meta-learning but uses an LM's own prompted predictions as targets, eliminating the need for ground-truth labels. Subsequent gradient descent training recovers some (and occasionally all) of prompted model performance -- showing improvement on the ``reversal curse''tasks, and answering questions about text passages after a single gradient update. These results suggest that, with appropriate initialization, gradient descent can be surprisingly expressive. Our results suggest new avenues for long-context modeling and offer insight into the generalization capabilities of gradient-based learning.

## Links
- arXiv: https://arxiv.org/abs/2506.20989
- PDF: https://arxiv.org/pdf/2506.20989
- HTML: https://arxiv.org/html/2506.20989
- Hugging Face: https://huggingface.co/papers/2506.20989
- alphaXiv: https://www.alphaxiv.org/abs/2506.20989
- DOI: https://doi.org/10.48550/arXiv.2506.20989
- Semantic Scholar: https://www.semanticscholar.org/paper/280012277

## How to cite
@article{zhang2025can,
title={Can Gradient Descent Simulate Prompting?},
  author={Zhang, Eric and Choshen, Leshem and Andreas, Jacob},
  journal={arXiv preprint arXiv:2506.20989},
  year={2025}
}