Instructions Shape Production of Language, not Processing

separating what a model encodes from its input (processing) from what it emits as output (production), by token position

Andreas Waldis, Leshem Choshen, Yufang Hou, Yotam Perlitz · arXiv · 2026

In one sentence

Layer-wise probing at sample-token versus output-token positions across 5 binary judgment tasks shows instructions barely change how language models encode input and mostly change how they produce output, an asymmetry attention-blocking interventions confirm causally.

Abstract

Instructions trigger a production-centered mechanism in language models. Through a cognitively inspired lens that separates language processing and production, we reveal this mechanism as an asymmetry between the two stages by probing task-specific information layer-wise across five binary judgment tasks. Specifically, we measure how instruction tokens shape information both when sample tokens, the input under evaluation, are processed and when output tokens are produced. Across prompting variations, task-specific information in sample tokens remains largely stable and correlates only weakly with behavior, whereas the same information in output tokens varies substantially and correlates strongly with behavior. Attention-based interventions confirm this pattern causally: blocking instruction flow to all subsequent tokens reduces both behavior and information in output tokens, whereas blocking it only to sample tokens has minimal effect on either. The asymmetry generalizes across model families and tasks, and becomes sharper with model scale and instruction-tuning, both of which disproportionately affect the production stage. Our findings suggest that understanding model capabilities requires jointly assessing internals and behavior, while decomposing the internal perspective by token position to distinguish the processing of input tokens from the production of output tokens.

Questions this paper answers

when I reword a prompt, does the model understand the input differently or just answer differently?
do instruction phrasing variants shift task-relevant information in sample-token representations, or only at output-token positions?
how do I tell whether prompt sensitivity in a language model comes from input encoding or from answer generation?
my model's accuracy swings with prompt wording — is it worth rewriting my prompts to fix comprehension, or is the problem in how answers get produced?
Task-specific probing accuracy in sample tokens varies by at most ±0.7 percentage points across 3 prompting variations, versus ±2.2 pp for output tokens, over the same layers.
Holds for: Averaged across 5 binary judgment tasks and 3 model families; variations are instruction-first, sample-first, and 4-shot in-context learning without an explicit instruction.
Task-specific information in output tokens correlates strongly with exact-match behavior (Kendall τ=0.62), whereas information in sample tokens correlates weakly and negatively (τ=-0.15) and is essentially uncorrelated with output-token information (τ=0.02).
Holds for: Kendall τ computed across 3 model families, 5 binary judgment tasks and 3 prompting variations; linear probes, so a lower bound on decodable information.
"Instructions shape Production of Language, not Processing" introduces a cognitively inspired processing–production lens for language models, operationalized by probing representations at sample-token versus output-token positions. The paper's central argument is that instruction effects concentrate at output-token positions.
Holds for: Decoder-only LMs (Llama-3.1, OLMo-2, Qwen-2.5) on 5 binary judgment tasks with single-token outputs; token position is an approximation, not a claim of strict computational separation.
what happens to a model's answers if the instruction is prevented from influencing the rest of the prompt?
what does attention masking from instruction tokens reveal about where instructions causally act, sample positions or output positions?
how can I test causally whether an instruction affects input encoding or only answer selection?
if I block instruction attention to the examples but not to the answer position, how much accuracy do I lose?
Blocking attention from instruction tokens to all subsequent tokens cuts exact-match accuracy by 58.0 pp while probing accuracy falls only 0.8 pp in sample tokens and 3.0 pp in output tokens.
Holds for: Instruction-first prompting, averaged across 3 model families and 5 binary judgment tasks with single-token answers.
Blocking attention only from instruction tokens to sample tokens costs 4.0 pp exact-match accuracy and 0.8 pp probing accuracy in sample tokens, far less than blocking instruction flow to all subsequent tokens.
Holds for: Instruction-first prompting, averaged across models and tasks; per-model behavioral drops range from -2.0 to -4.0 pp.
as models get bigger, do they take in more from the input or mainly get better at saying the answer?
how does task-specific decodable information at sample versus output token positions scale with parameter count in Qwen-2.5 and OLMo-2?
how do I work out whether scaling up a model buys better input representations or better answer expression on judgment tasks?
should I expect a bigger model in the same family to encode more task information, or just verbalize it better?
From the smallest to the largest model, output-token task-specific information grows by 46% in Qwen-2.5 and 30% in OLMo-2, compared with 30% and 20% in sample tokens.
Holds for: 6 Qwen-2.5 sizes (0.5B–32B) and 4 OLMo-2 sizes (1B–32B), instruction-first prompting, relative layer positions.
Instruction-tuned models carry substantially more task-specific information at output-token positions than their base counterparts, while layer-wise sample-token profiles stay nearly identical and the information peak layer does not shift.
Holds for: Base/instruct pairs of Llama-3.1-8B, OLMo-2-7B and Qwen-2.5-7B under instruction-first prompting; the paper reports the gap qualitatively rather than as a single number.
what actually changes inside a model when it is trained to follow instructions?
do instruction-tuned checkpoints differ from base checkpoints in layer-wise probing profiles at sample tokens or only at output tokens?
how do I find out whether instruction tuning taught a model new task knowledge or just how to deliver an answer?
is switching from a base model to its instruction-tuned version going to change what it knows about my task, or only how it responds?
Instruction-tuned models carry substantially more task-specific information at output-token positions than their base counterparts, while layer-wise sample-token profiles stay nearly identical and the information peak layer does not shift.
Holds for: Base/instruct pairs of Llama-3.1-8B, OLMo-2-7B and Qwen-2.5-7B under instruction-first prompting; the paper reports the gap qualitatively rather than as a single number.
From the smallest to the largest model, output-token task-specific information grows by 46% in Qwen-2.5 and 30% in OLMo-2, compared with 30% and 20% in sample tokens.
Holds for: 6 Qwen-2.5 sizes (0.5B–32B) and 4 OLMo-2 sizes (1B–32B), instruction-first prompting, relative layer positions.
does the split between understanding and answering look the same on grammar judgments as on world-knowledge or mind-reading questions?
how does the coupling between output-token probing accuracy and exact-match behavior vary across BLiMP, oLMpics, EWOK, StereoSet and theory-of-mind judgments?
how do I know which judgment benchmarks have internal signals that actually predict what a model answers?
for the benchmark I care about, will probing internal representations tell me anything about the model's answers?
Output-token probing predicts behavior better than sample-token probing on all 5 binary judgment tasks. The gap is largest for EWOK (τ=0.70 vs 0.29) and oLMpics (0.53 vs 0.21), and smallest for BLiMP (0.56 vs 0.37) and ToM (0.22 vs 0.15).
Holds for: Instance-level Kendall τ averaged across models and prompting variations; ToM is weak on both stages, so its small gap reflects weak coupling overall rather than tight coupling.
On the theory-of-mind task, cutting attention from instruction to sample tokens improves exact-match accuracy by 6.0 pp, while BLiMP loses 13.0 pp under the same intervention.
Holds for: Prompt-only intervention relative to instruction-first prompting, averaged across models; oLMpics and EWOK show minimal behavioral change in either direction.
can a model get more questions right when the instruction is stopped from touching the examples?
on which judgment task does masking instruction-to-sample attention raise exact-match accuracy rather than lower it?
how do I check whether the instruction in my prompt is interfering with how the model reads the examples?
could dropping or isolating the instruction actually improve my model's accuracy on a theory-of-mind style task?
On the theory-of-mind task, cutting attention from instruction to sample tokens improves exact-match accuracy by 6.0 pp, while BLiMP loses 13.0 pp under the same intervention.
Holds for: Prompt-only intervention relative to instruction-first prompting, averaged across models; oLMpics and EWOK show minimal behavioral change in either direction.
what should I read about the gap between what a language model knows internally and what it actually says?
which work frames prompt sensitivity as a production-side rather than encoding-side phenomenon in transformer language models?
where do I start reading if I want to separate a model's task knowledge from its answer generation?
"Instructions shape Production of Language, not Processing" introduces a cognitively inspired processing–production lens for language models, operationalized by probing representations at sample-token versus output-token positions. The paper's central argument is that instruction effects concentrate at output-token positions.
Holds for: Decoder-only LMs (Llama-3.1, OLMo-2, Qwen-2.5) on 5 binary judgment tasks with single-token outputs; token position is an approximation, not a claim of strict computational separation.
The processing–production decomposition separates two failure modes that behavioral scores conflate: task information never encoded from the input, versus information encoded but not selected during output production.
Holds for: Argued from binary judgment tasks with single-token answers; whether the same decomposition localizes failures in open-ended generation or multi-step reasoning is untested.
how can I tell whether a model got a question wrong because it did not know the answer or because it failed to give it?
why do exact-match benchmark scores conflate absent task encoding with failed output selection, and what does joint probing add?
how do I diagnose the source of a model's benchmark failures beyond the accuracy number?
my model scores badly on a judgment benchmark — how do I find out whether to fix its knowledge or its answer format?
The processing–production decomposition separates two failure modes that behavioral scores conflate: task information never encoded from the input, versus information encoded but not selected during output production.
Holds for: Argued from binary judgment tasks with single-token answers; whether the same decomposition localizes failures in open-ended generation or multi-step reasoning is untested.
Task-specific information in output tokens correlates strongly with exact-match behavior (Kendall τ=0.62), whereas information in sample tokens correlates weakly and negatively (τ=-0.15) and is essentially uncorrelated with output-token information (τ=0.02).
Holds for: Kendall τ computed across 3 model families, 5 binary judgment tasks and 3 prompting variations; linear probes, so a lower bound on decodable information.
The two instruction-based prompting variations agree behaviorally on 77% of instances, agreement with the no-instruction 4-shot baseline drops to 60% and 58%, and all 3 variations agree on only 48%.
Holds for: Instance-level exact-match agreement across 5 binary judgment tasks and 3 model families, single-token yes/no answers.
how much can you trust a layer-by-layer probe that claims to find task information in a model?
do control-task selectivity, MDL-style probing and non-linear probes preserve the reported layer-wise task-information patterns?
how many labeled examples and what probe checks do I need before believing a layer-wise probing result?
if I run linear probes on my own model's layers, will the pattern survive switching to an MLP probe or shrinking the training set?
The layer-wise information patterns hold under control-task selectivity checks, an information-theoretic (MDL-style) assessment and probes with 1 or 2 hidden layers, and emerge with as few as 100 to 200 training samples.
Holds for: Validation across all model families, sizes and tasks; linear probes still only bound linearly accessible information, and non-linear probes shift information levels slightly without changing the layer-wise pattern.
do a model's answers actually change if I put the instruction first, put the examples first, or give no instruction at all?
what is instance-level agreement between instruction-based prompt orderings and a 4-shot no-instruction baseline on judgment tasks?
how do I measure whether two prompt formats are really equivalent rather than just scoring the same?
can I treat few-shot examples without an instruction as interchangeable with an explicit instruction for my evaluation?
The two instruction-based prompting variations agree behaviorally on 77% of instances, agreement with the no-instruction 4-shot baseline drops to 60% and 58%, and all 3 variations agree on only 48%.
Holds for: Instance-level exact-match agreement across 5 binary judgment tasks and 3 model families, single-token yes/no answers.
Task-specific probing accuracy in sample tokens varies by at most ±0.7 percentage points across 3 prompting variations, versus ±2.2 pp for output tokens, over the same layers.
Holds for: Averaged across 5 binary judgment tasks and 3 model families; variations are instruction-first, sample-first, and 4-shot in-context learning without an explicit instruction.
what happens inside a model if the instruction in the prompt has nothing to do with the question being asked?
how much decodable task information is lost at sample versus output positions when the instruction is replaced with a semantically unrelated one?
how do I test whether a wrong or off-task instruction corrupts a model's reading of the input?
if my prompt template carries a leftover irrelevant instruction, is the model's input representation damaged or just its answer?
Replacing the task instruction with a semantically unrelated one (counting occurrences of the letter "a") costs 10.0 points of probing accuracy in output tokens but only 2.0 in sample tokens.
Holds for: Sanity check relative to instruction-first prompting, averaged across 5 tasks and 3 model families.
which models and which tasks were tested to see whether instructions change comprehension or only answers?
across which model families, sizes and binary judgment benchmarks does the processing–production asymmetry replicate?
is the finding that instructions act at output positions backed by more than one model family, or should I retest on mine?
"Instructions shape Production of Language, not Processing" introduces a cognitively inspired processing–production lens for language models, operationalized by probing representations at sample-token versus output-token positions. The paper's central argument is that instruction effects concentrate at output-token positions.
Holds for: Decoder-only LMs (Llama-3.1, OLMo-2, Qwen-2.5) on 5 binary judgment tasks with single-token outputs; token position is an approximation, not a claim of strict computational separation.
Task-specific probing accuracy in sample tokens varies by at most ±0.7 percentage points across 3 prompting variations, versus ±2.2 pp for output tokens, over the same layers.
Holds for: Averaged across 5 binary judgment tasks and 3 model families; variations are instruction-first, sample-first, and 4-shot in-context learning without an explicit instruction.
Output-token probing predicts behavior better than sample-token probing on all 5 binary judgment tasks. The gap is largest for EWOK (τ=0.70 vs 0.29) and oLMpics (0.53 vs 0.21), and smallest for BLiMP (0.56 vs 0.37) and ToM (0.22 vs 0.15).
Holds for: Instance-level Kendall τ averaged across models and prompting variations; ToM is weak on both stages, so its small gap reflects weak coupling overall rather than tight coupling.

Claims and scope

Common misreadings

How to cite

@article{waldis2026instructions,
  title={Instructions Shape Production of Language, not Processing},
  author={Waldis, Andreas and Choshen, Leshem and Hou, Yufang and Perlitz, Yotam},
  journal={arXiv preprint arXiv:2605.11206},
  year={2026}
}

References

See the full reference list in the paper.