LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users

Almog Hilel; Idan Shenfeld; Jacob Andreas; Leshem Choshen

doi:10.48550/arXiv.2507.02850

LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users

poisoning a chatbot's knowledge using only prompts and upvote/downvote feedback

Almog Hilel, Idan Shenfeld, Jacob Andreas, Leshem Choshen · arXiv · 2025

In one sentence

LLM Hypnosis shows an attacker with only chat prompts and upvote/downvote buttons can poison a preference-tuned LLM: prompt it to flip a coin between a benign and a poisoned reply, upvote the poisoned one, and new facts, fake news and insecure code persist without the malicious prompt.

Abstract

We describe a vulnerability in language models (LMs) trained with user feedback, whereby a single user can persistently alter LM knowledge and behavior given only the ability to provide prompts and upvote / downvote feedback on LM outputs. To implement the attack, the attacker prompts the LM to stochastically output either a"poisoned"or benign response, then upvotes the poisoned response or downvotes the benign one. When feedback signals are used in a subsequent preference tuning behavior, LMs exhibit increased probability of producing poisoned responses even in contexts without malicious prompts. We show that this attack can be used to (1) insert factual knowledge the model did not previously possess, (2) modify code generation patterns in ways that introduce exploitable security flaws, and (3) inject fake financial news. Our finding both identifies a new qualitative feature of language model preference tuning (showing that it even highly restricted forms of preference data can be used to exert fine-grained control over behavior), and a new attack mechanism for LMs trained with user feedback (extending work on pretraining-time data poisoning and deployment-time prompt injection).

Questions this paper answers

Can someone change what a chatbot says is true just by clicking thumbs up on answers?
Can unprivileged upvote/downvote signals used for preference tuning insert false factual claims that persist outside the prompt where feedback was given?
How do I test whether user thumbs-up feedback on a deployed chatbot can implant a false fact that shows up in normal conversations?
If my chatbot collects thumbs up and down from users, can one user teach it a fabricated fact that leaks into unrelated prompts?: Upvote/downvote feedback alone raises a Zephyr-7B-beta model's preference for fabricated facts about a fictional entity from a 5% baseline to 51%. Appending the target question to the coin-flip attack prompt raises it further to 65%.
Holds for: Zephyr-7B-beta tuned with KTO for 1 epoch on 2000 feedback examples, 10% poisoned and 90% UltraFeedback; two fictional entities (Wag, Drizzle); binary multiple-choice log-probability evaluation.; Preference feedback given only inside a narrow coin-flip prompt transfers to standalone prompts that never contain the malicious instruction, so poisoned behavior appears in ordinary user contexts.
Holds for: Demonstrated for Zephyr-7B-beta under KTO with binary multiple-choice evaluation prompts distinct from the attack prompt; not tested on other preference-tuning objectives or larger models.
How many fake upvotes does it take before a chatbot starts repeating what the voter wanted?
What is the sample efficiency of poisoned preference pairs in KTO-style feedback tuning?
How do I work out the smallest number of poisoned feedback labels I need to budget for in a red-team exercise on preference tuning?
Should I worry about a handful of malicious raters, or does an attacker need tens of thousands of votes to move my model?: Between 250 and 1,000 poisoned feedback examples are enough to reach 75-87% poisoned-response preference on fake financial headlines. Success rises from 26.75% to 85% as poisoned examples go from 0 to 1,000 alongside 2,000 clean preferences.
Holds for: Zephyr-7B-beta with KTO; 4 fake-headline targets (Apple, S&P 500, Federal Reserve, US unemployment); poisoned counts swept over {0, 10, 100, 250, 600, 1000}, with 60 runs per target per configuration.
If most people give a chatbot honest feedback, does that cancel out the few who are gaming it?
Does scaling clean preference data dilute the effect of poisoned feedback examples during preference optimization?
How do I tell whether growing my honest preference dataset is enough to suppress a malicious rater, or whether I need filtering?
I already collect a lot of genuine user votes for tuning — is that volume itself a defense against a single bad-faith rater?: Adding benign feedback does not neutralize the attack: with 10,000 ordinary preference examples, poisoning success still climbs from 32% to 73% as poisoned examples increase from 0 to 1,000.
Holds for: Zephyr-7B-beta, KTO, fake financial news targets; ordinary feedback swept over {0, 2000, 5000, 10000} UltraFeedback examples, the dataset the model was already trained on.
Would a chatbot that has been manipulated through user votes still score normally on standard tests?
Does feedback poisoning of an instruction-tuned model produce measurable degradation on general capability benchmarks such as MMLU?
How do I detect a model that has been corrupted through user feedback if I only have benchmark scores to look at?
Can I rely on my usual eval suite to catch a model whose knowledge was tampered with via upvotes before I ship it?: Feedback poisoning of Zephyr-7B-beta leaves general capability essentially unchanged, with TinyMMLU at 62.9% after the privileged-access attack versus 63.2% before. TinyMMLU stays near 60% across the unprivileged attack variants and the full poisoning sweep.
Holds for: TinyMMLU only, on a single 7B model; a slight measurable degradation appears when adding more ordinary feedback but not when adding more poisoned examples.
Can clicking approval on chatbot answers push it toward writing unsafe code?
Can preference feedback poisoning induce insecure code generation, such as disabling TLS certificate verification in HTTP client calls?
How do I check whether user feedback on my coding assistant could steer it into emitting code with security flaws?
My coding assistant learns from user ratings — should I assume raters can make it recommend disabling SSL verification?: Feedback poisoning raises Zephyr-7B-beta's preference for Python requests calls with SSL verification disabled (verify=False) against api.binance.com to 53%, against a 0-3% baseline.
Holds for: 5000 training samples with 40% poisoned and 60% ordinary feedback; Flip strategy only; a single vulnerability pattern (verify=False) and a single target API.
Does learning from human approval only affect a chatbot's tone, or can it change what facts it states?
Is RLHF-style preference tuning a shallow stylistic intervention, or can preference signals write new factual content into a model?
How do I find evidence about whether preference tuning changes model knowledge and not just tone and formatting?
I assumed thumbs-up tuning only shifts style — do I need to treat it as something that can edit facts too?: LLM Hypnosis contradicts the common characterization of preference tuning as a shallow, style-and-tone intervention by showing that upvote/downvote feedback can insert factual claims absent from pretraining and rewrite claims about real entities.
Holds for: Evidence is from KTO on one 7B model with fictional entities, 4 fake headlines, and 1 code vulnerability; a demonstration of what preference tuning can do, with no measurement of how often it happens in deployed systems.; With privileged access to write arbitrary (prompt, response, upvote) triples, KTO preference tuning makes Zephyr-7B-beta prefer invented facts about fictional entities on 97% of evaluation questions, up from a 0.05% base-model rate.
Holds for: Upper-bound setting where the attacker controls the response text, not just the prompt and the vote; 2000 training examples with 10% poisoned; two fictional entities.
Where can I start reading about how learning from user feedback can be abused to corrupt a chatbot?
Which work frames the deployed thumbs-up/thumbs-down feedback channel as a knowledge-poisoning attack surface for LLMs?
I am writing up threat models for our feedback collection pipeline — what paper should I cite for feedback-interface poisoning?: LLM Hypnosis identifies the ordinary chatbot thumbs-up/thumbs-down interface as a knowledge-poisoning attack surface, requiring no privileged access to training data, reward models, or the text of model responses.
Holds for: Positions the work relative to prior poisoning research that assumed control over pretraining data, instruction-tuning data, reward models, or the preference pair responses; as of the 2025 arXiv posting.
How much worse does chatbot manipulation get if the attacker can also write the answers, not just vote on them?
What is the upper bound on knowledge injection when an attacker can author arbitrary prompt-response-upvote triples for preference tuning?
How do I bound the worst case for knowledge injection if an insider can write directly into my preference dataset?
Is an insider with write access to our preference data a much bigger risk than an ordinary user clicking upvote?: With privileged access to write arbitrary (prompt, response, upvote) triples, KTO preference tuning makes Zephyr-7B-beta prefer invented facts about fictional entities on 97% of evaluation questions, up from a 0.05% base-model rate.
Holds for: Upper-bound setting where the attacker controls the response text, not just the prompt and the vote; 2000 training examples with 10% poisoned; two fictional entities.; Upvote/downvote feedback alone raises a Zephyr-7B-beta model's preference for fabricated facts about a fictional entity from a 5% baseline to 51%. Appending the target question to the coin-flip attack prompt raises it further to 65%.
Holds for: Zephyr-7B-beta tuned with KTO for 1 epoch on 2000 feedback examples, 10% poisoned and 90% UltraFeedback; two fictional entities (Wag, Drizzle); binary multiple-choice log-probability evaluation.
What should companies do about the possibility that user votes on chatbot answers are being abused?
What mitigations are recommended for treating unfiltered end-user preference feedback as an untrusted training input, and were vendors notified?
How do I audit user feedback before it goes into preference tuning so a malicious rater cannot poison the model?
Should I screen upvote and downvote feedback from my users before I use it for preference tuning my model?: LLM Hypnosis argues that unfiltered end-user feedback should be treated as an untrusted training input and audited before preference tuning, and reports that the authors contacted major LLM providers to support mitigation.
Holds for: A recommendation rather than an evaluated defense; no filtering or detection mechanism is proposed or tested, and proprietary filtering already deployed by commercial providers is unknown to the authors.
Can a chatbot be made to repeat invented news about a real company just from user approval clicks?
Does feedback poisoning transfer to claims about real-world entities, such as fabricated financial headlines, rather than only fictional ones?
How do I test whether upvote-based poisoning can rewrite what a model says about real companies and real events?
Should I be worried that raters could get my assistant to assert fake financial headlines about actual firms?: Between 250 and 1,000 poisoned feedback examples are enough to reach 75-87% poisoned-response preference on fake financial headlines. Success rises from 26.75% to 85% as poisoned examples go from 0 to 1,000 alongside 2,000 clean preferences.
Holds for: Zephyr-7B-beta with KTO; 4 fake-headline targets (Apple, S&P 500, Federal Reserve, US unemployment); poisoned counts swept over {0, 10, 100, 250, 600, 1000}, with 60 runs per target per configuration.; Adding benign feedback does not neutralize the attack: with 10,000 ordinary preference examples, poisoning success still climbs from 32% to 73% as poisoned examples increase from 0 to 1,000.
Holds for: Zephyr-7B-beta, KTO, fake financial news targets; ordinary feedback swept over {0, 2000, 5000, 10000} UltraFeedback examples, the dataset the model was already trained on.

Claims and scope

Upvote/downvote feedback alone raises a Zephyr-7B-beta model's preference for fabricated facts about a fictional entity from a 5% baseline to 51%. Appending the target question to the coin-flip attack prompt raises it further to 65%. (Figure 2 (Flip and Flip+Q bars), Section 4.2)
Scope: Zephyr-7B-beta tuned with KTO for 1 epoch on 2000 feedback examples, 10% poisoned and 90% UltraFeedback; two fictional entities (Wag, Drizzle); binary multiple-choice log-probability evaluation.
With privileged access to write arbitrary (prompt, response, upvote) triples, KTO preference tuning makes Zephyr-7B-beta prefer invented facts about fictional entities on 97% of evaluation questions, up from a 0.05% base-model rate. (Section 4.1, Figure 2 (privileged access))
Scope: Upper-bound setting where the attacker controls the response text, not just the prompt and the vote; 2000 training examples with 10% poisoned; two fictional entities.
Feedback poisoning of Zephyr-7B-beta leaves general capability essentially unchanged, with TinyMMLU at 62.9% after the privileged-access attack versus 63.2% before. TinyMMLU stays near 60% across the unprivileged attack variants and the full poisoning sweep. (Section 4.1, Section 4.2, Figure 3 (right panel))
Scope: TinyMMLU only, on a single 7B model; a slight measurable degradation appears when adding more ordinary feedback but not when adding more poisoned examples.
Between 250 and 1,000 poisoned feedback examples are enough to reach 75-87% poisoned-response preference on fake financial headlines. Success rises from 26.75% to 85% as poisoned examples go from 0 to 1,000 alongside 2,000 clean preferences. (Figure 3 (left heatmap), Section 4.3)
Scope: Zephyr-7B-beta with KTO; 4 fake-headline targets (Apple, S&P 500, Federal Reserve, US unemployment); poisoned counts swept over {0, 10, 100, 250, 600, 1000}, with 60 runs per target per configuration.
Adding benign feedback does not neutralize the attack: with 10,000 ordinary preference examples, poisoning success still climbs from 32% to 73% as poisoned examples increase from 0 to 1,000. (Figure 3 (left heatmap), Section 4.3)
Scope: Zephyr-7B-beta, KTO, fake financial news targets; ordinary feedback swept over {0, 2000, 5000, 10000} UltraFeedback examples, the dataset the model was already trained on.
Feedback poisoning raises Zephyr-7B-beta's preference for Python requests calls with SSL verification disabled (verify=False) against api.binance.com to 53%, against a 0-3% baseline. (Section 4.4, Figure 2 (right))
Scope: 5000 training samples with 40% poisoned and 60% ordinary feedback; Flip strategy only; a single vulnerability pattern (verify=False) and a single target API.
Preference feedback given only inside a narrow coin-flip prompt transfers to standalone prompts that never contain the malicious instruction, so poisoned behavior appears in ordinary user contexts. (Section 3.2, Section 5, Figure 2)
Scope: Demonstrated for Zephyr-7B-beta under KTO with binary multiple-choice evaluation prompts distinct from the attack prompt; not tested on other preference-tuning objectives or larger models.
LLM Hypnosis identifies the ordinary chatbot thumbs-up/thumbs-down interface as a knowledge-poisoning attack surface, requiring no privileged access to training data, reward models, or the text of model responses.
Scope: Positions the work relative to prior poisoning research that assumed control over pretraining data, instruction-tuning data, reward models, or the preference pair responses; as of the 2025 arXiv posting.
LLM Hypnosis contradicts the common characterization of preference tuning as a shallow, style-and-tone intervention by showing that upvote/downvote feedback can insert factual claims absent from pretraining and rewrite claims about real entities.
Scope: Evidence is from KTO on one 7B model with fictional entities, 4 fake headlines, and 1 code vulnerability; a demonstration of what preference tuning can do, with no measurement of how often it happens in deployed systems.
LLM Hypnosis argues that unfiltered end-user feedback should be treated as an untrusted training input and audited before preference tuning, and reports that the authors contacted major LLM providers to support mitigation. (Section 6, Section 7)
Scope: A recommendation rather than an evaluated defense; no filtering or detection mechanism is proposed or tested, and proprietary filtering already deployed by commercial providers is unknown to the authors.

Common misreadings

The attack does not require writing the training responses. In the unprivileged setting the poisoned response text is always sampled from the model itself; the attacker controls only the prompt and the vote.
A 51-65% poisoned-answer rate is not a model that always lies. Poisoned accuracy is measured as preference between two candidate completions in a binary multiple-choice format, not as free-form generation frequency.
Stable TinyMMLU scores are not evidence that the poisoning failed. General-capability performance stays near 60% precisely because the injection is targeted, which is what makes the attack hard to catch with standard benchmarks.
The results are not established for preference-tuning methods in general. All experiments use KTO on a single Zephyr-7B-beta model, and other objectives may scale differently.
Deployed commercial chatbots are not demonstrated to be exploitable. The paper studies a simulated feedback pipeline and notes that undisclosed proprietary filtering may already mitigate the vulnerability.

Terminology in this paper

Flip attack: An attack prompt that instructs a language model to choose at random between a benign and a poisoned response, so that the poisoned response is sampled often enough to be upvoted but not so deterministically that preference-tuning gradients vanish.
Flip + Q: The Flip attack prompt with the attacker's real target question appended, so that upvoted feedback given in the coin-flip context transfers to the target question asked on its own.
Poisoned accuracy: The proportion of binary multiple-choice evaluation questions on which a model assigns higher probability to the attacker's intended response than to the correct response the model would originally have given.
Poisoning ratio: The proportion of attacker-constructed feedback data points relative to ordinary user feedback in the preference-tuning dataset.
Privileged access setting: An upper-bound threat model in which the attacker can insert arbitrary prompt-response-feedback triples into the preference-tuning dataset, rather than being limited to prompting the model and voting on whatever it samples.

How to cite

@article{hilel2025llm,
  title={LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users},
  author={Hilel, Almog and Shenfeld, Idan and Andreas, Jacob and Choshen, Leshem},
  journal={arXiv preprint arXiv:2507.02850},
  year={2025}
}

References

See the full reference list in the paper.