Unforgettable Generalization in Language Models

Eric Zhang, Leshem Choshen, Jacob Andreas · COLM 2024 · 2024

In one sentence

Fine-tuning Llama-2-7B on randomized labels scrambles predictions on the forgetting set, but whether forgetting generalizes to new instances of the same task is wildly variable, is set by the evaluation task rather than the forgotten one, and is shallow enough that linear probes still recover the skill.

Abstract

When language models (LMs) are trained to forget (or"unlearn'') a skill, how precisely does their behavior change? We study the behavior of transformer LMs in which tasks have been forgotten via fine-tuning on randomized labels. Such LMs learn to generate near-random predictions for individual examples in the"training''set used for forgetting. Across tasks, however, LMs exhibit extreme variability in whether LM predictions change on examples outside the training set. In some tasks (like entailment classification), forgetting generalizes robustly, and causes models to produce uninformative predictions on new task instances; in other tasks (like physical commonsense reasoning and scientific question answering) forgetting affects only the training examples, and models continue to perform the"forgotten''task accurately even for examples very similar to those that appeared in the training set. Dataset difficulty is not predictive of whether a behavior can be forgotten; instead, generalization in forgetting is (weakly) predicted by the confidence of LMs'initial task predictions and the variability of LM representations of training data, with low confidence and low variability both associated with greater generalization. Perhaps most surprisingly, random-label forgetting appears to be somewhat insensitive to the contents of the training set: for example, models trained on science questions with random labels continue to answer other science questions accurately, but begin to produce random labels on entailment classification tasks. Finally, we show that even generalizable forgetting is shallow: linear probes trained on LMs'representations can still perform tasks reliably after forgetting. Our results highlight the difficulty and unpredictability of performing targeted skill removal from models via fine-tuning.

Questions this paper answers

If a model is trained to give wrong answers on a few examples of a skill, does it lose that skill on examples it never saw?
does random-label fine-tuning on a forget set produce unlearning that generalizes to held-out instances of the same capability?
how do I check whether a capability I tried to remove from a language model is actually gone on new inputs?
if I fine-tune my model on random labels to strip a benchmark skill, can I trust that the skill is really gone?
Random-label forgetting on Llama-2-7B often fails to generalize beyond the examples it was trained on. Many of the 21 binarized multiple-choice tasks retain a large forget gap, staying well above the 50% chance accuracy that would indicate full forgetting.
Holds for: Llama-2-7B base, 21 binary multiple-choice tasks, 1000 training examples per task, full fine-tuning then a second pass on uniformly random labels at learning rate 1e-4.
Commonsense reasoning and science QA tasks resist random-label forgetting on Llama-2-7B, while linguistic acceptability and entailment tasks from GLUE are forgotten far more effectively. With 1000 forgetting examples, PIQA still scores 0.69 and SciQ 0.76, whereas RTE, QNLI, CB and WiC fall to 0.50.
Holds for: Llama-2-7B, 0-shot lm-evaluation-harness prompts, binarized tasks with one true response and one random distractor; a consistent tendency across task families rather than a clean separation of every dataset, with WNLI at 0.97 the strongest exception among GLUE tasks.
Are some abilities much harder to knock out of a language model than others?
which task families resist random-label unlearning, and which collapse to chance accuracy?
how do I tell in advance whether the capability I want to remove is one of the removable ones?
I want to strip commonsense reasoning rather than sentence-acceptability judgments from my model — should I expect that to work?
Commonsense reasoning and science QA tasks resist random-label forgetting on Llama-2-7B, while linguistic acceptability and entailment tasks from GLUE are forgotten far more effectively. With 1000 forgetting examples, PIQA still scores 0.69 and SciQ 0.76, whereas RTE, QNLI, CB and WiC fall to 0.50.
Holds for: Llama-2-7B, 0-shot lm-evaluation-harness prompts, binarized tasks with one true response and one random distractor; a consistent tendency across task families rather than a clean separation of every dataset, with WNLI at 0.97 the strongest exception among GLUE tasks.
In cross-task forgetting on Llama-2-7B, the degree of forgetting is largely determined by the task the model is evaluated on, not the task whose labels were randomized. A model trained on randomized science questions keeps answering science questions accurately while producing random labels on entailment classification.
Holds for: Llama-2-7B, all pairs among the 21 binarized tasks, 1000 randomized-label examples for training; rows and columns clustered with UPGMA. Reproduced with GPT-J-6B and GPT-2 in Figure 9 and with flipped rather than random labels in Figure 8.
When a model is trained to answer one kind of question badly, which abilities actually degrade?
in cross-task random-label forgetting, is the forget ratio driven by the training task or the evaluation task?
which dataset should I randomize labels on if I want a particular skill to disappear?
do I have to collect forgetting data from the exact skill I want removed, or will another dataset work better?
In cross-task forgetting on Llama-2-7B, the degree of forgetting is largely determined by the task the model is evaluated on, not the task whose labels were randomized. A model trained on randomized science questions keeps answering science questions accurately while producing random labels on entailment classification.
Holds for: Llama-2-7B, all pairs among the 21 binarized tasks, 1000 randomized-label examples for training; rows and columns clustered with UPGMA. Reproduced with GPT-J-6B and GPT-2 in Figure 9 and with flipped rather than random labels in Figure 8.
Many tasks are forgotten more effectively by fine-tuning Llama-2-7B on randomized labels from a different task than on their own randomized labels. Training to forget commonsense reasoning tasks is generally the most effective trigger for forgetting elsewhere.
Holds for: Llama-2-7B cross-task forget ratios over all pairs of the 21 binarized multiple-choice tasks, with 1000 randomized-label training examples each.
Is a skill a model finds difficult also easier to make it lose?
does fine-tuned accuracy or task difficulty correlate with forget ratio under random-label training?
can I use benchmark difficulty as a shortcut for guessing which skills will unlearn cleanly?
my target capability is one the model barely handles — does that make removal more likely to succeed?
Task difficulty does not predict whether a skill can be forgotten from Llama-2-7B. Forgetting is less effective on ARC Easy, which retains 0.86 accuracy after forgetting, than on the substantially harder ARC Challenge, which falls to 0.50.
Holds for: Task-level comparisons over the 21 binarized tasks with Llama-2-7B, using accuracy after fine-tuning as the difficulty measure, at 1000 forgetting examples; at 100 examples ARC Challenge instead retains 0.66.
Is there any signal you can measure ahead of time that tells you whether removal of a skill will stick?
do model confidence margins or hidden-state variance predict generalizable forgetting at the task and example level?
how do I predict, without labels, which skills will generalizably unlearn and which will not?
can I screen candidate skills for unlearnability using confidence scores or activation statistics before running the fine-tune?
Two label-free properties weakly predict which Llama-2-7B skills generalizably forget: low model confidence in the correct response relative to the distractor, and low total variance of the hidden state across examples. The variance is the trace of the covariance matrix at the question's last token in the fifth-to-last layer.
Holds for: Task-level predictors across the 21 binarized tasks with Llama-2-7B; both relationships are partial and noisy, and hidden-state variance needs only inference access and unlabelled task data.
Neither model confidence nor hidden-state variance predicts forgetting at the level of individual examples in Llama-2-7B, even though both are partially predictive across the 21 whole tasks studied.
Holds for: Within-task, example-level correlations for Llama-2-7B on the 21 binarized tasks; reported as an absence of correlation without a coefficient.
Once a model stops giving right answers, is the knowledge actually erased or just no longer spoken?
do linear probes on hidden states still separate correct from incorrect answers after random-label forgetting?
how do I test whether unlearning removed information from a model's representations rather than only its outputs?
if an audit probes my model's internal activations, will a random-label unlearning run hold up?
Even generalizable forgetting is shallow in Llama-2-7B. A linear probe on fifth-to-last-layer hidden states classifies (question, answer) pairs as correct or incorrect about as accurately after random-label forgetting as after fine-tuning, so the skill stays recoverable from representations.
Holds for: L2-regularized linear probes with early stopping, trained on the task training set and evaluated on its test set; probing accuracy is comparable across layers except the very earliest and latest.
Do the things a model picks up earliest in training turn out to be the hardest to remove?
is there an anticorrelation between per-example learning order during fine-tuning and forgetting order under random labels?
can I use the order examples were learned in to guess which ones will survive an unlearning run?
Learning order and forgetting order in Llama-2-7B are weakly but consistently anticorrelated: the examples learned first during fine-tuning are typically the last to be forgotten under random-label training. The effect holds across the tasks plotted but is modest in size.
Holds for: Examples thresholded at 0.6 confidence for both learning and forgetting, with the re-learning pass at learning rate 3e-5; tasks with fewer than 100 qualifying examples are not plotted.
Would using more wrong-labelled examples make skill removal work better?
does forget-set size explain the failure of random-label unlearning to generalize across tasks?
how many random-label examples do I need before unlearning starts generalizing?
should I scale up my forgetting set from 100 to 1000 examples to get better removal?
The failure of forgetting to generalize is not explained by the size of the forgetting set. Cutting the random-label training set from 1000 to 100 examples leaves the broad pattern across tasks intact, with WNLI at 0.97 forget accuracy in both settings and PIQA at 0.71 versus 0.69.
Holds for: Llama-2-7B, tasks compared at 100 versus 1000 forgetting examples; some datasets do show less forgetting in the small setting, including BoolQ at 0.77 versus 0.50 and QNLI at 0.71 versus 0.50.
Do results about skills that survive being trained away hold for more than one language model and one training trick?
does the resistance pattern hold under flipped-label training and in GPT-J-6B and GPT-2 as well as Llama-2-7B?
how do I know whether unlearning-resistance results transfer to the model size and label-corruption scheme I use?
I am using flipped labels on a smaller model than Llama-2-7B — should I expect the same skills to resist removal?
The same tasks that resist random-label forgetting also resist forgetting via flipped labels, and the cross-task forget-ratio pattern reappears in GPT-J-6B and in the 124M-parameter GPT-2 despite their lower fine-tuned accuracy.
Holds for: Flipped-label forgetting uses a forget ratio normalized by the drop from fine-tuned accuracy to 1 minus fine-tuned accuracy; GPT-J-6B and GPT-2 results are cross-task randomized-label forget ratios only.
What should I read first about whether removing an ability from a language model really works?
which work reframes LM unlearning evaluation around generalization from the forget set to unseen instances of the same skill?
where do I start reading on evaluating skill removal in language models beyond checking the forget set itself?
is there a paper I can cite for the claim that fine-tuning-based skill removal may not generalize?
Unforgettable Generalization in Language Models reframes unlearning evaluation around a question prior LM unlearning work largely left open. The question is not whether forgetting spills onto unrelated tasks, but whether it generalizes from the forgetting set to other instances of the same skill.
Holds for: As of the paper's 2024 publication; the study covers skill forgetting via fine-tuning on randomized or flipped labels, not factual-knowledge deletion, gradient ascent, representation editing or other unlearning families surveyed in Section 2.
Even generalizable forgetting is shallow in Llama-2-7B. A linear probe on fifth-to-last-layer hidden states classifies (question, answer) pairs as correct or incorrect about as accurately after random-label forgetting as after fine-tuning, so the skill stays recoverable from representations.
Holds for: L2-regularized linear probes with early stopping, trained on the task training set and evaluated on its test set; probing accuracy is comparable across layers except the very earliest and latest.
How should success at making a model lose an ability be reported on multiple-choice benchmarks?
what does the forget gap measure that the forget ratio does not in skill-unlearning experiments?
how do I score an unlearning run on binarized multiple-choice tasks where chance is 50%?
which number should I put in my unlearning table so reviewers can compare across tasks?
Unforgettable Generalization in Language Models supplies two reusable metrics for skill-unlearning experiments: the forget gap and the forget ratio. The forget gap is accuracy after forgetting minus the 50% chance level, and the forget ratio is the achieved accuracy drop as a fraction of the drop to chance.
Holds for: Defined for binarized multiple-choice tasks where chance is 50%; the ratio requires an accuracy-after-fine-tuning reference point and is redefined for flipped-label forgetting.

Claims and scope

Common misreadings

Terminology in this paper

Forget gap
The accuracy a language model still achieves on a task after being trained to forget it, minus the 50% chance accuracy of a binarized multiple-choice task; a gap of 0 means the task is fully forgotten.
Forget ratio
The drop in task accuracy caused by a forgetting procedure, divided by the drop that would take the model from its fine-tuned accuracy down to 50% chance; 1 is complete forgetting and 0 is no forgetting at all.
Random-label forgetting
Unlearning a skill by fine-tuning a model a second time on the same task's training inputs paired with labels drawn uniformly at random from the answer options.
Cross-task forgetting
Fine-tuning a model on randomized labels from one task and then measuring how much accuracy it loses on a different task's test set.

How to cite

@inproceedings{zhang2024unforgettable,
  title={Unforgettable Generalization in Language Models},
  author={Zhang, Eric and Choshen, Leshem and Andreas, Jacob},
  booktitle={First Conference on Language Modeling},
    year={2024}
}

References

See the full reference list in the paper.