Safety & Society / NeurIPS Workshop 2024

Forget a Passage Without Breaking the Rest of the Model

An unlearning study targets unwanted tokens with a bounded loss and explores where to place a small set of trainable parameters.

Towards Robust and Cost-Efficient Knowledge Unlearning for Large Language Models

Sungmin Cha, Sungjun Cho, Dasol Hwang, Moontae Lee

Overview

Trying to make a model forget a memorized passage can make unrelated text worse. Simply pushing the probability of an unwanted token down may disturb the distribution the model uses everywhere else. This study tests losses and adapter initializations that aim to suppress a target more selectively.

Key Findings

During training, a model can learn to finish a passage with the very text a user later asks it to forget. An obvious edit is to reduce the probability of that continuation. This paper follows two consequences of that edit. What happens to the alternatives at that position, and how much of the model has to change?

Give the model a particular alternative

Ordinary gradient ascent pushes down the probability of the target token by pushing up the scores of every other token. Its objective can also keep improving as the target probability goes toward zero. The paper's inverted hinge loss sets a bounded goal and gives special weight to the strongest alternative. Let ptp_t be the probability of the target token in the given context and p⋆p_\star the highest probability of any other token. The per-token term is

LIHL=max⁡(0,1+pt−p⋆).\mathcal L_{\mathrm{IHL}}=\max(0,1+p_t-p_\star).

Minimizing this term moves probability away from the target and toward a concrete alternative. The authors train with an auxiliary retain set as well, so the update still receives a signal to keep predicting ordinary text. Loss and update, Sections 3.3 and 3.5

The paper frames unlearning as reducing access to selected examples while retaining useful behavior on other inputs.
The paper frames unlearning as reducing access to selected examples while retaining useful behavior on other inputs.

Start the adapters where the forget set has more influence

LoRA edits a large weight matrix with a product of two small matrices while the original weights stay frozen. That lowers the number of trainable parameters. But in the preliminary tests, simply using standard LoRA could take longer or damage general language performance by the time it suppressed the targeted text. FLoRA uses the squared gradients of the language-model loss to estimate parameter importance, comparing the forget set with the retain set. It uses those weights to initialize the low-rank components. The paper aggregates importance by row and obtains the initialization through a singular value decomposition of the rescaled matrix, avoiding a separate iterative fitting procedure. Adapter initialization, Section 3.4

For GPT-Neo 125M, the paper compares compute spent reaching the forgetting criterion with classification, dialogue and perplexity scores after the edit.
For GPT-Neo 125M, the paper compares compute spent reaching the forgetting criterion with classification, dialogue and perplexity scores after the edit.

Check both extraction and the abilities that survive

The team evaluated three GPT-Neo sizes on memorized text from the Training Data Extraction Challenge. The stopping rule compares two measures against a held-out set: overlap with the target text in ten-token sequences and accuracy when predicting its next tokens. General reasoning, dialogue and language-modeling tests separately track what survives the update.

GPT-Neo 2.7B, Table 1Ten-token extraction likelihood (%)Memorization accuracy (%)Reasoning accuracy (%)Pile perplexityEpochs
Before editing70.493.452.310.4—
LoRA0.120.445.961.114.0
LoRA + inverted hinge loss0.026.749.622.217.8
FLoRA + inverted hinge loss0.128.549.616.010.3

With the hinge loss held fixed, FLoRA reaches the criterion in fewer epochs and leaves perplexity closer to the original model's value. Plain LoRA leaves a lower memorization accuracy on this test, but the accompanying perplexity increase shows the cost of optimizing for forgetting alone. The apparent 0.0% is a rounded measure under the paper's extraction protocol. Probing under different prompts or sampling rules would be a separate experiment. GPT-Neo results and evaluation protocol, Table 1 and Section 4

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why use a low-rank adapter?

A: It limits how many parameters are trainable. The initialization determines where the resulting update begins.

Q: Does low extraction prove the data are erased?

A: It measures what the evaluation can recover. A different attack may still elicit the content, and other forgetting metrics can remain nonzero.

Q: What counts toward the reported efficiency of adapter unlearning?

A: The final comparison combines the inverted hinge loss and FLoRA. How much time the method saves depends in part on when the experiment stops. An end-to-end comparison also needs to count the computation required to estimate Fisher information for initialization. That separates a faster optimization phase from a lower total cost.

Future Ideas

Audit a synthetic deletion request before retraining

Plant fictional records and related decoys in a small model’s training data. Compare a full retrain, ordinary LoRA editing, and the proposed adapter variants. Score extraction from exact and paraphrased prompts at several attempt budgets, plus unrelated and neighboring facts. Compare the protocol with the paper’s cited TOFU benchmark and count Fisher estimation on the forget and retain sets in the total editing cost. Keep the planted records synthetic to avoid reproducing private data.

Count Fisher estimation in the total compute and compare with a reference trained without the planted targets. Probe related records as well as the exact target.

Sources