Safety & Society / ACL 2023

Can a Language Model Forget a Passage It Once Knew?

A targeted update makes selected training passages harder to reproduce. The result depends on the attack and on how much useful behavior the update preserves.

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo

Overview

A language model may continue a passage word for word when given enough of its opening. That ability is useful for understanding memorization and troublesome when the passage contains something someone wants removed. Moontae and his colleagues test a direct intervention. They take a pretrained model, make specific passages less likely under it, and measure whether it still performs ordinary language tasks.

Key Findings

Start where the passage gives itself away

A model that has memorized a passage may complete it word for word when given enough of the beginning. The team made that behavior measurable before trying to change it. Extraction likelihood averages the overlap between generated continuations and the true suffix across prefix lengths. Memorization accuracy instead asks whether the model’s top next-token prediction matches the original at each position. One measure watches the model finish a passage. The other tests how many of its small predictions line up.

Figure 1. The edit takes place after pretraining, when a removal request can identify a passage that a data filter never saw.
Figure 1. The edit takes place after pretraining, when a removal request can identify a passage that a data filter never saw.

Give the familiar tokens a worse score

Ordinary training lowers the negative log likelihood of examples. The researchers update the weights to raise that same loss on the selected passage. In Equation 1, x<tx_{<t} is the prefix before the next token, and pθp_\theta is the model’s predicted probability.

LUL(fθ,x)=−∑t=1Tlog⁡pθ(xt∣x<t).\mathcal L_{\mathrm{UL}}(f_\theta,\boldsymbol x) = -\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t}).

The algorithm climbs the loss instead of descending it. The authors use unseen test passages to establish reference thresholds for the two memorization measures. They can then stop when the selected text behaves more like text the model never trained on. That reference makes forgetting testable under this protocol. Other prompts and approximate reconstructions could still reveal information.

A larger model keeps more of its other abilities

The study edits GPT-Neo models of several sizes and checks nine classification and four dialogue benchmarks. The table shows selected results from Table 2 for the 2.7-billion-parameter model, averaged across five selections of 32 passages. UL+ uses the stronger stopping criterion.

SettingExtraction likelihood, 10-token overlap ↓Memorization accuracy ↓Classification average ↑Dialogue F1 average ↑
Before editing70.4%93.4%52.311.5
UL13.0%66.0%52.312.5
UL+1.6%31.0%51.911.1

The 125-million-parameter model paid a higher price. Its dialogue F1 fell from 9.4 to 2.6 with UL+. The comparison ties the practicality of the procedure to both the amount removed and the model that must absorb the change.

The order of removal is part of the result

In Figure 2, editing 128 passages in four rounds of 32 preserves classification performance better than editing the same 128 at once. The authors also run the same number of updates on groups from eight domains. GPT-Neo 1.3B retains an average classification score of 49.4 after the Enron email edit and 45.3 after editing U.S. patent background sections, from 49.8 initially. The same update budget therefore produces different side effects depending on what it is asked to erase. Table 3

Questions & Answers

Q: What information does the method need before it can edit the model?

A: It needs the target token sequence and access to the model’s parameters for gradient updates. The study starts from selected training passages. Finding everything a person might consider sensitive is a separate problem.

Q: If exact continuations become rare, can a prompt still recover the same information?

A: The tested measurements concern recovery of target tokens or spans. An attacker can ask about facts in other ways, seek related passages, or adapt prompts to the updated model. Those probes can give different results.

Q: Why compare with text the model never trained on?

A: Unseen text gives an empirical baseline for how often the model predicts or completes ordinary passages from patterns it learned elsewhere. The baseline depends on which unseen passages are chosen and how they resemble the target.

Q: What did the unlearning update cost outside the target passages?

A: Appendix B reports that Pile perplexity rises from 10.4 to 17.9 for the 2.7B model after UL+. The smaller model also showed a dialogue decline in the reported evaluation. Comparing before and after within a checkpoint helps isolate the edit. A comparison across models with different training corpora can mix the effect of the edit with what their original models knew.

Future Ideas

A repeatable deletion test using planted records

Train a small internal language model on a synthetic support corpus with known, unique identifiers planted in a subset of fictional tickets. For each removal request, compare gradient ascent, retraining without the ticket, and a no-edit checkpoint. Test exact continuations, paraphrased queries, and adaptive searches with the same query budget for each model. Keep related tickets that should survive the edit in a separate retention set. Repeat the tests after subsequent edits and model updates. Use the trial to establish measurable acceptance thresholds for that pipeline before any production use.

Add a retained-record probe at several attack budgets. Track whether passages that share prefixes, names, or neighboring context actually become easier to elicit after the update.

Sources

Original abstract

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for LMs has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger-sized LMs. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with previous methods known to mitigate privacy risks for LMs, we show that our approach can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust.