Structure & Discovery / arXiv 2017

Recover the Topics Inside One Document

An early PADD manuscript uses topic-pair statistics from a collection to estimate the mixture of topics in each document.

Prior-aware Dual Decomposition: Document-specific Topic Inference for Spectral Topic Models

Moontae Lee, David Bindel, David Mimno

Overview

A word-pair model can find themes across a collection while leaving a question unanswered about an individual document. How much of this review is about its main subject, and how much is about a second topic? Prior-aware Dual Decomposition, or PADD, reconstructs document mixtures while asking them to agree with the topic relationships already learned from the collection.

Key Findings

The topic model has an index but no entry for this document

A spectral topic model can learn that words tend to occur together and estimate how topics co-occur across a collection. The next reader still wants to know what a particular article is about. Its few observed words can fit many different mixtures, especially when the article is short. PADD uses collection-level topic relationships to help resolve that ambiguity.

Figure 1. A conventional topic model gives each document a composition of topics, which in turn explain its words.
Figure 1. A conventional topic model gives each document a composition of topics, which in turn explain its words.

Fit the words and keep the collection’s relationships

Let BB give the word distribution of each topic, h~m\widetilde h_m be observed word frequencies in document mm, and wmw_m be its topic proportions. Equation 1 chooses nonnegative proportions summing to one and seeks to match the topic-pair matrix AA across the collection:

min⁡w1,…,wM∑m=1M∥Bwm−h~m∥22subject to1M∑m=1Mwmwm⊤=A.\min_{w_1,\ldots,w_M}\sum_{m=1}^{M}\|Bw_m-\widetilde h_m\|_2^2 \quad\text{subject to}\quad \frac{1}{M}\sum_{m=1}^{M} w_m w_m^\top=A.

The objective asks whether the inferred topics explain the document’s own words. The outer-product constraint asks whether their co-occurrence across documents matches what the corpus model learned. A shared adjustment matrix separates the resulting optimization into document-level subproblems. The coordinator updates the adjustment after seeing their combined topic pairs, and the document problems can run in parallel. Equations 1–2 and algorithm

A generous shortlist works until topics correlate

The authors first test constructed mixtures where they know the answers. Simple probabilistic inference often finds nearly all candidate topics but includes many extras. That approach works well as the number of sparse, weakly correlated topics grows in one artificial setting. Its precision deteriorates in the more realistic simulation with substantial topic correlations. PADD uses those correlations and performs comparably to Gibbs sampling in the paper’s semi-real experiments. Figure 3 · Figure 4

Test new articles against a reference that can still disagree

CorpusThresholded Linear InverseSPIPADD
NIPS0.5360.3070.744
Blog0.5270.2890.715
New York Times0.4270.2810.668

These F1 scores measure prominent-topic identification in unseen documents and are averaged over tested topic counts. The real documents have no known true mixture. The comparison uses Gibbs sampling as a reference with the topic parameters supplied. SPI’s recall exceeds 0.96 in all three unseen collections, but precision is below 0.20. PADD makes a tighter shortlist. The gain reflects closer agreement with a probabilistic reference and with the corpus’s topic relationships. Both can be uncertain for a very short document or a small collection. Table 1

Questions & Answers

The opening answers are simulated author responses based on the manuscript.

Q: Why use more than the words in the document?

A: Several mixtures can explain a short text. A corpus-level relationship can supply another clue about which topics plausibly occur together.

Q: Are the reference mixtures observed?

A: The constructed settings have known mixtures. The fully real setting uses estimates from Gibbs as a comparison.

Q: Where might a shared topic prior hurt an otherwise correct assignment?

A: A short document provides limited evidence of its own, so a strong prior can give unusual or newly emerging topic combinations too little weight. Gibbs agreement measures agreement with a reference inference procedure. Known mixtures and independent human review test whether the resulting labels are actually useful when patterns change over time.

Future Ideas

Flag document labels dominated by the prior

Fit an archive’s topic labels with and without the shared prior. Show a curator the documents whose prominent topics change most, with supporting words for each label. Use an earlier collection to fit the prior and a later collection for the curator trial. Compare against random review at the same curator budget and measure which disagreements expose prior assumptions that no longer fit the collection.

Include documents with controlled length and known topic mixtures alongside real examples. Hold the learned vocabulary fixed while varying topic correlations in the test data, so disagreements caused by genuine prior drift can be distinguished from short-text uncertainty.

Sources

Original abstract

Spectral topic modeling algorithms operate on matrices/tensors of word co-occurrence statistics to learn topic-specific word distributions. This approach removes the dependence on the original documents and produces substantial gains in efficiency and provable topic inference, but at a cost: the model can no longer provide information about the topic composition of individual documents. Recently Thresholded Linear Inverse (TLI) is proposed to map the observed words of each document back to its topic composition. However, its linear characteristics limit the inference quality without considering the important prior information over topics. In this paper, we evaluate Simple Probabilistic Inverse (SPI) method and novel Prior-aware Dual Decomposition (PADD) that is capable of learning document-specific topic compositions in parallel. Experiments show that PADD successfully leverages topic correlations as a prior, notably outperforming TLI and learning quality topic compositions comparable to Gibbs sampling on various data.