Safety & Society / INFORMS Workshop 2017 (DMDA)

The Next Helpful Vote Has a History

A workshop entry from the Chinese Voting Process project puts a question to readers of product reviews and online answers. How much does today’s vote depend on the votes that came before it?

Reading note · The CV’s workshop file has been deleted. Method and results from the separate NeurIPS paper are identified in the article.

The Chinese Voting Process: The Evolution of Online Helpfulness Evaluations

Moontae Lee, Seok Hyun Jin, David Mimno

Overview

A product review with a long voting history offers two kinds of evidence. It contains advice about the product, and it shows how other shoppers reacted. The workshop title identifies helpfulness evaluations as the subject. The same authors’ NeurIPS paper supplies the technical background for the question.

Key Findings

A tally hides its sequence

Imagine two product reviews that finish with the same helpfulness score. One drew strong approval first and criticism much later. The other collected alternating votes all along. Their final counts are identical, but earlier readers encountered two different pages. They may have seen different rankings or different signals about what other readers thought. That is the question behind this workshop entry on the evolution of helpfulness evaluations.

Why position and previous reactions are different signals

The authors’ 2016 NeurIPS paper gives a technical account of the same research question. It models two choices. A reader first selects a response to evaluate and then decides whether to endorse it. Display rank affects which responses readers encounter. An existing tally provides a different signal to those who are already reading the response.

In the NeurIPS model, the probability of a positive vote for response jj on item ii has the form

p(v=1)=σ ⁣(qij+gi(t−1)(j)),p(v=1)=\sigma\!\left(q_{ij}+g_i^{(t-1)}(j)\right),

where σ\sigma is the logistic function, qijq_{ij} is the response-specific quality term, and gi(t−1)(j)g_i^{(t-1)}(j) represents the effect of the response’s observable voting history before time tt. The quality term is an estimate under the model. It gives researchers a way to account for prior votes while studying which responses readers find helpful. NeurIPS model, Table 2

What a simple vote reversal does

The companion paper illustrates the effect with sequences containing three approvals and three disapprovals each. Three approvals followed by three disapprovals and three disapprovals followed by three approvals both finish at 50% helpful. Their fourth votes push against different prior majorities. The paper uses this example to explain why a model can learn different quality estimates from identical final counts. These constructed sequences illustrate the modeling choice. NeurIPS example, Table 1

What survives in the workshop record

The CV and Seok Hyun Jin’s publication list identify the INFORMS DMDA workshop contribution. The longer title in Jin’s publication list specifies product reviews and answers to questions. The CV’s two workshop links point to one another’s records. The Dropbox target named for this workshop says the file was deleted, so its datasets, equations, figures, and numerical results remain unverified. Moontae Lee’s CV · Seok Hyun Jin’s publications

Questions & Answers

Q: What could make two reviews with the same final tally different?

A: Their vote sequences can differ, and early feedback can affect what later readers see. The companion model uses this history as information.

Q: What evidence would identify the effect of visible counts?

A: Randomly showing or hiding the tally while holding the item, position, and voting opportunity constant would test whether visibility changes behavior.

Q: Which explanation comes from the companion paper?

A: The NeurIPS manuscript distinguishes position bias, which changes what readers encounter, and presentation bias, which changes how visible evaluations influence them. Its analysis of trendiness and conformity belongs to that companion paper. The workshop entry establishes the contribution’s identity and topic. The unavailable workshop manuscript prevents a more detailed account of its own experiments.

Future Ideas

Replicate a vote-visibility experiment in a product-review sandbox. Keep the order of reviews fixed and randomly hide visible vote counts. Track vote direction and voting frequency per impression. Have blinded readers rate the reviews separately so the study can distinguish influence from agreement about quality.

Sources