Safety & Society / NeurIPS 2016

The Order of Votes Can Change the Story They Tell

A model of Amazon reviews and 82 StackExchange forums separates the choice of which response to vote on from the decision to vote it up or down.

Beyond Exchangeability: The Chinese Voting Process

Moontae Lee, Seok Hyun Jin, David Mimno

Overview

Picture two answers with three helpful votes and three unhelpful votes apiece. One rose to the top before the negative votes arrived. The other drew mixed reactions from the beginning. Their totals agree, but the readers who cast those votes saw different displays and different opinions from earlier readers. The Chinese Voting Process models that history to estimate how position, past votes, and the response itself shape the next vote.

Key Findings

Two answers, six votes each

Take two answers that each end with three positive and three negative votes. In the paper's first example (Table 1), one answer collects three positives and then three negatives, while another alternates. Moontae argues that the endings differ in meaning. The fourth vote on the first answer goes against a clear majority, which they treat as a harder decision that deserves more weight. Alternating votes each express weaker disagreement, so that answer looks closer to neutral. Classic self-reinforcing models, including the Chinese Restaurant Process and the Pólya urn, give both orderings the same probability, because they are exchangeable. The authors built the Chinese Voting Process (CVP) to read the order.

A reader's two decisions

The CVP splits each visit into two phases. In the selection phase, a reader either picks an existing response or writes a new one. In the voting phase, a reader who picked a response votes it up or down. Table 2 gives the rank-based weight of response jj at time tt, with τ\tau measuring the community's sensitivity to high-ranked responses:

fi(t)(j)=(1+rank⁡i(t)(j))−τ.f_i^{(t)}(j) = \bigl(1+\operatorname{rank}_i^{(t)}(j)\bigr)^{-\tau}.

Normalizing the existing response weights together with the weight α\alpha for writing a new one gives the selection probabilities. The authors fixed α\alpha at 0.5 after a search. A negative τ\tau lets the model describe readers who skip the top, which the paper says happens early when few responses exist.

Why the urn ratios carry history

Each response starts with pseudo-votes. As votes arrive, the model updates two features, rr and ss, the positive and negative shares of votes recorded so far. With uu denoting response length relative to other responses to item ii, Table 2 gives the probability that the next vote on response jj is positive:

pijt=σ ⁣(qij+λrij(t−1)+μsij(t−1)+νiuij(t−1)).p_{ijt} = \sigma\!\left(q_{ij}+\lambda r_{ij}^{(t-1)} +\mu s_{ij}^{(t-1)}+\nu_i u_{ij}^{(t-1)}\right).

Here σ\sigma is the logistic function and qijq_{ij} is the response's estimated intrinsic quality. The weights λ\lambda and μ\mu describe the effect of earlier positive and negative shares. The per-item weight νi\nu_i captures length preference. A positive vote contributes probability pijtp_{ijt}, and a negative vote contributes 1−pijt1-p_{ijt}. In the paper's toy example the regression omits relative length, starts at the second vote, and uses an ℓ2\ell_2 penalty to keep estimates finite even for a one-sided history.

These are the estimates in Table 3 after each vote, with both responses starting at one positive and one negative pseudo-vote.

Vote receivedFirst responseIts quality estimateSecond responseIts quality estimate
1Positive—Positive—
2Positive0.363Negative−0.363
3Positive0.574Positive0.004
4Negative0.237Negative−0.230
5Negative0.004Positive0.007
6Negative−0.175Negative−0.166

The first response's estimate drops from 0.574 to 0.237 when a reader votes against the early consensus. The two responses finish with the same totals and different estimates. The authors report that using raw vote counts as features would keep the first estimate rising through its three negative votes.

Fitting a process that depends on its own past

A non-exchangeable process looks hard to fit. The authors note that the full trajectories are observed, including the rank each reader saw. Conditional on the observed history, the likelihood splits into a voting piece and a selection piece. The voting piece becomes regularized logistic regression, and with α\alpha fixed the selection piece is convex in τ\tau.

For community comparison they define Trendiness as τ\tau. Conformity κ\kappa is the geometric mean of odds ratios for following versus opposing the current majority. It uses parameters refit at each time step, since a reader cannot see future votes.

Data and the first test

The data include 363 Amazon products, kept from 595 after filtering, and 82 StackExchange forums with at least 100 qualifying questions. The prediction task fits the model up to time tt and scores action t+1t+1.

Table 5 reports average negative log-likelihood over the first 50 votes, where lower is better. The baseline is a Chinese Restaurant Process that weights responses by vote count.

CommunityCRP selectionCVP selectionResidual, display rankResidual, quality
StackOverflow2.1521.989.005.003
Mathematics1.8411.876.014.008
Amazon2.5972.261.023.016
Meta StackOverflow1.4111.575.018.013

The CVP wins in most forums. Mathematics and Meta StackOverflow are two exceptions where the baseline performs better. The advantage shrinks as votes accumulate, and for items past 50 timesteps the authors report no significant difference. In the voting phase, positive votes dominate every community, which makes gains hard. Even so, the authors report that every reduced model differs significantly from the full model by one-way ANOVA in the major communities. Quality, opinion ratios, and length each contributed.

Checking quality against comments

Intrinsic quality has no direct ground truth, so the authors used comments as a second signal. Figure 2 covers StackOverflow, with 337k comments on 104k responses.

Figure 2 shows comments and sentiment against display rank and against estimated quality for StackOverflow, with prediction curves for selection and voting at right.
Figure 2 shows comments and sentiment against display rank and against estimated quality for StackOverflow, with prediction curves for selection and voting at right.

Higher-ranked responses draw more comments, and responses at both high and low quality draw more comments. Sentiment follows quality more cleanly. Across all 83 communities, the paper reports weighted average residuals of 0.012 for quality rank and 0.028 for display rank, and bumpiness falling from 0.391 to 0.226. The residual columns in Table 5 show the same direction in each listed community. Comments can still be affected by who sees a response, so this supports the quality scores without proving them.

Mapping community cultures

Trendiness and Conformity place each community on a plane in Figure 1.

Figure 1 places 83 communities by Trendiness on the horizontal axis and Conformity on the vertical axis, with clusters by topic.
Figure 1 places 83 communities by Trendiness on the horizontal axis and Conformity on the vertical axis, with clusters by topic.

Religion, scholarship, and meta forums sit at low Trendiness, where readers spread votes across many opinions. Theoretical computer science has low Trendiness and high Conformity. Applied communities such as physics and math are trendier, and Amazon readers are the most trend-sensitive and the least afraid of minority opinions.

The authors also asked whether Trendiness merely reflects size. Splitting StackOverflow by language (Figure 3) yields distinguishable averages that all stay in one region. JavaScript users are more satisfied with trendy answers than C and C++ users, mobile developers lean conformist, and Perl users argue more. The authors conclude that community character shapes voting more than technical similarity. They also treat τ\tau as community-level, noting it could be extended to per-item values.

Figure 3 applies the same two coordinates within StackOverflow. The mobile language tags cluster high on conformity, and Perl sits near the lower-left corner.
Figure 3 applies the same two coordinates within StackOverflow. The mobile language tags cluster high on conformity, and Perl sits near the lower-left corner.

Questions & Answers

Q: Why can’t we use the final helpful-vote percentage?

A: The percentage discards the history of what each voter saw. When someone votes against an early consensus, the model treats that vote in light of the previous opinions. The same final ratio can emerge from very different histories.

Q: What if I have the votes but no record of which answer was shown first?

A: The selection model depends on the display rank at each decision. The paper uses observed trajectories and discusses limitations where ordering rules or tie breaks are uncertain. A deployment would benefit from recording impressions and rank directly.

Q: Is the quality score a verdict about an answer?

A: It is a model-based estimate of how the response would fare after accounting for the included variables. A correctness audit would require an independent way to assess the answer, such as blinded expert annotation or a reproducible test.

Q: Do the votes establish that display position causes the apparent quality gap?

A: The historical data let the model estimate how rank and earlier votes relate to later behavior. Rank, response age, and who reads a thread can also change together. A randomized ordering with recorded impressions would help isolate the effect of position, while blinded usefulness judgments provide a measure outside the same voting process.

Future Ideas

Measure the answers a forum overlooks

For an opt-in sample of low-risk question threads, rotate eligible answers among a small set of positions and record their impressions. Ask independent reviewers to rate a stratified sample of answers while hiding their vote totals. Use the observed votes to compare a score based on vote history with the current ordering and an impression-adjusted vote baseline.

The key outcome is whether the score surfaces more independently verified useful answers per review, especially among recent or previously low-exposure posts. Keep the trial small enough for manual quality checks and estimate performance separately on new and established answers before using the score to reorder the site.

Predeclare the observation window for new answers. Compare their estimates at equal age and separately by impression count, so an apparent early gain does not simply reflect more time on screen.

Sources

Original abstract

Many online communities present user-contributed responses such as reviews of products and answers to questions. User-provided helpfulness votes can highlight the most useful responses, but voting is a social process that can gain momentum based on the popularity of responses and the polarity of existing votes. We propose the Chinese Voting Process (CVP) which models the evolution of helpfulness votes as a self-reinforcing process dependent on position and presentation biases. We evaluate this model on Amazon product reviews and more than 80 StackExchange forums, measuring the intrinsic quality of individual responses and behavioral coefficients of different communities.