Overview
Mary picks up a football in the kitchen, walks to the garden, and drops it. To say where the football is, a reader must connect possession and location, then notice when that connection changes. This paper builds those connections with vectors. It asks what happens when a model has a clean record of the events and can devote its work to reasoning over them.
Key Findings
Where is the football? Sentences become matrix products
Mary goes to the kitchen, picks up a football, walks to the garden, and drops it. Someone asks where the football is. Answering takes two linked facts, one about who holds the ball and one about where that person went. This scenario comes from bAbI, a set of twenty synthetic question categories from Facebook. Moontae argues that earlier systems on bAbI were all trained end to end, so a wrong answer could trace back to misreading the sentence, storing it badly, or reasoning poorly. Their study separates those three stages.
Binding things with outer products
The authors build on Tensor Product Representation, which Smolensky proposed in 1990. Every actor, object, and location becomes a unit vector, meaning a list of numbers with length one. An outer product binds a contained item to its container. The football held by Mary becomes the matrix , and Mary in the garden becomes . Multiplying the two matrices applies transitivity, because equals one.
The result says the football is in the garden. The paper gives this equation unnumbered, and the same simplification covers Categories 1, 2, 6, 8, 9, 10, 11, 12, and 13 with different answer formats. Statements go into ordered memory slots. To answer a question, the model multiplies each stored matrix by the transpose of the queried entity and keeps the most recent result whose length is near 1.
Remembering where things used to be
Questions such as "Where was the apple before the hallway?" in Category 3 need history. The authors define a binding , where is a matrix that packs a next and previous location into one vector. A pseudo-inverse of unpacks the pair later. The model rebuilds a trajectory such as nowhere, bedroom, hallway. Category 5 uses a parallel operation for ownership transfers. Conjunctions and "or" use a third binding, . Category 14 binds locations to four named times of day.
Deduction, induction, and honest labels
Categories 15, 16, 18, and 20 turn out to be the same transitive chaining. Putting the general concept on the left of the outer product lets one observation about a lion transfer to the whole category. Category 20 connects "bored" to a later trip to the garden. The authors note a caveat in Category 16. The answer for Brian could be green or yellow depending on which evidence counts, so their 99.5% test accuracy reflects ambiguous labels. In Category 5 they find the 8th training story and stories 63 and 186 of the test set contain answers that ignore the latest transfer.
Two categories that need more machinery
Path finding (Category 19) uses four invertible matrices with and . The model starts from a random vector for one place, propagates the rest through direction matrices, and queues statements it cannot yet resolve. It then picks the two-direction pair that best satisfies the query.
Positional reasoning (Category 17) cannot use invertible matrices, because "below" followed by "below" is still "below". The authors use four idempotent matrices, so , and embed them in block matrices. Equation form of the first two blocks follows.
The paper gives these matrices explicitly. The compact diagonal notation displays the same four blocks. Each block records one direction independently, and a check such as "is above " inspects both the above and below blocks.
What the numbers show
Table 4 in the paper lists the best earlier accuracy per category and Table 5 lists this model's test accuracy, with all numbers from the 1k training-set setting. The selected rows below compare them.
| Category | Best earlier | Earlier model | This paper (test) |
|---|---|---|---|
| 5, three arguments | 99.3% | Dynamic MemNN | 99.8% |
| 16, basic induction | 100% | Strongly-supervised MemNN | 99.5% |
| 17, positional reasoning | 72% | Multitask | 100% |
| 18, size reasoning | 95% | Strongly-supervised MemNN | 100% |
| 19, path finding | 36% | Strongly-supervised MemNN | 100% |
On these templates and their small vocabulary, the authors report 100% accuracy for the semantic parsing stage, which uses a dependency parser and named-entity recognition. The parameter-setting procedure ignores answers and clues, which lets the authors report accuracy on the training data too. Their own conclusion is that bAbI is mostly uni-relational, so strong scores there may overstate reasoning ability.
A hypothesis about multiple relations
The authors hypothesize that positional reasoning and path finding are hard for memory networks because each direction pair has its own transitivity structure. They discuss Neural Reasoner by Peng and colleagues, which reached 97.9% and 87.0% on these categories with 10k training examples but only 66.4% and 17.3% with 1k. The authors suggest that evolving question representations across layers could explain this, and they say it remains a conjecture. They defer the broader theory to a supplementary report and plan to generalize the approach to multi-relational problems.
Questions & Answers
Q: What does using vectors buy us here?
A: It provides one mathematical format in which to represent entities, bind them into facts, and apply operations that correspond to inferences. The worked football example shows how an intermediate entity can be eliminated by multiplication.
Q: Is this an end-to-end question answering model?
A: The study separates text parsing, relation encoding, and inference. The parser identifies the entities and relations in the restricted language. The final reasoning operations work on their vector encodings.
Q: What happens when two entities have similar vectors?
A: Their dot product can introduce cross-talk when the model tries to retrieve a relation. The experimental setting has a small vocabulary, allowing the model to distinguish the entities. A larger system would have to measure binding and retrieval errors when the number of entities or relations grows.
Q: What would the vector operations do with ambiguous names or conflicting timestamps?
A: The experiments start from a controlled representation with unambiguous entities and task-specific relation rules. A name-resolution error could bind a fact to the wrong entity before any of the vector operations begin. Testing the inference engine on correctly resolved facts and then introducing ambiguous input would show which stage needs improvement.
Future Ideas
A trace checker for event timelines
For a small set of warehouse movements or software deployments, translate event logs into typed facts and answer questions about where an item or version was at a given time. Show the sequence of events that supports each answer. Implement both a vector-based relation engine and a conventional graph query for the same facts.
Stress the system with conflicting events, duplicate names, and missing timestamps. Compare correctness, support-trace validity, and time to inspect failures against the graph baseline. The vector implementation would earn a role if it can maintain trace correctness under conflicting or stale events. The paper assumes unique and unambiguous entities. Duplicate-name tests would separately evaluate an extension to entity resolution.
Sources
Original abstract
Question answering tasks have shown remarkable progress with distributed vector representation. In this paper, we investigate the recently proposed Facebook bAbI tasks which consist of twenty different categories of questions that require complex reasoning. Because the previous work on bAbI are all end-to-end models, errors could come from either an imperfect understanding of semantics or in certain steps of the reasoning. For clearer analysis, we propose two vector space models inspired by Tensor Product Representation (TPR) to perform knowledge encoding and logical reasoning based on common-sense inference. They together achieve near-perfect accuracy on all categories including positional reasoning and path finding that have proved difficult for most of the previous approaches. We hypothesize that the difficulties in these categories are due to the multi-relations in contrast to uni-relational characteristic of other categories. Our exploration sheds light on designing more sophisticated dataset and moving one step toward integrating transparent and interpretable formalism of TPR into existing learning paradigms.