Learning & Reasoning / EMNLP 2023

A Correct Answer Can Hide a Broken Explanation

Heuristic-analytic reasoning lets a model make a quick judgment and then check the physical details. On two story benchmarks, the check improves the evidence behind the answer more than it changes overall accuracy.

From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning

Zheyuan Zhang, Shane Storks, Fengyuan Hu, Sungryull Sohn, Moontae Lee, Honglak Lee, Joyce Chai

Overview

If a story says that someone put a donut in the garbage and then ate it, a reader may spot the contradiction before naming the physical state that changed. A language model can make the same quick judgment and still give the wrong reason. This paper tests a way to turn that initial judgment into a more coherent explanation.

Key Findings

Follow the object through both stories

A language model can choose the plausible story while pointing to the wrong contradiction. The study makes that gap measurable. TRIP asks for the plausible story, the two sentences that conflict in the other story, and the physical states that account for the conflict. The authors also turn ProPara, a dataset about processes, into Tiered-ProPara. There the model chooses which passage contains a transformation, identifies the sentence describing it, and names what the entity becomes.

Figure 1. Both tasks require a broad choice, a location in the text, and a more specific explanation.
Figure 1. Both tasks require a broad choice, a location in the text, and a more specific explanation.

Accuracy scores the broad choice. Consistency also requires locating the right evidence. Verifiability requires the underlying state or entity prediction to check out. On TRIP, at least one non-default physical state must be predicted in each conflicting sentence, and the predicted non-default states must be correct. The measures separate a lucky judgment from an explanation that survives inspection. Task definitions

Make the next step read less

The intervention takes inspiration from the way a quick judgment can guide closer analysis. In the fine-tuned system, a high-level classifier chooses a story and the next classifier sees a narrower context. In Tiered-ProPara, the process narrows from two stories to one story and then to the selected sentence. The authors apply this structure during training and inference. In the prompting version, demonstrations instead teach a large model to commit to the higher-level judgment, find evidence, and then explain the underlying event. Figure 2 · Figure 4

A small accuracy change can hide a large explanation change

Fine-tuned approachTRIP accuracyTRIP consistencyTRIP verifiabilityTiered-ProPara accuracyTiered-ProPara consistencyTiered-ProPara verifiability
FCGLI baseline93.766.233.894.556.736.2
FCGLI with heuristic-analytic reasoning94.375.441.195.183.657.4

The fine-tuned results show gains in consistency and verifiability on both datasets. Prompting can trade a higher-level decision against a better explanation. On the filtered TRIP subset, LLaMA’s verifiability rises from 19.7 with a chain-of-thought baseline to 35.2 with the top-down prompt. Its accuracy drops from 74.6 to 55.6. Early commitment works only as well as the decision that directs it. Table 1 · Table 2

Look at where the model spends its attention

The attention analysis measures whether LLaMA assigns more attention to relevant context at different steps. On Tiered-ProPara, the attentional ratio rises from 1.21 for the unstructured prompting baseline to 2.20 for heuristic-analytic prompting. The paper shows examples in which a selected passage or contradictory sentence gets more attention. That pattern supports the idea that the prompt changes where the model looks. It leaves open whether changing attention by itself would produce the same gains. Table 3 and Figures 5–6

Questions & Answers

Q: Why start with the story instead of listing every possible state?

A: The initial story decision narrows the context. The model can then focus on the sentences most likely to explain the conflict, instead of making detailed predictions for every entity and action in both stories.

Q: What does verifiability add to accuracy?

A: It asks the system to support the conclusion through the task's specified intermediate evidence. A system can guess the right story while naming the wrong sentences or physical states.

Q: What happens when the first judgment is wrong?

A: The later predictions can inherit that error. The paper reports a trade-off in some prompt comparisons. A follow-up system could let a failed state check send the process back to the first decision.

Q: Does better verifiability mean that a model reasoned more faithfully?

A: Verifiability checks whether the submitted story, cited conflict, and predicted physical states fit together under the benchmark. This can expose a lucky top-level prediction. It still leaves open how the model arrived at that prediction. The single heuristic-analytic prompt lowers LLaMA accuracy on both benchmarks in the reported setting, while the separate prompt chain recovers accuracy in Appendix A.2.

Future Ideas

Make a procedure checker reopen its first decision when a state fails

Test a model on paired maintenance instructions that differ by one physically incompatible step. Have it select the suspect procedure, cite the sentence, and predict a state such as whether a valve is open or a part is attached. A small rules engine checks the predicted states. If the check fails, the model may revise the initial judgment and must record what changed.

Compare this two-pass check against a single-pass heuristic-analytic prompt and an accuracy-only classifier at matched token budgets. Use held-out procedures and have technicians verify the relevant states. Report final accuracy, verified explanations, and how often reopening the judgment fixes or introduces an error.

Sources

Original abstract

Pre-trained language models (PLMs) have shown impressive performance in various language tasks. However, they are prone to spurious correlations, and often generate illusory information. In real-world applications, PLMs should justify decisions with formalized, coherent reasoning chains, but this challenge remains under-explored. Cognitive psychology theorizes that humans are capable of utilizing fast and intuitive heuristic thinking to make decisions based on past experience, then rationalizing the decisions through slower and deliberative analytic reasoning. We incorporate these interlinked dual processes in fine-tuning and in-context learning with PLMs, applying them to two language understanding tasks that require coherent physical commonsense reasoning. We show that our proposed Heuristic-Analytic Reasoning (HAR) strategies drastically improve the coherence of rationalizations for model decisions, yielding state-of-the-art results on Tiered Reasoning for Intuitive Physics (TRIP). We also find that this improved coherence is a direct result of more faithful attention to relevant language context in each step of reasoning. Our findings suggest that human-like reasoning strategies can effectively improve the coherence and reliability of PLM reasoning.