Overview
A local assistant can keep a question on a device, but it cannot always answer it well. Thinking longer costs time and may still produce the wrong result. HARISSA trains a model to estimate which answering method is likely to work, then checks its confidence in the actual answer before deciding whether to respond, try again, or defer to a person.
Key Findings
Give the fastest answer a chance to earn its place
A small model on a device can answer a question immediately or take time to work through it. On a server the analogous choice is which model size to run. An inexpensive first answer can save time. If the system has to discard it and start again with a larger model, that first attempt has become an extra cost. HARISSA asks the model two different questions. Before generation, is this option worth trying? After generation, does this particular answer look good enough to deliver?
One signal is available before writing and another after
The first signal comes from the hidden representation at the last token of the prompt, after the model has processed the question. The second comes from the hidden state at the last token of the generated answer. During offline training, each option answers each training question and receives a correctness label. The authors fine-tune the model with LoRA and a binary prediction objective, using both states. Small logistic probes fitted on those states then estimate correctness. A calibration step aligns each probe's score with observed success rates on development data.
The deployment considers the options in order of cost. The prompt-state probe can skip an option before it generates. If a generated answer receives a low score, the system tries the next option. After the last option it can hand a question back to a person. Equation 1 gives the cost used to select thresholds. The indicators and record a delivered error and a deferral, while is latency.
The weights and price a second of waiting and one deferral in units of a wrong answer delivered. That makes the trade-off explicit. A service that cares more about the cost of an error can choose a different point on the curve.
The device and the server save time in different ways
On the device, the cascade switches between direct answers and chain-of-thought generation from the same model. On the server it can try models of 2B, 4B, 9B, and 27B parameters. Figure 1 plots device results across thresholds. Figure 2 does the same for the server. The following points in Table 1 answer every query, with no deferral.
| Device policy, Table 1 | MedQA accuracy / latency | MedMCQA accuracy / latency | BBH accuracy / latency |
|---|---|---|---|
| Direct | 78.8% / 9.8 s | 63.6% / 7.8 s | 82.2% / 5.7 s |
| Chain-of-thought | 86.2% / 81.0 s | 68.6% / 66.4 s | 90.6% / 44.9 s |
| HARISSA | 85.7% / 33.0 s | 68.5% / 38.4 s | 88.7% / 11.7 s |
At the reported server thresholds for MedQA, HARISSA reaches 81.8% at 11.6 seconds. FrugalGPT and Self-REF reach 69.4% and 69.7% at 11.4 seconds. The 27B model alone is more accurate and slower. In server MedMCQA, the three cascades are close at the selected price on latency. The device and server experiments thus show different benefits from the same policy, and each benefit depends on the cost and competence of the available options.
An answer already written is easier to judge
The authors find that the end-of-answer probe distinguishes correct from incorrect answers more reliably than the prompt probe, and both improve after adaptation. Figure 3 compares their AUROC, a measure of how well a score ranks correct answers above incorrect ones. The prompt probe is still valuable because it can save the generation entirely. The answer probe sees more evidence and can make a stronger decision about whether to deliver the result.
When the policy defers 10% of device BBH questions using the answer probe, wrong answers are delivered on 4.9% of all queries. Deferring the least confident using sequence likelihood leaves 7.6%, and Self-REF confidence leaves 7.9%. The answer probe leaves the fewest errors in five of six task and deployment pairs at that deferral rate. This is a benchmark of correctness-based deferral, including on medical questions. It does not measure what a real person would do with the handoff or establish the safety of clinical use.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Why use two confidence scores?
A: Before generating, the system can estimate whether a method is likely to work. After generating, it has information about the particular answer it produced. The two decisions can differ, even for the same request.
Q: Does the model need a cloud evaluator?
A: The inference policy reads local model states. The training protocol still needs correctness labels and calibration data. We measure the probes and routing after fitting them on the relevant training and development splits.
Q: Do the medical benchmarks demonstrate clinical safety?
A: They test performance on benchmark questions. The deferral study measures how often incorrect answers are delivered under that protocol. Clinical deployment would require independent validation, domain-specific policies, and accountable human review.
Q: Why measure accuracy at the same coverage when testing deferral?
A: A system can improve the fraction of correct answers it delivers by refusing more requests. At matched coverage, a lower error rate shows that it deferred worse answers more effectively. Also check how deferrals affect the queue and end-to-end tail latency. The paper’s timing estimates from GPU rates leave those workload effects open.
Future Ideas
A local help desk that knows which requests need a longer check
Collect questions about a device manual, with independently verified answers and common out-of-scope requests. Train both probes to route between a brief answer, a slower local evidence check, and human review. Define a verifiable correctness label for open-ended answers, since the paper grades a single multiple-choice letter. Test whether the prefill probe adds value beyond the answer probe. Hold out firmware versions and hardware families.
Compare direct answering, always checking, and learned routing at matched review budgets. Record the answer error rate, coverage, worst-case latency, and unresolved deferral queue. Include firmware changes that make a previously correct answer obsolete, so the evaluation tests whether a confident answer remains reliable when the underlying facts move.
Measure errors at matched delivered-answer coverage and report the 95th-percentile delay including human review. Check whether a deferral stays unresolved when the reviewer has no more evidence than the local model.
Sources
Original abstract
Running a language model locally offers advantages in privacy, latency, and cost, but local hardware fits only small models, which are less capable than frontier models. The usual remedy for a hard query, escalating it to a cloud model, gives up the privacy and cost advantages of running locally. A deployment that stays local faces two decisions for hard queries instead. First, it can spend more computation on a query, e.g., reasoning before answering, which raises accuracy at a cost in latency, so it must decide which queries are worth the extra computation (efficiency). Second, some queries are beyond the local model, and delivering a wrong answer is worse than deferring the query to a human in the loop, so it must decide which answers are safe to deliver (safety). We show that both decisions can be made from the model's own hidden states. The prefill state, computed before any token is generated, predicts whether the model will answer correctly, and the answer state, at the end of the generated answer, predicts whether that answer is correct. HARISSA fine-tunes the model so that both states predict correctness, then makes both decisions with one policy that cascades through the ways of answering from cheapest to most expensive, skipping a way the prefill state predicts will fail and deferring the query when the answer it stops with is predicted wrong. On a device running a single model, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency. On a server holding four sizes of one model, HARISSA is more accurate than the FrugalGPT and Self-REF cascades at the same latency, and at the same deferral rate the answer state leaves fewer wrong answers than the standard confidence signals in five of six task and setting pairs.