Overview
Tell a model that its research summary is vague and it might add details unrelated to the weak claim. A stronger model might fix the claim without following the critique at all. Both changes can produce a better final answer. This study asks whether training can identify which advice actually helped a particular model make the right edit.
Key Findings
An editor can be right and still be unhelpful
Suppose a draft report promises a practical recommendation but never explains how a reader would carry it out. An editor can accurately flag the gap and still write a note the author cannot use. The authors of this preprint call the system that writes and revises the answer the actor, and the system that supplies feedback the critic. They ask whether a good critique is a property of the feedback on its own or of the relationship between that feedback and the model asked to act on it.
Their controlled WritingBench experiment changes the critic while keeping the first answer and the refiner fixed. In the reverse comparison, it keeps the critique fixed and changes the refiner. Larger critics received better scores as standalone editors. The change in downstream quality was much smaller than when the researchers gave the same advice to a larger refiner. It mattered whether the recipient could follow the advice.
Give the critic credit for the change it set in motion
The proposed Targeted Actionable Improvement Score, or TAIScore, is built around a short record of what happened. For a prompt , the actor writes an initial response , the critic writes feedback , and the same actor produces the revised response . A judge assesses whether the critique found a real and important issue, whether the actor followed it, whether that part of the answer improved, and whether the edit remained faithful to the original instruction. The judge then emits an overall score between 1 and 10. The four diagnostics guide and explain that decision. The scalar is the critic’s training reward.
Section 4.2 compares critiques of the same answer, all executed by the same actor. If is the score of the th critique-and-revision, the relative advantage for critic training is
The group has a common starting answer. An advantage above zero therefore identifies guidance that helped this actor more than the other critiques did on that response. The comparison is meant to avoid giving a generic note credit when an actor happens to improve the answer in a different way.
Change the writer and the editor gets a new job
After the critic adapts, the actor uses it to produce candidate revisions. A judge that does not see the critique decides which revisions improved on their first drafts. The improved revision and its initial answer become a preference pair for Direct Preference Optimization. The aim is for the actor to internalize good changes and write stronger first drafts, even without a critic at inference. The critic then starts the next round with the new actor. In the experiments, the authors compare three alternating rounds against one-stage approaches at the same total counts of critic-training queries and actor preference pairs.
The test moves from writing to research reports
The authors evaluate the final Qwen3-8B actor on WritingBench, two HelloBench subsets, and DeepResearch-Gym. The excerpt below is from Table 1. OEQA is open-ended question answering, and KPR is recall of reference key points in a report. These are benchmark means.
| Training signal | WritingBench overall | HelloBench OEQA | DeepResearch-Gym KPR | Report quality |
|---|---|---|---|---|
| Initial Qwen3-8B | 72.33 | 34.86 | 71.93 | 81.89 |
| Frozen gpt-oss-120B critic | 75.41 | 36.01 | 73.46 | 82.51 |
| Trained on outcome gain | 75.63 | 35.35 | 74.37 | 82.38 |
| Trained on critique quality | 75.18 | 35.51 | 74.19 | 82.32 |
| TAIScore | 75.96 | 36.48 | 75.21 | 82.63 |
| TAIScore with co-evolution | 76.72 | 39.84 | 76.14 | 83.15 |
A smaller, adapted critic supplies better training pairs than the large frozen critic in this comparison. Table 3 also tests what happens before the actor learns anything new. With the starting answers and actor fixed, TAIScore guidance improves the WritingBench score by 2.78 points, compared with 1.49 for the frozen critic. In the same table, moving a TAIScore critique to the wrong draft reduces the gain to 0.55 points. A second pass with no critique gains 0.97. The comparison helps isolate the value of feedback that addresses a particular draft. In Table 4, critics trained for different actors still help Qwen3-8B, while the one tailored to that actor gives the greatest gain. These checks connect the final benchmark improvement to the quality of the guidance itself. They also set a boundary. Better performance under the study’s judges makes a case for actor-specific feedback in these tasks. It leaves open how the same method would handle professional writing with different editorial standards.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: What makes advice actionable?
A: It identifies something specific the actor can change and gives enough direction for that actor to make the change. We look for the targeted improvement in the revision, rather than scoring the tone or apparent sophistication of the advice.
Q: Why train the writer after training the critic?
A: A good revision can become a preferred answer in a training pair. Using those pairs teaches the actor to produce stronger responses. Updating the critic again makes its feedback responsive to the actor’s changing capabilities.
Q: Could a fluent revision hide an unfixed problem?
A: Yes. That is why the judge sees which weakness the critique targeted and checks whether that part improved. Human assessment is still valuable when the judge overlooks a persistent factual or stylistic problem.
Q: Does the evidence show that one critic will work with any writer?
A: The feedback experiments hold the first answer and actor fixed in places, which helps identify the effect of the critique. The full co-evolution loop was tested with Qwen-family actors, and repeated decoding from a final checkpoint cannot measure variability across independently trained critics. Test different actor families and checkpoints before relying on an existing critic after a model replacement.
Future Ideas
A revision coach that knows which edits a local model can make
Collect draft bug reports that omit evidence, expected behavior, or reproduction steps. Train a small critic to point out one repair at a time, pairing it with the exact model used to edit the report. Compare with a larger generic critic and with a version that optimizes only the final quality rating.
Hold out projects and defect families. Ask blinded maintainers whether the missing item was repaired, whether the edit introduced unsupported details, and whether the report became usable. Repeat after upgrading the editor to see whether updating the critic restores any lost benefit.
Compare the fixed update cadence with a critic refresh triggered by a measurable drop in adoption after an actor change. Run independent checkpoints and charge the trigger system for the revisions used to detect drift.
Sources
Original abstract
Natural-language critiques provide supervision beyond scalar rewards for non-verifiable generation, which lacks deterministic verifiers. In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it. However, final revision quality does not reveal whether the critique was actually useful: a capable actor may improve without following the feedback, while valid feedback may fail if the actor cannot execute it. We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness. We introduce TAIScore (Targeted Actionable Improvement Score), a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves. We use this reward to train an actor-tailored critic with GRPO, and use critique-guided refinements to construct DPO preference pairs for the actor, forming a co-evolving critic-actor loop where the critic adapts to the actor's changing capability. Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward signals. Co-evolving the critic and actor further improves performance, suggesting that effective critique supervision should adapt as the actor changes.