Learning & Reasoning / arXiv 2026

Let a Program Check the Answer, One Visual Claim at a Time

A vision-language model turns an image into a structured record. Small programs write training questions and compute their answers from the checked facts.

Program-Verified Self-Evolution for Vision-Language Models

Ahmed Heakl, Sungik Choi, Moontae Lee, Salman Khan

Overview

A model looking at a bar chart may give the same wrong answer several times. A majority vote then converts that mistake into a training label. This study tries to make the label depend on a smaller set of claims. If a model can read the two bars and their labels, an ordinary comparison program can determine which is higher.

Key Findings

A computer can count the mugs once it knows what a mug is

A picture shows a red mug, a pear and a blue book. Ask a model how many mugs are on the table and several of its answers may agree on the wrong number. A training system that uses that majority as the answer can turn a mistake into a lesson. Program-Verified Self-Evolution, or VQS, changes the source of the answer. It has one model turn the picture into a structured record, verifies the visual claims that a question depends on, and lets a small program compute the answer from that record.

Figure 2 from the preprint. The top panel illustrates how majority-vote labels can reinforce a wrong visual answer. The bottom panel computes an answer from a checked parse and filters the question before training.
Figure 2 from the preprint. The top panel illustrates how majority-vote labels can reinforce a wrong visual answer. The bottom panel computes an answer from a checked parse and filters the question before training.

Teach the parser to leave a record that can be checked

The parser and solver start from the same pretrained vision-language model. The parser samples several candidate readings of an unlabeled image. A checker verifies individual claims, such as the presence of the book and its color. It scores each parse by the fraction of its claims that hold. The researchers keep the best parse only when it beats the candidate mean by a margin and makes enough claims to be informative. They then teach the parser to reproduce that selected record. The margin avoids learning indiscriminately from ambiguous candidates, while the minimum claim count discourages an easy high score from a nearly empty parse.

Figure 3 from the preprint. The parser proposes structured readings. A checker verifies individual claims and selects a sufficiently detailed parse that clearly outperforms the alternatives.
Figure 3 from the preprint. The parser proposes structured readings. A checker verifies individual claims and selects a sufficiently detailed parse that clearly outperforms the alternatives.

The schema depends on the source. Photographs become objects and relations. Charts become named series with values and axis information. Diagrams become nodes and directed links. Infographics become entries with fields such as a number and its unit. In Figure 4, the same system can describe an object on a plate, an investment series, a life-cycle diagram, or an animal comparison. Programs can then ask how far apart two values are or which nodes connect, using several checked fields together.

Figure 4 from the preprint. The structured parse changes shape with the material, so a diagram can support a connectivity question and a chart can support an arithmetic one.
Figure 4 from the preprint. The structured parse changes shape with the material, so a diagram can support a connectivity question and a chart can support an arithmetic one.

A good question requires both the picture and a useful challenge

Each program template generates a question, answer and number of operations from a parse. The parser turns the question into natural language while retaining the computed answer. Three checks decide whether the example survives. A fact checker validates the fields used by the program. A text-only model screens out questions that can be answered without seeing the picture. Finally, the current solver tries the question repeatedly, and the training set keeps questions the solver can sometimes get right. This gives reinforcement learning a mixture of successes and failures to learn from.

The solver’s Equation 2 rewards exact agreement with the computed answer aa and gives a smaller reward for using the required format. Here yy is the solver’s generated answer, and both checks return zero or one.

r(y;a)=0.9 eq(y,a)+0.1 fmt(y).r(y;a)=0.9\,\mathrm{eq}(y,a)+0.1\,\mathrm{fmt}(y).

The supervision still rests on visual interpretation. A program can calculate the wrong number flawlessly if the parse contains the wrong inputs. A paraphrase can also change the question. The authors found four such paraphrase errors in their 8B audit, all involving swapped years in a comparison.

Fewer labels survive, and a greater share of them are correct

Human evaluators checked 500 shared questions across four visual domains. Table 2 reports correctness among retained questions. The stricter method keeps fewer examples, which is the cost of improving the labels.

Answer source, Table 2Correct among retainedQuestions kept
Templates plus checker94.4%76.0%
Solver majority vote76.4%92.0%
Model judge82.2%90.8%

The ten-benchmark average in Table 1 improves by 3.18 points for the 2B backbone, 2.32 for 4B and 2.43 for 8B. At 2B, Table 4 shows that removing parser training drops average solver accuracy from 62.4 to 58.0. Removing the claim check while keeping parser training gives 60.9. In Table 5, the largest template-family gains come from chart differences and diagram connectivity, which require combining several reads. Simple object existence barely changes. Repeating the pipeline three times raises the 2B average gain to 3.84 points over the original model. The case for the method rests on the connection between what the model sees and what the program computes. Extending it beyond the four domains still requires useful schemas and programs for the material at hand.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What does the program verify?

A: It computes an answer from the parsed image record. The visual checker checks the individual image claims the program consumes. Both steps matter because correct execution over an incorrect record can still produce a wrong label.

Q: Why verify individual claims instead of the whole image description?

A: The facts needed for one question are often easier to inspect than a long description. A whole-description judgment can hide which statement the checker accepted or missed.

Q: Does the system need people to label its training images?

A: The reported training drops the source dataset’s questions, answers, and labels. Human raters assess selected outputs for the evaluation. Their judgments help measure the labeling method without serving as training targets in the reported setup.

Q: Which visual claims still need special checking?

A: Chart values and diagram arrows were among the weaker types in the checker audit. Those cases can fail because the parser and checker make a similar mistake or because a relevant fact lies outside the template. Compare systems at matched retention and report which visual categories are being dropped, since filtering more examples can improve the measured accuracy of what survives.

Future Ideas

Train a chart assistant on changes it can point to

Generate training questions from operational dashboards whose chart style varies by team. Use a parser to recover a few needed values and units, a visual checker to confirm those claims, and a program to calculate comparisons. Save the data points and crop that support each generated answer.

Hold out dashboard vendors and chart templates. Compare majority vote, whole-answer judging, and claim-level checking at several matched retention levels. Include a test where the image is hidden to identify questions that can be answered from wording alone. Track label correctness, unit mistakes, and errors after charts are resized or redrawn. A rollout should preserve enough labels for uncommon chart types rather than improving the average by discarding them.

Send questions that do not fit the fact schema to a separate review queue and measure coverage for those questions too. This makes the cost of abstention visible and avoids silently collecting labels only for easy charts.

Sources

Original abstract

Self-evolving vision-language models train on questions they generate from unlabeled images. Since these questions have no gold answers, prior methods label them by majority vote over sampled answers or by a model judge. In a human evaluation, we find that 24% of majority-vote labels and 18% of model-judge labels produced during self-evolution are wrong. To address this problem, we present Verifiable QA Generation for Self-Evolving Models (VQS), which changes how the model judges answers. Instead of voting on an answer, the model parses each image into a structured record, such as a scene graph, a chart table, or a diagram graph. Fixed programs then write a question from the record and compute its answer. The model still acts as a visual checker, but it only confirms the individual facts the program reads, one short claim at a time. These claim-level checks select the parser's training targets, so the parser also improves without labels. Human raters find 94% of VQS answers correct, against 76% for majority voting. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at the 2B, 4B, and 8B scales and outperforms the strongest self-evolving baseline at each. Gains keep growing over three training rounds, reaching 3.84 points at 2B. Code is released at https://github.com/ahmedheakl/VQS