#2 Double Helix of Intelligence (Part II)
Controlled Generator-Evaluator Co-evolution under Underspecified Rewards
Controlled Generator-Evaluator Co-evolution under Underspecified Rewards
Mina got two stickers, but the classroom machine’s notebook shows one. On a normal run, two sets of instructions pass the teacher’s test. One reuses the same worksheet number after a restart. The other invents a new number each time. To tell them apart, switch the machine off just after it hands out the sticker and turn it back on. Count the stickers Mina actually got.
That is the same test the refund service needs. Interrupt it after the payment goes through but before it saves the receipt. On retry, did the bank send money again? The machine’s message saying “Done” is one piece of evidence. The record of what it did is another. The bank’s ledger can tell us what reached the customer.
Part I gave the writer and checker different jobs. Here we ask when a check is good enough to change what the machine will try next. A little bit of evidence can help. Reward the wrong clue repeatedly and the machine will get better at producing that clue.
There are three useful kinds of evidence. You can test an outcome, as when you count the stickers Mina received. You can read a trace, a record of the steps the machine took. Or you can ask for a judgment, as when a teacher chooses which set of instructions is easier to follow. An answer can be easier to read and still fail the restart test. Ask what each kind of evidence can actually tell you.
| Evidence | What it can tell us about the refund | What still has to be established |
|---|---|---|
| Executed check | A controlled retry caused one recorded debit | Coverage of other interruptions and concurrent workers |
| Trace-based estimate | The retry read the identifier from durable storage | Whether the provider retained the identifier and deduplicated it |
| Preference judgment | Operators can review and reconcile the recovery procedure | Whether a clearer procedure also changes safety or correctness |
The notation packages two pieces of a trace. Here is the feedback at the end, such as “retry succeeded.” records the events and how they connect. A line showing that the machine read the old worksheet number before requesting another sticker gives us a clue. The stronger check is to see whether it used that number and whether the machine recognized it.
Sometimes we get a test score and still need to ask whether the pass was lucky. Imagine the machine gives one sticker because the teacher happened to catch it before the retry. Write for the task, for the proposed instructions, and for the visible score, for a pass and for a fail. A separate model with settings (psi) gives a score between and for the chance of accidental success. sets how much to subtract for that chance, and is the result we use for learning.
Both refund patches pass the ordinary test, so both get . Suppose the checker estimates an chance of a lucky pass for the fragile patch and for the robust one. With , their learning scores are and . To see whether this penalty helps, train with it and test the resulting policy on new cases. The original pass labels alone cannot tell us whether the checker learned to spot luck.
Suppose the teacher cannot see Mina’s hand and asks three students what happened. If they give scores of , , and , they agree. If they say , , and , the average is still , but the group has more to investigate. Both groups could be wrong if everyone read the same notebook.
Let be the number of evaluators and be evaluator ’s estimate of correctness. The hat signals an estimate. The bar in means average all estimates. The variance is the average squared distance from that mean. The weight is when all scores agree and falls as they spread out. sets how quickly it falls. In , the negative exponent makes the weight shrink. is the label we use for training after this discount.
Set and compare the two groups. The second group’s differences from its mean are , , and . Squaring and averaging them gives about . Its weight is about , so we use .
| Scores | Mean | Population variance | Confidence weight | Label used in training |
|---|---|---|---|---|
| 0.8, 0.8, 0.8 | 0.80 | 0.0000 | 1.000 | 0.800 |
| 0.6, 0.8, 1.0 | 0.80 | 0.0267 | 0.513 | 0.411 |
The smaller label is a training choice that makes uncertain successes count less. It can pull an otherwise accurate average below the actual correctness rate. All three students could also give the same wrong score, in which case the weight stays at . Keep some cases where you can count the stickers yourself, and check how well the group does when it agrees as well as when it argues.
A teacher sometimes has both a notebook estimate and a real count. Compare the two on the same cases. Let here stand for the checked label, for the trace label, and (epsilon) for the average size of their difference. Absolute value bars turn a difference into a size, so an error of counts just as much as an error of . takes the average.
Hold the generator and evaluator settings fixed. Their scores are and , and the labels “trace” and “true” indicate which labels each version uses. is the evaluator’s share of the generator’s blended score, so is the fraction affected by replacing the checked label. The evaluator uses a loss that compares its prediction to the label . The number bounds how quickly that loss can change when the label moves.
If and the average label error is , the generator score can move by at most . For the evaluator’s binary cross-entropy loss, limiting predictions to bounds its sensitivity by . With , predictions stay between and , the sensitivity is at most , and the evaluator score can move by at most about .
Imagine 100 classroom trials, with a restart in just one trial and a wrong label on that very trial. The average error can look small while every restart is judged incorrectly. If the generator starts producing more restart cases, it visits a different mix of situations and sees a different error rate. Check the failure families separately as well as the overall average.
Suppose a classroom timer can be off by three seconds on every run. Yesterday it said 60 seconds and today it said 56. Yesterday could really have been 57 and today 59. The four-second measured improvement could hide a two-second slowdown. To be sure, allow the old and new measurements to be wrong in opposite directions.
An AI policy is the rule that decides what the system will try and with what probabilities. is its average score when we use reward rule . is the reward we wish we could measure and is the stand-in reward at round . is the old policy and is the new one. If the stand-in is off by at most on every possible run under consideration, the real gain is at least the measured gain minus two copies of that error.
If the old score is , the new score is , and each can be off by , the real scores could be and . With a measured gain of under the same error bound, at least survives. The bound assumes a limit on error for every run in scope. A small average error on validation cases does not establish that limit.
Now imagine that the teacher changes the marking rules between the two days because she discovers a flaw in the old test. Regrade the same work under both rules to measure how much the grading changed. In the comparison below, is a reference policy and is the refreshed reward. The bracket compares the same new policy under the two sets of rules.
The bracket can be negative because the new grader finally notices the extra sticker. It measures a change in scoring rather than a change in what the machine did. An independent count tells us which score is closer to reality. If the refreshed reward also has its own uniform error bound , we can compare the reference and new policy using the first score difference and an error allowance of .
Put six slips labeled A and four labeled B into a bag. Drawing a slip is one way to choose a repair. A new score can change the odds of drawing A or B. Checking the evidence matters because the update makes a previously trusted repair more common or less common, even if it has never seen a real restart.
We combine an observed result with the learned verifier score to get a shaped reward . is a baseline shared by all proposed repairs for the task , and decides how much the verifier’s score matters. The sharp symbol marks the reward after shaping.
For one task, is the old probability of choosing repair and is a possible replacement. Let be the repair’s expected shaped reward after averaging over different runs of the same repair. The symbol in means all probabilities are nonnegative and add to . means add over repairs. The first sum is the new policy’s expected reward. The second is the KL cost of moving away from the old probabilities, with setting how much movement the update will tolerate. Larger makes that cost smaller.
For a finite collection where each repair starts with positive probability, the best update has a simple form. Multiply each repair’s old weight by , then divide by the total new weight so the probabilities again add to one. The sum over in the denominator adds all the new weights. This update is called advantage-verification policy improvement.
Here A is the fragile repair and B is the robust one. Both pass the original test. Start at 60 percent A and 40 percent B. Set , , and . Compare what happens if the checker sees only the original record, then what happens if it sees an independent crash replay.
| Evidence supplied to the verifier | Verifier scores A / B | Shaped rewards A / B | Updated probabilities A / B |
|---|---|---|---|
| Original trace, crash untested | 0.90 / 0.70 | 0.70 / 0.60 | 64.7% / 35.3% |
| Audit replay tests the crash | 0.20 / 0.80 | 0.35 / 0.65 | 45.2% / 54.8% |
At first the odds of A versus B are . With the original record, A leads B by in shaped reward, so its odds grow by a factor . With the crash replay, A trails by and its odds shrink by a factor . The common baseline cancels in each comparison. For a small step, the change in a repair’s probability is approximately , where is the old policy’s average reward. The gain depends on how far above average the repair scores and how often it was tried before. A smaller shrinks the size of the move. The replay changes its direction.
Turning down the volume on a questionable clue and taking a smaller step are two different controls. A smaller gives a checker’s score less say in the reward. A smaller limits how much one update changes the odds.
Let be the ideal verifier score and let be the shaped reward using it in place of . Suppose the verifier’s error has size at most on every case in scope. Compare policies using with the same KL cost to the current generator for both rewards. Then each policy’s score can shift by at most . Let maximize the score under the learned verifier and maximize it under the ideal verifier. As with the timer, apply the error allowance to both policies.
With and , at most is lost in the regularized objective. If the optimizer itself misses the best estimated score by at most , add that too. The total bound is . This compares policies for the same task and regularization under the assumed error bound.
Suppose a quarter of the slips in a box describe a repair that works. Draw two slips, putting the first back before drawing the second. Both draws miss a good repair with probability . Therefore the chance that at least one works is , about 44 percent. You still have to tell which slip works.
In general, independent attempts with chance of success each give as the chance of finding at least one good proposal. Now spend some of the budget on checking. is the number of evaluators testing one bad proposal. Let be an upper bound on one evaluator approving it by mistake. If their decisions are independent given the bad proposal, the chance that a majority approves it by mistake has this upper bound.
At , even this loose upper bound falls below 1 percent once . It concerns one bad proposal and only false approvals. If you inspect more proposals, add their individual bounds to get a conservative limit on the chance that any one gets through. That is another cost of searching more widely.
| Illustrative allocation | Compute with , | Chance a correct candidate exists at | Majority bound per invalid candidate at |
|---|---|---|---|
| 4 candidates, 9 evaluators each | 48 | 68.4% | 19.8% |
| 8 candidates, 3 evaluators each | 48 | 90.0% | 58.3% |
Both rows spend 48 units because one proposal costs and one check costs . The second row is more likely to produce at least one good repair but has a weaker bound on mistakenly approving one bad repair. These are two distinct probabilities. If all the evaluators consult the same notebook, more votes may reproduce the same mistake. With average pairwise error correlation , is a diagnostic for the number of effectively independent votes. It gives 3 for nine evaluators with . It cannot certify a majority vote with correlated errors. A fresh look at the bank ledger may buy more than another reader of the same log.
Rename Mina’s worksheet number everywhere and replay the same instructions. The outcome should stay the same. Now change just one step so the machine creates a new number upon retry. The crash replay should expose a duplicate. Train or calibrate the checker on pairs like these, then test it on new kinds of failures. The checker should change its score when the repair becomes unsafe and stay steady when only the label changes. Measure both behaviors. A checker that ignores every edit can look steady while missing the failure that matters.
Suppose the checker has nearly tied the scores for A and B, and a second look might reverse their order. The decision matters most when the next update is about to put substantial weight on one of them. If the current estimates could be off by at most and , the uncertainty radius for the change in log odds is at most . The two errors can pull in opposite directions. Ask for an independent replay when that range includes both directions and the update would matter. Compare this rule with random audits at the same budget, counting how much probability each approach moves toward independently verified failures.
Keep copies of the old and new sets of repairs. Score each set with both the old and the new checker and run an independent test where possible. Those four comparisons let you see whether the repairs changed, the checker changed, or both. Try several update intervals on one set of tasks and test the resulting schedule on separate ones. Vary the verifier’s influence, the number of repairs, and the amount of score error. The numerical examples show what can happen in controlled cases. What remains open is how reliably the checks steer learning when evidence is sparse and the system keeps changing.