Research essay / Learning & reasoning

#2 Double Helix of Intelligence (Part II)

Controlled Generator-Evaluator Co-evolution under Underspecified Rewards

The Motivations

Mina got two stickers, but the classroom machine’s notebook shows one. On a normal run, two sets of instructions pass the teacher’s test. One reuses the same worksheet number after a restart. The other invents a new number each time. To tell them apart, switch the machine off just after it hands out the sticker and turn it back on. Count the stickers Mina actually got.

That is the same test the refund service needs. Interrupt it after the payment goes through but before it saves the receipt. On retry, did the bank send money again? The machine’s message saying “Done” is one piece of evidence. The record of what it did is another. The bank’s ledger can tell us what reached the customer.

Part I gave the writer and checker different jobs. Here we ask when a check is good enough to change what the machine will try next. A little bit of evidence can help. Reward the wrong clue repeatedly and the machine will get better at producing that clue.

The Reasoning

Work outward from what can actually be checked

There are three useful kinds of evidence. You can test an outcome, as when you count the stickers Mina received. You can read a trace, a record of the steps the machine took. Or you can ask for a judgment, as when a teacher chooses which set of instructions is easier to follow. An answer can be easier to read and still fail the restart test. Ask what each kind of evidence can actually tell you.

EvidenceWhat it can tell us about the refundWhat still has to be established
Executed checkA controlled retry caused one recorded debitCoverage of other interruptions and concurrent workers
Trace-based estimateThe retry read the identifier from durable storageWhether the provider retained the identifier and deduplicated it
Preference judgmentOperators can review and reconcile the recovery procedureWhether a clearer procedure also changes safety or correctness

The notation τ=(f,g)\tau=(f,g) packages two pieces of a trace. Here ff is the feedback at the end, such as “retry succeeded.” gg records the events and how they connect. A line showing that the machine read the old worksheet number before requesting another sticker gives us a clue. The stronger check is to see whether it used that number and whether the machine recognized it.

Sometimes we get a test score and still need to ask whether the pass was lucky. Imagine the machine gives one sticker because the teacher happened to catch it before the retry. Write xx for the task, zz for the proposed instructions, and cc for the visible score, 11 for a pass and 00 for a fail. A separate model with settings ψ\psi (psi) gives a score qψ(x,z,τ)q_\psi(x,z,\tau) between 00 and 11 for the chance of accidental success. λspur\lambda_{\mathrm{spur}} sets how much to subtract for that chance, and rmetar_{\mathrm{meta}} is the result we use for learning.

rmeta=c−λspurqψ.r_{\mathrm{meta}}=c-\lambda_{\mathrm{spur}}q_\psi .

Both refund patches pass the ordinary test, so both get c=1c=1. Suppose the checker estimates an 0.80.8 chance of a lucky pass for the fragile patch and 0.10.1 for the robust one. With λspur=0.5\lambda_{\mathrm{spur}}=0.5, their learning scores are 1−0.5×0.8=0.601-0.5\times0.8=0.60 and 1−0.5×0.1=0.951-0.5\times0.1=0.95. To see whether this penalty helps, train with it and test the resulting policy on new cases. The original pass labels alone cannot tell us whether the checker learned to spot luck.

Give agreement a job it can do

Suppose the teacher cannot see Mina’s hand and asks three students what happened. If they give scores of 0.80.8, 0.80.8, and 0.80.8, they agree. If they say 0.60.6, 0.80.8, and 1.01.0, the average is still 0.80.8, but the group has more to investigate. Both groups could be wrong if everyone read the same notebook.

Let MM be the number of evaluators and c^m\hat c_m be evaluator mm’s estimate of correctness. The hat signals an estimate. The bar in c^‾\overline{\hat c} means average all MM estimates. The variance s2s^2 is the average squared distance from that mean. The weight atra_{\mathrm{tr}} is 11 when all scores agree and falls as they spread out. σ2\sigma^2 sets how quickly it falls. In exp⁡(−s2/σ2)\exp(-s^2/\sigma^2), the negative exponent makes the weight shrink. c~\tilde c is the label we use for training after this discount.

atr=exp⁡(−s2/σ2),c~=atr c^‾.a_{\mathrm{tr}}=\exp(-s^2/\sigma^2),\qquad \tilde c=a_{\mathrm{tr}}\,\overline{\hat c}.

Set σ2=0.04\sigma^2=0.04 and compare the two groups. The second group’s differences from its mean are −0.2-0.2, 00, and 0.20.2. Squaring and averaging them gives about 0.02670.0267. Its weight is about 0.5130.513, so we use 0.513×0.8≈0.4110.513\times0.8\approx0.411.

ScoresMeanPopulation varianceConfidence weightLabel used in training
0.8, 0.8, 0.80.800.00001.0000.800
0.6, 0.8, 1.00.800.02670.5130.411

The smaller label is a training choice that makes uncertain successes count less. It can pull an otherwise accurate average below the actual correctness rate. All three students could also give the same wrong score, in which case the weight stays at 11. Keep some cases where you can count the stickers yourself, and check how well the group does when it agrees as well as when it argues.

How far can a trace label move the objective?

A teacher sometimes has both a notebook estimate and a real count. Compare the two on the same cases. Let cc here stand for the checked label, c~\tilde c for the trace label, and ϵtr\epsilon_{\mathrm{tr}} (epsilon) for the average size of their difference. Absolute value bars ∣⋅∣|\cdot| turn a difference into a size, so an error of −0.1-0.1 counts just as much as an error of 0.10.1. E\mathbb E takes the average.

Hold the generator and evaluator settings fixed. Their scores are UGU_G and UEU_E, and the labels “trace” and “true” indicate which labels each version uses. α\alpha is the evaluator’s share of the generator’s blended score, so 1−α1-\alpha is the fraction affected by replacing the checked label. The evaluator uses a loss ℓ(v,c)\ell(v,c) that compares its prediction vv to the label cc. The number LℓL_\ell bounds how quickly that loss can change when the label moves.

UGtrace−UGtrue=(1−α)E[c~−c],∣UGtrace−UGtrue∣≤(1−α)ϵtr,∣UEtrace−UEtrue∣≤Lℓϵtr.\begin{aligned} U_G^{\mathrm{trace}}-U_G^{\mathrm{true}} &=(1-\alpha)\mathbb E[\tilde c-c],\\ |U_G^{\mathrm{trace}}-U_G^{\mathrm{true}}| &\le(1-\alpha)\epsilon_{\mathrm{tr}},\\ |U_E^{\mathrm{trace}}-U_E^{\mathrm{true}}| &\le L_\ell\epsilon_{\mathrm{tr}}. \end{aligned}

If α=0.5\alpha=0.5 and the average label error is 0.040.04, the generator score can move by at most 0.020.02. For the evaluator’s binary cross-entropy loss, limiting predictions to [δ,1−δ][\delta,1-\delta] bounds its sensitivity by log⁡((1−δ)/δ)\log((1-\delta)/\delta). With δ=0.05\delta=0.05, predictions stay between 0.050.05 and 0.950.95, the sensitivity is at most log⁡19≈2.94\log 19\approx2.94, and the evaluator score can move by at most about 0.1180.118.

Imagine 100 classroom trials, with a restart in just one trial and a wrong label on that very trial. The average error can look small while every restart is judged incorrectly. If the generator starts producing more restart cases, it visits a different mix of situations and sees a different error rate. Check the failure families separately as well as the overall average.

How much improvement survives reward error?

Suppose a classroom timer can be off by three seconds on every run. Yesterday it said 60 seconds and today it said 56. Yesterday could really have been 57 and today 59. The four-second measured improvement could hide a two-second slowdown. To be sure, allow the old and new measurements to be wrong in opposite directions.

An AI policy π\pi is the rule that decides what the system will try and with what probabilities. Jr(π)J_r(\pi) is its average score when we use reward rule rr. R⋆R^\star is the reward we wish we could measure and rkr_k is the stand-in reward at round kk. πk\pi_k is the old policy and πk+1\pi_{k+1} is the new one. If the stand-in is off by at most ϵk\epsilon_k on every possible run under consideration, the real gain is at least the measured gain minus two copies of that error.

JR⋆(πk+1)−JR⋆(πk)  ≥  Jrk(πk+1)−Jrk(πk)−2ϵk.J_{R^\star}(\pi_{k+1})-J_{R^\star}(\pi_k) \;\ge\; J_{r_k}(\pi_{k+1})-J_{r_k}(\pi_k)-2\epsilon_k .

If the old score is 0.600.60, the new score is 0.640.64, and each can be off by 0.030.03, the real scores could be 0.630.63 and 0.610.61. With a measured gain of 0.100.10 under the same error bound, at least 0.040.04 survives. The bound assumes a limit on error for every run in scope. A small average error on validation cases does not establish that limit.

Now imagine that the teacher changes the marking rules between the two days because she discovers a flaw in the old test. Regrade the same work under both rules to measure how much the grading changed. In the comparison below, π⋆\pi^\star is a reference policy and rk+1r_{k+1} is the refreshed reward. The bracket compares the same new policy under the two sets of rules.

JR⋆(π⋆)−JR⋆(πk+1)≤Jrk(π⋆)−Jrk+1(πk+1)+[Jrk+1(πk+1)−Jrk(πk+1)]+2ϵk.\begin{aligned} J_{R^\star}(\pi^\star)-J_{R^\star}(\pi_{k+1}) &\le J_{r_k}(\pi^\star)-J_{r_{k+1}}(\pi_{k+1})\\ &\quad+[J_{r_{k+1}}(\pi_{k+1})-J_{r_k}(\pi_{k+1})] +2\epsilon_k. \end{aligned}

The bracket can be negative because the new grader finally notices the extra sticker. It measures a change in scoring rather than a change in what the machine did. An independent count tells us which score is closer to reality. If the refreshed reward also has its own uniform error bound ϵk+1\epsilon_{k+1}, we can compare the reference and new policy using the first score difference and an error allowance of ϵk+ϵk+1\epsilon_k+\epsilon_{k+1}.

What Emerges

Watch a weak signal change what the model will try

Put six slips labeled A and four labeled B into a bag. Drawing a slip is one way to choose a repair. A new score can change the odds of drawing A or B. Checking the evidence matters because the update makes a previously trusted repair more common or less common, even if it has never seen a real restart.

We combine an observed result cc with the learned verifier score Vϕ(x,z,τ)V_\phi(x,z,\tau) to get a shaped reward r♯r^\sharp. b(x)b(x) is a baseline shared by all proposed repairs for the task xx, and α\alpha decides how much the verifier’s score matters. The sharp symbol marks the reward after shaping.

r♯(x,z,τ)=(1−α)c(x,z)+α(Vϕ(x,z,τ)−b(x)).r^\sharp(x,z,\tau)=(1-\alpha)c(x,z)+\alpha\bigl(V_\phi(x,z,\tau)-b(x)\bigr).

For one task, pk(z)p_k(z) is the old probability of choosing repair zz and p(z)p(z) is a possible replacement. Let rzr_z be the repair’s expected shaped reward after averaging over different runs of the same repair. The symbol Δ\Delta in p∈Δp\in\Delta means all probabilities are nonnegative and add to 11. ∑z\sum_z means add over repairs. The first sum is the new policy’s expected reward. The second is the KL cost of moving away from the old probabilities, with η\eta setting how much movement the update will tolerate. Larger η\eta makes that cost smaller.

max⁡p∈Δ[∑zp(z)rz−1η∑zp(z)log⁡p(z)pk(z)].\max_{p\in\Delta}\left[ \sum_z p(z)r_z-\frac1\eta \sum_z p(z)\log\frac{p(z)}{p_k(z)} \right].

For a finite collection where each repair starts with positive probability, the best update has a simple form. Multiply each repair’s old weight by eηrze^{\eta r_z}, then divide by the total new weight so the probabilities again add to one. The sum over jj in the denominator adds all the new weights. This update is called advantage-verification policy improvement.

pk+1(z)=pk(z)eηrz∑jpk(j)eηrj.p_{k+1}(z)=\frac{p_k(z)e^{\eta r_z}}{\sum_j p_k(j)e^{\eta r_j}} .

Here A is the fragile repair and B is the robust one. Both pass the original test. Start at 60 percent A and 40 percent B. Set α=0.5\alpha=0.5, b=0.5b=0.5, and η=2\eta=2. Compare what happens if the checker sees only the original record, then what happens if it sees an independent crash replay.

Evidence supplied to the verifierVerifier scores A / BShaped rewards A / BUpdated probabilities A / B
Original trace, crash untested0.90 / 0.700.70 / 0.6064.7% / 35.3%
Audit replay tests the crash0.20 / 0.800.35 / 0.6545.2% / 54.8%

At first the odds of A versus B are 0.6/0.4=1.50.6/0.4=1.5. With the original record, A leads B by 0.100.10 in shaped reward, so its odds grow by a factor e2×0.10e^{2\times0.10}. With the crash replay, A trails by 0.300.30 and its odds shrink by a factor e2×(−0.30)e^{2\times(-0.30)}. The common baseline cancels in each comparison. For a small step, the change in a repair’s probability is approximately ηpk(z)(rz−rˉ)\eta p_k(z)(r_z-\bar r), where rˉ=∑jpk(j)rj\bar r=\sum_j p_k(j)r_j is the old policy’s average reward. The gain depends on how far above average the repair scores and how often it was tried before. A smaller η\eta shrinks the size of the move. The replay changes its direction.

Price an unreliable verifier separately from a large update

Turning down the volume on a questionable clue and taking a smaller step are two different controls. A smaller α\alpha gives a checker’s score less say in the reward. A smaller η\eta limits how much one update changes the odds.

Let A⋆A^\star be the ideal verifier score and let r⋆r^\star be the shaped reward using it in place of VϕV_\phi. Suppose the verifier’s error has size at most ϵV\epsilon_V on every case in scope. Compare policies using JrJ_r with the same KL cost to the current generator GkG_k for both rewards. Then each policy’s score can shift by at most αϵV\alpha\epsilon_V. Let G^\widehat G maximize the score under the learned verifier and G⋆G^\star maximize it under the ideal verifier. As with the timer, apply the error allowance to both policies.

Jr⋆(G^)≥Jr♯(G^)−αϵV≥Jr♯(G⋆)−αϵV≥Jr⋆(G⋆)−2αϵV.\begin{aligned} J_{r^\star}(\widehat G) &\ge J_{r^\sharp}(\widehat G)-\alpha\epsilon_V\\ &\ge J_{r^\sharp}(G^\star)-\alpha\epsilon_V\\ &\ge J_{r^\star}(G^\star)-2\alpha\epsilon_V. \end{aligned}

With α=0.4\alpha=0.4 and ϵV=0.08\epsilon_V=0.08, at most 2×0.4×0.08=0.0642\times0.4\times0.08=0.064 is lost in the regularized objective. If the optimizer itself misses the best estimated score by at most δopt=0.01\delta_{\mathrm{opt}}=0.01, add that too. The total bound is 0.0740.074. This compares policies for the same task and regularization under the assumed error bound.

Spend verification compute on independent evidence

Suppose a quarter of the slips in a box describe a repair that works. Draw two slips, putting the first back before drawing the second. Both draws miss a good repair with probability 0.75×0.75=0.56250.75\times0.75=0.5625. Therefore the chance that at least one works is 1−0.5625=0.43751-0.5625=0.4375, about 44 percent. You still have to tell which slip works.

In general, KK independent attempts with chance pp of success each give 1−(1−p)K1-(1-p)^K as the chance of finding at least one good proposal. Now spend some of the budget on checking. MM is the number of evaluators testing one bad proposal. Let q<1/2q<1/2 be an upper bound on one evaluator approving it by mistake. If their decisions are independent given the bad proposal, the chance Pr⁡\Pr that a majority approves it by mistake has this upper bound.

Pr⁡(false accept by majority)≤exp⁡[−2M(1/2−q)2].\Pr(\mathrm{false\ accept\ by\ majority}) \le \exp[-2M(1/2-q)^2].

At q=0.20q=0.20, even this loose upper bound falls below 1 percent once M=26M=26. It concerns one bad proposal and only false approvals. If you inspect more proposals, add their individual bounds to get a conservative limit on the chance that any one gets through. That is another cost of searching more widely.

Illustrative allocationCompute with cG=3c_G=3, cV=1c_V=1Chance a correct candidate exists at p=0.25p=0.25Majority bound per invalid candidate at q=0.20q=0.20
4 candidates, 9 evaluators each4868.4%19.8%
8 candidates, 3 evaluators each4890.0%58.3%

Both rows spend 48 units because one proposal costs cG=3c_G=3 and one check costs cV=1c_V=1. The second row is more likely to produce at least one good repair but has a weaker bound on mistakenly approving one bad repair. These are two distinct probabilities. If all the evaluators consult the same notebook, more votes may reproduce the same mistake. With average pairwise error correlation ρ\rho, M/[1+(M−1)ρ]M/[1+(M-1)\rho] is a diagnostic for the number of effectively independent votes. It gives 3 for nine evaluators with ρ=0.25\rho=0.25. It cannot certify a majority vote with correlated errors. A fresh look at the bank ledger may buy more than another reader of the same log.

Open Questions

Can a verifier show which event changed its mind?

Rename Mina’s worksheet number everywhere and replay the same instructions. The outcome should stay the same. Now change just one step so the machine creates a new number upon retry. The crash replay should expose a duplicate. Train or calibrate the checker on pairs like these, then test it on new kinds of failures. The checker should change its score when the repair becomes unsafe and stay steady when only the label changes. Measure both behaviors. A checker that ignores every edit can look steady while missing the failure that matters.

Could an audit follow the probability the update is about to move?

Suppose the checker has nearly tied the scores for A and B, and a second look might reverse their order. The decision matters most when the next update is about to put substantial weight on one of them. If the current estimates could be off by at most uAu_A and uBu_B, the uncertainty radius for the change in log odds is at most ηα(uA+uB)\eta\alpha(u_A+u_B). The two errors can pull in opposite directions. Ask for an independent replay when that range includes both directions and the update would matter. Compare this rule with random audits at the same budget, counting how much probability each approach moves toward independently verified failures.

Can the controller tell a bad verifier from a late verifier?

Keep copies of the old and new sets of repairs. Score each set with both the old and the new checker and run an independent test where possible. Those four comparisons let you see whether the repairs changed, the checker changed, or both. Try several update intervals on one set of tasks and test the resulting schedule on separate ones. Vary the verifier’s influence, the number of repairs, and the amount of score error. The numerical examples show what can happen in controlled cases. What remains open is how reliably the checks steer learning when evidence is sparse and the system keeps changing.