Research essay / Learning & reasoning

#1 Double Helix of Intelligence (Part I)

Controlled Generator-Evaluator Co-evolution under Underspecified Rewards

The Motivations

Imagine a classroom sticker machine. Its job is to give one sticker to each child who finishes a worksheet. Mina finishes hers. The machine drops a sticker into her hand and freezes before it writes her name in its notebook. When the teacher presses Retry, the machine gives Mina another sticker. A check that only asks “Did Mina get a sticker?” gives it full credit. The classroom rule was one sticker per worksheet.

Now imagine two students improving the machine. One writes instructions for it. The other tries those instructions and gives feedback. If the checker always asks whether a sticker came out, the writer will get better at making stickers come out. To learn to handle a restart, the pair needs a test that turns the machine off at that exact moment. In an AI system, the generator proposes an answer or an action. The evaluator checks it and supplies a score, a repair, or a reason to ask for help.

The same mistake can send a customer two refunds. A service can send the money and lose its connection before recording the receipt. A safe retry uses the same refund number, and the payment provider must recognize that number. An underspecified reward gives credit for part of the job while leaving out a rule that matters. Every time the two models learn, they change what the other one sees next. That is the double helix: better answers need better ways to check them, and better checks change which answers get made.

The Reasoning

Give the loop a workable starting point

Think of each letter as a label on a box. xx is the task, such as “give Mina one sticker.” zz is one proposed set of instructions. The Greek letter τ\tau (tau) is the trace, an ordered record of what the machine did. The generator GθG_\theta produces a proposal, with θ\theta (theta) standing for its adjustable settings. The evaluator VϕV_\phi (phi) reads the task, proposal, and trace, then estimates how likely the proposal is to be correct.

A reliable test gives us cc, the observed correctness label: 11 for a pass and 00 for a fail. If a test is unavailable, cc is unknown. VϕV_\phi is an estimate between 00 and 11. In a run with an available test, a pass and a checker score of 0.60.6 can be blended. With evaluator weight α=0.25\alpha=0.25, the blended score is 0.75×1+0.25×0.6=0.900.75\times1+0.25\times0.6=0.90. The two weights add to 11. Turning up α\alpha gives the checker more influence.

The two models have different scores to improve, UGU_G for the generator and UEU_E for the evaluator. E\mathbb E means take an average, including over the proposals the generator tends to make. The subscript xx means averaging over tasks alone. The generator pays for drifting from a trusted reference GrefG_{\mathrm{ref}}. The distance measure DKLD_{\mathrm{KL}} compares how two policies spread their probabilities. The evaluator pays a loss ℓ\ell when its predictions miss the checked labels and a calibration penalty Ωcal\Omega_{\mathrm{cal}} when its confidence does not match its track record. The two λ\lambda terms set the strengths of those penalties.

UG(θ,ϕ)=Eθ[(1−α)c+αVϕ]−λKL ExDKL(Gθ∥Gref),UE(θ,ϕ)=−Eθ[ℓ(Vϕ,c)]−λcalΩcal(ϕ).\begin{aligned} U_G(\theta,\phi) &=\mathbb E_\theta[(1-\alpha)c+\alpha V_\phi] -\lambda_{\mathrm{KL}}\,\mathbb E_x D_{\mathrm{KL}}(G_\theta\|G_{\mathrm{ref}}),\\ U_E(\theta,\phi) &=-\mathbb E_\theta[\ell(V_\phi,c)] -\lambda_{\mathrm{cal}}\Omega_{\mathrm{cal}}(\phi). \end{aligned}

Before the pair starts teaching itself, give each model a minimum score to meet. If a checker approves three of ten known bad solutions, its false approval rate is 30 percent. A 10 percent cap allows only one. A strong generator score cannot make that checker pass its own gate. Test these gates on cases set aside from training, including retries and restarts.

Pack both models’ settings into y=(θ,ϕ)y=(\theta,\phi). We minimize the loss ff by taking the negative of the two scores, weighted by ωG\omega_G and ωE\omega_E. The minimum scores are τG\tau_G and τE\tau_E. Those are thresholds, whereas τ\tau without a subscript is the trace. A constraint written as g(y)≤0g(y)\le 0 passes when its value is zero or negative. Each hjh_j is an extra limit, such as the cap on false approvals or a limit on how far the generator can drift.

min⁡y=(θ,ϕ)f(y)=−ωGUG(y)−ωEUE(y)subject tog1(y)=τG−UG(y)≤0,g2(y)=τE−UE(y)≤0,hj(y)≤0.\begin{aligned} \min_{y=(\theta,\phi)} \quad & f(y)=-\omega_GU_G(y)-\omega_EU_E(y)\\ \text{subject to}\quad &g_1(y)=\tau_G-U_G(y)\le0,\\ &g_2(y)=\tau_E-U_E(y)\le0,\qquad h_j(y)\le0. \end{aligned}
Gate on separate validation dataTargetIllustrative resultConsequence
Generator passes unseen workflow checksAt least 75%82%Meets its floor
Evaluator approves independently identified invalid patchesAt most 10%18%Reliability constraint is violated
Average KL divergence from reference policyAt most 0.050.03Meets the drift budget

Once these gates are separate, we can ask two questions. With a generator fixed, how strong is the best checker we can train for it? With a checker fixed, how well does it do on the hardest generator that meets the capability threshold? The first asks whether a new skill can still be checked. The second asks whether the checker will keep up as the writer improves.

Give the evaluator four distinct jobs

The students can use the same afternoon in four ways. They can try to break the machine by switching it off at the worst moment. They can help fix it by showing how to reuse the same worksheet number. They can teach that idea on easy examples and try it later on a different machine. Or they can ask a teacher to count the real stickers when the machine’s own notebook is unclear. Those are four different jobs because each answers a different question.

ModeQuantity it acts onRefund example
AdversarialIndependently verified failures the evaluator detectsKill the service after the debit and before it records the receipt
CooperativeA shared potential for generator progress and evaluator qualityPropose persisting the operation identifier before sending the debit
CurriculumImprovement on new tasks after a feedback-driven updateTeach durable identifiers on one failure family and test transfer to concurrent refunds
AuditingSeverity of missed failures, false blocks, and abstentionsReconcile the ledger on unusual restart cases and send uncertain cases to a human

Find a failure. In adversarial mode, the checker earns credit for catching a mistake that a separate test confirms. The discovery score Dϕ(θ)=Eθ[(1−c)(1−Vϕ)]D_\phi(\theta)=\mathbb E_\theta[(1-c)(1-V_\phi)] averages this credit. The first factor switches the credit on only for a verified failure. An evaluator score of 0.20.2 earns 0.80.8 if the test says c=0c=0, and earns zero if the test says c=1c=1. Calling a working proposal broken still increases the evaluator’s prediction loss.

Help with a repair. In cooperative mode, both sides try to raise a shared score Ψ\Psi (psi). The weights βG\beta_G and βE\beta_E set the value of improving each model. The symbol ∇\nabla means the direction in which a score increases fastest. Γ\Gamma converts the evaluator’s direction into the generator’s coordinates so the two directions can be compared. The last term charges for disagreement, with ρ\rho setting the price and the squared length measuring its size.

Ψ=βGUG+βEUE−ρ2∥∇θUG−Γ∇ϕUE∥2.\Psi=\beta_GU_G+\beta_EU_E -\frac{\rho}{2}\left\|\nabla_\theta U_G-\Gamma\nabla_\phi U_E\right\|^2.

If each side could find the best update to its own settings while holding the other fixed, the old settings would still be an available choice. The shared score could only stay the same or go up. A noisy training step needs a fresh check to see whether it actually helped.

Teach for the next task. A hint aa might say “keep the old worksheet number when you retry.” We want to know whether it helps on a new task, such as a library checkout that freezes before saving. The superscript ++ means after an update. ηG\eta_G (eta) is the size of the teaching step, and the hat on the training gradient means it is estimated. The labels “train” and “val” refer to the practice cases and the separate validation cases. Δlearn\Delta_{\mathrm{learn}} is the change on those new cases.

θ+(a)=θ+ηG∇θUGtrain^(θ,ϕ;a),Δlearn(a)=UGval(θ+(a),ϕ)−UGval(θ,ϕ).\begin{aligned} \theta^+(a)&=\theta+\eta_G\widehat{\nabla_\theta U_G^{\mathrm{train}}}(\theta,\phi;a),\\ \Delta_{\mathrm{learn}}(a) &=U_G^{\mathrm{val}}(\theta^+(a),\phi)-U_G^{\mathrm{val}}(\theta,\phi). \end{aligned}

For a small step, the gain depends on the dot product of the teaching direction and the direction that improves the new tasks. Two arrows pointing the same way have a positive dot product. In symbols, the gain is approximately ηG⟨∇θUGval,∇θUGtrain^(a)⟩\eta_G\langle\nabla_\theta U_G^{\mathrm{val}},\widehat{\nabla_\theta U_G^{\mathrm{train}}}(a)\rangle, where the angle brackets denote that dot product. Giving a student the answer to today’s worksheet can make today’s score jump while doing little for tomorrow’s worksheet. Give the hint credit when the student can solve a new problem.

Audit serious misses. An audit can accept, reject, or pause for help. Call that decision dd. A missed failure has loss Rmiss=s 1{c=0, d=accept}R_{\mathrm{miss}}=s\,\mathbf 1\{c=0,\,d=\mathrm{accept}\}, where ss measures its severity and 1\mathbf 1 switches on only when both conditions are true. CVaR⁡β\operatorname{CVaR}_\beta is the average loss in the worst fraction 1−β1-\beta of cases. The formula tries a loss cutoff ν\nu (nu) and charges for losses beyond it. (a)+(a)_+ means the larger of aa and zero.

CVaR⁡β(Rmiss)=min⁡ν[ν+E(Rmiss−ν)+1−β].\operatorname{CVaR}_\beta(R_{\mathrm{miss}}) =\min_{\nu}\left[\nu+\frac{\mathbb E(R_{\mathrm{miss}}-\nu)_+}{1-\beta}\right].

Suppose 99 of 100 patches cause no harm and one causes a loss of 100. The overall average is 1. In the worst five cases the average is (100+0+0+0+0)/5=20(100+0+0+0+0)/5=20. The second number gives the missed failure the attention it deserves even when the batch looks good on average.

What a violated constraint costs

Picture ten minutes to split between writing new instructions and checking them. If checking must get at least six minutes, spending eight minutes on writing breaks the rule. A multiplier acts like a price for continuing to break a rule. Its price can rise each time the same problem appears.

Put the constraint values into a list g(y)g(y) and give each its own nonnegative price in the list λ\lambda (lambda). L\mathcal L is the original loss ff plus the sum of prices times constraint values, written λ⊤g\lambda^\top g. The index tt counts updates. The step sizes ηp\eta_p and ηd\eta_d set how far the model settings and the prices move. ΠY\Pi_{\mathcal Y} keeps the settings inside their allowed region, and [⋅]+[\cdot]_+ stops any price from dipping below zero.

L(y,λ)=f(y)+λ⊤g(y),yt+1=ΠY[yt−ηp∇yL(yt,λt)],λt+1=[λt+ηd g(yt+1)]+.\begin{aligned} \mathcal L(y,\lambda)&=f(y)+\lambda^\top g(y),\\ y_{t+1}&=\Pi_{\mathcal Y}[y_t-\eta_p\nabla_y\mathcal L(y_t,\lambda_t)],\\ \lambda_{t+1}&=[\lambda_t+\eta_d\,g(y_{t+1})]_+ . \end{aligned}

If false approvals stay eight percentage points above the cap for five steps and ηd=0.5\eta_d=0.5, the price rises by 5×0.5×0.08=0.205\times0.5\times0.08=0.20. The rising price tells the optimizer to take the violation more seriously. Getting the failure rate down still depends on examples that teach the checker what it missed.

Use uu for the generator score and vv for the evaluator score. Both would ideally reach 0.90.9, measured by the squared loss (u−0.9)2+(v−0.9)2(u-0.9)^2+(v-0.9)^2. With a shared budget u+v≤1.4u+v\le1.4 and a checker floor v≥0.75v\ge0.75, the best feasible pair is (0.65,0.75)(0.65,0.75). At that point the gradient of the loss is (−0.50,−0.30)(-0.50,-0.30). The budget and floor exert prices of 0.500.50 and 0.200.20 along directions (1,1)(1,1) and (0,−1)(0,-1). Their sum cancels the loss gradient.

(−0.50,−0.30)+0.50(1,1)+0.20(0,−1)=(0,0).(-0.50,-0.30) + 0.50(1,1)+0.20(0,-1)=(0,0).

Both rules are tight. Under convexity, bounded gradients, and the usual feasibility and step-size assumptions, the averaged gap in this procedure shrinks on the order of 1/T1/\sqrt T over TT updates. A trained checkpoint still needs to pass the original gates on fresh cases.

When two local updates have to agree

Suppose two students are building opposite ends of the same bridge and each gets to choose a height. One wants eight blocks and the other wants four. They can work separately, but their ends must eventually meet. Simply telling them to agree gets them no closer to knowing which adjustments cost each side the least.

In the alternating direction method of multipliers, or ADMM, aa and bb are the two copies of the shared setting. Each side has its own loss, fGf_G or fEf_E, with its local rules included. arg⁡min⁡\arg\min means choose the input that makes the expression smallest. A superscript ++ indicates the newest value. ρ\rho prices disagreement, and dd remembers accumulated disagreement from earlier rounds.

a+=arg⁡min⁡a{fG(a)+(ρ/2)∥a−b+d∥2},b+=arg⁡min⁡b{fE(b)+(ρ/2)∥a+−b+d∥2},d+=d+a+−b+.\begin{aligned} a^+&=\arg\min_a\{f_G(a)+(\rho/2)\|a-b+d\|^2\},\\ b^+&=\arg\min_b\{f_E(b)+(\rho/2)\|a^+-b+d\|^2\},\\ d^+&=d+a^+-b^+. \end{aligned}

Take local losses 12(a−0.8)2\tfrac12(a-0.8)^2 and 12(b−0.4)2\tfrac12(b-0.4)^2. With ρ=1\rho=1 and starting values b=0.4b=0.4, d=0d=0, the first round gives (a,b)=(0.60,0.50)(a,b)=(0.60,0.50) and the next gives (0.60,0.55)(0.60,0.55). The gap falls from 0.100.10 to 0.050.05, and the second side moves by 0.050.05 on that next round. Track both the remaining gap a+−b+a^+-b^+ and the movement ρ(b+−b)\rho(b^+-b). A small gap tells us the two sides match today, while a small movement tells us the target is settling. The convergence result requires convex, closed local problems, an existing saddle point, and exact solves or errors whose sum stays finite.

Why looking ahead changes the game

Imagine two kids moving a marker on graph paper. One controls the left–right move and one controls the up–down move. Each reacts to the other’s last position. Even when both want to follow the rules, their moves can send the marker circling away from the middle.

In a two-model game, yy is their current joint position. F(y)F(y) stacks the negatives of the two gradients, one for each player’s own score. With step size η\eta, an ordinary update moves using the directions at yy. Extragradient first takes a trial step to y~\tilde y (the tilde marks a provisional value). It measures the directions there, then uses them to make the real move from yy to y+y^+.

y~=y−ηF(y),y+=y−ηF(y~).\tilde y=y-\eta F(y),\qquad y^+=y-\eta F(\tilde y).

Try a game in which one player minimizes uvuv and the other maximizes it. For y=(u,v)y=(u,v) the update direction is F(y)=(v,−u)F(y)=(v,-u). Starting at (1,0)(1,0) with η=0.5\eta=0.5, an ordinary update goes to (1,0.5)(1,0.5), farther from the center. Extragradient visits that same point as a trial, sees F(1,0.5)=(0.5,−1)F(1,0.5)=(0.5,-1), and finishes at (0.75,0.5)(0.75,0.5), closer to the center. The distance factors are 1+η2\sqrt{1+\eta^2} and 1−η2+η4\sqrt{1-\eta^2+\eta^4}. The example isolates rotation. A general linear convergence claim needs a direction field that changes at a bounded rate, called Lipschitz continuity, and a restoring effect that stays strong throughout the region, called strong monotonicity. The steps must stay inside that region and a stopping point must exist there.

What if we mix several ways of updating? Let FmF_m be the direction field for mode mm, u,vu,v any two states, and pmp_m the fixed share given to that mode. The inner product ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle measures how much the difference in directions points along the displacement between the states. The constant μm>0\mu_m>0 is the minimum strength of this restoring effect, and LmL_m caps how quickly the direction can vary.

⟨Fm(u)−Fm(v),u−v⟩≥μm∥u−v∥2,\langle F_m(u)-F_m(v),u-v\rangle\ge\mu_m\|u-v\|^2,

Taking the weighted sum gives a mixture Fˉ=∑mpmFm\bar F=\sum_m p_mF_m with restoring strength at least ∑mpmμm\sum_m p_m\mu_m and variation at most ∑mpmLm\sum_m p_mL_m times the distance moved. Its stopping point can land somewhere new. The directions 2(y−1)2(y-1) and 4(y+1)4(y+1) stop at 11 and −1-1. Their equal mixture is 3y+13y+1, which stops at −1/3-1/3. Alternating the two updates in time changes which state each direction sees.

What Emerges

Decide where another batch is worth spending

If a class has time for ten trials, should it use all ten to try to break the machine? A crash test may reveal more per attempt than a repair session, but we might be less sure of that estimate. The controller can spend some of its budget on the more predictable work.

The list qq holds the shares, all at least zero and together summing to one. Δ\boldsymbol\Delta holds the estimated gains on new cases. Σ\Sigma is the covariance matrix. Its diagonal entries measure the uncertainty in each estimate, and the other entries measure how much the estimates move together. The dot product Δ⊤q\boldsymbol\Delta^\top q is the expected gain of the mix. The quadratic term q⊤Σqq^\top\Sigma q is the mix’s variance. λmix\lambda_{\mathrm{mix}} sets how much that uncertainty costs.

max⁡q≥0,  1⊤q=1Δ⊤q−λmix2q⊤Σq.\max_{q\ge0,\;\mathbf 1^\top q=1} \boldsymbol\Delta^\top q-\frac{\lambda_{\mathrm{mix}}}{2}q^\top\Sigma q.

For two modes, let ww be the share for finding failures and 1−w1-w the share for repairs. Suppose their expected gains are 0.120.12 and 0.100.10 and their independent variances are 0.040.04 and 0.010.01. With λmix=1\lambda_{\mathrm{mix}}=1, the objective peaks at w=0.60w=0.60. Six of ten trials go to finding failures. Here is what those illustrative numbers buy.

Adversarial / cooperative shareExpected gainVariance of weighted estimateObjective
100% / 0%0.1200.04000.1000
60% / 40%0.1120.01600.1040
0% / 100%0.1000.01000.0950

When two estimates rise and fall together, the variance includes 2w(1−w)Σ122w(1-w)\Sigma_{12}. That can happen when both sets of practice cases omit the same restart bug. Holding out an independent audit is useful because a budget chosen from those estimates may never notice that omission.

Let the evidence update the allocation

A schedule can change when new evidence arrives. Let yty_t be the pair’s settings at round tt. The score J(yt)J(y_t) measures remaining trouble, such as missed failures or broken limits, so smaller is better. TmT_m means try one block of mode mm. The net gain rt(m)r_t(m) is the fall in trouble minus the compute cost cmc_m and any charge for switching from the previous mode mt−1m_{t-1}. The weights λc\lambda_c and λs\lambda_s price compute and switching. The indicator 1\mathbf 1 switches on the latter charge when the mode changes. The hats denote estimated gains, pt,mp_{t,m} is the current share, and η\eta controls how strongly shares react.

rt(m)=J(yt)−J(Tm(yt))−λccm−λs1{m≠mt−1},pt+1,m=pt,mexp⁡(η r^t(m))∑jpt,jexp⁡(η r^t(j)).\begin{aligned} r_t(m)&=J(y_t)-J(T_m(y_t))-\lambda_c c_m-\lambda_s\mathbf 1\{m\ne m_{t-1}\},\\ p_{t+1,m}&=\frac{p_{t,m}\exp(\eta\,\widehat r_t(m))} {\sum_jp_{t,j}\exp(\eta\,\widehat r_t(j))}. \end{aligned}

If auditing has two of every ten turns, its odds against cooperation are 0.20/0.80=0.250.20/0.80=0.25. Suppose an extra audit is estimated to help by 0.300.30 more than a cooperative turn. At η=1\eta=1, multiply its odds by e0.30≈1.35e^{0.30}\approx1.35. That takes the audit share to about 25 percent. This schedule change is gradual. A confirmed serious failure can still trigger a required audit right away.

For a more formal check, compare the controller with a fixed mix p⋆p^\star that can reliably reduce trouble on the states it visits. Suppose the mix’s expected gain p⋆⋅rtp^\star\cdot r_t is at least γJ(yt)−ε\gamma J(y_t)-\varepsilon. Here γ>0\gamma>0 sets how strongly the gain scales with trouble, and ε\varepsilon allows for a gap. Let δ\delta bound the per-mode estimation error and ξt\xi_t be the extra drift or noise in a round’s progress. With bounded estimates available for all modes in the set M\mathcal M, the update falls behind the fixed mix by order Tlog⁡∣M∣\sqrt{T\log|\mathcal M|} on the estimated gains over TT rounds. The O(⋅)O(\cdot) term below represents this comparison cost up to a constant.

1T∑t<TEJ(yt)  ≤  J(y0)+O(Tlog⁡∣M∣)+∑t<TEξtγT+ε+2δγ.\frac{1}{T}\sum_{t<T}\mathbb E J(y_t) \;\le\; \frac{J(y_0)+O(\sqrt{T\log|\mathcal M|})+\sum_{t<T}\mathbb E\xi_t}{\gamma T} +\frac{\varepsilon+2\delta}{\gamma}.

The first fraction counts the starting trouble, the cost of learning which mix to use, and accumulated noise. The second fraction shows why better scheduling can hit a floor if feedback stays wrong. A class can keep rearranging its schedule and still miss the crash if nobody ever switches off the machine.

Check the full cycle for interference

Suppose crash testing fixes a bug, then the next repair accidentally brings the bug back. Check the machine after both activities. Call the remaining combined trouble WtW_t after cycle tt. It includes the adversarial gap and the amount by which the cooperative score Ψ\Psi falls short of its target Ψ⋆\Psi^\star, weighted by λΨ\lambda_\Psi. If Wt+1≤κWt+ξtW_{t+1}\le\kappa W_t+\xi_t, then κ\kappa is the fraction that remains after useful work and ξt\xi_t is what the cycle adds back. If ξt≤δWt\xi_t\le\delta W_t and κ+δ<1\kappa+\delta<1, then Wt≤(κ+δ)tW0W_t\le(\kappa+\delta)^tW_0. With κ=0.8\kappa=0.8, δ=0.1\delta=0.1, and W0=1W_0=1, at most 0.910≈0.3490.9^{10}\approx0.349 remains after ten cycles. To apply that reasoning, measure the whole cycle, including what the next update undoes.

Open Questions

What if a teaching move pays off five updates later?

A hint about reusing a worksheet number might do nothing on the next exercise. Its value shows up later, when the machine learns to handle library loans. Start two copies from the same settings. Give one the curriculum hint and the other a repair session of equal cost. Then give both the same four follow-up batches. Compare them after one update and again after five on unseen failures. Include the cost of running both copies when judging whether the longer lookahead was worth it.

Can we detect a bad sequence before either model looks worse?

Change a small part of one model at a time and measure how both models respond. A two-by-two matrix AmA_m can record those four responses for mode mm. For successive modes m1,…,mhm_1,\ldots,m_h, the product Amh⋯Am1A_{m_h}\cdots A_{m_1} estimates the combined effect of the whole sequence near that checkpoint. Its largest singular value measures the largest factor by which it can magnify a small change. Plant a known blind spot and test whether that estimate warns early enough to justify calling for an audit.

Where should the stability claim stop?

Compare four runs with the same cases and compute budget: initialization only, one mode at a time, fixed alternation, and a learned schedule. Keep a second audit set untouched until the settings are chosen. Measure false approvals on severe failure families as well as ordinary task success. The calculations here give conditions under which specific updates make progress. The comparison would show how often real training satisfies those conditions.