#1 Double Helix of Intelligence (Part I)
Controlled Generator-Evaluator Co-evolution under Underspecified Rewards
Controlled Generator-Evaluator Co-evolution under Underspecified Rewards
Imagine a classroom sticker machine. Its job is to give one sticker to each child who finishes a worksheet. Mina finishes hers. The machine drops a sticker into her hand and freezes before it writes her name in its notebook. When the teacher presses Retry, the machine gives Mina another sticker. A check that only asks “Did Mina get a sticker?” gives it full credit. The classroom rule was one sticker per worksheet.
Now imagine two students improving the machine. One writes instructions for it. The other tries those instructions and gives feedback. If the checker always asks whether a sticker came out, the writer will get better at making stickers come out. To learn to handle a restart, the pair needs a test that turns the machine off at that exact moment. In an AI system, the generator proposes an answer or an action. The evaluator checks it and supplies a score, a repair, or a reason to ask for help.
The same mistake can send a customer two refunds. A service can send the money and lose its connection before recording the receipt. A safe retry uses the same refund number, and the payment provider must recognize that number. An underspecified reward gives credit for part of the job while leaving out a rule that matters. Every time the two models learn, they change what the other one sees next. That is the double helix: better answers need better ways to check them, and better checks change which answers get made.
Think of each letter as a label on a box. is the task, such as “give Mina one sticker.” is one proposed set of instructions. The Greek letter (tau) is the trace, an ordered record of what the machine did. The generator produces a proposal, with (theta) standing for its adjustable settings. The evaluator (phi) reads the task, proposal, and trace, then estimates how likely the proposal is to be correct.
A reliable test gives us , the observed correctness label: for a pass and for a fail. If a test is unavailable, is unknown. is an estimate between and . In a run with an available test, a pass and a checker score of can be blended. With evaluator weight , the blended score is . The two weights add to . Turning up gives the checker more influence.
The two models have different scores to improve, for the generator and for the evaluator. means take an average, including over the proposals the generator tends to make. The subscript means averaging over tasks alone. The generator pays for drifting from a trusted reference . The distance measure compares how two policies spread their probabilities. The evaluator pays a loss when its predictions miss the checked labels and a calibration penalty when its confidence does not match its track record. The two terms set the strengths of those penalties.
Before the pair starts teaching itself, give each model a minimum score to meet. If a checker approves three of ten known bad solutions, its false approval rate is 30 percent. A 10 percent cap allows only one. A strong generator score cannot make that checker pass its own gate. Test these gates on cases set aside from training, including retries and restarts.
Pack both models’ settings into . We minimize the loss by taking the negative of the two scores, weighted by and . The minimum scores are and . Those are thresholds, whereas without a subscript is the trace. A constraint written as passes when its value is zero or negative. Each is an extra limit, such as the cap on false approvals or a limit on how far the generator can drift.
| Gate on separate validation data | Target | Illustrative result | Consequence |
|---|---|---|---|
| Generator passes unseen workflow checks | At least 75% | 82% | Meets its floor |
| Evaluator approves independently identified invalid patches | At most 10% | 18% | Reliability constraint is violated |
| Average KL divergence from reference policy | At most 0.05 | 0.03 | Meets the drift budget |
Once these gates are separate, we can ask two questions. With a generator fixed, how strong is the best checker we can train for it? With a checker fixed, how well does it do on the hardest generator that meets the capability threshold? The first asks whether a new skill can still be checked. The second asks whether the checker will keep up as the writer improves.
The students can use the same afternoon in four ways. They can try to break the machine by switching it off at the worst moment. They can help fix it by showing how to reuse the same worksheet number. They can teach that idea on easy examples and try it later on a different machine. Or they can ask a teacher to count the real stickers when the machine’s own notebook is unclear. Those are four different jobs because each answers a different question.
| Mode | Quantity it acts on | Refund example |
|---|---|---|
| Adversarial | Independently verified failures the evaluator detects | Kill the service after the debit and before it records the receipt |
| Cooperative | A shared potential for generator progress and evaluator quality | Propose persisting the operation identifier before sending the debit |
| Curriculum | Improvement on new tasks after a feedback-driven update | Teach durable identifiers on one failure family and test transfer to concurrent refunds |
| Auditing | Severity of missed failures, false blocks, and abstentions | Reconcile the ledger on unusual restart cases and send uncertain cases to a human |
Find a failure. In adversarial mode, the checker earns credit for catching a mistake that a separate test confirms. The discovery score averages this credit. The first factor switches the credit on only for a verified failure. An evaluator score of earns if the test says , and earns zero if the test says . Calling a working proposal broken still increases the evaluator’s prediction loss.
Help with a repair. In cooperative mode, both sides try to raise a shared score (psi). The weights and set the value of improving each model. The symbol means the direction in which a score increases fastest. converts the evaluator’s direction into the generator’s coordinates so the two directions can be compared. The last term charges for disagreement, with setting the price and the squared length measuring its size.
If each side could find the best update to its own settings while holding the other fixed, the old settings would still be an available choice. The shared score could only stay the same or go up. A noisy training step needs a fresh check to see whether it actually helped.
Teach for the next task. A hint might say “keep the old worksheet number when you retry.” We want to know whether it helps on a new task, such as a library checkout that freezes before saving. The superscript means after an update. (eta) is the size of the teaching step, and the hat on the training gradient means it is estimated. The labels “train” and “val” refer to the practice cases and the separate validation cases. is the change on those new cases.
For a small step, the gain depends on the dot product of the teaching direction and the direction that improves the new tasks. Two arrows pointing the same way have a positive dot product. In symbols, the gain is approximately , where the angle brackets denote that dot product. Giving a student the answer to today’s worksheet can make today’s score jump while doing little for tomorrow’s worksheet. Give the hint credit when the student can solve a new problem.
Audit serious misses. An audit can accept, reject, or pause for help. Call that decision . A missed failure has loss , where measures its severity and switches on only when both conditions are true. is the average loss in the worst fraction of cases. The formula tries a loss cutoff (nu) and charges for losses beyond it. means the larger of and zero.
Suppose 99 of 100 patches cause no harm and one causes a loss of 100. The overall average is 1. In the worst five cases the average is . The second number gives the missed failure the attention it deserves even when the batch looks good on average.
Picture ten minutes to split between writing new instructions and checking them. If checking must get at least six minutes, spending eight minutes on writing breaks the rule. A multiplier acts like a price for continuing to break a rule. Its price can rise each time the same problem appears.
Put the constraint values into a list and give each its own nonnegative price in the list (lambda). is the original loss plus the sum of prices times constraint values, written . The index counts updates. The step sizes and set how far the model settings and the prices move. keeps the settings inside their allowed region, and stops any price from dipping below zero.
If false approvals stay eight percentage points above the cap for five steps and , the price rises by . The rising price tells the optimizer to take the violation more seriously. Getting the failure rate down still depends on examples that teach the checker what it missed.
Use for the generator score and for the evaluator score. Both would ideally reach , measured by the squared loss . With a shared budget and a checker floor , the best feasible pair is . At that point the gradient of the loss is . The budget and floor exert prices of and along directions and . Their sum cancels the loss gradient.
Both rules are tight. Under convexity, bounded gradients, and the usual feasibility and step-size assumptions, the averaged gap in this procedure shrinks on the order of over updates. A trained checkpoint still needs to pass the original gates on fresh cases.
Suppose two students are building opposite ends of the same bridge and each gets to choose a height. One wants eight blocks and the other wants four. They can work separately, but their ends must eventually meet. Simply telling them to agree gets them no closer to knowing which adjustments cost each side the least.
In the alternating direction method of multipliers, or ADMM, and are the two copies of the shared setting. Each side has its own loss, or , with its local rules included. means choose the input that makes the expression smallest. A superscript indicates the newest value. prices disagreement, and remembers accumulated disagreement from earlier rounds.
Take local losses and . With and starting values , , the first round gives and the next gives . The gap falls from to , and the second side moves by on that next round. Track both the remaining gap and the movement . A small gap tells us the two sides match today, while a small movement tells us the target is settling. The convergence result requires convex, closed local problems, an existing saddle point, and exact solves or errors whose sum stays finite.
Imagine two kids moving a marker on graph paper. One controls the left–right move and one controls the up–down move. Each reacts to the other’s last position. Even when both want to follow the rules, their moves can send the marker circling away from the middle.
In a two-model game, is their current joint position. stacks the negatives of the two gradients, one for each player’s own score. With step size , an ordinary update moves using the directions at . Extragradient first takes a trial step to (the tilde marks a provisional value). It measures the directions there, then uses them to make the real move from to .
Try a game in which one player minimizes and the other maximizes it. For the update direction is . Starting at with , an ordinary update goes to , farther from the center. Extragradient visits that same point as a trial, sees , and finishes at , closer to the center. The distance factors are and . The example isolates rotation. A general linear convergence claim needs a direction field that changes at a bounded rate, called Lipschitz continuity, and a restoring effect that stays strong throughout the region, called strong monotonicity. The steps must stay inside that region and a stopping point must exist there.
What if we mix several ways of updating? Let be the direction field for mode , any two states, and the fixed share given to that mode. The inner product measures how much the difference in directions points along the displacement between the states. The constant is the minimum strength of this restoring effect, and caps how quickly the direction can vary.
Taking the weighted sum gives a mixture with restoring strength at least and variation at most times the distance moved. Its stopping point can land somewhere new. The directions and stop at and . Their equal mixture is , which stops at . Alternating the two updates in time changes which state each direction sees.
If a class has time for ten trials, should it use all ten to try to break the machine? A crash test may reveal more per attempt than a repair session, but we might be less sure of that estimate. The controller can spend some of its budget on the more predictable work.
The list holds the shares, all at least zero and together summing to one. holds the estimated gains on new cases. is the covariance matrix. Its diagonal entries measure the uncertainty in each estimate, and the other entries measure how much the estimates move together. The dot product is the expected gain of the mix. The quadratic term is the mix’s variance. sets how much that uncertainty costs.
For two modes, let be the share for finding failures and the share for repairs. Suppose their expected gains are and and their independent variances are and . With , the objective peaks at . Six of ten trials go to finding failures. Here is what those illustrative numbers buy.
| Adversarial / cooperative share | Expected gain | Variance of weighted estimate | Objective |
|---|---|---|---|
| 100% / 0% | 0.120 | 0.0400 | 0.1000 |
| 60% / 40% | 0.112 | 0.0160 | 0.1040 |
| 0% / 100% | 0.100 | 0.0100 | 0.0950 |
When two estimates rise and fall together, the variance includes . That can happen when both sets of practice cases omit the same restart bug. Holding out an independent audit is useful because a budget chosen from those estimates may never notice that omission.
A schedule can change when new evidence arrives. Let be the pair’s settings at round . The score measures remaining trouble, such as missed failures or broken limits, so smaller is better. means try one block of mode . The net gain is the fall in trouble minus the compute cost and any charge for switching from the previous mode . The weights and price compute and switching. The indicator switches on the latter charge when the mode changes. The hats denote estimated gains, is the current share, and controls how strongly shares react.
If auditing has two of every ten turns, its odds against cooperation are . Suppose an extra audit is estimated to help by more than a cooperative turn. At , multiply its odds by . That takes the audit share to about 25 percent. This schedule change is gradual. A confirmed serious failure can still trigger a required audit right away.
For a more formal check, compare the controller with a fixed mix that can reliably reduce trouble on the states it visits. Suppose the mix’s expected gain is at least . Here sets how strongly the gain scales with trouble, and allows for a gap. Let bound the per-mode estimation error and be the extra drift or noise in a round’s progress. With bounded estimates available for all modes in the set , the update falls behind the fixed mix by order on the estimated gains over rounds. The term below represents this comparison cost up to a constant.
The first fraction counts the starting trouble, the cost of learning which mix to use, and accumulated noise. The second fraction shows why better scheduling can hit a floor if feedback stays wrong. A class can keep rearranging its schedule and still miss the crash if nobody ever switches off the machine.
Suppose crash testing fixes a bug, then the next repair accidentally brings the bug back. Check the machine after both activities. Call the remaining combined trouble after cycle . It includes the adversarial gap and the amount by which the cooperative score falls short of its target , weighted by . If , then is the fraction that remains after useful work and is what the cycle adds back. If and , then . With , , and , at most remains after ten cycles. To apply that reasoning, measure the whole cycle, including what the next update undoes.
A hint about reusing a worksheet number might do nothing on the next exercise. Its value shows up later, when the machine learns to handle library loans. Start two copies from the same settings. Give one the curriculum hint and the other a repair session of equal cost. Then give both the same four follow-up batches. Compare them after one update and again after five on unseen failures. Include the cost of running both copies when judging whether the longer lookahead was worth it.
Change a small part of one model at a time and measure how both models respond. A two-by-two matrix can record those four responses for mode . For successive modes , the product estimates the combined effect of the whole sequence near that checkpoint. Its largest singular value measures the largest factor by which it can magnify a small change. Plant a known blind spot and test whether that estimate warns early enough to justify calling for an audit.
Compare four runs with the same cases and compute budget: initialization only, one mode at a time, fixed alternation, and a learned schedule. Keep a second audit set untouched until the settings are chosen. Measure false approvals on severe failure families as well as ordinary task success. The calculations here give conditions under which specific updates make progress. The comparison would show how often real training satisfies those conditions.