Abstract
Agents that write code also grade it, and that is a problem.
We built a verifier that takes a proposed change, applies it to a real source tree, and runs tests the generating model never saw, inside a sealed sandbox. We used it to measure fifty cross-file coding tasks in Python and TypeScript, where the correct fix depends on a contract stated somewhere other than the file being edited.
Two findings. A model configuration costing 2.57 times more was not measurably better at this work: 38 of 50 against 36 of 50, with overlapping confidence intervals. And when a model wrote its own tests alongside its fix, roughly a quarter of its self-reported successes were wrong, with its own tests passing on the mistake. The more expensive configuration did this too.
Below: the method, the numbers, the results that did not hold up, and where the evidence is thin.
Research
Fifty repo-scale tasks, two model configurations (cheap and expensive), 95% Wilson intervals. The live tool tries the change against your files and tells you whether it fits. The numbers below come from a separate measurement setup that graded against hidden suites the generating model never saw.
Cheap passed 36/50 = 0.720 [0.583, 0.825]. Expensive passed 38/50 = 0.760 [0.626, 0.857]. The intervals overlap. McNemar p = 0.50. Expensive cost 2.57×. Paying more was not distinguishable from cheap on this catalog.
When the agent reported success, a share of those reports were wrong, and the agent's own tests agreed with the mistake: cheap 10/38 = 0.263 [0.150, 0.420]; expensive 6/26 = 0.231 [0.110, 0.421]. The intervals overlap. Spending more did not remove it.
On this catalog, verification mean 0.34s, p50 0.20s, p99 0.82s (n = 50). That is this catalog. A large existing suite can be much slower; the sandbox wall used in measurement is 8s.
Routing among configurations is frozen. Each call stores enough to compute a pre-registered reward later. The policy is not updated from those rows. A learner that retunes after seeing production results is out of scope.
On a handful of uncovered cross-file contracts, the repo's own suite passed a wrong fix at least once. That is an observed failure mode, not a rate: the honest set after polarity was six tasks, which is too small to quote as a frequency. The live product tries the change you send against your files; it does not replay that hidden-suite grade.
Uses the catalog figures on this page. It does not choose a model for you, and it does not claim the hosted tool prevents the losses it displays. The hosted path tries the change against your files and tells you whether it fits.
Comparison of published model settings on this catalog. This product does not pick anyone's model, so the difference is not a saving it delivers.
Cheap: 1,000 × $3.90 / 1000 = $3.90
Expensive: 1,000 × $10.01 / 1000 = $10.01
Difference: $10.01 − $3.90 = $6.11
On this catalog of 50 tasks, the more expensive configuration was not measurably better (McNemar p = 0.50).
The rate 10/38 = 0.263 comes from 50 repo-scale cross-file contract tasks (self-reported successes that failed a hidden suite). It will differ on other codebases. This sizes a problem. It does not promise a saving. The hosted path tries the change you send against your files; it does not run that hidden suite.
1,000 × 10/38 (0.263) ≈ 263.2 changes that reach a reviewer already wrong while reporting success.
No review time entered, so no time total is shown.
