記事一覧

Two Quiet Results, One Shared Lesson: Where Capability Gains Are Coming From Now

2026年6月3日

#AI#Software Engineering#claude#Formal Verification#agents
Two Quiet Results, One Shared Lesson: Where Capability Gains Are Coming From Now

Two pieces of news showed up this week that are easy to miss in isolation but worth reading together. The first is a developer demonstration: an apparent Opus 4.8 model produced a formally verified proof of a polygon intersection algorithm, a task that earlier Claude generations had failed for more than a year. The second is an arXiv paper (2605.29682) that proposes scaling laws for agent harnesses, introducing a quantity the authors call Effective Feedback Compute.

Each result is interesting on its own. Read together, they outline where capability gains in production AI systems are actually coming from in 2026, and why the picture is more complicated than "bigger model equals better outcomes."

It helps to start with what formal verification actually means, because the term gets used loosely. A formally verified proof is not a passing test suite. It is a machine-checkable argument, written in a language like Lean or Coq, that a piece of code satisfies a mathematical specification in every case, not just the cases your tests happened to cover. For geometry primitives like polygon intersection, the proofs involve case analysis over a large number of geometric configurations and require the prover to maintain a consistent set of invariants across each branch. The reason earlier Claude models failed at this is that they could produce plausible-looking proof sketches but could not maintain the structural discipline that a verifier demands. The verifier does not care about plausibility. It either accepts the proof or rejects it.

If Opus 4.8 is succeeding here, the meaningful claim is not "the model is smarter in a fuzzy sense." It is that a measurable capability — sustained logical discipline across a multi-step structured proof — has reached a threshold that earlier models did not clear. This is a useful kind of capability to track because it cannot be gamed by training on benchmark answers. The verifier is independent.

The arXiv paper is a different kind of result. The authors study what happens when you fix the base model and scale the surrounding harness: the number of self-critique passes, the number of independent samples, the depth of the verification step, and so on. They find that, for a class of complex tasks, the relationship between harness compute and task success rate fits a clean scaling law. Doubling the compute spent on iteration and checking produces predictable improvements in success rate, and for many tasks the curve is steeper than the corresponding curve for scaling up the base model.

If that result holds up under replication — which is a meaningful caveat for any single arXiv paper — it changes how teams should reason about where to invest engineering effort. For most of the last three years the implicit assumption has been that smarter agents require a smarter base model. The paper suggests that the harness is its own first-class capability surface, and that investment in feedback loops, verifiers, and structured iteration scales in a way that is comparable to investment in the underlying model.

The connection between the two results is worth being explicit about. The formal verification result demonstrates that single-pass capability has moved upward. The scaling-laws result demonstrates that the multi-pass architecture around the model is itself an axis of capability. Either one alone is a data point. Together they describe a frontier with two dimensions, both of which are moving and both of which are worth instrumenting.

For practitioners, two reasonable habits follow. First, when a new model generation appears, re-run your hardest synthesis tasks against it before assuming the previous-best model is still the right default. The geometry result suggests that some thresholds genuinely shift, and you may be paying for a fallback you no longer need. Second, when you are budgeting engineering time, do not treat the harness as commodity glue. The evidence is accumulating that the harness is one of the highest-leverage places to put senior attention. Spec-driven development, verifier-in-the-loop patterns, and structured self-critique are not stylistic choices anymore. They are the architectural features that the scaling law is rewarding.

This week's two results are quiet ones. They do not have the splash of a new model launch. They are, however, the kind of foundational evidence that tends to reshape architectural decisions over the following two or three quarters. Worth reading the underlying material directly rather than just the summaries.