
We think the AI SDLC is a phase transition in how software gets built, more than another tooling upgrade. Think ice turning to water, or the familiar innovation S-curve: a stable baseline, a steep transition, then a new baseline. We're in the middle of that curve right now. Things are changing fast, nobody has all the answers, and everyone is learning.
Teams are working through it in different ways. Most agentic coding today still happens in a 15-turn chat loop where a human inspects every diff live. Others use frameworks like ECC, Superpowers, GSD, BMAD, or Spec Kit. Those are good tools, and much better than nothing, but they take a different approach than we do to getting consistent, disciplined leverage out of coding agents. At Heavy Chain Engineering, we've been focused on building bigger, better, and more reliably at scale for years. Our projects routinely run for days, across thousands of agent turns, with a person stepping in only a handful of times. For example, over the past 55 hours, on a single project, our engineering harness ran 783 autonomous coding, testing, and review passes across 65 tasks, spent over 250 million tokens on independent code and architecture reviews alone, and integrated 32,000 lines across 339 files on just 14 human inputs.
When you sit in the loop every thirty seconds, your own judgment hides a lot of sins; when agents run unattended at that scale, naive metrics will burn hundreds of millions of tokens while quietly wrecking your architecture. So we extracted our metrics, analyzed them, and are sharing a few things we learned.
What We Found
As AI compute and token spend grow into a material R&D and COGS line item, standard engineering activity metrics—lines of code generated, commits pushed, and vendor "agent success rates"—create a dangerous accounting blind spot. We took a beat to audit 637 production tasks and hundreds of millions of tokens across multi-day autonomous software builds to see what we could learn. We found four rules for long-running agentic builds that bear directly on the cost of delivering value:
Effort ≠ Outcomes: When we audited an earlier weekend of work, back in September, our AI agents reported success on 98% of their individual steps, yet only 51% of the work they produced made it into the shipped product and stayed there! Four of those runs completed 507 steps at 99.1% success and shipped nothing, so when an AI coding vendor quotes a 97–99% success rate, ask what it measures. Gaps like that are why we measure constantly, and why every significant failure gets an extra, deeper look. By our November builds, the 55-hour build verified 127 of 129 of its requirements. Financial governance requires auditing Return on Token ()—verified product requirements delivered per dollar of compute—never raw token burn or code volume.
Upfront Specification Spend Acts as Compounding CapEx, Not Overhead: Across long, 50-to-65-task initiatives, tokens spent upfront on locked architecture and an initial end-to-end reference slice behave like capital expenditure in a Discounted Cash Flow (DCF) model. Underfunded upfront design leads to compounding rework debt that stalls large builds completely; disciplined upfront design coincided with our first-pass forward efficiency increasing from 57% to 89% even after adding lines of code.
Quality Slips Carry an Inverse-Square Cost Penalty (): In autonomous engineering, rework does not add linear cost—it degrades unit economics quadratically because failed cycles reduce forward yield while consuming additional compute. Even with heavy prompt-cache discounts (89%+ cache hit rates), tasks that drag into a third review round deliver lower return per review token than clean first-pass work.

Account for Human Time Separately: Track your engineers' time separately from your AI bill. Then you can choose to spend more on cheap AI work (reused, or "cached," AI input costs about 90% less than new input) so your senior engineers spend their time on decisions instead of routine work. On our builds, agents ran to for every time a person had to step in, and on the previous 72-task project, roughly 830 AI steps for every human decision.
What I Would Ask Engineering to Track and Share
If your teams run agents on real delivery work, these are the numbers I would want to see regularly, next to the token bill:
- Verified requirements delivered. Count acceptance criteria that an independent check verified as met, not tasks closed, commits, or lines of code. Our 65-task build verified 127 of 129 criteria.
- Cost per verified requirement. The review agents alone read 251 million input tokens, plus the building agents' tokens, which our harness tracks but we didn't capture for this build. To turn tokens into dollars, multiply each model's input, cached-input, and output tokens by that model's published price, add them up, and divide by the number of verified requirements. Keep cached input separate: 89.2% of our review input came from cache, which is billed at roughly a tenth of the normal rate, so a raw token count overstates the real cost.
- Human inputs per build. How many times a person had to step in. Ours for this project: 14 inputs across 783 autonomous runs.
Two Controls a Controller Will Recognize
- Segregation of duties. Never let the agent that did the work write the report on the work. Our build scorecards are computed read-only from git history, test results, and an append-only ledger, and they print "could not compute" instead of
0when a check never ran. Ask whether your team's AI metrics come from an independent record like that, or from the agent's own summary. - Spending limits on rework. When AI-written code fails review, the agents fix it and resubmit, and every round costs money. A hard cap of three review rounds per task, shared across attempts, keeps one flawed task from turning into an open-ended token sink; 96% of our 637 audited tasks passed inside that cap. When a task goes off track, we stop, rewrite its instructions, and start it over from a clean copy of the code. Those restarts succeeded 91.2% of the time (83 of 91), which costs far less than paying agents to keep patching work built on bad instructions.
What We're Improving
- Weighting requirements by value. Today every acceptance criterion counts the same, so a requirement that unlocks a customer contract and one that renames a settings field each count as one. Our harness already captures the raw material to fix this: work starts from user journeys, and every spec carries a value statement from the product manager describing what the work is expected to be worth. The next step is to carry that value statement through every stage, from spec to tasks to merged code, so we can see where value stalls or gets lost along the way, and so cost per verified requirement becomes cost per unit of value.
The math, the observations, and the gory details are coming in a series of more technical posts, Measuring Long-Running Coding Agents, for those who want them.
AI attribution: 3/4 — Human-originated, AI-shaped. What this means
Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email info@heavychain.org.

