記事一覧

Measuring Long-Running Coding Agents, Part 1: Surviving Code, Outside Scorecards, and the Micro-Task Trap

2026年11月19日

#Artificial Intelligence#Software Engineering#ai agents#Devops#metrics
Measuring Long-Running Coding Agents, Part 1: Surviving Code, Outside Scorecards, and the Micro-Task Trap

In our last post, What Are We Actually Getting for Our AI Coding Spend?, we wrote for CEOs and CFOs about measuring value delivery and what it costs. This three-part series is for the people who build and run these systems: engineering leads, staff engineers, and anyone wiring up their own agent harness. It covers where those numbers came from, how we compute them, and what surprised us along the way.

Some context on scale. Our projects routinely run for days, across thousands of agent turns, with a person stepping in only a handful of times. Over one 55-hour stretch on a single project, ETC, our engineering harness, ran 783 autonomous coding, testing, and review passes across 65 tasks, spent over 250 million tokens on independent code and architecture reviews alone, and integrated 32,000 lines across 339 files on 14 human inputs. Over the past three months we audited 10,265 ledger events, 637 production tasks, and 105,791 automated gate checks as ETC built and extended itself. When you sit in the loop every thirty seconds, your own judgment hides a lot of sins; when agents run unattended at that scale, the wrong metrics will burn hundreds of millions of tokens while quietly wrecking your architecture.

What This Series Covers

  • Part 1 (this post): why activity metrics mislead on multi-day builds, how we score a build from evidence instead of an agent's summary, and how splitting work too finely makes review numbers look better while delivery gets worse.
  • Part 2: Return on Token, the metric we built from those lessons; why rework costs grow quadratically; and why we cap automated review at three rounds.
  • Part 3: why upfront design pays back across a long build, how we account for human attention separately from token spend, and what we're instrumenting next.

The 99.1% Mirage

In September, we audited 212 autonomous coding agent runs across a weekend of real feature work. Across the whole fleet, agents succeeded on 98.2% of their tool turns: reading files cleanly, editing code cleanly, invoking test runners without crashing. Yet only 51% of total effort ever merged and stayed merged. Four rejected runs burned 507 turns at 99.1% efficacy and left zero surviving lines.

To understand why busy, locally green agents fail so reliably on multi-day horizons—and what to measure instead—you have to look at the physical structure of a 50-task build.

The Tunnel Problem: Pulling Threads of Value Through 50 Heavy Blocks

Start with what actually happens during a large software build.

You begin on the left with a clear value hypothesis: two or three continuous threads of real behavior that a user or system needs. Neither a human engineer nor a frontier model can land a 30,000-line change in one shot, so engineering has to decompose that whole idea into pieces—specifications, module interfaces, construction stages, and 30 to 50 bounded implementation tasks.

Picture 50 heavy concrete tunnel segments lined up in a row, with three thin, glowing fiber-optic threads of user value trying to make it from Segment 1 all the way out Segment 50.

Every boundary between two tasks is a splice in that fiber-optic cable:

  • Local green, global zero: Left alone, AI workers optimize locally. Worker 14 bends a return type slightly to turn its unit test green. Worker 28 mocks around the mismatch. All 50 workers report green checkmarks—and the thread of user value snapped silently back at Segment 14.
  • More cuts create more splice loss: Every extra cut introduces another seam where intent attenuates, temporary shims creep in, and tokens burn re-reading context. Cutting a coherent 4-task change into 41 micro-tasks puts 40 splices into the cable instead of 3.

Two rows of concrete blocks with three gold threads of value running through them. Top: 41 micro-tasks with 40 splices; every block shows a green check, but one thread snaps early, so 2 of 3 threads exit. Bottom: 4 right-sized tasks with 3 locked splices; all 3 threads exit the far end.

Seen this way, an engineering harness works less like a task queue and more like a tensioning rig, built to pull those thin threads of value through 50 heavy blocks without snapping them:

  1. Name each thread going in as numbered, observable acceptance criteria and a recorded value hypothesis.
  2. Pull one thin thread all the way through first (Stage 1's vertical reference slice) and prove light exits the far end before building breadth.
  3. Lock every splice with explicit module interface contracts—and give workers a zero-penalty contract-conflict savepoint (ADR-0049) when two ends refuse to mate, instead of rewarding them for taping over the gap with a shim.
  4. Cut as few blocks as an agent can reliably carry using an architect's 1–10 complexity dial, minimizing splice loss.
  5. Test the threads exiting Segment 50, ignoring how busy the 50 workers were inside the tunnel.

Never Let an Agent Write Its Own Scorecard

Because an LLM asked to summarize its own build will always write good news, we banned agents from authoring scorecards (ADR-0041).

In ETC, etc tasks scorecard is a read-only command computed strictly from task records, the append-only ledger, verification files, and git commits. An automated invariant (INV-027 in invariants.md) enforces three rules on that output:

  • No composite 0–100 health scores and no color badges: Single grades hide trade-offs and invite Goodhart gaming.
  • Lines of code are a receipt, never progress: Net lines reward code bloat when high and reward unreadable code-golf compression when capped.
  • Never print 0 when evidence is missing: 0 blocking findings means the architecture check ran and found none. If the check never ran, the scorecard prints could not compute (and null with a reason in JSON), because a 0 for an unrun check masquerades as a pass.

Here is an unedited etc tasks scorecard output from Feature 207.1 (transactional-scope-harmony), the opening feature of our current Epic B build:

Criteria met: 15 of 15 (the verifier's verdicts in F-2026-11-15-030711-transactional-scope-harmony-and/verification.md)
Exceptions:
  002-shared-scope-module review rounds 3 of 3
Receipt:
  lines: 4620 added, 389 removed (git diff of 11 accepted attempts, start to accepted commit)
  new files: 17 (the same 11 diffs)
  unauthorized boundary crossings: 0 blocking findings (the architecture check in the work's check run of 2026-11-15T15:15:16Z)
  escalations decided: none (ledger answers)
Task flow (Build -> Check -> Review):
  tasks completed: 11 of 11 tasks (100%) · 11 on 1st attempt, 0 retried, attempt cap 2 (task records)
  review cycles: 15 total cycles across 11 tasks · 1.36 cycles/task (ideal 1.00) · 73% cycle pass rate, 11 of 15 (task records)
  Cycle 1 (initial): 11 entered -> 8 accepted (73%) · 3 sent back · cumulative 8 of 11 (73%) (task records)
  Cycle 2 (rework 1): 3 entered -> 2 accepted (67%) · 1 sent back · cumulative 10 of 11 (91%) (task records)
  Cycle 3 (rework 2, cap 3): 1 entered -> 1 accepted (100%) · 0 sent back · cumulative 11 of 11 (100%) (task records)
  Attempt 2 (retry, cap 2): 0 entered -> 0 accepted (—) · cumulative 11 of 11 (100%) (task records)
Intent proof (verdict, evidence, test):
  AC-1 met · verification.md:134 · tests/architecture/test_etc_model.py::test_etcs_model_holds_the_ten_components_each_in_its_own_capability_file
  AC-2 met · verification.md:135 · tests/integration/test_layer_rules_architecture.py::test_rule_23_should_pass_an_engine_module_when_it_imports_the_package
  AC-3 met · verification.md:136 · tests/engine/test_moves_scope.py::test_should_return_no_owners_when_the_project_has_no_system_file
  ... and 12 more
ETC leverage:
  operator inputs: 3 (1 request, 1 go-ahead, 1 answer) (work state, ledger)
  autonomous runs: 116 (11 worker attempts, 4 rework cycles, 70 check runs, 31 judge runs) · 38.7x per operator input (task records, work state)
  tokens: 32,007,482 input (27,797,166 cached), 424,488 output (judge records)

Notice how separating first-attempt completion (11 of 11, 100%) from Cycle-1 review pass rate (8 of 11, 73%) immediately exposes rework that "100% tasks completed" would hide: three tasks needed review fixes before merging, and Task 002 went all the way to Round 3 of 3.

The Review Funnel Paradox: Why Micro-Tasks Feel Fast and Ship Slow

Even that review funnel can fool you if you look at it without token cost and plan size.

In October, we asked ETC to swap our terminal UI layer (~900 lines inside one package) onto Textual—about a 20-minute job for one experienced engineer. Because our old task-size formula only ever forced splits and never pushed toward fewer, direct tasks, the planner turned a 900-line library swap into a 595-line spec, a 1,622-line design, and 23 tasks that ballooned into 41 packets over 35 hours—including two later tasks whose sole job was deleting temporary layouts that earlier tasks had left behind just to pass isolated tests.

Look at the trap when you compare 207.1 against that Textual swap:

Build Accepted Tasks Build Elapsed Cycle-1 Review Pass Rate Cycles / Task
207.1 (Cohesive scope) 11 2h 55m 73% 1.36
Textual swap (Over-shredded) 41 35h 00m 85% 1.24

The wasteful 35-hour build posted better review-funnel numbers (85% vs. 73%)! When you chop a 20-minute change into 41 trivial shards, each tiny shard breezes past review while the project burns two days and tens of millions of tokens in splice overhead.

That paradox led directly to our equation for Return on Token, which is where Part 2 picks up: what it measures, why rework hurts it quadratically, and why we stop patching after three review rounds.


AI attribution: 3/4 — Human-originated, AI-shaped. What this means

Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email info@heavychain.org.