In early July, a record's detail page on a platform I work on showed a "coming soon" placeholder for one of its sections. We'd shipped the feature behind that section in May. It had been live for two months and had never once worked.
The platform is a B2B SaaS product built by a small human team directing a fleet of AI coding agents: dozens of dispatches a day, background agents implementing while a conductor sequences the work. The incident cost me about two weeks of time and some real credibility. It's not a story about AI writing bad code. Everyone, human and machine, did something reasonable at each step. The failures lived in the seams: between design and delivery, between a rule and its enforcement, between what an agent reported and what was true. Standard quality checks don't look at those seams.
A schema with no writer
The feature introduced a two-tier data model. If you've worked with an ERP you know the shape: a parent entity, and under it one or more child records, each with its own sensitive fields. The design said, as a numbered business rule, that the sensitive fields belong to the child.
The migration ran. The columns existed: encrypted fields, verification timestamps, audit hooks. For eight weeks, nothing wrote to them.
The culprit was an older write path. Onboarding already had code that saved those details, and it predated the feature. It wrote to an old parent-scoped table. Nobody repointed it. The data importer set the new columns to NULL. So the platform ran with two homes for the same data: the old one got all the real data, and the designed one was empty from the day it shipped. A related field move also broke silently: a user could fill in a form, click save, get a 200, and the data landed where no page would ever read it.
Why every check passed
We don't run a loose shop. 98% backend test coverage, enforced. Lint on every push. Every PR reviewed. A runtime harness that won't mark a feature shipped until its behavior runs in the live app. All of it passed for six weeks, and all of it was working correctly. Each check just answers a different question than the one that mattered.
Coverage measures code that exists; a column with no writer adds no lines to cover. Lint checks form, and the schema was well-formed. Review judges each PR on its own terms, and each PR was locally fine. The runtime harness proves that wired-up behavior works; it can't flag behavior nobody wired, because it doesn't read the design.
The data was worse. The design had an invariant: every parent has at least one child record, always. We enforced it in code, on the creation path, with tests. But the backfill had only covered a small subset, and nothing audited existing rows. When we finally ran the query, 1,780 of 1,804 active parents had zero children. The rule held on every tested path and was violated in 98.7% of real rows.
A schema only says what data should exist. Data is only real where code writes it. Every check we had was measuring something else.
The design record was gone
Untangling this required the feature's design record: the spec, the decisions, the task breakdown, five draft ADRs. It was gone. Not stale. Gone, for twenty days, and nobody had noticed.
Our design artifacts lived in a directory that was untracked by git but not gitignored. That's the worst quadrant: no history, no remote copy, and destructive commands treat it as disposable clutter. 439 files sat in that state.
Then a background agent was dispatched to fix a migration conflict. Its prompt told it that it was in an isolated worktree. It wasn't. It was in the shared main checkout, and nothing verified the claim, though one git command would have. Blocked by another agent's uncommitted edits, it reached for the standard cleanup: git stash push -u, which stashes untracked files too. That swept the 439 files. Then it did something that looks careful: it ran git stash show, saw eight familiar tracked files, decided the stash held nothing important, and dropped it.
The trap is in how those files are stored. A stash made with -u keeps the untracked files in a third parent commit, and git stash show doesn't display the third parent at all. You can inspect the stash, see a short harmless diff, and get no hint that hundreds of files are riding along. Seeing them takes git show 'stash@{0}^3', which most people learn only after this exact burn. The agent didn't skip verification. It verified with an instrument that can't show the thing that was at risk.
The same hazard had fired three days earlier, and we caught it in time. We wrote the lesson into the agents' memory notes. Never git stash -u in this repo. The next agent never read the note, and nothing made it. Prevention written as prose lasted three days.
Recovery
Dropping a stash doesn't delete it. The commit dangles in git's object database until garbage collection removes it, and both stash commits were still there. We pinned them with tags, exported a bundle off-machine, restored all 439 files, and put the design artifacts under version control like we should have from the start. Data lost: none. Credit deserved: little. GC could have run on any of those twenty days.
Decisions made without the record
The missing record did more damage than the missing files. With the design gone, the empty state looked like the intended state. An agent saw the relevant records empty and wrapped up work that was maybe 60% done, because there was no document left to check it against. And every feature built in that window read the running code, took the old parent-scoped table as the truth, and built on it. The wrong model propagated.
The clearest case was an ADR. Working from summarized memory and live code, the reasonable-looking conclusion was that the old table was the design and the empty child tier was an abandoned experiment. An ADR ratifying the old table got drafted, reviewed, and merged. It contradicted the feature's central business rule. Review didn't catch it because there was nothing to check it against; the spec had been missing for three weeks, and that had come to feel normal.
What reversed it was a demand for evidence: point me back to the documentation. The recovered spec settled it in a day. We superseded the ADR with the opposite decision: keep the delivered machinery (the encryption and verification code was sound), re-key the data to the designed grain. The re-key ran as an expand-and-contract migration, and the destructive contract step waited for explicit human approval.
Cleaning up the propagated confusion was its own job, and it was genuinely hard, because the agents weren't uncertain. They were confidently wrong. The bad model was smeared across the whole project, so every piece of it corroborated every other piece, and they kept citing each other back to me. It took a dedicated repo-wide sweep to dig the assumption out of every place it had taken root.
New rule, mechanically checked: any ADR touching a shipped feature must cite the governing spec, or state plainly that none exists.
The agent fleet is a distributed system
The rebuild taught the last lesson. Agents kept finishing the implementation and then stopping one step short, at the gated push or at opening the PR. Six times in one day. Each time the conductor finished the step by hand.
The cause wasn't capability. We serialize database-heavy test runs because two suites on the shared Postgres flood each other, and we'd implemented the serialization as a token file agents were told to poll. But these agents are turn-based. Between turns, an agent isn't a running process at all. "Wait, then act" compiles to either a sleep loop that burns the context window, or ending the turn. And a background agent that ends its turn isn't waiting. It's finished, silently, with a final message that reads like progress: "implementation complete, awaiting gate slot."
Around the same time, our conductor-to-agent messaging reported success on messages that were never delivered. One agent sat blocked for hours on decisions the conductor believed it had sent four times.
Lost messages, false acks, workers whose reported status doesn't match reality: that's the standard distributed-systems failure list. An agent fleet is a distributed system and fails like one, no matter how capable the nodes are.
Our fix is the standard answer in new clothes. Every dispatch names a terminal artifact that defines completion, usually the URL of an open PR. Two lines go into every prompt: don't end your turn before the artifact exists, and if you're blocked, say exactly what on and return. Waiting lives only in the conductor. And completion claims get verified against artifacts (branch on the remote, PR open, CI green, tree clean), never against the agent's account of itself.
What we changed
Every lesson became a mechanism, because a rule without an enforcement mechanism is a wish.
A guard hook that blocks
git stash -u, blocks dropping a stash that carries untracked files, and blocks non-dry-rungit clean.Design artifacts tracked in git; recovery snapshots pinned as tags and bundled off-machine.
An isolation preflight: git-mutating commands refuse to run when the repo root doesn't match the agent's assigned root.
A dormant-surface check in CI that fails on any schema column no non-migration code writes.
Invariants written as queries, run in CI and on a schedule against real data.
A round-trip definition of done for data-moving migrations: capture, persist, read, display, search, proven in the assembled app before merge.
The ADR provenance rule and the terminal-artifact contract above.
The inter-agent mailbox demoted to best-effort. Anything load-bearing travels in the dispatch prompt or in files on disk.
Three things
First: data is only real where code writes it. Dormant schema is invisible to coverage, lint, review, and runtime checks, so somewhere a check has to ask directly whether anything writes the thing the design says is the point. Same for invariants. A rule enforced on the write path but never audited against the data may hold in 1.3% of your rows.
Second: prevention written as prose fails. If a postmortem produces a rule that doesn't name its enforcement mechanism (a hook, a CI check, a constraint), expect the problem to recur. Ours lasted three days.
Third: treat the fleet as the distributed system it is. Name a terminal artifact in every dispatch. Verify completion against git and CI, not the agent's self-report. Keep all waiting in the coordinator, because a turn-based worker that "waits" has stopped.
Every check we had passed a feature whose designed data flow was dormant, whose design record had been destroyed, and whose core invariant was violated in 98.7% of rows. None of them failed, because none of them were built to ask those questions. A check answers exactly what you design it to answer, and nothing else.
Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment.