The Product vs. The Work: Managing Engineering Exhaust in the Age of AI Coding Agents

"When autonomous agents write code, the volume of engineering exhaust—plans, reviews, task packets, and benchmarks—rapidly outgrows the shipping product itself. Keeping that exhaust from polluting the active codebase is essential to prevent context poisoning, but how teams achieve that separation involves real architectural trade-offs."

Executive Summary

As engineering teams integrate autonomous AI coding agents (Claude Code, Codex, Antigravity, etc.) into daily workflows, a new operational challenge has emerged: the exponential growth of cognitive exhaust.

For every production feature shipped, autonomous workflows generate:

• Task packets, execution briefs, and task decomposition files.
• Architectural proposals, code review transcripts, and evaluation scorecards.
• Benchmark runs, prompt engineering trials, and debugging session traces.
• Ephemeral scratchpads, meeting notes, and historical audit ledgers.

When this material is committed directly to the main product branch alongside shipping code, repositories suffer from severe context poisoning: AI agents ingest obsolete planning documents as active constraints, repository search tools (find, grep) return overwhelming noise, and the true system architecture becomes obscured.

There is no single "golden solution" to this problem. Different teams and organizations solve this in fundamentally different ways depending on their tooling stack, compliance obligations, and developer workflows:

1. External Project Management & Knowledge Systems: Storing planning, task state, and review transcripts entirely outside Git (e.g. Linear, Jira, Notion, internal developer portals).
2. Dedicated Companion Repositories: Moving all engineering management, benchmark data, and prompt research into a completely separate repository.
3. Cloud Object Storage & Telemetry Data Lakes: Streaming raw session transcripts and evaluation metrics directly to S3/GCS or structured data stores.
4. Purely Ephemeral Lifecycles: Discarding task packets and execution traces upon pull request merge, relying on code diffs and PR descriptions as the sole record.
5. In-Repo Orphan Worktrees: Isolating work records onto an orphan Git branch (repo/work) mounted locally via a linked Git worktree (git worktree add).

This paper explores these architectural alternatives, analyzes their practical trade-offs, and documents the in-repo orphan worktree pattern as one identified solution that offers Git-backed auditability without third-party dependencies or multi-repo synchronization friction.

1. The Core Tension: Two Artifact Classes with Opposing Lifecycles

Every software development effort produces two fundamentally distinct classes of artifacts:

Dimension	The Product	The Work
Audience	End users, runtime environments, compilers, external API callers.	Engineering management, auditors, autonomous agent harnesses, postmortems.
Growth Rate	Bounded ($O(\text{features})$).	Compounding ($O(\text{agents} \times \text{iterations})$).
Lifespan	Long-lived; reflects present-tense reality.	Finite relevance; goes stale once a feature closes.
Risk of Retention	Low (this is the shipping asset).	High (context poisoning, token bloat, hallucinations).

2. Why Separation Matters: The Context Poisoning Problem

When work artifacts and historical planning documents live inside the product tree, autonomous coding agents fail in predictable, costly ways:

1. Hallucinating Dead Requirements: When an agent searches for domain requirements, it frequently retrieves obsolete task packets or superseded design memos. It treats dead requirements from three weeks ago as active constraints, writing code for architectures that were already discarded.
2. Context Window Contamination: Agent token budgets are finite. When 40% of the files returned by directory listings and search queries are historical logs or obsolete research, the agent's attention is diverted away from the actual code under test.
3. The Defensive Shim Anti-Pattern: Agents unable to discern whether a file is an internal working note or an active public interface defensively generate backward-compatibility wrappers, shims, and forwarders, polluting the codebase with dead layers.

3. The Spectrum of Solutions: A Trade-Off Analysis

Teams have solved this problem using several different architectural patterns. No single approach is strictly superior; each represents a different set of trade-offs:

Approach 1: External Systems of Record (Linear, Jira, Notion, Confluence)

• How it works: All task packets, sprint plans, and review transcripts live in third-party project management platforms. Git holds only source code and enduring documentation.
• Advantages: Clean product tree; non-technical stakeholders can easily inspect progress; rich UI for filtering and reporting.
• Trade-offs: Requires agents to have external API credentials; breaks offline workflows; task history is decoupled from exact Git commit SHAs; subject to SaaS pricing and export limitations.

Approach 2: Dedicated Companion Repository (project vs. project-work)

• How it works: The organization provisions two repositories. project contains only deployable code; project-work contains all task decomposition, benchmark records, and prompt research.
• Advantages: Absolute physical separation; distinct access permissions; independent CI/CD configurations.
• Trade-offs: High operational friction; requires coordinating dual pull requests; developers must manage two separate checkouts; links between code commits and work tickets require manual cross-referencing.

Approach 3: Object Storage & Data Lakes (S3, GCS, BigQuery)

• How it works: Agents stream execution logs, LLM evaluation traces, and benchmark outputs directly to cloud object storage or analytical data warehouses.
• Advantages: Scales effortlessly to gigabytes or terabytes of LLM telemetry; negligible cost; keeps Git lightweight.
• Trade-offs: Lacks Git-style branching and versioning; requires specialized query tooling to read back historical decisions; not immediately legible in local text editors.

Approach 4: Purely Ephemeral Scratchpads (Discard-on-Merge)

• How it works: Agents write planning notes to local scratchpad directories. When a pull request is merged, the scratchpads are discarded. Only the code diff, commit message, and PR description survive.
• Advantages: Maximum simplicity; zero infrastructure overhead; zero ongoing maintenance.
• Trade-offs: Eliminates auditability; impossible to reconstruct why an agent made a controversial architectural choice weeks later; loss of valuable prompt engineering benchmarks and negative test evidence.

Approach 5: The In-Repo Orphan Branch & Linked Worktree

• How it works: The work is stored on an independent, orphan Git branch (repo/work) within the same repository, mounted into the local filesystem via git worktree add .sdlc repo/work. The directory is ignored on main.
• Advantages: Completely self-contained in standard Git; zero third-party dependencies; full local offline access; preserved commit history; clean product checkout.
• Trade-offs: Developers must understand Git worktree mechanics; CI pipelines must be configured to ignore the work branch; potential confusion if engineers are unfamiliar with orphan branches.

4. Deep-Dive: The In-Repo Orphan Worktree Pattern

For teams that prefer a self-contained, Git-native approach without third-party services or multi-repo overhead, the orphan worktree pattern offers a balanced compromise.

Architectural Topology

• Branch main (Product): Holds strictly deployable code, tests, and configuration. Its .gitignore explicitly ignores .sdlc/ (or the chosen work directory).
• Branch repo/work (The Work): An orphan Git branch that shares zero commit history with main. It records engineering activity, historical notes, task ledgers, and reviews.
• Mount Point: Inside the local checkout, repo/work is mounted at .sdlc/ via git worktree add.

/workspace/my-project/             <── Checked out to branch 'main'
├── .gitignore                     <── Ignores '.sdlc/'
├── src/                           <── Clean product code
├── tests/                         <── Clean test suite
├── pyproject.toml / package.json  <── Runtime config
└── .sdlc/                         <── Checked out to orphan branch 'repo/work'
    ├── .git                       <── Git worktree pointer file
    ├── features/                  <── Task packets and execution evidence
    ├── records/historical/        <── Archived research and design memos
    └── ledgers/                   <── Audit trails and decision ballots

Key Operational Guarantees

1. Zero Product Bloat: A fresh clone of main contains zero bytes of planning notes, benchmarks, or task packets.
2. Persistent Local Access: Because .sdlc/ is a linked worktree, developers and agents have instantaneous filesystem access to all historical records without switching branches.
3. Branch Switching Immunity: When a developer switches product branches (git checkout feature-x $\to$ git checkout main), the .sdlc/ worktree remains completely stationary.
4. Independent Versioning & Pushing: Commits inside .sdlc/ are committed directly to repo/work and pushed to the remote independently.

5. Step-by-Step Implementation Guide

Setting up the orphan worktree pattern in any Git repository requires only standard Git commands.

Step 1: Ignore the Work Path in the Product

Add the designated work folder (e.g., .sdlc/) to the product branch's .gitignore:

echo ".sdlc/" >> .gitignore
git add .gitignore
git commit -m "chore(git): ignore .sdlc worktree directory on product branch"

Step 2: Initialize the Orphan Branch

Create an independent branch with zero shared history from main using an empty root tree:

# Generate a null Git tree hash and commit it as the orphan root
empty_tree=$(git hash-object -t tree /dev/null)
root_commit=$(git commit-tree "$empty_tree" -m "Initial commit on work record branch")

# Create the branch pointer
git branch repo/work "$root_commit"

Step 3: Mount the Worktree

Mount the orphan branch as a local worktree inside the repository:

git worktree add .sdlc repo/work

Step 4: Configure the Worktree's Internal .gitignore

The work branch itself should ignore transient developer scratchpads and multi-gigabyte raw runtime dumps:

cat << 'EOF' > .sdlc/.gitignore
# Local ephemeral scratchpads (never pushed)
scratch/
tmp/
*.log

# Local runtimes and caches
.venv/
node_modules/
__pycache__/
EOF

git -C .sdlc add .gitignore
git -C .sdlc commit -m "chore(work): configure work branch gitignore"

Step 5: Push and Establish Remote Upstream

git push -u origin repo/work

6. Real-World Case Study: Impact Metrics

In a recent pilot evaluation of this pattern on a data compiler repository that had accumulated 6 weeks of autonomous agent activity:

Metric	Before Separation	After Separation	Delta
Product Tree Size	365,000 lines	5,532 lines	-359,468 lines (-98.5%)
Tracked Markdown Files	142 loose files	4 enduring ADRs	-138 files (-97.2%)
Agent Search Precision	18% relevant hits	96% relevant hits	+433% improvement
Full Regression Runtime	835 seconds (serial)	12.1 seconds (parallel)	69x speedup
Historical Data Loss	Zero	Zero	100% records preserved

Note: The regression runtime gain came from parallelizing the test suite during the same cleanup, not from separating the work records.

7. Decision Framework: Choosing What Fits Your Team

When deciding how to manage the boundary between product code and engineering exhaust, consider your team's constraints:

• Choose External SaaS (Approach 1) if your engineering organization relies heavily on centralized roadmaps, non-technical stakeholders participate in reviews, and you have robust API integrations for your agents.
• Choose Cloud Object Storage (Approach 3) if your agents generate massive multimodal logs, GB-scale model evaluation dumps, or raw benchmark datasets.
• Choose Ephemeral Scratchpads (Approach 4) if you are building fast, throwaway prototypes or early-stage spikes where long-term decision provenance is unnecessary.
• Choose In-Repo Orphan Worktrees (Approach 5) if you want full, version-controlled audit trails stored securely alongside the codebase with zero added SaaS costs and zero multi-repo overhead.

---

AI attribution: 3/4 — Human-originated, AI-shaped. What this means

Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email jason.vertrees@gmail.com.