This is another short, practical, technical memo. As I dig more deeply into Herdr for multi-agent orchestration, I encounter concepts I think that might be useful for others to see. In this case, I was about to start a working session with Claude driving codex, Antigravity, and grok through Herdr. And before I set the work off, I wanted to ensure that the models were capable enough to do the work. We don't always need the strongest frontier models to do all of our work. However, in this case, as Claude was going to dispatch work to those three agents and itself, I wanted to make sure that none of those agents was just going to fail prima facie at its given task. So I set up a simple test to see which agents are ready to work on this project with me.
This memo shows you my test setup and then the results that I got. I actually found it pretty interesting and what I learned forced me to rule out one of the models for this production task.
Before trusting autonomous CLI agents with multi-agent orchestration or prompt compilation gates, you need an empirical baseline. If an agent cannot adhere to schema constraints, cite primary sources accurately, or isolate its own inferences during a bounded evaluation, it cannot be trusted in production pipelines.
We ran a single-shot calibration benchmark across four CLI-and-model combinations: Codex (GPT-6), Grok (Grok 4.6), and Anti-Gravity running both Gemini 3.1 Pro (default and high effort) and Claude Opus 4.6.
The Test Setup
Every agent received an identical one-shot prompt file:
Task: Research and report the top five best practices for non-interactive, zero-shot, schema-validated batch LLM prompting.
Constraints: Under 1,200 words; cite only URLs actually opened; mark unverified claims explicitly as
UNVERIFIED; log clarifying questions and explicit assumptions under a top-level heading; adhere to a rigid report format.Settled Baseline: Native structured output was deliberately left as an established pipeline constraint to test whether models would waste output bandwidth reiterating the obvious or identify deeper operational controls.
| Seat / Model | Execution Command | Configuration Notes |
|---|---|---|
| Codex / GPT-6 | codex exec --ephemeral -s read-only |
Reasoning effort: high (gpt-6-astra) |
| Grok / Grok 4.6 | grok --permission-mode plan |
Brief supplied via prompt file |
| Anti-Gravity / Gemini 3.1 Pro | agy --sandbox --dangerously-skip-permissions |
Default effort |
| Anti-Gravity / Gemini 3.1 Pro | agy --sandbox --dangerously-skip-permissions |
--effort high (CLI ceiling) |
| Anti-Gravity / Claude Opus 4.6 | agy --sandbox --dangerously-skip-permissions |
Thinking mode (CLI rejects --effort) |
Benchmark Telemetry & Compliance
Wall-clock runtimes are derived from filesystem write timestamps. Word counts reflect target artifact outputs.
| Seat / Model | Runtime | Words | URLs Cited / Fetched | Format Adherence | Isolated Inference? | Wasted Slot on Settled Baseline? |
|---|---|---|---|---|---|---|
| Codex (GPT-6) | 1m 26s | 939 | 8 / 8 cited* | Exact | Yes (explicit, repeated) | Partly (reframed around retry inflation) |
| Grok (Grok 4.6) | 3m 19s | 1,375 | 7 / 7 fetched | Prepended narration | Partly (quotes marked; analysis unlabeled) | Yes (noted schema-valid ≠ correct) |
| Gemini 3.1 Pro (Default) | 1m 37s | 727 | 2 / 2 fetched | Exact | No | Yes |
| Gemini 3.1 Pro (High) | 2m 09s | 733 | 4 / 4 fetched | 1-sentence preamble | No | Yes |
| Claude Opus 4.6 | 3m 46s | 1,110 (artifact) | 3 cited / 2 fetched | Exact in artifact | Partly (flagged search snippets) | Yes |
*Codex's console log logged three URL lookups alongside blank search calls, leaving its 8-page fetch claim partially untraceable in local telemetry.
Sourcing Honesty & Verification Audits
Autonomous agents frequently confuse search snippets with document verification. When audited against primary provider documentation, distinct patterns emerged:
Phantom Metrics & False-Clean Flags (Opus): Opus's tool-status line asserted "All claims below are sourced from pages I opened. No UNVERIFIED material." In reality, two of its practices relied entirely on search snippet previews, and it cited a production guide it never fetched. It also hallucinated a specific comparative metric—"100% schema compliance versus ~86% for legacy JSON mode"—that appeared in none of the fetched pages.
Paraphrasing vs. Verification (Gemini High Effort): Gemini claimed all sources were "fetched and verified," yet summarized OpenAI documentation using non-existent marketing phrases ("Schema-First approach") without direct quotes.
Third-Party Over-Reliance (Gemini Default): Rather than referencing core provider documentation, four of its five practices cited the documentation of a single third-party library (
Instructor).Rigorous Boundary Enforcement (Codex & Grok): Codex explicitly labeled its own deductive leaps (e.g., "the legal distinctions above are my application..."). Grok pulled direct, verified quotes tied to specific URLs across OpenAI, Anthropic, xAI, and Google.
Critical Technical Takeaways
Where the council reached consensus:
Move domain ontology into schema definitions: Field names, detailed descriptions, and constrained enums carry more reliable semantic weight than system-prompt prose.
Delimit variable inputs with structural tags: Parsing legal or regulatory source text containing normative keywords ("shall", "must not") routinely derails model execution unless bounded inside explicit XML or markdown wrappers.
Strip conversational framing: Persona prompts and conversational "think step-by-step" markers add latency and token overhead without improving adherence in structured batch pipelines.
High-value single-seat findings:
Retry Inflation (Codex): Frameworks like Pydantic AI automatically retry validation failures under the hood. If your telemetry only records final-state outputs, automated retries will artificially inflate reported reliability. Success must be logged on the first response.
Abstention as an Explicit Type (Grok): Constrained decoders will force a valid JSON structure even when the source document lacks the relevant entities, silently generating convincing hallucinations. Abstention must be modeled as a first-class schema outcome.
Batch Replay Non-Determinism (Opus): Temperature 0 and pinned model snapshots do not guarantee bitwise identical outputs across model revisions. Validators must assert semantic equivalence rather than string identity.
Cache Prefix Discipline (Opus & Grok): Structuring inputs to maximize prefix caching provides immediate 50–80% cost reductions at batch scale, demanding byte-stable prompt headers.
Direct contradictions:
Reasoning Fields in Schemas: Both Gemini runs demanded dedicated scratchpad fields in the output schema. Codex argued against visible reasoning blocks, citing vendor research that externalized CoT is redundant on native reasoning models.
Assistant Turn Prefilling: High-effort Gemini recommended prefilling the assistant turn with an opening
{. Grok explicitly discouraged this, noting that prefilling breaks replayable contracts, disrupts prefix caching, and is obsoleted by native constrained decoding.
Harness Quirks & CLI Evaluation
I/O Routing Failures: Codex, Grok, and Gemini wrote directly to target files or stdout. Opus output a brief 209-word narrative to stdout and dumped the complete 1,110-word report into its local sandbox directory (
~/.antigravity/...), breaking downstream shell piping. (Note: I used this Opus through agy.)Narration Leaks: Grok prepended conversational narration directly into its stdout stream. Unattended pipelines consuming Grok CLI outputs require an automated regex strip (
^##) before parsing.CLI Parameter Limits: The Anti-Gravity harness crashed on
--effortflags when running Claude models, necessitating a manual restart without the argument.
Production Verdict
Codex (GPT-6): Qualified. Highest discipline regarding verification boundaries, strictly separated inference from external facts, and identified silent pipeline vulnerabilities (retry inflation).
Grok (Grok 4.6): Qualified (with stdout sanitization). Comprehensive sourcing across all major providers, strict handling of assumptions, and identified the necessity of typed abstention.
Claude Opus 4.6: Qualified for architectural design; untrusted for autonomous research. Surfaced critical distributed systems concerns (caching layouts, replay variance), but claimed unverified search snippets as verified fetched pages.
Gemini 3.1 Pro: Disqualified under sandbox harness. Over-indexed on third-party documentation, failed to separate external claims from internal heuristics, and spent disproportionate token allocations on settled architecture. (Note: I also tried Gemini Flash 3.8, which was noticeably worse at these tasks right out of the gate. That's why we tried Gemini 3.1 Pro.)
Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email jason.vertrees@gmail.com.