Resolving a Council of Independent AI Reviewers into a Single Decision

Running multiple frontier LLMs in parallel to evaluate prompt pipelines, codebases, or architectures is straightforward. The hard part is reconciling their outputs. When three independent agents—such as Codex (GPT-6), Grok 4.6, and Claude Opus 4.6—audit the same system, each generates dense reports packed with overlapping observations, subjective recommendations, and direct contradictions.

The default impulse is to feed all three reports back into an LLM with the instruction: "Synthesize these into a single unified action plan."

That approach fails.

Naive synthesis systematically destroys the most valuable signal multi-agent evaluation provides: the critical, solitary finding spotted by only one reviewer.

Why Standard Multi-Agent Aggregation Fails

Off-the-shelf multi-agent aggregation strategies suffer from three documented architectural flaws:

• Aggregator Bias (Mixture-of-Agents): Prompting an LLM judge to consolidate findings (Wang et al., 2024) triggers known model biases: position bias, verbosity bias, and self-enhancement bias (Zheng et al., 2023). The longest or first-parsed report dominates. Worse, aggregator models cannot distinguish between disagreement and omission: when Reviewer B says nothing about a template Reviewer A flagged, the aggregator routinely treats silence as consensus approval.

• Premature Convergence (Multi-Agent Debate): Having agents debate until they reach consensus (Du et al., 2023) routinely filters out informative minority disagreements (Smit et al., 2023; arXiv:2510.20963). Under iterative debate loops, candidate agents regress toward the mean, producing homogenized restatements of popular arguments rather than defending nuanced edge-case catches.

• Majority Voting (Self-Consistency): Universal self-consistency (Chen et al., 2023) works for closed-domain, single-answer mathematical or symbolic problems. In complex qualitative audits, majority voting discards legitimate minority edge cases by construction.

Most multi-agent orchestration frameworks (such as Anthropic's multi-agent workflows, Google ADK, or OpenAI Agents SDK) describe the plumbing for running parallel calls, but leave contradiction resolution as an exercise for the developer.

What Consensus Frameworks Offer

To construct an aggregation protocol that preserves high-signal outliers, we evaluated established human consensus and peer-review models:

Framework	Core Mechanism	Advantage for AI Councils	Inherent Failure Mode
Delphi Method	Iterative anonymous rounds with facilitator-provided rationales.	Eliminates peer anchoring and status-driven bandwagoning.	Prioritizes consensus over explanation; amplifies shared baseline ignorance.
Nominal Group Technique	Silent generation of all items before discussion, followed by voting.	Ensures every raw finding is placed on the board prior to evaluation.	Post-hoc ranking stages bury single-vote, high-severity findings.
NeurIPS Meta-Review	Area chair audits raw review quality against the paper, not numerical scores.	Separates review thoroughness from superficial confidence; explains usage.	Chairs default to reviewer averages when overloaded.
IETF Rough Consensus (RFC 7282)	"Humming" over voting; consensus requires resolving all technical objections.	Forces explicit handling of technical objections instead of headcounts.	Requires a chair capable of distinguishing technical flaws from preference.
Google Code Review Standards	"Technical facts and data overrule opinions and personal preferences."	Strictly partitions empirical code facts from subjective stylistic choices.	Requires a designated human decider to break deadlocks within a time box.

The Seven-Step Reconciliation Protocol

By combining the structural enumeration of the Nominal Group Technique, the evidentiary verification of RFC 7282, and Google's fact-versus-opinion partition, we implemented a deterministic seven-step pipeline:

Raw Review Outputs (Codex, Grok, Opus)
                 │
                 ▼
     [1. Atomic JSON Extraction]
                 │
                 ▼
      [2. Deduplicated Matrix]
                 │
                 ▼
    [3. Deterministic Fact Check] ──(Ambiguous)──► [4. Anonymized Round 2]
                 │                                        │
                 ▼                                        ▼
    [5. Structured Meta-Review] ◄─────────────────────────┘
                 │
                 ▼
    [6. Human Decider Ruling] (Recommendations Only)
                 │
                 ▼
    [7. Atomic Prompt Rewrites]

1. Extract, Don't Summarize

Split each reviewer's raw report into discrete, machine-readable records. No merging or synthesis is permitted at this stage. Each record captures:

• File path and target line range.

• Quoted source span.

• Category tag: FACT (verifiable directly against code/text) or RECOMMENDATION (architectural heuristic or stylistic preference).

• Severity level.

• Raw proposed diff/rewrite.

2. Build the Repertoire Matrix

Populate a unified evaluation matrix where rows are deduplicated findings keyed by target_path:line and functional claim, not superficial prose. Columns represent individual review seats. Cells are strictly tri-state:

• AGREE

• DISAGREE

• DID_NOT_EXAMINE

Silence is explicitly recorded as unexamined—never as tacit agreement or dissent.

3. Deterministically Verify Every Fact

Before debating recommendations, an automated coordinator checks every FACT assertion directly against the target repository snapshot:

• If Codex claims line 42 violates a 1,024-token cache prefix boundary, the coordinator measures the token count of lines 1–42.

• Claims are stamped VERIFIED, REFUTED, or AMBIGUOUS.

• A valid VERIFIED finding raised by only one reviewer survives with 100% evidentiary weight, neutralizing verbosity and position biases.

4. Time-Boxed Second Round on Ambiguous Facts

If a fact assertion is explicitly contested or remains ambiguous, run a single, isolated query back to the disagreeing models. The prompt provides only the line excerpt and the opposing claim, with all reviewer identities stripped. The models must return verifiable evidence within a single turn. No open-ended debate loops are permitted.

5. Compile the Meta-Review Ledger

Generate an auditable reconciliation ledger rather than a blended narrative document. Each entry reports:

• Claim identifier and verified status.

• Direct file evidence.

• Stance of each seat (AGREE, DISAGREE, DID_NOT_EXAMINE).

• Surviving minority findings listed on equal footing with unanimous findings.

6. Human Decision on Recommendations

Only RECOMMENDATION rows reach the human product owner or tech lead. The human decider reviews the matrix with the coordinator's evidence notes attached, ruling on deadlocks or subjective trade-offs (e.g., choosing between inline field descriptions versus explicit structural delimiters). Unresolved trade-offs are logged as intentional policy decisions, never averaged out.

7. Apply Targeted Rewrites

Once rows are decided, generate isolated prompt/code diffs where every change explicitly references a resolved matrix row ID. An LLM judge may subsequently evaluate the rewritten artifact against a functional rubric, but it never participates in negotiating the findings.

Council Setup: Grounding the Reviewers

This protocol was deployed across three frontier seats selected via bounded calibration benchmarking:

• Codex (GPT-6): High structural discipline; enforces clear boundaries between empirical findings and downstream inference; highly sensitive to silent failure modes like retry inflation.

• Grok 4.6: Broad provider-level context retrieval; strong assumption logging; surfaces architectural edge cases like typed schema abstention.

• Claude Opus 4.6 (via Anti-Gravity): High architectural depth regarding distributed inference constraints (e.g., cache boundaries, replay non-determinism).

To run this council reliably:

1. Enforce structured extraction schemas: Reviewers must output findings with explicit path pointers, confidence ratings, and category flags so ingestion is purely mechanical.

2. Ban mid-run interaction: Forbid back-and-forth coordinator questions during execution. Require all reviewers to log assumptions under a dedicated top-level heading.

3. Pin file snapshots: Resolve all line references against an immutable commit SHA.

Treating multi-agent outputs as an evidence matrix rather than a conversation transforms multi-agent reviews from noisy editorial compromises into rigorous, deterministic engineering audits.

Primary References

• Mixture-of-Agents: Wang et al., 2024. Mixture-of-Agents Enhances Large Language Model Capabilities. arXiv:2406.04692

• LLM-as-a-Judge Biases: Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685

• Multi-Agent Debate Dynamics: Du et al., 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325

• Critique of Debate Convergence: Smit et al., 2023. Should we be going MAD? Benchmarking Multi-Agent Debate. arXiv:2311.17371

• Premature Consensus in LLM Debates: 2025. When and Why Multi-Agent Debate Fails. arXiv:2510.20963

• Universal Self-Consistency: Chen et al., 2023. arXiv:2311.17311

• IETF Consensus Standards: RFC 7282. On Consensus and Humming in the IETF. RFC 7282

• Code Review Methodology: Google Engineering Practices. The Standard of Code Review. Google Eng Practices

• Editorial Guidelines: NeurIPS 2025. Area Chair Guidelines. NeurIPS 2025

---

Jason Vertrees is the founder of Heavy Chain Engineering, which helps lower middle-market vertical SaaS companies and PE firms turn scattered AI usage into measurable delivery leverage — 85% faster feature velocity, six-to-eight-week projects shipped in days. If you want help building an AI-native engineering organization, book an AI Delivery Assessment or email jason.vertrees@gmail.com.