Multi-Agent Debate: The Research Behind Better AI Code Review

The paper in one paragraph

One model reviewing its own code doesn't work. There's a paper about this. It's been around since May 2023 and most of the industry hasn't read it.

"Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate" by Liang et al. (arXiv:2305.19118, reference implementation on GitHub). The protocol: two or more LLMs answer a question independently, then exchange and challenge each other's answers across a few rounds. A judge — either a separate model or a deterministic aggregation rule — writes the final answer from the transcript. The paper shows MAD beats both a single LLM answering once and a single LLM reflecting on its own answer. It also names the failure mode of self-reflection — Degeneration of Thought.

That's the claim. The rest of this article walks through the setup, the results, the mechanism, what Joint Chiefs takes from it, and the caveats the paper flags itself.

The experimental setup

The paper runs MAD on tasks that are hard for single-model chain-of-thought — counter-intuitive arithmetic (Common MT and a small math set) and commonsense reasoning benchmarks. The choice is deliberate. Easy tasks don't tell you anything because every model gets them right. Hard tasks, where models disagree, are where a debate layer has room to help.

Each task runs under three conditions:

• Direct: one model, one answer, no reflection.
• Self-Reflection: one model answers, then reads its own answer and revises.
• MAD: two models — called "affirmative" and "negative" in the paper — answer independently, swap, respond to each other over a small number of rounds, then hand the transcript to a third model acting as judge.

The judge's job is not to vote. It reads the full debate and picks the side with the stronger argument — or synthesizes a third answer if neither side is fully right. That detail matters more than the two-debater setup, and it's the part that carries over most directly to code review.

What the paper found

Multi-Agent Debate beats single-model self-reflection

Across the benchmarks in the paper, MAD produces higher accuracy than direct answering or self-reflection. The effect is consistent enough that the paper calls self-reflection harmful relative to MAD — not just weaker. A model reflecting on its own output gets more confident without getting more correct. So self-reflection is worse than useless when a better option exists.

The size of the gain depends on the task. On tasks where the base model is already near-ceiling, there's little room to improve. On tasks where the base model is making systematic reasoning errors, MAD recovers a meaningful fraction of those errors by bringing in a second viewpoint. Code review behaves like the second kind. There's a long tail of subtle issues that single-model review misses.

Convergence is usually fast

The useful number of debate rounds is small — typically two to four before positions either agree or stop moving. Continuing past that point doesn't improve the final answer and can degrade it by introducing noise. Joint Chiefs' adaptive early break on convergence comes straight from this.

The judge matters

The judge's reasoning — not a majority count — is what produces the gain. A weak judge that picks whichever side wrote more text underperforms a strong judge that reads the arguments and picks the answer with better grounding. That's why MAD is not voting. It's why Joint Chiefs picks the moderator model explicitly instead of tallying.

Degeneration of Thought, the central mechanism

The paper's core diagnostic is Degeneration of Thought (DoT). The claim: when a single LLM reflects on its own previous output, the reflection is sampled from the same distribution as the original answer. Same biases, same training data priors, same mistakes. The model can't notice what it wasn't trained to notice. Ask it to double-check and you get more confident agreement with its earlier self. Not a second opinion.

DoT is what makes single-model code review fundamentally limited. The first pass and the self-review pass come from the same well. They share blind spots. The failure isn't a bug in any one model — it's a property of self-reflection as a mechanism.

MAD breaks DoT by putting two models with different weights in dialogue — ideally different architectures and different training corpora. Their blind-spot distributions don't overlap, so each one can notice what the other missed. The gain comes from the diversity of the participants, not the count.

The four design principles Joint Chiefs inherits

Joint Chiefs takes four principles straight from the paper:

1. Adaptive break on convergence

Debate stops when positions stop moving. The paper showed that forcing more rounds after convergence degrades output. Joint Chiefs watches cross-round agreement on individual findings and terminates when the panel reaches unanimous or near-unanimous positions. Max rounds is a safety net, not a target. Real debates often converge in two or three rounds.

2. Tit-for-tat engagement

Every round must address the prior round's findings by title — agree, challenge, or revise. Not just restate the model's original position. Without this, a "debate" turns into parallel monologues that ignore each other. The paper's protocol enforces direct engagement. Joint Chiefs' prompts do the same, so skipping a prior finding becomes a visible omission.

3. Multiple independent models (DoT prevention)

The panel must include models from different labs and ideally different architectures. A panel of OpenAI + OpenAI fails in the same way a single model reflecting on itself does. Joint Chiefs ships with OpenAI, Gemini, Grok, Anthropic, and optional Ollama. Five independent teams training on five different data blends. Per-provider weights run from 0.0 to 3.0 — set a provider to 0 to exclude it without breaking the independence property.

4. Judge arbitration

A deciding model reads the full transcript and writes the final synthesis. The judge evaluates reasoning quality — a well-argued minority position can override a weakly-justified majority. That's why Joint Chiefs' default consensus mode is moderator decides, and why findings get anonymized before the moderator reads the transcript. The judge should weigh arguments, not brands.

What MAD does not prove

Caveats the paper names itself, which matter if you're building on the work:

• MAD was evaluated on reasoning and factuality, not code. The mechanism generalizes. The specific accuracy numbers don't. Follow-up work on MAD-style debate for code exists (see Du et al., 2023 and later), but the gains are task-dependent.
• Compute cost scales with participants and rounds. A three-model, five-round debate is roughly 15× the tokens of a single-model review. For trivial changes that's wasted. For high-stakes review it's cheap insurance. The right trade-off is a product choice, not a research conclusion.
• The judge can introduce its own bias. If the judge consistently sides with one debater, you inherit that model's blind spots at the decision layer. Joint Chiefs' anonymization step and tiebreaker slot bound this risk. They don't eliminate it.
• Scaling behavior isn't fully characterized. MAD with many more than three or four participants hasn't been studied as thoroughly. Joint Chiefs' default panel size is small on purpose.