Technology Aug 30, 2026 · 16 min read

The Same Model Debating Itself Was More Self-Critical Than Two Different Models

v0.2.1 RELEASED — Aug 28, 2026. Release notes · Field test report v0.2.1 Key Finding: DeepSeek+GPT (0.246 convergence, no Mistral) performed the same as GPT+GPT (0.273, homogeneous control). The distinction is not "diversity vs homogeneity" — it is Mistral vs no-Mistral. The v0.2.1 separating expe...

DE
DEV Community
by Debashish Ghosal
The Same Model Debating Itself Was More Self-Critical Than Two Different Models

v0.2.1 RELEASED — Aug 28, 2026. Release notes · Field test report

v0.2.1 Key Finding: DeepSeek+GPT (0.246 convergence, no Mistral) performed the same as GPT+GPT (0.273, homogeneous control). The distinction is not "diversity vs homogeneity" — it is Mistral vs no-Mistral. The v0.2.1 separating experiment reframes this article's thesis.

AdversarialDebate v0.2.0 is released — v0.2.1 shipped Aug 28.

I included a homogeneous pair as a control because I expected it to fail.

If the whole thesis behind AdversarialDebate is that diversity helps two models pressure each other's reasoning, then the same model debating itself should be the weak baseline. Same training distribution. Same safety tuning. Same blind spots. Same defaults.

That was the theory.

The data said something much stranger.

In the full field test, GPT + GPT outperformed both GPT + Gemini and Gemini + Mistral.

Not by a rounding error. By enough that I had to stop and rethink what "diversity" actually means in a multi-agent system.

This post is about the most counterintuitive result in the whole project: weak diversity can be worse than no diversity at all.

v0.2.0 did not fully resolve this question because the homogeneous control stayed a partial sample, but it did reinforce the broader lesson:

  • weak diversity (GPT + Gemini) is still bad
  • moderate diversity (GPT + Mistral) is the best full-corpus default
  • homogeneous control can still be more self-critical than a weak heterogeneous pair

Here is the shortest version of the result:

Pair Avg Score Verdict Rate Avg Concessions per Debate
GPT + GPT 0.688 57% 20.9
GPT + Gemini 0.357 4% 10.5
Gemini + Mistral 0.512 4% 15.6

The same model did not just hold up as a control. It behaved like a better debate partner than two supposedly more diverse pairings.

Relevant v0.1.0 sources:

Relevant v0.2.0 sources:

The Result That Should Not Have Happened

Here is the comparison that forced the issue:

Pair Diversity Avg Score Verdict Rate Concessions Capitulation
GPT + GPT Same model 0.688 57% 1,444 21
GPT + Gemini US + US 0.357 4% 727 0
Gemini + Mistral US + EU 0.512 4% 1,073 1

The homogeneous pair, the one I expected to be the most boring and least useful, beat two heterogeneous pairs.

That result is still the clean v0.1.0 story.

In v0.2.0, I only ran a partial homogeneous control sample:

  • 7 debates
  • 0 verdicts
  • 0.273 average convergence score
  • 0 theater
  • 0 capitulation

So v0.2.0 does not replace the original article claim. It leaves it as a strong v0.1.0 result with a smaller supporting signal in the next release.

That is not supposed to happen if your mental model is just "more diversity = better debate."

What the field test actually suggests is more uncomfortable:

  • no diversity can still produce useful self-criticism
  • weak diversity can produce stubborn deadlock
  • moderate diversity can produce productive debate
  • maximum diversity can collapse into capitulation

That is not a clean ladder. It is a spectrum with a bad middle and a dangerous far end.

Why I Included A Homogeneous Pair In The First Place

The control was there to test whether the whole architecture was doing something real.

If two copies of the same model reviewed the same PR and then debated, I expected one of two outcomes:

  1. trivial agreement
  2. shallow disagreement with very fast convergence

If the system had leaked context between the two passes, the homogeneous pair would have exposed it immediately. If both sides quietly saw the same reasoning too early, the outputs would collapse into fake consensus.

So GPT + GPT was useful for two reasons:

  • it gave me a baseline against heterogeneous pairs
  • it stress-tested the isolation invariant

I did not expect it to become one of the most interesting pairings in the entire run.

The Homogeneous Pair Was Not Boring

The strongest evidence that the homogeneous pair was doing something real is that it did not converge all the time.

From the field test:

  • 57% verdict rate
  • 43% disputed
  • 1,444 concessions across 69 debates
  • about 21 concessions per debate

If the same model were simply echoing itself, that should not happen.

If the isolation were fake, that should not happen.

If the homogeneous control were trivial, that should not happen.

Instead, the same model in two isolated contexts produced enough variance to create real disagreement, and enough recognizability to create meaningful concession.

That is a very different dynamic from what I saw in the weakest heterogeneous pair.

And the v0.2.0 negative-control rerun reinforced the weak-pair side of that comparison:

  • GPT + Gemini negative control: 0.033 average score, 0/24 verdicts
  • GPT + GPT partial control: 0.273 average score, 0/7 verdicts

That is a smaller sample than I would like, but it still points in the same direction: weak diversity can be worse than no diversity.

And it matters for product design. If your evaluation only compares different models to each other, you can miss the fact that one model in two isolated contexts may actually produce cleaner self-correction than two loosely similar models forced into a debate loop.

Compare It To The Weak Pair

Now look at GPT + Gemini:

  • 0.357 average score
  • 4% verdict rate
  • 727 concessions across 69 debates
  • about 10 concessions per debate
  • 0% capitulation
  • 2.0 average rounds

This pair almost always used the full round budget and still rarely converged.

That does not mean the pair was useless. In many cases the claims still aligned with the real issue. But as a debate pairing, it was much less productive.

This was one of the most important differences in the entire field test:

GPT + GPT produced disagreement plus self-correction.

GPT + Gemini produced disagreement plus stubbornness.

That is a materially different product behavior.

My Best Hypothesis: The Same Model Can Recognize Its Own Mistakes

I cannot prove this yet, but I think this is what is happening.

When GPT-4o-mini debates itself, both sides speak in reasoning patterns the other side can fully evaluate. They share not just training distribution, but the shape of their errors.

That means one side can say, in effect:

"I know exactly how you got here, and I can see where that line of reasoning fails."

That is different from heterogeneous debate. With a different model, the other side may disagree without ever fully recognizing the internal move that created the original error.

So the same-model pairing can do something surprising: it can become more self-critical because it is arguing in a language it understands perfectly.

The best analogy I have is editing your own draft the next day.

You are the same person. You still have the same habits. But the separation in context is enough to let you spot flaws that were invisible in the first pass.

That is what the isolation architecture is buying here: two rooms for the same brain.

Why Weak Diversity Can Be Worse

The GPT + Gemini result forced me to stop thinking about diversity as a binary property.

They are different models. They are not the same provider. They are not identical systems.

And yet the debates were much less productive than GPT + GPT.

My current read is that weak diversity creates the worst of both worlds:

  • different enough that the models do not easily recognize each other's internal reasoning patterns
  • similar enough that both still produce strong, plausible arguments on the same side of the issue

That creates a nasty equilibrium.

Neither side yields.
Neither side collapses.
Neither side fully persuades the other.

You get evidence-backed rebuttal without convergence.

That can still be useful for a human reader, but it is a terrible default if your product promise is "structured debate that improves resolution quality."

That is why I no longer think of model pairing as a model-selection problem alone. It is an interaction-design problem.

That conclusion survived v0.2.0 cleanly.

The pair roles in the current release are now explicit:

  • pair3_gpt_mistral = primary / positive pair
  • pair5_deepseek_mistral = validation pair
  • pair1_gpt_gemini = negative control
  • homogeneous_gpt = partial homogeneous control

That makes this result easier to place in the broader system story.

The Diversity Spectrum Looks Nothing Like I Expected

The clean thesis would have been: more diversity, more useful debate.

The actual spectrum coming out of the field test looks closer to this:

Diversity level Example Avg Score Dominant behavior
None GPT + GPT 0.688 Self-criticism
Weak GPT + Gemini 0.357 Stubborn disagreement
Moderate GPT + Mistral 0.754 Productive debate
Strong DeepSeek + Mistral 0.982 Capitulation risk

That spectrum explains a lot of the confusion in multi-agent product design.

If you test only one weak heterogeneous pair, you may conclude the system is not very good.

If you test only the strongest pair, you may conclude the system is amazing while missing that some of the "success" is surrender.

If you do not include a homogeneous control at all, you miss the fact that the middle of the spectrum may be worse than one side of it.

That is why this result matters beyond this one project. It changes how I would design multi-agent evaluations in general.

The lesson is not "same model good, different model bad." The lesson is that pair behavior has to be measured, not assumed from provider diversity.

The Homogeneous Pair Also Validated The Isolation Architecture

This result was not just interesting from a pairing perspective. It also strengthened my confidence in the system design.

If the isolation architecture were leaking, the same-model pair would have been the first place to see it.

Instead, the opposite happened. The pair produced enough genuine disagreement that 43% of the debates stayed disputed.

That means the two independent passes were really independent.

The same model, given the same artifact, still produced meaningfully different reviews across contexts often enough to create real debate.

That is a big deal.

It means the delayed revelation rule did more than prevent contamination. It preserved natural model variance as a source of useful pressure.

That is one of the few places where the architecture itself made me more optimistic than the initial product thesis did.

There Is Still A Catch

I do not want to oversell the homogeneous pair.

It had 21 capitulation cascades, which is about 30% of the pair's debates.

So this was not some magical perfect control. The same-model pairing still showed the same failure mode as the strongest diverse pair, just at a lower rate.

And I did not run the flakiness sweep on the homogeneous pair specifically.

That matters.

One possible explanation for the result is not self-criticism but variance noise. If GPT + GPT flips more often than the stronger cross-model pairs, the 57% verdict rate may be less meaningful than it looks.

I do not think that is the whole story, but I cannot rule it out yet.

This is one of the places where v0.1.0 gave me a valuable result and an unfinished question at the same time.

The Gemini Problem Is Real

This article is partly about homogeneous control. It is also partly about Gemini.

Across the field test, Gemini looked like the most stubborn model in the pool:

Model Avg concessions per debate
GPT-4o-mini 19.4
Mistral Small 3.2 25.3
DeepSeek-V3 27.9
Gemini 2.5 Flash 15.9

That does not mean Gemini was worse at identifying issues.

It means Gemini was less willing to yield in debate.

That distinction matters a lot.

The practical implication from this run is simple: do not pair Gemini with a model that is close enough to fight forever but not different enough to break the stalemate.

That pairing pattern gave me some of the least productive debates in the whole experiment.

What Could Have Gone Better

Three things stand out here.

1. I should have planned a stronger control strategy up front

I added the homogeneous pair as a sanity check. In hindsight, it was one of the most important experimental conditions in the entire test. It deserved more explicit planning, more reporting emphasis, and its own dedicated stability check.

2. I should have run the flakiness sweep on the homogeneous pair too

I have strong evidence that the engine is stable overall. I do not yet have strong evidence that the homogeneous self-criticism result is stable specifically. That should have been part of the first release.

3. I should have treated "similar labs" as a hypothesis, not a category

US vs US was a useful shorthand while testing. It is not a complete explanation. OpenAI and Google are not the same model family, and I do not want to pretend geography alone explains pair behavior. The interaction could be about alignment style, response incentives, or something else entirely.

What I Learned

This is what I take from the result.

1. Homogeneous pairs are not automatically bad

If the isolation architecture is real, the same model can still produce useful disagreement and meaningful self-correction.

2. Weak diversity may be the worst place to sit

Too similar to unlock real blind-spot discovery. Too different to easily self-correct. Just enough overlap to argue forever.

3. Pairing is not a model-quality question alone

It is a behavioral systems question. You are not choosing the best model. You are choosing the best interaction.

4. Controls matter more than people think in agent evaluations

Without the homogeneous pair, this whole result disappears. I would have told a much cleaner and much less truthful story.

5. The right default is probably moderate diversity

Right now the safest practical recommendation looks like GPT + Mistral, not GPT + Gemini and not automatically DeepSeek + Mistral.

Different enough to find blind spots. Similar enough to evaluate each other. Not so different that the weaker model just folds.

That is a much more useful product insight than "use the pair with the highest score."

What I’d Change Next

This result changes the roadmap more than I expected.

The next version needs:

  1. pair-level guidance in docs and config recommendations
  2. stability testing for homogeneous controls, not just the highest-scoring pair
  3. better pair-quality metrics beyond convergence and verdict rate
  4. a more explicit diversity taxonomy based on observed interaction behavior, not just provider labels

The long-term goal is not just to know whether a pair converges.

It is to know whether a pair debates well.

Those are not the same problem.

Questions I Want Developers To Argue With Me About

This result is weird enough that I want people to push on it.

  1. If the same model can outperform a weak heterogeneous pair, what does "diversity" actually mean in agent evaluation?
  2. How much of this result is model behavior versus prompt behavior versus protocol design?
  3. Should multi-agent systems recommend pairings explicitly the way database tools recommend query plans?
  4. If homogeneous pair performance is stable, does that make self-critique more promising than most multi-model ensembles?

If you have seen similar behavior in judge systems, critic loops, or verifier chains, I want to hear about it.

Because this is exactly the kind of result that makes a product more useful and a blog post less clean.

AdversarialDebate is live here:

Next in the series: the field test design mistake that turned out to be one of the best decisions in the whole release, adding a fourth model mid-run and changing what the data could say.

v0.2.1 Update: The Mistral Effect Explains the Homogeneous Control

This article argued that "weak diversity can be worse than no diversity at all" — GPT+GPT (0.688) outperformed GPT+Gemini (0.357). The v0.2.1 separating experiment deepens that finding.

v0.2.1 added DeepSeek+GPT — two models from different labs, different training regimes, different regions. If diversity of training objective drives debate quality, this pair should outperform the homogeneous control.

It did not.

Pair Convergence Mistral? v0.2.0 Baseline
GPT+Mistral 0.536 Yes 0.536
DeepSeek+Mistral 0.572 Yes 0.572
DeepSeek+GPT 0.246 No New in v0.2.1
GPT+GPT 0.273 No 0.273 (partial)
GPT+Gemini 0.033 No 0.033

DeepSeek+GPT (0.246) performed at the same level as GPT+GPT (0.273). Two models from different labs converged like two copies of the same model.

This reframes the article's thesis. The distinction is not "diversity vs homogeneity." It is Mistral vs no-Mistral. The GPT+GPT control looked surprisingly strong because the non-Mistral pairs were all weak — not because homogeneity is inherently good. The v0.2.1 data confirms: without Mistral, lab diversity alone does not produce productive debate.

The v0.2.1 release also added:

  • Row-count invariants at all 5 pipeline seams — the 2,333→359 collapse is now structurally prevented
  • False-negative measurement — 1.7-3.4% missed-issue rate across 59 known-bad PRs, the first recall data ever reported

Full details in the v0.2.1 field test report.

v0.2.2 Update: Noise Floor and Shared RLHF Priors

Two v0.2.2 findings deepen this article's thesis:

Noise-floor baseline. The homogeneous_gpt pair (n=7) has a 95% CI of 0.129–0.432 — its true convergence score could be anywhere from "worse than GPT+Gemini" to "competitive with Mistral pairs." The GPT+GPT comparison in this article should have carried a confidence interval, not just a point estimate.

Shared RLHF priors. The Mistral effect is confirmed, but the causal mechanism has two competing explanations. The positive story (Mistral-specific training) is the one this article tells. The negative story — non-Mistral models share RLHF priors and rubber-stamp each other — is now documented as an equally consistent alternative. The next experiment should test RLHF-distance directly.

Full details in the v0.2.2 field test report.

DE
Source

This article was originally published by DEV Community and written by Debashish Ghosal.

Read original article on DEV Community
Back to Discover

Reading List