Skip to content
What we doTech ExchangeTrack recordResearchProductsFor GoodBeliefsStaff LoginContact us
Research · September 2026

Who Should Review Whom?

Response-conditioned disassortative peer review for test-time LLM reasoning

Rui Li · Chenglong Sun · Xueyan Zheng · Dan Liu

Gi (Beijing) Tech Design Ltd. · 2026-09

Having several AI instances grade each other's work is now standard practice. The question almost nobody asks is who should grade whom. We ran a controlled experiment — 3,500 problems, about 140,000 API calls, two public benchmarks — pairing the same set of answers for mutual review in different ways. The answer can save you money: the pairing makes no difference. Seating the two most disagreeing agents together is no better than a random draw.

Three findings

Most different ≠ most instructive

Farthest pairing does make agents spot a peer's errors more often — detection rises monotonically with semantic distance, an order of magnitude between the bottom and top deciles. But detecting an error is not correcting it. Whether a correct fix is proposed, and whether it is adopted, shows no distance effect. The system got better at finding faults, not at fixing them.

Review is active, but net-zero

Every corrected error is matched by another, originally correct answer talked out of position by false opposition — the two directions cancel almost exactly. On the harder benchmark, skipping review entirely and handing the candidates straight to the synthesizer is significantly better (+1.5pp, p=0.010). Review is not a free neutral step; in the wrong place it is a net cost.

The bottleneck is credit assignment

Reviewers occasionally produce correct answers absent from all four initial candidates — genuinely new information — yet under 1.5% of it survives to the final answer. The system cannot tell corrective disagreement from false opposition. Turning disagreement into a small verifiable question is what the next generation of systems should solve.

If you build these systems

Majority voting plus direct synthesis matches every review topology at one fifth of the token cost. Spend the savings on calibration and verification, not on routing cleverness.

Dual funnels: forward chain (error exposure → detection → fix → final correct) and reverse chain (correct answer → false opposition → destabilization → final wrong). Detection rises with distance; resolution does not; the reverse chain grows heavier.
Dual funnels: forward chain (error exposure → detection → fix → final correct) and reverse chain (correct answer → false opposition → destabilization → final wrong). Detection rises with distance; resolution does not; the reverse chain grows heavier.

Full text & materials

Cite

@misc{fprr2026,
  title  = {Who Should Review Whom? Response-Conditioned Disassortative Peer Review for Test-Time LLM Reasoning},
  author = {Li, Rui and Sun, Chenglong and Zheng, Xueyan and Liu, Dan},
  year   = {2026},
  month  = sep,
  note   = {Gi (Beijing) Tech Design Ltd.}
}

What's next

The next experiment no longer asks who should talk to whom, but how to tell who is credible inside a disagreement: candidates and budget fixed, we vary only whether the verifier can see the candidate answers, testing whether blindness blocks false opposition. The design is frozen; results will be published the same way.