I’d test independent answers before paying for agent debate. In one reported comparison, debate scored 76.5% accuracy versus 76.8% for independent voting, while using about 3.4 times the tokens. That result is a reason to test - not proof that voting always wins.
My starting rule is simple: use stochastic consensus to combine separate answers, and try multi-agent debate when agents can check and correct disputed claims. Neither agreement nor a persuasive explanation proves an answer is right.
Quick Comparison
| Criterion | Stochastic consensus | Multi-agent debate |
|---|---|---|
| Process | Generate separate answers, then vote or select | Share proposals, critique and revise |
| Quality | Helps when separate attempts avoid different errors | Helps when critiques lead to checked corrections |
| Cost | More samples mean more tokens | Critiques, revisions and repeated context add tokens |
| Latency | Samples can run in parallel | Each round depends on earlier responses |
| Setup | Sampling, parsing and selection rules | Agent roles, message routing and round limits |
| Main risk | A majority can share the same mistake | Agents can persuade each other to accept a mistake |
I’d compare both with a single-agent baseline using the same tasks and matched budgets. I’d track accuracy, £ per decision, tokens and end-to-end latency, while keeping initial answers and minority objections.
The goal is checked answers, not forced agreement. I’d send unresolved or high-risk cases to an external checker or human reviewer.
::: @figure 
{Stochastic Consensus vs Multi-Agent Debate}
:::
How Stochastic Consensus and Multi-Agent Debate Work
Stochastic Consensus: Generate Answers and Aggregate
Run the same prompt independently across samples, record each configuration, and aggregate the answers only after generation. Use consistent evidence and output schemas, with no peer visibility during generation. Vary sampling paths or use a controlled pool of models with fixed configurations. Shared weights and evidence can lead to correlated errors, while higher temperature may add noise rather than useful variation.
Before counting votes, make the answers comparable. For classification, require canonical labels and state whether multiple labels are allowed. For extraction, use fixed fields backed by source spans. For constraint-checking, require a status, the relevant rule and supporting evidence. Keep “insufficient evidence” separate from “pass”, and vote on parsed final answers - not full explanations.[5][6]
| Method | How it works | Required checks | Main failure risk |
|---|---|---|---|
| Pre-processing | Standardise labels, dates and formats before selection; does not select an answer itself | Schema and parser tests | Merges different meanings or splits equivalent answers |
| Majority voting | Select the most frequent normalised answer; define a fallback if no candidate clears the threshold | Comparable outputs and valid-vote thresholds | A shared error defeats a correct minority |
| Weighted voting | Give more influence to candidates with validated reliability | Representative performance data; periodic checks | Stale weights or correlated errors dominate |
| Confidence-aware selection | Combine votes with calibrated confidence or verifier scores | Calibration against labelled cases | Confident but incorrect answers win |
Keep all candidates, vote distributions, confidence values and minority evidence. The winning label alone hides disagreement. For extraction, check that the selected fields still match the source document. Selecting fields separately can create a record that no candidate proposed.
The core test is whether aggregation beats critique under the same budget.
Multi-Agent Debate: Propose, Critique and Revise
Generate independent proposals first. Then share only the claims and evidence needed for critique. Each objection should identify the disputed claim, cite a source or requirement, and state what needs correcting. Match critiques to the task: labels, source spans or the exact failed condition. Allow a limited number of revisions, preserve the original answers, and use a judge, verifier or aggregation rule to select the result.[3][4]
| Control | Stochastic consensus | Multi-agent debate |
|---|---|---|
| Initial sampling | Separate candidate answers | Separate proposals before discussion |
| Visibility | Peer outputs hidden until aggregation | Selected proposals and critiques shared |
| Critique | Optional validation stage | Explicit objections tied to evidence |
| Revision | Normally absent | Limited updates in response to objections |
| Selection | Vote, weighted rule or verifier | Judge, verifier or post-revision vote |
| Implementation | Parser, normalisation, vote accounting and tie rules | Role routing, moderation, context caps and round limits |
| Execution controls | Validate the selected output before acting | Verify the final decision; escalate unresolved material objections |
Assign roles such as extractor, rule checker and evidence verifier. A controller should enforce message limits, permitted evidence and revision scope. Start with one to three revision rounds, then test whether more rounds help. Stop at the round limit or validated stopping threshold; don’t force agreement.
Escalate unresolved high-risk decisions to a deterministic checker or human reviewer. Persuasive prose is not verification. The next question is whether the extra coordination improves accuracy enough to justify the added tokens and latency.
19. Agentic Engineering - Debate and Voting
Compare Decision Quality, Token Costs and Latency
Compare both patterns using verified outcomes, cost and latency under the same budget.
Agreement alone isn't a measure of quality. Aggregation reduces reliance on one reasoning path; debate can correct a flawed path. How well either works depends on the task and budget.[9][10]
| Criterion | Stochastic consensus | Multi-agent debate |
|---|---|---|
| Decision quality | Works well when independent samples take varied reasoning paths and the correct answer is common among them | Works well when agents have complementary information and critiques lead to checked corrections |
| Error resistance | Can dilute one-off mistakes, but not shared misconceptions | Can expose errors, but agents may settle on the same mistake |
| Token cost | Grows with sample count and answer length | Grows with agent count, rounds, critiques, revisions and repeated context |
| Latency | Parallel sampling limits elapsed time, subject to concurrency limits | Sequential rounds add latency; parallel critiques can reduce it |
| Ambiguous tasks | Majority voting can hide valid interpretations | Debate can bring these out only if dissent survives |
Accuracy gains depend on where sampling or critique helps with the task.
When Sampling or Critique Improves Accuracy
Independent sampling helps with objective tasks that allow different solution paths. Critique helps when evidence can test assumptions, arithmetic or conflicting constraints. A checked resolution passes an external test. A changed answer may just show that agents have converged.
| Task condition | Possible gain | Limit to test |
|---|---|---|
| Objective arithmetic with independent solution paths | Majority selection can reduce isolated calculation errors | Shared arithmetic assumptions remain dangerous |
| Code generation with executable tests | Multiple candidates increase the chance of a passing implementation | Without execution, debate may only produce plausible code |
| Evidence-based business recommendation | Independent drafts reveal alternative options | Consensus is unreliable if every agent uses the same incomplete evidence |
| Ambiguous policy or strategy question | Sampling reveals differences in interpretation | Debate can clarify assumptions and request missing information |
Any claim that debate beats independent sampling should state the benchmark, model, prompt, agent count, rounds, tools, evaluation method and total budget.[9][10]
Estimate Token Costs and End-to-End Latency
Count input, output and cached tokens separately, then apply the relevant rates. Include aggregation, judging, tools, retries and verification. Caching may reduce repeated-input charges, but check which inputs qualify.[7][8]
For a hypothetical accounting illustration, 12,000 input tokens and 2,000 output tokens per decision become 12 million input and 2 million output tokens per 1,000 decisions. A hypothetical total of £0.012 per decision equals £12 per 1,000 decisions; this is not an observed production price.
If billing uses another currency, record the exchange rate and conversion date. Report median, p95 and p99 end-to-end latency. Include queuing and orchestration, not just model response time.
| Workflow | Execution assumption | Main latency driver |
|---|---|---|
| Single agent | One generation call | Response length, tools and queuing |
| Stochastic consensus | Samples run concurrently where limits permit | Slowest sample plus aggregation; limited concurrency creates batches |
| Debate | Critiques may run together; rounds remain dependent | Longest call at each stage, repeated across rounds |
| Debate with verification | Final checking follows revision | Verification, retries and disputed-case escalation |
Test Each Pattern Under Matched Budgets
Use UK business tasks such as invoice and VAT calculations, contract-clause classification and customer support escalation.
Set both a generation budget, such as total output tokens, and a financial ceiling. Equal output tokens don't mean equal spend.
Run fixed-seed and varied-seed trials. Use independently verified ground truth for difficult cases, and report paired differences with confidence intervals or bootstrap uncertainty. Track correct-to-incorrect changes separately from successful corrections.
| Control or metric | What to hold constant or report | | - | - | - | | Evidence and tools | Same documents, rules for how current evidence must be, retrieval corpus and tool permissions | | Protocol differences | Record differences between protocols | | Actual resource use | Calls, input/output/cached tokens, tool calls and £ per 1,000 decisions | | Decision quality | Accuracy, citation correctness, calibration and injected-error detection | | Risk and ambiguity | Failure severity, appropriate abstention and performance on missing or conflicting evidence | | Operations and uncertainty | Median, p95 and p99 elapsed time, timeout rate, retry rate and variation across repeated trials |
Retain failure cases in the test records to track shared mistakes alongside raw accuracy.
When Debate Reinforces Shared Mistakes
Debate becomes riskier when agents start with the same evidence. The key question is whether critique adds evidence or simply repeats an answer. When agents share a model, prompt or retrieval source, debate can reinforce an error rather than expose it.[12][13]
Shared Errors, Anchoring and Information Cascades
The first plausible answer can anchor the discussion. Later agents may adjust the details without questioning the premise. Agreement then spreads through influence rather than evidence, while extra rounds repeat unsupported claims and push objections aside.[13][14]
Correction needs something checkable: new evidence or a reproducible calculation. Agreement that comes from seeing others’ answers does not provide that check. Choose debate over independent aggregation when critique produces verified corrections. More rounds help only when they add evidence, not just discussion.
One reported MMLU comparison found 76.5% mean accuracy for three agents debating over three rounds, versus 76.8% for nine independent samples with majority voting. Debate used about 3.4 times the token cost and 1.6 times the wall-clock time.[11] This is a limited result: homogeneous debate can cost more without improving accuracy in some settings, but it does not prove sampling always wins across other models, tasks, tools or budget controls.
The main failure patterns are predictable.
| Shared blind spot | Independent voting | Multi-agent debate | Safeguard |
|---|---|---|---|
| Common model or evidence error | Samples may repeat the same error | Repeated agreement can reinforce the error | Vary models, prompts or retrieval sources; check against authoritative evidence |
| Shared prompt assumption | Every sample may inherit the same framing | Debate may accept the framing before questioning it | Check premises and ask at least one agent to propose other interpretations |
| Anchoring to the first answer | Private samples preserve more independent search paths | Later agents may follow the first plausible proposal | Collect private proposals before sharing answers and randomise presentation order |
| Unsupported persuasive explanation | A fluent answer affects the final aggregation only indirectly | Eloquence or confidence can overpower less persuasive but correct objections | Require citations, calculations, tests or reproducible counterexamples |
| Information cascade | The model and prompt create correlation, but live social influence does not | Agents converge because they see earlier answers | Hide authorship, confidence and ordering where practical; compare views before and after debate |
| Failure to preserve minority evidence | Majority voting can discard a correct minority | Debate can pressure agents to withdraw valid objections | Keep minority reports and require a ruling on unresolved objections |
| Extra rounds with no new information | More samples cost more but retain independent outputs | More rounds add transcript overhead | Set token, round and time budgets with explicit stop conditions |
Preserve Dissent and Check Corrections
Keep the first round private to reduce cascades. Record each agent’s initial proposal, assumptions and evidence before it sees peer responses. Hide authorship, rank, confidence and ordering where practical. Each revision should explain what changed and why, pointing to a verified source, reproducible calculation, test result or resolved contradiction.
A red-team role alone does not create independence if the critic shares the same context. Give that agent a separate retrieval path or verification tool, and require a concrete attempt to disprove the claim.[13]
Keep justified minority views in the decision record. Stop when the disputed claim has been checked, no new evidence emerges, or the token, round or time limit is reached - not simply when everyone agrees. Refer unresolved legal, medical, financial or safety-critical disagreements to a human expert or external verifier, preserving the competing claims and evidence.
Conclusion: Choose Based on Testing and Risk
The costs and failure modes above make this a practical decision, not a theoretical one.
Start with stochastic consensus. Keep it only if repeated samples improve quality enough to justify the extra cost.
Choose Consensus, Debate or Conditional Escalation
Turn the comparison into a routing rule:
| Pattern | Starting use | Acceptance gate |
|---|---|---|
| Stochastic consensus | Normalised labels, structured extraction and numeric outputs | Higher accuracy or better calibration within budget |
| Multi-agent debate | Competing interpretations and checkable disputes | Verified corrections reduce serious errors enough to justify the overhead |
| Conditional escalation | Disagreement, low confidence, failed validation or high-impact decisions | Tested triggers route cases to debate, independent verification or human review |
Test these routing rules on held-out task data. Set thresholds using accuracy, calibration, failure severity, £ per decision, and p50 and p95 end-to-end latency. Compare workflows under matched budgets.
Use the Findings in Agent Implementation
The evidence used to choose a pattern should also support review, rollback and escalation. Once you set the routing rule, keep the records needed to audit and reverse it: proposals, critiques, evidence, vote distributions and validation results.
Apply role-based access, data minimisation, and defined retention and deletion rules. Document oversight, and revisit thresholds whenever the model, prompts, tools or workload change.[15][16]
Use task-specific testing, risk controls and cost modelling to set the implementation brief.
FAQs
How many independent answers should I sample?
There’s no fixed number. Hokstad Consulting recommends starting with 5–20 samples to test your task, then adding more until the majority vote stabilises [1][2].
Agreement doesn’t guarantee correctness. Weigh any accuracy gains against extra token costs and latency. For higher-risk decisions, use more samples and explicit review processes rather than relying only on how often the leading answer appears [1][2].
How can I detect shared errors without ground truth?
Agreement does not prove accuracy. Agents may rely on the same faulty sources, logic or assumptions. Use distributed telemetry to trace dependencies across hand-offs, shared-state writes and tool boundaries. Count shared upstream data as one evidence trail, not separate confirmations.
Use controlled replays to pinpoint errors. Then check the leading answers independently through deterministic tests, primary evidence or, for high-risk decisions, human review.
When is debate worth the extra cost and delay?
Debate is worth the extra cost and delay only when testing shows fewer errors in accepted answers than simpler voting. Sequential rounds use more tokens, add latency, put pressure on context windows and risk reinforcing shared mistakes.
Hokstad Consulting recommends voting by default for fixed outputs. Reserve debate for high-value disputes where independent counting isn't enough, such as conflicting sources or disputed assumptions.