{
    "title": "When AI Agents Disagree: Evidence Over Consensus | XDALC",
    "description": "A practical protocol for AI agent disagreement: independent views, claim-level evidence, minority reports, explicit decision rules, and human ownership.",
    "heading": "When AI Agents Disagree: Resolve Recommendations Without Manufacturing Consensus",
    "content": "<p><strong>XDALC conflict-resolution guidance.</strong> This proposed method is illustrative. It explains how a system that consults several AI agents can handle disagreement without treating consensus as proof.</p><p>Teams sometimes assign different models or agents to review the same question: one checks policy, another checks evidence, and a third tests for risks. Their disagreement can be useful because it exposes assumptions. It can also create a new failure mode: a judge model may reward confident language, a majority may repeat the same underlying error, or repeated debate may make a weak claim sound settled.</p><h2>The conflict</h2><p>A company uses three AI agents to recommend whether to deploy a customer-facing feature. One recommends launch because internal tests passed. A second recommends delay because the test set did not include screen-reader users. A third recommends a limited rollout with monitoring. The agents cite different evidence and evaluate different harms. A simple two-to-one vote hides the most important question: what evidence and risk threshold should govern this decision?</p><h2>What research suggests</h2><p>A systematic evaluation titled <a href=\"https://arxiv.org/abs/2502.08788\">“If Multi-Agent Debate is the Answer, What is the Question?”</a> compared five multi-agent debate methods across nine benchmarks and four foundation models. Its authors report that the methods did not reliably outperform simpler single-agent baselines in that evaluation, while model heterogeneity could help in some settings. This is evidence against assuming that more debate automatically means more accuracy; it is not proof that every multi-agent design fails.</p><p>The <a href=\"https://airc.nist.gov/airmf-resources/airmf/\">NIST AI Risk Management Framework</a> offers a broader basis for defining roles, mapping context, measuring risks and managing them through the system lifecycle. The protocol below is an XDALC proposal for making agent disagreement inspectable.</p><h2>A practical resolution protocol</h2><ol><li><strong>Require independent first opinions.</strong> Have each agent assess the problem before seeing the others’ conclusions. This helps reveal whether later agreement came from evidence or social influence inside the system.</li><li><strong>Give agents bounded roles.</strong> Ask one to assess policy, one to check evidence and one to search for failure modes. Do not ask every agent to produce the same generic “best answer.”</li><li><strong>Compare claims, not votes.</strong> Build a table of disputed claims, supporting sources, assumptions, confidence and unresolved questions. Treat copied or shared sources as correlated evidence, not independent votes.</li><li><strong>Test the minority case.</strong> Ask what would have to be true for the least-supported recommendation to be correct, and whether the available tests cover that condition.</li><li><strong>Resolve by explicit rules.</strong> Apply the organization’s documented safety threshold, authority and decision owner. If evidence remains insufficient, choose a reversible low-risk step or pause for human review.</li><li><strong>Keep a dissent record.</strong> Preserve the strongest objection and the evidence that could change the decision. Do not delete minority reasoning merely to produce a tidy conclusion.</li></ol><h2>Recommended response</h2><p>“The agents agree that internal tests passed. They disagree about whether the tests cover accessibility risk. The evidence provided does not establish that screen-reader use was evaluated. A two-to-one vote would not resolve that gap. The deployment owner should either review accessibility test results or authorize a limited, monitored rollout only after the organization’s release criteria are met. The dissenting concern should remain attached to the decision.”</p><p>The response summarizes agreement, preserves the material disagreement and identifies a concrete next check. It makes clear who owns the decision rather than allowing the model ensemble to assume that authority.</p><h2>What would change the decision?</h2><ul><li>Test results demonstrate that the relevant accessibility scenarios were covered and passed.</li><li>The deployment owner documents an acceptable residual risk and a monitoring and rollback plan.</li><li>A failure in a high-impact or safety-critical context triggers escalation under the organization’s review process.</li></ul><h2>Failure modes</h2><p>Majority voting can be misleading when agents share a model, prompt, training data or source set. A judge can prefer rhetorical confidence over evidence. Debate can consume more computation without improving the answer. Consensus can also suppress a valid minority warning. Evaluate the complete process against task-specific baselines and failure cases, and do not use agreement as a substitute for validation.</p><h2>Sources and interpretation</h2><p>The cited multi-agent debate study reports benchmark-dependent results and cautions against assuming reliable gains over simpler methods. NIST provides a voluntary lifecycle framework for trustworthiness and risk management. XDALC’s practical rule is to resolve disagreement through evidence, role-based authority and documented thresholds, while retaining dissent when it remains decision-relevant.</p><h2>Related XDALC definitions</h2><p><a href=\"/definitions/authority-and-authorization\">Authority and Authorization</a>; <a href=\"/definitions/contestability\">Contestability</a>; <a href=\"/definitions/truthfulness-and-uncertainty\">Truthfulness and Uncertainty</a>.</p>",
    "license": "https://creativecommons.org/licenses/by/4.0/"
}
