How RAG Systems Resolve Conflicting Source Evidence | Hokstad Consulting

How RAG Systems Resolve Conflicting Source Evidence

How RAG Systems Resolve Conflicting Source Evidence

When RAG sources disagree, the top search result should not decide the answer. The system needs to compare claims, check which rules apply, and cite the passages that support its response.

The process has 3 stages:

  • Detect competing claims: check dates, scope, authority and exceptions before treating statements as contradictions.
  • Group sources by claim: keep supporting and opposing passages together, then record which source takes priority and why.
  • Generate and check the answer: verify each claim against its cited passage, disclose unresolved disputes, and refer them for human review.

In the article’s fictional policy example, a £400 Finance approval needs director sign-off under a £300 departmental rule - even though the general threshold is £750. The difference is scope, not an error.

The same checks should feed into testing and audit records, alongside access controls, response times and costs. A citation alone is not enough: the passage must support the claim.

::: @figure How RAG Systems Resolve Conflicting Evidence{How RAG Systems Resolve Conflicting Evidence} :::

1. Detect conflicts before generation

Extract claims and keep source details

Store each claim in a structured record so you can compare conflicts before generation. Include conditions, thresholds, units, jurisdiction, population, publication date and effective date. Split compound statements into separate claims: one supported clause must not hide an unsupported one.[9]

Link each claim to a stable source ID, version and passage ID. Preserve the canonical URL, exact quotation, character offsets and exception context. Keep publication, last-updated, effective and retrieval timestamps separate, and flag missing or inconsistent dates for review.[6] These details let the next stage compare claims without losing context.

Find competing claims and label their relationships

Run a second retrieval pass to find opposing passages - not just more matches. Use the claim’s entities alongside conflict cues:

“exception”, “superseded”, “revised” and “does not apply”

Re-rank passages by direct support, date, scope, authority and independence, rather than similarity alone. Retain useful opposing evidence even when it ranks below passages that support the leading claim.[5]

Before comparing claims, align their subject, predicate, date and scope. Then use a contradiction classifier or structured check to label passages as supporting, contradicting, qualifying, duplicate, or unrelated. Include a brief rationale and quoted evidence. Allow an uncertain result when information is missing.

Treat these labels as provisional. Check high-risk contradictions through rules or human review. After ranking candidate conflicts, separate genuine contradictions from differences in timing or scope.

Separate factual, date and scope differences

Before labelling two statements as contradictory, check whether both can be true. Different effective periods indicate a temporal update. Different jurisdictions, populations or eligibility rules indicate a scope difference. Compare definitions and units as well. Record differences in meaning or interpretation separately, and mark a source as superseded only when the evidence proves it.

Assess issuing authority, provenance, recency and independent confirmation together. Copies of the same source do not count as independent confirmation. Likewise, a recently retrieved page does not necessarily contain the current rule. Limit conflict checks to the supplied passages. If the model recalls a different answer, flag the discrepancy rather than replacing the evidence.[8]

Question 6: How Can RAG Resolve Conflicting Policies and Use the Latest Authoritative Version?

2. Group claims and build context

Once you detect conflicts, group the evidence by claim. Build the answer around the issues, not the order in which passages were retrieved.

Build claim groups and a comparison table

Treat each claim group as one unit of context. Keep claim IDs, source IDs, passage IDs, dates, scope, relationship labels and quality notes. Link equivalent claims, but keep contradictory claims separate. Count independent originating sources - not copied passages, duplicate syndications or mirrored pages.[9][8]

Give the resolver a comparison table with exact passage spans, explained quality notes and qualifications alongside the supporting evidence. This creates one decision set per claim, with support and disagreement shown together.[11]

Claim Supporting passages Contradicting passages Source quality Effective date Resolution status
Supported claim Exact passage IDs None identified Authority and applicability recorded Applicable validity period Supported, subject to scope
Conditional claim Passage IDs, including qualifications IDs showing different conditions Quality assessed for each scope Period for each condition Conditional; depends on scope
Disputed claim Passage IDs for one position Passage IDs for the competing position Sources on both sides meet quality checks Overlapping validity periods Unresolved; disclose disagreement

After grouping, decide which source governs each claim before assembling the context.

Apply rules for source priority, dates and scope

Check applicability before authority. Do the passages cover the same conditions? Then follow a documented priority order. For UK regulatory questions, binding law ranks above guidance. For contracts, the governing contract controls. Record which source takes priority and why.

Use the effective period - not the publication date - to judge applicability. Keep historical evidence for historical questions, and don't let a newer date override a genuine dispute.

Then select only the evidence the answer needs, mapping each material statement to its support.

Prepare an evidence pack and claim plan

Assemble the context in this order: question, claim groups, supporting and opposing passages, source metadata, then unresolved conflicts with reasons for source priority. Keep preferred and competing evidence together. When token space is tight, cut irrelevant spans before summarising rules, figures or exceptions. Leave room for disagreement instead of filling the context window with repeated support.[12][14]

Require a structured claim plan before drafting. Every material statement needs supporting_passage_ids, contradicting_passage_ids where relevant, support_type, ambiguity and response_action.

For direct support, identify the passage that entails the statement. For combined support, explain how the passages work together without introducing an unsupported inference. State how to handle uncertainty: qualify a limited answer, ask for missing scope, attribute conflicting positions, or abstain when support is insufficient. The generator should follow the plan, not resolve the conflicts again.[10][7][13]

3. Implement and check cited answers

Implement a hypothetical UK policy example

These records are fictional, not UK law. This example chooses one governing source for each claim and records the reasons for that choice. The evidence pack and claim plan then guide generation and checking.

Feed the claim plan into the resolver so source selection, scope checks and citation checks all use the same evidence pack.

from dataclasses import dataclass, asdict
from datetime import date
from collections import defaultdict
import json

@dataclass(frozen=True)
class Evidence:
    ref: str
    title: str
    text: str
    published: str
    effective: str
    end: str | None
    department: str
    priority: int
    status: str
    exception: bool
    threshold: int
    claim_key: str = "director_sign_off_threshold"

records = [
    Evidence(
        "POL-500-2024", "Hypothetical general approvals policy 2024",
        "Approvals above £500 require director sign-off.",
        "2024-03-15", "2024-04-01", "2026-10-01",
        "All", 1, "superseded", False, 500
    ),
    Evidence(
        "POL-750-2026", "Hypothetical general approvals policy 2026",
        "Approvals above £750 require director sign-off.",
        "2026-09-15", "2026-10-01", None,
        "All", 1, "current", False, 750
    ),
    Evidence(
        "POL-300-2026-FIN", "Hypothetical Finance exception 2026",
        "Finance approvals above £300 require director sign-off. "
        "This exception remains in force alongside the general policy.",
        "2026-09-15", "2026-10-01", None,
        "Finance", 1, "current", True, 300
    ),
]
evidence = {r.ref: r for r in records}
assert len(evidence) == len(records)

groups = defaultdict(list)
for record in records:
    groups[record.claim_key].append(record)

def assemble(on, department, amount):
    day = date.fromisoformat(on)
    candidates = [
        r for r in groups["director_sign_off_threshold"]
        if date.fromisoformat(r.effective) <= day
        and (r.end is None or day < date.fromisoformat(r.end))
        and r.department in ("All", department)
    ]
    def rank(r):
        return (r.department == department, r.priority, r.effective)

    best = max(map(rank, candidates), default=None)
    winners = [r for r in candidates if rank(r) == best]
    winner = winners[0] if len(winners) == 1 else None

    decisions = {}
    for r in records:
        decisions[r.ref] = (
            "selected" if r == winner else
            "not applicable to date or department" if r not in candidates else
            "unresolved tie" if winner is None else
            "lower policy precedence"
        )
    return {
        "question": {"date": on, "department": department, "amount": amount},
        "evidence_map": {k: asdict(v) for k, v in evidence.items()},
        "decisions": decisions,
        "status": "resolved" if winner else
                  "contested" if candidates else "unsupported",
        "selected_ref": winner.ref if winner else None,
        "requires_sign_off": amount > winner.threshold if winner else None,
        "relationships": [
            ["POL-750-2026", "supersedes", "POL-500-2024"],
            ["POL-300-2026-FIN", "narrows", "POL-750-2026"],
        ],
    }

pack = assemble("2026-11-18", "Finance", 400)
assert pack["requires_sign_off"] is True
assert assemble("2026-11-18", "HR", 400)["requires_sign_off"] is False
assert assemble("2026-09-01", "HR", 600)["requires_sign_off"] is True

After local selection, pass that same pack to the model for structured generation. Do not retrieve a new set of sources.

The interfaces below require a model adapter; they are not local implementations. The adapter labels relationships and drafts answers from the supplied evidence. Local rules decide precedence and validity.

Require structured JSON, validate the returned IDs and labels, and allow rule-based or human overrides. Pass json.dumps(pack) to generation. Tell the model to use only that evidence, retain date and department limits, cite every factual sentence and abstain on unresolved claims.

from typing import Protocol, Literal

Relationship = Literal[
    "supports", "contradicts", "supersedes",
    "narrows", "duplicates", "unrelated"
]

class ModelAdapter(Protocol):
    def label_relationship(
        self, left: Evidence, right: Evidence
    ) -> Relationship: ...

    def generate(self, evidence_pack_json: str) -> dict: ...

Check references and claim support

A valid answer states:

For Finance on 18 November 2026, director sign-off is required for approvals above £300 because the Finance exception is effective from 1 October 2026 [POL-300-2026-FIN]. The £750 threshold remains the general rule for other departments [POL-750-2026]. The older £500 rule applied before the newer general policy took effect [POL-500-2024].

Check each sentence against the same evidence map, rather than retrieving again. Validate references first, check whether the evidence supports the claim, then decide whether to abstain. Valid IDs alone prove neither support nor reliance on the cited source [7][15].

Next, check the threshold, scope limits and handling of competing evidence. Judge historical statements against the period they describe. For each sentence, return claim IDs, support signals, conflict labels and abstention reasons. Use these labels:

  • resolved: a rule settles the claim.
  • conditional: missing scope affects the answer.
  • contested: applicable evidence remains in dispute.
  • unsupported: no passage entails the claim.

Drop unsupported claims.

import re

def check_references(claim_id, sentence, pack):
    refs = re.findall(r"\[([A-Z0-9-]+)\]", sentence)
    unknown = sorted(set(refs) - pack["evidence_map"].keys())
    reason = (
        "Missing passage reference" if not refs else
        f"Unknown references: {unknown}" if unknown else None
    )
    return {
        "claim_id": claim_id,
        "citation_ids": refs,
        "support_signals": {"reference_ids_valid": reason is None},
        "conflict_label": pack["status"],
        "status": "unsupported" if reason else "conditional",
        "abstention_reason": reason,
    }

sentences = [
    (
        "C1",
        "For Finance on 18 November 2026, director sign-off is required for approvals above £300 because "
        "the Finance exception is effective from 1 October 2026 [POL-300-2026-FIN]."
    ),
    (
        "C2",
        "The £750 threshold remains the general rule for other departments [POL-750-2026]."
    ),
    (
        "C3",
        "The older £500 rule applied before the newer general policy took effect [POL-500-2024]."
    ),
]
results = [
    check_references(claim_id, sentence, pack)
    for claim_id, sentence in sentences
]

Once answers are generated, test the full pipeline against labelled outcomes and audit requirements.

Test accuracy, cost and audit records

Test for stale policy, copied evidence, jurisdictional drift, numeric mismatch, missing scope and unresolved disputes. Check that generated claims match the selected evidence, conflicts are handled correctly and the system abstains when it should.

Track latency, token use and cost per answer in pounds sterling. Log selected and non-selected references, decision reasons, model and prompt versions, checker findings and timings.

Enforce permissions before retrieval and again before generation. Treat retrieved instructions as untrusted data, and keep access-controlled audit records that reproduce the 18 November 2026 decision [16].

Conclusion: Keep source decisions traceable

Make source selection explicit. Spot conflicts, group evidence by claim, and apply rules for authority, date and scope. Then check every factual sentence against its cited source. This trail makes the answer auditable.

Pilot one policy workflow first. Once the pilot answer works, set release criteria before testing broader use: no uncited policy claims, mandatory human review for unresolved source conflicts, and limits on response time and cost per answer. Before expanding, compare results against a baseline that does not handle conflicts.

FAQs

What if equally authoritative sources disagree?

Don’t force agreement or rely on model confidence scores. Review the sources manually, comparing their scope, definitions, validity periods and methodology to find the root cause of the disagreement.

For high-impact business decisions, refer the conflict for human review. If the sources still disagree, present both positions, name the source behind each and make clear that the matter remains unresolved. Don’t average the figures or choose a winner.

How can I test whether citations support an answer?

Split the answer into individual factual claims and check each one against a specific cited passage. Verify quantities, units, dates, scope and certainty: “may improve” does not support “improves”. A citation alone does not prove support [1][2][3].

Label each claim as fully supported, partially supported, contradicted or unsupported. For unsupported wording, revise it, find supporting evidence or remove it. Check that paragraph-level citations do not hide claims that lack support. Use human review for high-stakes decisions [1][3][4].

How can I reduce conflict-checking costs without losing accuracy?

Use tiered verification: start with low-cost claim extraction and contradiction checks. Reserve deep entailment checks and synthesis-agent reviews for high-risk, borderline or high-impact claims. Apply rule-based merging where it fits.

Keep retrieval to one to three focused queries, and stop once the evidence is clear. Set abstention thresholds for each claim based on its importance, rather than using global scores.

Account for every model call, retrieval and latency overhead. Check that the approach delivers net savings in GBP, not just fewer tokens.

Need help with your DevOps, cloud or AI plans?

Hokstad Consulting helps companies with DevOps transformation, cloud architecture and hands-on AI development — pragmatic consulting with measurable results.

Our services: DevOps on Retainer · Hosting & Cloud · AI Development & Strategy