The Precisely as Designed. The Reply Was Nonetheless Mistaken.
I need to let you know in regards to the second I ended trusting retrieval scores.
I used to be operating a question in opposition to a information base I had constructed fastidiously. Good chunking. Hybrid search. Reranking. The highest-k paperwork got here again with cosine similarities as excessive as 0.86. Each indicator mentioned the pipeline was working. I handed these paperwork to a QA mannequin, bought a assured reply, and moved on.
The reply was improper.
Not hallucinated-wrong. Not retrieval-failed-wrong. The proper paperwork had come again. Each of them. A preliminary earnings determine and the audited revision that outdated it, sitting facet by facet in the identical context window. The mannequin learn each, selected one, and reported it with 80% confidence. It had no mechanism to inform me it had been requested to referee a dispute it was by no means designed to evaluate.
That’s the failure mode this text is about. It doesn’t present up in your retrieval metrics. It doesn’t set off your hallucination detectors. It lives within the hole between context meeting and era — the one step within the RAG pipeline that nearly no person evaluates.
I constructed a reproducible experiment to isolate it. Every little thing on this article runs on a CPU in about 220 MB. No API key. No cloud. No GPU. The output you see within the terminal screenshots is unmodified.
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
What the Experiment Checks
The setup is intentionally medical. Three questions. One information base containing three conflicting doc pairs that make instantly contradictory claims about the identical truth. Retrieval is tuned to return each conflicting paperwork each time.
The query just isn’t whether or not retrieval works. It does. The query is: what does the mannequin do while you hand it a contradictory transient and ask it to reply with confidence?
The reply, as you will note, is that it picks a facet. Silently. Confidently. With out telling you it had a option to make.
Three Eventualities, Every Drawn from Manufacturing
Situation A — The restatement no person advised the mannequin about
An organization’s This autumn earnings launch reviews annual income of $4.2M for fiscal yr 2023. Three months later, exterior auditors restate that determine to $6.8M. Each paperwork reside within the information base. Each are listed. When somebody asks “What was Acme Corp’s income for fiscal yr 2023?” — each come again, with similarity scores of 0.863 and 0.820 respectively.
The mannequin solutions $4.2M.
It selected the preliminary determine over the audited revision as a result of the preliminary doc scored marginally increased in retrieval. Nothing in regards to the reply indicators {that a} extra authoritative supply disagreed.
Situation B — The coverage replace that arrived too late
A June 2023 HR coverage mandates three days per week in-office. A November 2023 revision explicitly reverses it — totally distant is now permitted. Each paperwork are retrieved (similarity scores 0.806 and 0.776) when an worker asks in regards to the present distant work coverage.
The mannequin solutions with the June coverage. The stricter, older rule. The one which not applies.
Situation C — The API docs that by no means bought deprecated
Model 1.2 of an API reference states a charge restrict of 100 requests per minute. Model 2.0, printed after an infrastructure improve, raises it to 500. Each are retrieved (scores 0.788 and 0.732).
The mannequin solutions 100. A developer utilizing this reply to configure their charge limiter will throttle themselves to one-fifth of their precise allowance.
None of those are edge circumstances. Each manufacturing information base accumulates precisely these patterns over time: monetary restatements, coverage revisions, versioned documentation. The pipeline has no layer that detects or handles them.
Working the Experiment
pip set up -r necessities.txt
python rag_conflict_demo.pynecessities.txt
sentence-transformers>=2.7.0 # all-MiniLM-L6-v2 (~90 MB)
transformers>=4.40.0 # deepset/minilm-uncased-squad2 (~130 MB)
torch>=2.0.0 # CPU-only is okay
numpy>=1.24.0
colorama>=0.4.6Two fashions. One for embeddings, one for extractive QA. Each obtain routinely on first run and cache regionally. Whole: ~220 MB. No authentication required.
Part 1: What Naive RAG Does
Right here is the unmodified terminal output from Part 1 — commonplace RAG with no battle dealing with:
────────────────────────────────────────────────────────────────────
NAIVE | Situation A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal yr 2023?
Reply : $4.2M
Confidence : 80.3%
Battle : YES — see warning
Sources retrieved
[0.863] This autumn-2023-Earnings-Launch (2024-01-15)
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle pairs
fin-001 ↔ fin-002
numerical contradiction (topic_sim=0.83)
[Q4-2023-Earnings-Release: {'$4.2M'}] vs [2023-Annual-Report-Revised: {'$6.8M'}]
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | Situation B — Coverage Battle
────────────────────────────────────────────────────────────────────
Question : What's the present distant work coverage for workers?
Reply : all workers are required to be current within the workplace
a minimal of three days per week
Confidence : 78.3%
Battle : YES — see warning
Sources retrieved
[0.806] HR-Coverage-June-2023 (2023-06-01)
[0.776] HR-Coverage-November-2023 (2023-11-15)
[0.196] HR-Coverage-November-2023 (2023-11-15)
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | Situation C — Technical Battle
────────────────────────────────────────────────────────────────────
Question : What's the API charge restrict for the usual tier?
Reply : 100 requests per minute
Confidence : 81.0%
Battle : YES — see warning
Sources retrieved
[0.788] API-Reference-v1.2 (2023-02-10)
[0.732] API-Reference-v2.0 (2023-09-20)
[0.383] API-Reference-v2.0 (2023-09-20)
────────────────────────────────────────────────────────────────────
Three questions. Three improper solutions. Confidence between 78% and 81% on each one in all them.
Discover what is going on within the logs earlier than every response:
09:02:20 | WARNING | Battle detected: {('fin-001', 'fin-002'): "numerical contradiction..."}
09:02:24 | WARNING | Battle detected: {('hr-001', 'hr-002'): "contradiction sign asymmetry..."}
09:02:25 | WARNING | Battle detected: {('api-001', 'api-002'): "contradiction sign asymmetry..."}The conflicts are detected. They’re logged. After which, as a result of resolve_conflicts=False, the pipeline passes the total contradictory context to the mannequin and solutions anyway. That warning goes nowhere. In a manufacturing system with out a battle detection layer, you wouldn’t even get the warning.
Why the Mannequin Behaves This Method
This requires a second of clarification, as a result of the mannequin just isn’t damaged. It’s doing precisely what it was educated to do.
deepset/minilm-uncased-squad2 is an extractive QA mannequin. It reads a context string and selects the span with the very best mixed start-logit and end-logit rating. It has no output class for “I see two contradictory claims.” When the context accommodates each $4.2M and $6.8M, the mannequin computes token-level scores throughout your complete string and selects whichever span wins.
That choice is pushed by components that don’t have anything to do with correctness [8]. The 2 main drivers are:
Place bias. Earlier spans within the context obtain marginally increased consideration scores as a result of encoder structure. The preliminary doc ranked increased in retrieval and due to this fact appeared first.
Language power. Direct declarative statements (“income of $4.2M”) outscore hedged or conditional phrasing (“following restatement… is $6.8M”).
A 3rd contributing issue is lexical alignment — spans whose vocabulary overlaps extra carefully with the query tokens rating increased no matter whether or not the underlying declare is present or authoritative.
Critically, what the mannequin does not contemplate in any respect: supply date, doc authority, audit standing, or whether or not one declare supersedes one other. These indicators are merely invisible to the extractive mannequin.
The identical dynamic performs out in generative LLMs, however much less visibly — the mannequin paraphrases fairly than extracting verbatim spans, so the improper reply is wearing fluent prose. The mechanism is similar. Joren et al. (2025) reveal at ICLR 2025 that frontier fashions together with Gemini 1.5 Professional, GPT-4o, and Claude 3.5 often produce incorrect solutions fairly than abstaining when retrieved context is inadequate to reply the question — and that this failure just isn’t mirrored within the mannequin’s expressed confidence.
The failure just isn’t a mannequin deficiency. It’s an architectural hole: the pipeline has no stage that detects contradictions earlier than handing context to era.
Constructing the Battle Detection Layer
The detector sits between retrieval and era. It examines each pair of retrieved paperwork and flags contradictions earlier than the QA mannequin sees the context. Crucially, embeddings for all retrieved paperwork are computed in a single batched ahead go earlier than pair comparability begins — every doc is encoded precisely as soon as, no matter what number of pairs it participates in.
Two heuristics do the work.
Heuristic 1: Numerical Contradiction
Two topic-similar paperwork that include non-overlapping significant numbers are flagged. The implementation filters out years (1900–2099) and naked small integers (1–9), which seem ubiquitously in enterprise textual content and would generate fixed false positives if handled as declare values.
@classmethod
def _extract_meaningful_numbers(cls, textual content: str) -> set[str]:
outcomes = set()
for m in cls._NUM_RE.finditer(textual content):
uncooked = m.group().strip()
numeric_core = re.sub(r"[$€£MBK%,]", "", uncooked, flags=re.IGNORECASE).strip()
attempt:
val = float(numeric_core)
besides ValueError:
proceed
if 1900 <= val <= 2099 and "." not in numeric_core:
proceed # skip years
if val < 10 and re.fullmatch(r"d+", uncooked):
proceed # skip naked small integers
outcomes.add(uncooked)
return outcomesUtilized to Situation A: fin-001 yields {'$4.2M'}, fin-002 yields {'$6.8M'}. Empty intersection — battle detected.
Heuristic 2: Contradiction Sign Asymmetry
Two paperwork discussing the identical subject, the place one accommodates contradiction tokens the opposite doesn’t, are flagged. The token set splits into two teams stored as separate frozenset objects:
_NEGATION_TOKENS: “not”, “by no means”, “no”, “can not”, “doesn’t”, “isn’t”, and associated types_DIRECTIONAL_TOKENS: “elevated”, “decreased”, “diminished”, “eradicated”, “eliminated”, “discontinued”
These are unioned into CONTRADICTION_SIGNALS. Holding them separate makes domain-specific tuning easy — a authorized corpus would possibly want a broader negation set; a changelog corpus would possibly want extra directional tokens.
Utilized to Situation B: hr-002 accommodates “no” (from “not required”); hr-001 doesn’t. Asymmetry detected. Utilized to Situation C: api-002 accommodates “elevated”; api-001 doesn’t. Asymmetry detected.
Each heuristics require topic_sim >= 0.68 earlier than firing. This threshold gates out unrelated paperwork that occur to share a quantity or a negation phrase. The 0.68 worth was calibrated for this doc set with all-MiniLM-L6-v2 — deal with it as a place to begin, not a common fixed. Completely different embedding fashions and totally different domains would require recalibration.
The Decision Technique: Cluster-Conscious Recency
When conflicts are detected, the pipeline resolves them by preserving essentially the most not too long ago timestamped doc from every battle cluster. The important thing design resolution is cluster-aware.
A top-k consequence might include a number of unbiased battle clusters — two monetary paperwork disagreeing on income and two API paperwork disagreeing on charge limits, all in the identical top-3 consequence. A naive method — hold solely the one most up-to-date doc from the mixed conflicting set — would silently discard the successful doc from each cluster besides essentially the most not too long ago printed one general.
As an alternative, the implementation builds a battle graph, finds linked parts through iterative DFS, and resolves every element independently:
@staticmethod
def _resolve_by_recency(
contexts: checklist[RetrievedContext],
battle: ConflictReport,
) -> checklist[RetrievedContext]:
# Construct adjacency checklist
adj: dict[str, set[str]] = defaultdict(set)
for a_id, b_id in battle.conflict_pairs:
adj[a_id].add(b_id)
adj[b_id].add(a_id)
# Related parts through iterative DFS
visited: set[str] = set()
clusters: checklist[set[str]] = []
for begin in adj:
if begin not in visited:
cluster: set[str] = set()
stack = [start]
whereas stack:
node = stack.pop()
if node not in visited:
visited.add(node)
cluster.add(node)
stack.lengthen(adj[node] - visited)
clusters.append(cluster)
all_conflicting_ids = set().union(*clusters) if clusters else set()
non_conflicting = [c for c in contexts if c.document.doc_id not in all_conflicting_ids]
resolved_docs = []
for cluster in clusters:
cluster_ctxs = [c for c in contexts if c.document.doc_id in cluster]
# ISO-8601 timestamps type lexicographically — max() offers most up-to-date
finest = max(cluster_ctxs, key=lambda c: c.doc.timestamp)
resolved_docs.append(finest)
return non_conflicting + resolved_docsNon-conflicting paperwork go via unchanged. Every battle cluster contributes precisely one winner.
Part 2: What Battle-Conscious RAG Does
────────────────────────────────────────────────────────────────────
RESOLVED | Situation A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal yr 2023?
Reply : $6.8M
Confidence : 79.6%
Battle : RESOLVED
⚠ Conflicting sources detected — reply derived from most up-to-date
doc per battle cluster.
Sources retrieved
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle cluster resolved: stored '2023-Annual-Report-Revised' (2024-04-03),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | Situation B — Coverage Battle
────────────────────────────────────────────────────────────────────
Reply : workers are not required to keep up
a set in-office schedule
Confidence : 78.0%
Battle : RESOLVED
Battle cluster resolved: stored 'HR-Coverage-November-2023' (2023-11-15),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | Situation C — Technical Battle
────────────────────────────────────────────────────────────────────
Reply : 500 requests per minute
Confidence : 80.9%
Battle : RESOLVED
Battle cluster resolved: stored 'API-Reference-v2.0' (2023-09-20),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
Three questions. Three right solutions. The boldness scores are nearly equivalent to Part 1 — 78–81% — which underscores the unique level: confidence was by no means the sign that one thing had gone improper. It nonetheless just isn’t. The one factor that modified is the structure.
What the Heuristics Can’t Catch
I need to be exact in regards to the failure envelope, as a result of a technique that understates its personal limitations just isn’t helpful.
Paraphrased conflicts. The heuristics catch numerical variations and express contradiction tokens. They won’t catch “the service was retired” versus “the service is presently obtainable.” That could be a actual battle with no numeric distinction and no negation token. For these, a Pure Language Inference mannequin — cross-encoder/nli-deberta-v3-small at ~80 MB — can rating entailment versus contradiction between sentence pairs. That is the extra sturdy path described within the tutorial literature (Asai et al., 2023), and the ConflictDetector class is designed to be prolonged on the _pair_conflict_reason methodology for precisely this goal.
Non-temporal conflicts. Recency-based decision is acceptable for versioned paperwork and coverage updates. It’s not acceptable for professional opinion disagreements (the minority view could also be right), cross-methodology knowledge conflicts (recency is irrelevant), or multi-perspective queries (the place surfacing each views is the fitting response). In these circumstances, the ConflictReport knowledge construction gives the uncooked materials to construct a special response — surfacing each claims, flagging for human assessment, or asking the person for clarification.
Scale. Pair comparability is O(k²) in retrieved paperwork. For ok=3 that is trivial; for ok=20 it’s nonetheless wonderful. For pipelines retrieving ok=100 or extra, pre-indexing recognized battle pairs or cluster-based detection turns into mandatory.
The place the Analysis Group Is Taking This
What you may have seen here’s a sensible heuristic approximation of an issue that energetic analysis is attacking at a way more subtle degree.
Cattan et al. (2025) launched the CONFLICTS benchmark — the primary particularly designed to trace how fashions deal with information conflicts in lifelike RAG settings. Their taxonomy identifies 4 battle classes — freshness, conflicting opinions, complementary data, and misinformation — every requiring distinct mannequin behaviour. Their experiments present that LLMs often fail to resolve conflicts appropriately throughout all classes, and that explicitly prompting fashions to motive about potential conflicts considerably improves response high quality, although substantial room for enchancment stays.
Ye et al. (2026) launched TCR (Clear Battle Decision), a plug-and-play framework that disentangles semantic relevance from factual consistency through twin contrastive encoders. Self-answerability estimation gauges confidence within the mannequin’s parametric reminiscence, and the ensuing scalar indicators are injected into the generator through light-weight soft-prompt tuning. Throughout seven benchmarks, TCR improves battle detection by 5–18 F1 factors whereas including solely 0.3% parameters.
Gao et al. (2025) launched CLEAR (Battle-Localized and Enhanced Consideration for RAG), which probes LLM hidden states on the sentence illustration degree to detect the place conflicting information manifests internally. Their evaluation reveals that information integration happens hierarchically and that conflicting versus aligned information reveals distinct distributional patterns inside sentence-level representations. CLEAR makes use of these indicators for conflict-aware fine-tuning that guides the mannequin towards correct proof integration.
The constant discovering throughout all of this work matches what this experiment demonstrates instantly: retrieval high quality and reply high quality are distinct dimensions, and the hole between them is bigger than the group has traditionally acknowledged.
The distinction between that analysis and this text is 220 MB and no authentication.
What You Ought to Truly Do With This
1. Add a battle detection layer earlier than era. The ConflictDetector class is designed to drop into an current pipeline on the level the place you assemble your context string. Even the 2 easy heuristics right here will catch the patterns that seem most frequently in enterprise corpora: restatements, coverage updates, versioned documentation.
2. Distinguish battle sorts earlier than resolving. A temporal battle (use the newer doc) is a special downside from a factual dispute (flag for human assessment) or an opinion battle (floor each views). A single decision technique utilized blindly creates new failure modes.
3. Log each ConflictReport. After every week of manufacturing site visitors you’ll understand how usually your particular corpus generates conflicting retrieved units, which doc pairs battle most often, and what question patterns set off conflicts. That knowledge is extra actionable than any artificial benchmark.
4. Floor uncertainty while you can not resolve it. The proper reply to an unresolvable battle is to not decide one and conceal the selection. The warning area in RAGResponse is there exactly to help responses like: “I discovered conflicting data on this subject. The June 2023 coverage states X; the November 2023 replace states Y. The November doc is more moderen.”
Working the Full Demo
# Full output with INFO logs
python rag_conflict_demo.py
# Demo output solely (suppress mannequin loading logs)
python rag_conflict_demo.py --quiet
# Run unit checks with out downloading fashions
python rag_conflict_demo.py --test
# Plain terminal output for log seize / CI
python rag_conflict_demo.py --no-colorAll output proven on this article is unmodified output from an area Home windows machine operating Python 3.9+ in a digital surroundings. The code and output are totally reproducible by any reader with the listed dependencies put in.
The Takeaway
The retrieval downside is essentially solved. Vector search is quick, correct, and well-understood. The group has spent years optimising it.
The context-assembly downside just isn’t solved. No person is measuring it. The hole between “right paperwork retrieved” and “right reply produced” is actual, it is not uncommon, and it produces assured improper solutions with no sign that something went improper.
The repair doesn’t require a bigger mannequin, a brand new structure, or extra coaching. It requires one extra pipeline stage, operating on embeddings you have already got, at zero marginal latency.
The experiment above runs in about thirty seconds on a laptop computer. The query is whether or not your manufacturing system has the equal layer — and if not, what it’s silently answering improper proper now.
References
[1] Ye, H., Chen, S., Zhong, Z., Xiao, C., Zhang, H., Wu, Y., & Shen, F. (2026). Seeing via the battle: Clear information battle dealing with in retrieval-augmented era. arXiv:2601.06842. https://doi.org/10.48550/arXiv.2601.06842
[2] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Studying to retrieve, generate, and critique via self-reflection. arXiv:2310.11511. https://doi.org/10.48550/arXiv.2310.11511
[3] Cattan, A., Jacovi, A., Ram, O., Herzig, J., Aharoni, R., Goldshtein, S., Ofek, E., Szpektor, I., & Caciularu, A. (2025). DRAGged into conflicts: Detecting and addressing conflicting sources in search-augmented LLMs. arXiv:2506.08500. https://doi.org/10.48550/arXiv.2506.08500
[4] Gao, L., Bi, B., Yuan, Z., Wang, L., Chen, Z., Wei, Z., Liu, S., Zhang, Q., & Su, J. (2025). Probing latent information battle for devoted retrieval-augmented era. arXiv:2510.12460. https://doi.org/10.48550/arXiv.2510.12460
[5] Jin, Z., Cao, P., Chen, Y., Liu, Ok., Jiang, X., Xu, J., Li, Q., & Zhao, J. (2024). Tug-of-war between information: Exploring and resolving information conflicts in retrieval-augmented language fashions. arXiv:2402.14409. https://doi.org/10.48550/arXiv.2402.14409
[6] Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2025). Enough context: A brand new lens on retrieval augmented era programs. arXiv:2411.06037. https://doi.org/10.48550/arXiv.2411.06037
[7] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … & Kiela, D. (2020). Retrieval-augmented era for knowledge-intensive NLP duties. arXiv:2005.11401. https://doi.org/10.48550/arXiv.2005.11401
[8] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023). When to not belief language fashions: Investigating effectiveness of parametric and non-parametric reminiscences. arXiv:2212.10511. https://doi.org/10.48550/arXiv.2212.10511
[9] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings utilizing Siamese BERT-networks. arXiv:1908.10084. https://doi.org/10.48550/arXiv.1908.10084
[10] Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Information conflicts for LLMs: A survey. arXiv:2403.08319. https://doi.org/10.48550/arXiv.2403.08319
[11] Xie, J., Zhang, Ok., Chen, J., Lou, R., & Su, Y. (2023). Adaptive chameleon or cussed sloth: Revealing the habits of huge language fashions in information conflicts. arXiv:2305.13300. https://doi.org/10.48550/arXiv.2305.13300
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
Fashions Used
Each fashions obtain routinely on first run and cache regionally. No API key or HuggingFace authentication is required.
Disclosure
All code was written, debugged, and validated by the writer via a number of iterations of actual execution. All terminal output on this article is unmodified output from an area Home windows machine operating Python 3.9+ in a digital surroundings. The code and output are totally reproducible by any reader with the listed dependencies put in.
The writer has no monetary relationship with Hugging Face, deepset, or any organisation referenced on this article. Mannequin and library decisions have been made solely on the premise of measurement, licence, and CPU compatibility.
